# Project export: This X-Ray Does Not Exist

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: A generative AI pipeline for synthesis of realistic medical images. Tested by expert radiologists. Industry applications for data enrichment of AI models, & overcoming clinical data-sharing barriers.
- Devpost: https://devpost.com/software/this-x-ray-does-not-exist
- GitHub: https://github.com/jayanthspratap/this-xray-does-not-exist/
- Team: 1 GitHub contributor(s) — Jayanth Pratap (7 commits)

## Devpost submission (written by the team)

### Overview

💭 Motivation Medical images play a crucial role in training artificial intelligence (AI) algorithms for various diagnostic tasks. However, limited access to large-scale datasets poses a challenge in developing accurate and robust models. Recent advances in latent diffusion models have demonstrated the generation of novel, realistic images using text prompts. Generating high-fidelity medical images could potentially solve the medical data gap. We present an approach for generating synthetic chest X-rays (CXRs) with domain adaptation of Stable Diffusion (SD), a state-of-the-art latent diffusion model for text-to-image generation. We also present a method for removal of text and artifacts from the resulting synthetic images, and evaluated images with two expert radiologists. 🔨 Methods To generate novel CXRs, we fine-tuned the SD v2-1-base model for 10 epochs on an Intel® Max Series GPU, using low-rank adaptation. We used a dataset of 32,532 CXRs paired with clinical reports, a proprietary dataset obtained in a research partnership with Segmed. We then performed GPU inference and generated 1,000 synthetic CXRs using classifier-free guidance of 4, and 75 inference steps. We implemented a text removal step on images, to eliminate textual artifacts on the generated images. This pipeline leverages text segmentation using Meta’s Segment Anything Model, and inpainting with Stability.AI’s Stable Diffusion 2. The inference is run on an Intel® Max Series GPU and can be executed through a Gradio web application. 📈 Evaluation and Results The synthetic images showed quantitative concordance with real CXRs. The Fréchet Inception Distance between the synthetic and real dataset was 68.4, calculated using InceptionV3 pretrained on ImageNet. A random sample of 50 generated CXRs and 50 real CXRs was shuffled, given to a board-certified radiologist with >10 years of clinical experience, and labeled as either "synthetic" or "real". The reader had an accuracy of 0.81, sensitivity 0.86, and specificity 0.76. Notably, Reader 1 misidentified 7 out of 50 synthetic images as real images. A random sample of 50 generated CXRs and 50 real CXRs was shuffled, given to a board-certified radiologist with >10 years of clinical experience, and labeled as either "synthetic" or "real". The reader had an accuracy of 0.81, sensitivity 0.86, and specificity 0.76. Notably, Reader 1 misidentified 7 out of 50 synthetic images as real images. A random sample of 100 generated CXRs was examined for overall quality (anatomy, style, artifacts) by the same radiologist expert. Some feedback, both positive and negative, is included below in reference to the synthetic X-rays: A random sample of 100 generated CXRs was examined for overall quality (anatomy, style, artifacts) by the same radiologist expert. Some feedback, both positive and negative, is included below in reference to the synthetic X-rays: A few [synthetic images] stand out due to their sharpness and defined details, making them more closely resemble real X-rays. The rib cage and heart have a "box-y" shape. However, the lung parenchyma looks good. Looks good, except for the ECG marker with a weird position (also the only one). Costo-phrenic angles too deep, esp on the left. In total, there were 36 positively scoring samples that were deemed by the radiologist to be comparable to real X-rays, in both anatomy and appearance. Of the remainder, many were labeled as moderate quality, with the only identified downside being the presence of text artifacts. In the majority of these cases, the AI-enabled text removal is able to rectify this, creating a clean synthetic CXR. 🏆 Conclusion Our pipeline is able to generate synthetic CXRs that resemble real CXRs, with further improvement in performance to be expected as we scale up the training time and data. The proposed method holds promise for augmenting limited medical image datasets, facilitating the development of robust ML algorithms in chest radiography. The ability to generate realistic synthetic CXRs has significant implications for the field of medical imaging. By augmenting limited datasets, our approach can improve training ML algorithms for diagnostic tasks, holding tremendous potential for advancing healthcare outcomes.

## README (from the GitHub repository)

# This X-Ray Does Not Exist
![image](https://github.com/jayanthspratap/this-xray-does-not-exist/assets/33165884/f9508827-787b-468e-84fd-9191aaa097d5)

*A generative AI pipeline for synthesis of realistic medical images. Tested by expert radiologists. Industry applications for data enrichment of AI models, and overcoming clinical data-sharing barriers.*

## 💭 Motivation
Medical images play a crucial role in training artificial intelligence (AI) algorithms for various diagnostic tasks. However, limited access to large-scale datasets poses a challenge in developing accurate and robust models. Recent advances in latent diffusion models have demonstrated the generation of novel, realistic images using text prompts. Generating high-fidelity medical images could potentially solve the medical data gap. We present an approach for generating synthetic chest X-rays (CXRs) with domain adaptation of Stable Diffusion (SD), a state-of-the-art latent diffusion model for text-to-image generation. We also present a method for removal of text and artifacts from the resulting synthetic images, and evaluated images with two expert radiologists.

## 🔨 Methods
To generate novel CXRs, we fine-tuned the SD v2-1-base model for 10 epochs on an Intel® Max Series GPU, using low-rank adaptation. We used a dataset of 32,532 CXRs paired with clinical reports, a proprietary dataset obtained in a research partnership with Segmed. We then performed GPU inference and generated 1,000 synthetic CXRs using classifier-free guidance of 4, and 75 inference steps.

We implemented a text removal step on images, to eliminate textual artifacts on the generated images. This pipeline leverages text segmentation using Meta’s Segment Anything Model, and inpainting with Stability.AI’s Stable Diffusion 2. The inference is run on an Intel® Max Series GPU and can be executed through a Gradio web application.

## 📈 Evaluation and Results
1. The synthetic images showed quantitative concordance with real CXRs. The Fréchet Inception Distance between the synthetic and real dataset was 68.4, calculated using InceptionV3 pretrained on ImageNet.
2. A random sample of 50 generated CXRs and 50 real CXRs was shuffled, given to a board-certified radiologist with >10 years of clinical experience, and labeled as either "synthetic" or "real". The reader had an accuracy of 0.81, sensitivity 0.86, and specificity 0.76. **Notably, Reader 1 misidentified 7 out of 50 synthetic images as real images.**

3. A random sample of 100 generated CXRs was examined for overall quality (anatomy, style, artifacts) by the same radiologist expert. Some feedback, both positive and negative, is included below in reference to the synthetic X-rays:

> A few [synthetic images] stand out due to their sharpness and defined details, making them more closely resemble real X-rays.

> The rib cage and heart have a "box-y" shape. However, the lung parenchyma looks good.

> Looks good, except for the ECG marker with a weird position (also the only one).

> Costo-phrenic angles too deep, esp on the left.

**In total, there were 36 positively scoring samples that were deemed by the radiologist to be comparable to real X-rays, in both anatomy and appearance.** Of the remainder, many were labeled as moderate quality, with the only identified downside being the presence of text artifacts. In the majority of these cases, the AI-enabled text removal is able to rectify this, creating a clean synthetic CXR.

## 🏆 Conclusion
Our pipeline is able to generate synthetic CXRs that resemble real CXRs, with further improvement in performance to be expected as we scale up the training time and data. The proposed method holds promise for augmenting limited medical image datasets, facilitating the development of robust ML algorithms in chest radiography. The ability to generate realistic synthetic CXRs has significant implications for the field of medical imaging. By augmenting limited datasets, our approach can improve training ML algorithms for diagnostic tasks, holding tremendous potential for advancing healthcare outcomes.

## Footnotes
This project was completed in the 2024 TreeHacks hackathon at Stanford University. Computation was done on the Intel Developer Cloud platform. Data acquisition was done in partnership with Segmed, and no patient data is made available in this repository. Hence, all scripts and notebooks are included for transparency as opposed to direct replicability. Please contact jaypratap@college.harvard.edu with any questions.


## Detected evidence (automated analysis)

Indexed codebase: 4 recognized source files, 46 KB.
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (7 of 7)

```
finetune_model.sh
inference_text_to_image_lora.py
OCR_SAM_text_removal.ipynb
prep_segmed_data.ipynb
README.md
stable_diffusion_finetuning.ipynb
train_text_to_image_lora.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Update README.md
- Add all files from Intel Cloud workspace
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### finetune_model.sh

```shell
export MODEL_NAME="stabilityai/stable-diffusion-2-1-base"
export DATA_DIR="../datasets/segmed_filtered_images"
export OUT_DIR="../models/sd-cxr-healthy-2024_02_17"

accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$DATA_DIR \
  --caption_column="text" \
  --resolution=512 \
  --train_batch_size=5 \
  --num_train_epochs=10 --checkpointing_steps=1000 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir=$OUT_DIR \
  --validation_prompt="cxr healthy" --report_to="wandb" >> ../output/output_2024_02_17.txt 2>&1
```

### inference_text_to_image_lora.py

```python
from diffusers import StableDiffusionPipeline
import torch
from tqdm import tqdm
import os

model_name= "stabilityai/stable-diffusion-2-1-base"
model_path = "../models/sd-cxr-healthy-2024_02_17"
pipe = StableDiffusionPipeline.from_pretrained(model_name, torch_dtype=torch.float16)
pipe.unet.load_attn_procs(model_path)
pipe.to("cuda")

# number of images to generate
num_images = 10000

# prompts
prompts = []
prompts.append("view_position = PA. modality = CR. No findings.")
# prompts.append("view_position = PA. modality = CR. Small right pleural effusion with atelectasis versus developing airspace disease in the bilateral lung bases.")
# prompts.append("view_position = PA. modality = CR. ET tube in good position.")
# prompts.append("view_position = LATERAL. modality = CR. No findings.")

# inference parameters
# prompt = "cxr no finding"
guidance_scale = 4
num_steps = 75

# output directory
out_dir = "../output/results_2024_02_17/"
if not os.path.exists(out_dir): os.mkdir(out_dir)

# for i in tqdm(range(num_images)):
#     image = pipe(prompt, num_inference_steps=num_steps, guidance_scale=guidance_scale).images[0]
#     image.save(os.path.join(out_dir, f"synth_{i:04d}.png"))

for i, prompt in enumerate(prompts):
    for j in tqdm(range(num_images)):
        image = pipe(prompt, num_inference_steps=num_steps, guidance_scale=guidance_scale).images[0]
        # image.save(os.path.join(out_dir, f"prompt{i}_{j:03d}.png"))
        image.save(os.path.join(out_dir, f"{j:04d}.png"))
```

### train_text_to_image_lora.py

```python
# coding=utf-8
# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Fine-tuning script for Stable Diffusion for text2image with support for LoRA."""

import argparse
import logging
import math
import os
import random
from pathlib import Path

import datasets
import numpy as np
import torch
import torch.nn.functional as F
import torch.utils.checkpoint
import transformers
from accelerate import Accelerator
from accelerate.logging import get_logger
from accelerate.utils import ProjectConfiguration, set_seed
from datasets import load_dataset
from huggingface_hub import create_repo, upload_folder
from packaging import version
from torchvision import transforms
from tqdm.auto import tqdm
from transformers import CLIPTextModel, CLIPTokenizer

import diffusers
from diffusers import AutoencoderKL, DDPMScheduler, DiffusionPipeline, UNet2DConditionModel
from diffusers.loaders import AttnProcsLayers
from diffusers.models.attention_processor import LoRAAttnProcessor
from diffusers.optimization import get_scheduler
from diffusers.utils import check_min_version, is_wandb_available
from diffusers.utils.import_utils import is_xformers_available


# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
check_min_version("0.18.0.dev0")

logger = get_logger(__name__, log_level="INFO")


def save_model_card(repo_id: str, images=None, base_model=str, dataset_name=str, repo_folder=None):
    img_str = ""
    for i, image in enumerate(images):
        image.save(os.path.join(repo_folder, f"image_{i}.png"))
        img_str += f"![img_{i}](./image_{i}.png)\n"

    yaml = f"""
---
license: creativeml-openrail-m
base_model: {base_model}
tags:
- stable-diffusion
- stable-diffusion-diffusers
- text-to-image
- diffusers
- lora
inference: true
---
    """
    model_card = f"""
# LoRA text2image fine-tuning - {repo_id}
These are LoRA adaption weights for {base_model}. The weights were fine-tuned on the {dataset_name} dataset. You can find some example images in the following. \n
{img_str}
"""
    with open(os.path.join(repo_folder, "README.md"), "w") as f:
        f.write(yaml + model_card)


def parse_args():
    parser = argparse.ArgumentParser(description="Simple example of a training script.")
    parser.add_argument(
        "--pretrained_model_name_or_path",
        type=str,
        default=None,
        required=True,
        help="Path to pretrained model or model identifier from huggingface.co/models.",
    )
    parser.add_argument(
        "--revision",
        type=str,
        default=None,
        required=False,
        help="Revision of pretrained model identifier from huggingface.co/models.",
    )
    parser.add_argument(
        "--dataset_name",
        type=str,
        default=None,
        help=(
            "The name of the Dataset (from the HuggingFace hub) to train on (could be your own, possibly private,"
            " dataset). It can also be a path pointing to a local copy of a dataset in your filesystem,"
            " or to a folder containing files that 🤗 Datasets can understand."
        ),
    )
    parser.add_argument(
        "--dataset_config_name",
        type=str,
        default=None,
        help="The config of the Dataset, leave as None if there's only one config.",
    )
    parser.add_argument(
        "--train_data_dir",
        type=str,
        default=None,
        help=(
            "A folder containing the training data. Folder contents must follow the structure described in"
            " https://huggingface.co/docs/datasets/image_dataset#imagefolder. In particular, a `metadata.jsonl` file"
            " must exist to provide the captions for the images. Ignored if `dataset_name` is specified."
        ),
    )
    parser.add_argument(
        "--image_column", type=str, default="image", help="The column of the dataset containing an image."
    )
    parser.add_argument(
        "--caption_column",
        type=str,
        default="text",
        help="The column of the dataset containing a caption or a list of captions.",
    )
    parser.add_argument(
        "--validation_prompt", type=str, default=None, help="A prompt that is sampled during training for inference."
    )
    parser.add_argument(
        "--num_validation_images",
        type=int,
        default=4,
        help="Number of images that should be generated during validation with `validation_prompt`.",
    )
    parser.add_argument(
        "--validation_epochs",
        type=int,
        default=1,
        help=(
            "Run fine-tuning validation every X epochs. The validation process consists of running the prompt"
            " `args.validation_prompt` multiple times: `args.num_validation_images`."
        ),
    )
    parser.add_argument(
        "--max_train_samples",
        type=int,
        default=None,
        help=(
            "For debugging purposes or quicker training, truncate the number of training examples to this "
            "value if set."
        ),
    )
    parser.add_argument(
        "--output_dir",
        type=str,
        default="sd-model-finetuned-lora",
        help="The output directory where the model predictions and checkpoints will be written.",
    )
    parser.add_argument(
        "--cache_dir",
        type=str,
        default=None,
        help="The directory where the downloaded models and datasets will be stored.",
    )
    parser.add_argument("--seed", type=int
[truncated — 34290 more characters]
```