# Project export: XrayVision
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: Read You X-rays, Right Away
- Devpost: https://devpost.com/software/xrayvision-dgblcq
- GitHub: https://github.com/loufay/XrayVision
- Video: https://www.youtube.com/embed/i7mnssfbFjQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — lfay (30 commits), Clemence Mottez (21 commits)
## Devpost submission (written by the team)
### Inspiration
Hack the world a better place! - This motto drives our vision to make medical imaging more accessible, trustworthy, and reassuring for everyone, regardless of their background. Around the world, many patients - especially in less wealthy areas - face long waits for expert analysis, leading to anxiety and uncertainty. By using multiple AI models, we provide an instant, diverse opinions, increasing trust and transparency in AI-assisted healthcare. We aim to support the UN goals for health (Goal 3) and equality (Goal 10) by ensuring that people from all demographics, have better access to medical insights. We aim to empower patients with clarity, multiple perspectives, and a greater sense of safety in their healthcare journey.
### What it does
XrayVision is a web app that empowers patients by providing instant diagnostic insights from multiple AI models. By leveraging an ecosystem of diverse AI models, we ensure greater trust, reduced uncertainty, and more reliable results compared to a single-model approach. Our platform not only detects diseases and pathologies but also offers AI-driven explanations, generates comprehensive reports, and provides personalized recommendations for next steps. It’s designed to be a supportive tool for gaining feedback or a second opinion.
### How we built it
Our team developed XrayVision by combining cutting-edge AI technologies with robust software engineering principles to create a seamless and reliable platform. The front-end and back-end are built in Python using Streamlit, ensuring a user-friendly and efficient interface. We integrated multiple state-of-the-art foundation models (FM), trained on billions of text and image pairs—including models from Mistral-AI and Perplexity—to enhance diagnostic accuracy and trust. To power our AI models, we leverage the NVIDIA GeForce RTX 3090 (24GB), enabling high-performance inference for real-time disease detection and explanation.
### Challenges we ran into
Balancing high performance and real-time processing capabilities without compromising user experience was complex. We are new to front-end development, so creating a front-end which shows all features in a nice way took more time than expected. Combining models that have different environment requirements
### Accomplishments we're proud of
our teamwork and especially the result of our project. the development of the first patient-focused app that integrates an ecosystem of multiple foundation models to detect diseases from chest x-rays, but even more important help a patient to understand complicated reports and diagnosis We could test our platform on a friend's x-ray and IT WORKED -> disease detection and localization :) Contributing to global health with cutting edge technologies as our platform is easy accessible on any device.
### What we learned
Teamwork makes the dream work! <3 Push & Pull even more often if you work at the same time on the same project.
### What's next
While XrayVision is currently focusing on analysis of chest x-rays, the next step will go one step further from 2D to 3D, we want to include CT and MRI disease analysis.
## README (from the GitHub repository)
# XRayVision
## Hack the World a Better Place!
### 🌲 TREEHACKS - 2025
XRayVision is a cutting-edge web app designed to empower patients by providing instant diagnostic insights from multiple AI models. By harnessing an ecosystem of diverse AI-powered analyses, we enhance trust, reduce uncertainty, and deliver more reliable results than any single-model approach. Our platform not only detects diseases and pathologies but also provides AI-driven explanations, generates comprehensive reports, and offers personalized recommendations for next steps. Whether seeking a second opinion or validating AI-driven diagnostics, XRayVision is here to support you.
---
## AI Models in Use
- **Mistral-AI (Mistral-7B-Instruct-v0.3)** → Radiology report explanation
- **Perplexity** → Personalized recommendations based on detected diseases
- **TorchXRayVision, CheXagent, MedImageInsight** → Disease prediction
- **CheXagent** → Disease localization & radiology report generation
---
## Our Motivation
**Hack the world a better place!** This is the driving force behind our mission to make medical imaging more accessible, trustworthy, and reassuring for everyone, regardless of their background. Across the globe, many patients—especially in underprivileged regions—face long waits for expert analysis, leading to stress and uncertainty. By leveraging multiple AI models, we provide instant and diverse medical insights, fostering trust and transparency in AI-assisted healthcare.
We proudly align with the **UN Sustainable Development Goals**, particularly **Goal 3 (Good Health & Well-Being)** and **Goal 10 (Reduced Inequalities)**, ensuring that people from all demographics gain better access to critical medical insights.
**XRayVision empowers patients with clarity, multiple perspectives, and a greater sense of security in their healthcare journey.**
---
## Detected evidence (automated analysis)
Indexed codebase: 53 recognized source files, 271 KB.
- Hugging Face (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- Streamlit (technology) — detected in the code
- TensorFlow (technology) — detected in the code
- Mistral AI (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (54 of 54)
```
environment.yml
README.md
requirements.txt
src/main.py
src/models/CheXagent/chexagent.py
src/models/MedImageInsight/Distributed/__init__.py
src/models/MedImageInsight/Distributed/Utils.py
src/models/MedImageInsight/ImageDataLoader/__init__.py
src/models/MedImageInsight/ImageDataLoader/blob_storage.py
src/models/MedImageInsight/ImageDataLoader/build.py
src/models/MedImageInsight/ImageDataLoader/constants.py
src/models/MedImageInsight/ImageDataLoader/languages/__init__.py
src/models/MedImageInsight/ImageDataLoader/languages/prompt_engineering.py
src/models/MedImageInsight/ImageDataLoader/transforms/__init__.py
src/models/MedImageInsight/ImageDataLoader/transforms/autoaugment.py
src/models/MedImageInsight/ImageDataLoader/transforms/build.py
src/models/MedImageInsight/ImageDataLoader/transforms/threeaugment.py
src/models/MedImageInsight/ImageDataLoader/tsv_file.py
src/models/MedImageInsight/ImageDataLoader/tsv.py
src/models/MedImageInsight/ImageDataLoader/zipdata.py
src/models/MedImageInsight/ImageEncoder/__init__.py
src/models/MedImageInsight/ImageEncoder/build.py
src/models/MedImageInsight/ImageEncoder/coswin.py
src/models/MedImageInsight/ImageEncoder/davit_v1.py
src/models/MedImageInsight/ImageEncoder/registry.py
src/models/MedImageInsight/LangEncoder/__init__.py
src/models/MedImageInsight/LangEncoder/build.py
src/models/MedImageInsight/LangEncoder/registry.py
src/models/MedImageInsight/LangEncoder/transformer.py
src/models/MedImageInsight/Utils/__init__.py
src/models/MedImageInsight/Utils/Arguments.py
src/models/MedImageInsight/Utils/GeneraUtils.py
src/models/MedImageInsight/Utils/GlobalExceptHook.py
src/models/MedImageInsight/Utils/MPIAdapter.py
src/models/MedImageInsight/Utils/Utils.py
src/models/medimageinsightmodel.py
src/models/model_loader.py
src/models/UniCLModel.py
src/perform_disease_localization.py
src/perform_disease_prediction_chexagent.py
src/perform_disease_prediction_mi2.py
src/perform_disease_prediction_torchxrayvision.py
src/perform_disease_recommendation_perplexity.py
src/perform_report_explanation_mistral.py
src/perform_report_generation_chexagent.py
src/perform_report_structuring_gpt.py
src/webapp/tab1_welcome.py
src/webapp/tab2_personalize.py
src/webapp/tab3_xray_analysis.py
src/webapp/tab4_locate_disease.py
src/webapp/tab4_radreport.py
src/webapp/tab5_radreport.py
src/webapp/utils.py
webapp.py
```
### Dependencies
- requirements.txt: absl-py@==2.1.0, aiohappyeyeballs@==2.4.3, aiohttp@==3.10.10, aiosignal@==1.3.1, altair@==5.4.1, asttokens@@ file:///home/conda/feedstock_root/build_artifacts/asttokens_1733175639022/work, astunparse@==1.6.3, async-timeout@==4.0.3, attrs@==24.2.0, backcall@@ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work, beautifulsoup4@==4.12.3, bleach@==6.1.0, blinker@==1.8.2, cachetools@@ file:///home/conda/feedstock_root/build_artifacts/cachetools_1724028158384/work, certifi@==2024.8.30, charset-normalizer@==3.4.0, click@==8.1.7, cloudpickle@==3.1.0, comm@@ file:///home/conda/feedstock_root/build_artifacts/comm_1710320294760/work, conda@==4.3.16, cssselect@==1.2.0, cssutils@==2.11.1, cubinlinker@@ file:///home/conda/feedstock_root/build_artifacts/cubinlinker_1684753555810/work, cuda-python@@ file:///opt/conda/conda-bld/cuda-python_1684125314189/work, cudf@==23.4.1, cupy@@ file:///home/conda/feedstock_root/build_artifacts/cupy_1677786719874/work, cycler@==0.12.1, dask@==2023.5.0, dataframe_image@==0.2.6, debugpy@@ file:///home/conda/feedstock_root/build_artifacts/debugpy_1722923746907/work, decorator@@ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work, defusedxml@==0.7.1, distributed@==2023.5.0, docker-pycreds@==0.4.0, et-xmlfile@==1.1.0, executing@@ file:///home/conda/feedstock_root/build_artifacts/executing_1725214404607/work, fastavro@@ file:///home/conda/feedstock_root/build_artifacts/fastavro_1720472303091/work, fastjsonschema@==2.20.0, fastrlock@@ file:///home/conda/feedstock_root/build_artifacts/fastrlock_1702696317520/work, filelock@==3.16.1, flatbuffers@==24.3.25, frozenlist@==1.5.0, fsspec@@ file:///home/conda/feedstock_root/build_artifacts/fsspec_1729608855534/work, gast@==0.4.0, gitdb@==4.0.11, GitPython@==3.1.43, google-auth@==2.35.0, google-auth-oauthlib@==1.0.0, google-pasta@==0.2.0, grpcio@==1.67.0, h5py@==3.11.0, html2image@==2.0.5, huggingface-hub@==0.28.1, idna@==3.10, imageio@==2.35.1, imbalanced-learn@==0.12.4, imblearn@==0.0, importlib_metadata@@ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1726082825846/work, importlib_resources@==6.4.5, ipykernel@@ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1719845459717/work, ipython@@ file:///home/conda/feedstock_root/build_artifacts/ipython_1683289033986/work, jedi@@ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work, Jinja2@==3.1.4, joblib@==1.4.2, jsonschema@==4.23.0, jsonschema-specifications@==2023.12.1, jupyter_client@@ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1726610684920/work, jupyter_core@@ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1727163409502/work, jupyterlab_pygments@==0.3.0, keras@==2.13.1, kiwisolver@==1.4.7, lazy_loader@==0.4, libclang@==18.1.1, lightning-utilities@==0.11.8, llvmlite@==0.39.1, locket@==1.0.0, lxml@==5.3.0, Markdown@==3.7, markdown-it-py@==3.0.0, MarkupSafe@==2.1.5, matplotlib-inline@@ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1713250518406/work, mdurl@==0.1.2, mistune@==3.0.2, more-itertools@==10.5.0, mpmath@==1.3.0, msgpack@==1.1.0, multidict@==6.1.0, narwhals@==1.26.0, nbclient@==0.10.0, nbconvert@==7.16.4, nbformat@==5.10.4, nest_asyncio@@ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work, networkx@==3.1, numba@@ file:///home/conda/feedstock_root/build_artifacts/numba_1680825382026/work, numpy@==1.24.4, nvidia-cublas-cu12@==12.1.3.1, nvidia-cuda-cupti-cu12@==12.1.105, nvidia-cuda-nvrtc-cu12@==12.1.105, nvidia-cuda-runtime-cu12@==12.1.105, nvidia-cudnn-cu12@==9.1.0.70, nvidia-cufft-cu12@==11.0.2.54, nvidia-curand-cu12@==10.3.2.106, nvidia-cusolver-cu12@==11.4.5.107, nvidia-cusparse-cu12@==12.1.0.106, nvidia-nccl-cu12@==2.20.5, nvidia-nvjitlink-cu12@==12.6.77, nvidia-nvtx-cu12@==12.1.105, nvtx@@ file:///home/conda/feedstock_root/build_artifacts/nvtx_1708093799057/work, oauthlib@==3.2.2, openpyxl@==3.1.5, opt_einsum@==3.4.0, packaging@@ file:///home/conda/feedstock_root/build_artifacts/packaging_1733203243479/work, pandas@==2.0.3, pandocfilters@==1.5.1, parso@@ file:///home/conda/feedstock_root/build_artifacts/parso_1712320355065/work, partd@==1.4.1, pexpect@@ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work, pickleshare@@ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work, pillow@==10.4.0, pkgutil_resolve_name@==1.3.10, platformdirs@@ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1726613481435/work, plotly@==6.0.0, prompt_toolkit@@ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1727341649933/work, propcache@==0.2.0, protobuf@==4.25.5, psutil@@ file:///home/conda/feedstock_root/build_artifacts/psutil_1719274595110/work, ptxcompiler@@ file:///home/conda/feedstock_root/build_artifacts/ptxcompiler_1697578507378/work, ptyprocess@@ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl, pure_eval@@ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1721585709575/work, pyarrow@==10.0.1, pyasn1@==0.6.1, pyasn1_modules@==0.4.1, pycosat@==0.6.6, pydeck@==0.9.1, Pygments@@ file:///home/conda/feedstock_root/build_artifacts/pygments_1714846767233/work, pyparsing@==3.1.4, python-dateutil@@ file:///croot/python-dateutil_1716495738603/work, pytorch-lightning@==2.4.0, pytz@@ file:///home/conda/feedstock_root/build_artifacts/pytz_1726055524169/work, PyWavelets@==1.4.1, PyYAML@==6.0.2, pyzmq@@ file:///croot/pyzmq_1705605076900/work, referencing@==0.35.1, regex@==2024.11.6, requests@==2.32.3, requests-oauthlib@==2.0.0, rich@==13.9.4, rmm@==23.4.1, rpds-py@==0.20.1, rsa@==4.9, ruamel.yaml@==0.18.6, ruamel.yaml.clib@==0.2.8, safetensors@==0.5.2, scikit-image@==0.21.0, scikit-learn@==1.3.2, scipy@==1.10.1, seaborn@==0.13.2, sentry-sdk@==2.17.0, setproctitle@==1.3.3, shap@==0.44.1, six@@ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work, slicer@==0.0.7, smmap@==5.0.1, sortedcontainers@==2.4.0, soupsieve@==2.6, stack-data@@ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work, streamlit@==1.40.1, sympy@==1.13.3, tblib@==3.0.0, tenacity@==9.0.0, tensorboard@==2.13.0, tensorboard-data-server@==0.7.2, tensorboardX@==2.6.2.2, tensorflow@==2.13.1, tensorflow-estimator@==2.13.0, tensorflow-io-gcs-filesystem@==0.34.0, termcolor@==2.4.0, threadpoolctl@==3.5.0, tifffile@==2023.7.10, tinycss2@==1.4.0, tokenizers@==0.20.3, toml@==0.10.2, toolz@==1.0.0, torch@==2.4.1, torchmetrics@==1.5.1, torchvision@==0.19.1, torchxrayvision@==1.2.4, tornado@@ file:///home/conda/feedstock_root/build_artifacts/tornado_1717722826518/work, tqdm@==4.66.5, traitlets@@ file:///home/conda/feedstock_root/build_artifacts/traitlets_1713535121073/work, transformers@==4.46.3, triton@==3.0.0, typing_extensions@==4.12.2, tzdata@==2025.1, urllib3@==2.2.3, wandb@==0.18.5, watchdog@==4.0.2, wcwidth@@ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work, webencodings@==0.5.1, websocket-client@==1.8.0, Werkzeug@==3.0.6, wrapt@==1.16.0, xgboost@==2.1.2, yarl@==1.15.2, zict@==3.0.0, zipp@@ file:///home/conda/feedstock_root/build_artifacts/zipp_1731262100163/work
### Recent commits (newest first)
- update
- Merge branch 'main' of https://github.com/loufay/aiXperts into main
- update
- Update README.md
- Update README.md
- tab1
- Update tab3_xray_analysis.py
- Merge branch 'main' of https://github.com/loufay/aiXperts
- update
- Merge branch 'main' of https://github.com/loufay/aiXperts
- push
- update tab1
- update on main
- update tab1
- update tab3
- update
- Merge branch 'main' of https://github.com/loufay/aiXperts
- Create perform_disease_recommendation_perplexity.py
- Merge branch 'main' of https://github.com/loufay/aiXperts into main
- update
## Key source files (fetched from GitHub, selected and truncated for size)
### requirements.txt
```
absl-py==2.1.0
aiohappyeyeballs==2.4.3
aiohttp==3.10.10
aiosignal==1.3.1
altair==5.4.1
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1733175639022/work
astunparse==1.6.3
async-timeout==4.0.3
attrs==24.2.0
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
beautifulsoup4==4.12.3
bleach==6.1.0
blinker==1.8.2
cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1724028158384/work
certifi==2024.8.30
charset-normalizer==3.4.0
click==8.1.7
cloudpickle==3.1.0
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1710320294760/work
conda==4.3.16
cssselect==1.2.0
cssutils==2.11.1
cubinlinker @ file:///home/conda/feedstock_root/build_artifacts/cubinlinker_1684753555810/work
cuda-python @ file:///opt/conda/conda-bld/cuda-python_1684125314189/work
cudf==23.4.1
cupy @ file:///home/conda/feedstock_root/build_artifacts/cupy_1677786719874/work
cycler==0.12.1
dask==2023.5.0
dataframe_image==0.2.6
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1722923746907/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml==0.7.1
distributed==2023.5.0
docker-pycreds==0.4.0
et-xmlfile==1.1.0
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1725214404607/work
fastavro @ file:///home/conda/feedstock_root/build_artifacts/fastavro_1720472303091/work
fastjsonschema==2.20.0
fastrlock @ file:///home/conda/feedstock_root/build_artifacts/fastrlock_1702696317520/work
filelock==3.16.1
flatbuffers==24.3.25
frozenlist==1.5.0
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1729608855534/work
gast==0.4.0
gitdb==4.0.11
GitPython==3.1.43
google-auth==2.35.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
grpcio==1.67.0
h5py==3.11.0
html2image==2.0.5
huggingface-hub==0.28.1
idna==3.10
imageio==2.35.1
imbalanced-learn==0.12.4
imblearn==0.0
importlib_metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1726082825846/work
importlib_resources==6.4.5
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1719845459717/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1683289033986/work
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
Jinja2==3.1.4
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1726610684920/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1727163409502/work
jupyterlab_pygments==0.3.0
keras==2.13.1
kiwisolver==1.4.7
lazy_loader==0.4
libclang==18.1.1
lightning-utilities==0.11.8
llvmlite==0.39.1
locket==1.0.0
lxml==5.3.0
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1713250518406/work
mdurl==0.1.2
mistune==3.0.2
more-itertools==10.5.0
mpmath==1.3.0
msgpack==1.1.0
multidict==6.1.0
narwhals==1.26.0
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work
networkx==3.1
numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1680825382026/work
numpy==1.24.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.77
nvidia-nvtx-cu12==12.1.105
nvtx @ file:///home/conda/feedstock_root/build_artifacts/nvtx_1708093799057/work
oauthlib==3.2.2
openpyxl==3.1.5
opt_einsum==3.4.0
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1733203243479/work
pandas==2.0.3
pandocfilters==1.5.1
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1712320355065/work
partd==1.4.1
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
pillow==10.4.0
pkgutil_resolve_name==1.3.10
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1726613481435/work
plotly==6.0.0
prompt_toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1727341649933/work
propcache==0.2.0
protobuf==4.25.5
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1719274595110/work
ptxcompiler @ file:///home/conda/feedstock_root/build_artifacts/ptxcompiler_1697578507378/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure_eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1721585709575/work
pyarrow==10.0.1
pyasn1==0.6.1
pyasn1_modules==0.4.1
pycosat==0.6.6
pydeck==0.9.1
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1714846767233/work
pyparsing==3.1.4
python-dateutil @ file:///croot/python-dateutil_1716495738603/work
pytorch-lightning==2.4.0
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1726055524169/work
PyWavelets==1.4.1
PyYAML==6.0.2
pyzmq @ file:///croot/pyzmq_1705605076900/work
referencing==0.35.1
regex==2024.11.6
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.9.4
rmm==23.4.1
rpds-py==0.20.1
rsa==4.9
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
safetensors==0.5.2
scikit-image==0.21.0
scikit-learn==1.3.2
scipy==1.10.1
seaborn==0.13.2
sentry-sdk==2.17.0
setproctitle==1.3.3
shap==0.44.1
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
slicer==0.0.7
smmap==5.0.1
sortedcontainers==2.4.0
soupsieve==2.6
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
streamlit==1.40.1
sympy==1.13.3
tblib==3.0.0
tenacity==9.0.0
tensorboard==2.13.0
tensorboard-data-server==
[truncated — 902 more characters]
```
### src/main.py
```python
import streamlit as st
import pandas as pd
import plotly.express as px
import numpy as np
from PIL import Image
#from perform_report_explanation_mistral import explain_radiology_report
from webapp.tab1_welcome import tab1_welcome
from webapp.tab2_personalize import tab2_add_personal_data
from webapp.tab3_xray_analysis import tab3_xray_analysis
from webapp.tab4_locate_disease import tab4_locate_disease
from webapp.tab5_radreport import tab5_radreport
# Init all models
st.set_page_config(page_title="Main", page_icon=":computer:", layout="wide")
def main():
# CSS to inject custom styles
st.markdown("""
""", unsafe_allow_html=True)
st.markdown("Welcome to XRayVision
", unsafe_allow_html=True)
st.markdown("Read Your X-rays, Right Away 🩺
", unsafe_allow_html=True)
# Tabs
tabs = st.tabs(["Home", "Personal Information", "Read Your X-rays","Locate Disease", "Explain Report"])
tab1, tab2, tab3, tab4, tab5 = tabs
with tab1:
tab1_welcome()
with tab2:
tab2_add_personal_data()
with tab3:
tab3_xray_analysis()
with tab4:
tab4_locate_disease()
with tab5:
tab5_radreport()
if __name__ == "__main__":
main()
```
### webapp.py
```python
import streamlit as st
import plotly.express as px
import numpy as np
def main():
st.set_page_config(page_title="aiXpert", page_icon=":computer:", layout="wide")
st.markdown("aiXpert
", unsafe_allow_html=True)
# Create Tabs
tabs = st.tabs(["Home", "Data Distribution Explorer", "Data Correlation Explorer"])
# Access each tab using indexing
with tabs[0]: # Home tab
st.markdown("## Welcome to the Treehack!")
st.markdown("2025")
st.markdown("C+L <3 hi")
if __name__ == "__main__":
if "processComplete" not in st.session_state:
st.session_state["processComplete"] = None
main()
```
### environment.yml
```yaml
name: treeHack
name: medical-dl
channels:
- conda-forge
- defaults
dependencies:
- python=3.8 # Python version, ensure compatibility with all libraries
- numpy
- pandas
- matplotlib # For plotting and visualizations
- scikit-learn # For machine learning utilities
- scipy # For scientific computing tools
# Deep Learning Frameworks
- tensorflow-gpu=2.9 # or just tensorflow if not using GPU
- keras
# Medical image specific libraries
- pydicom # Handling DICOM files
- SimpleITK # For advanced image processing
- opencv # For general image processing tasks
# Visualization of medical images
- matplotlib
- seaborn # Enhanced visualization features
# For data augmentation in deep learning
- imgaug # Augmentation for image data
- albumentations # Fast augmentation library
# Additional tools for better experimentation
- jupyter
- ipykernel
- nb_conda # Manage conda environments in Jupyter notebooks
# Other useful libraries
- h5py # To handle large image data files efficiently
- pillow # Image processing library
# Environment management
- pip
- pip:
- tensorflow-addons>=0.15.0 # For additional functionalities in TensorFlow
- efficientnet # Pretrained models for deep learning
```
### src/perform_disease_localization.py
```python
import io
import requests
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from models.CheXagent.chexagent import CheXagent
# huggingface-cli login
# if other gpu set torch.float16 back to torch.bfloat16
def perform_disease_localization(path_to_image, disease=""):
##TODO: Remove path
# path_to_image = "/mnt/data2/datasets_lfay/MedImageInsights/data/CheXpert-v1.0-512/images/train/patient04905/study4/view1_frontal.jpg"
if path_to_image is not None:
# Save it to a temp file
with open(path_to_image.name, "wb") as f:
f.write(path_to_image.getbuffer())
file_path = path_to_image.name
print(file_path)
chexagent = CheXagent()
phrase = f"There is a {disease}."
response = chexagent.phrase_grounding(file_path, phrase)
print(f'Result: {response}')
print(f'=' * 42)
if __name__ == "__main__":
perform_disease_localization(path_to_image="")
```
### src/perform_disease_recommendation_perplexity.py
```python
import requests
def perform_disease_recommendation_perplexity(message):
api_key = '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'
url = "https://api.perplexity.ai/chat/completions"
payload = {
"model": "sonar-pro",
"messages": [
{
"role": "user",
"content": message
}
],
"max_tokens": 600,
"temperature": 0.2,
"top_p": 0.9,
"search_domain_filter": None,
"return_images": False,
"return_related_questions": False,
# "search_recency_filter": "",
"top_k": 0,
"stream": False,
"presence_penalty": 0,
"frequency_penalty": 1,
"response_format": None
}
headers = {
"Authorization": "Bearer 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",
# "Content-Type": "application/json"
}
response = requests.request("POST", url, json=payload, headers=headers)
print(response.json()['choices'][0]['message']['content'])
return response.json()['choices'][0]['message']['content']
```
### src/perform_report_generation_chexagent.py
```python
import io
import requests
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from models.CheXagent.chexagent import CheXagent
# huggingface-cli login
# if other gpu set torch.float16 back to torch.bfloat16
def perform_report_generation_chexagent(path_to_image):
##TODO: Remove path
# path_to_image = "../data/mimic.jpg"
print(path_to_image)
file_path = path_to_image
if path_to_image is not None:
# Save it to a temp file
with open(path_to_image.name, "wb") as f:
f.write(path_to_image.getbuffer())
file_path = path_to_image.name
print(file_path)
chexagent = CheXagent()
responses = chexagent.findings_generation_section_by_section([file_path])
result_string = ""
for response in responses:
result_string += f'{response[0]}: {response[1]}\n'
return result_string
if __name__ == "__main__":
perform_report_generation_chexagent(path_to_image="")
```
### src/perform_report_structuring_gpt.py
```python
# install pip install openai streamlit
import os
from openai import OpenAI
API_KEY = os.getenv("OPENAI_API_KEY")
client = OpenAI(api_key=API_KEY)
def structure_radiology_report(report_text):
prompt = f"""
You are an expert radiologist assistant. Structure the following radiology report into clear sections:
1. **Findings** - Describe detailed observations from the X-ray.
2. **Impression** - Summarize the key takeaways in a clinical manner.
3. **Recommendations** (if applicable) - Suggest any next steps.
Report:
{report_text}
Return only the structured text.
"""
response = openai.ChatCompletion.create(
model="gpt-4o",
messages=[{"role": "system", "content": "You are a radiology AI assistant."},
{"role": "user", "content": prompt}],
api_key=api_key
)
return response["choices"][0]["message"]["content"]
# Example usage
report_text = """There is evidence of right lower lobe consolidation with air bronchograms, suggestive of pneumonia.
No pleural effusion is seen. The cardiac silhouette is normal in size."""
structured_report = structure_radiology_report(report_text)
print(structured_report)
```
### src/perform_disease_prediction_chexagent.py
```python
import io
import requests
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from models.CheXagent.chexagent import CheXagent
import streamlit as st
# huggingface-cli login
# if other gpu set torch.float16 back to torch.bfloat16
@st.cache_resource
def load_chexagent():
return CheXagent()
def perform_disease_prediction_chexagent(path_to_image):
if path_to_image is not None:
# Save it to a temp file
with open(path_to_image.name, "wb") as f:
f.write(path_to_image.getbuffer())
file_path = path_to_image.name
print(file_path)
# chexagent = CheXagent()
chexagent = load_chexagent()
diseases = [
"Enlarged Cardiomediastinum",
"Cardiomegaly",
"Lung Opacity","Lung Lesion",
"Edema","Consolidation","Pneumonia","Atelectasis","Pneumothorax","Pleural Effusion","Fracture","Support Devices"]
#response = model.binary_disease_classification([path_to_image], "Pneumothorax")
prompt = f'Does this chest X-ray contain a Pneumothorax?'
response = chexagent.disease_identification([file_path], diseases)
# create dict with all diseases set to 1 if present in response else 0
results = {}
for disease in diseases:
results[disease] = 1 if disease in response else 0
if all(value == 0 for value in results.values()):
results['No Finding'] = 1
else:
results['No Finding'] = 0
# responses = chexagent.findings_generation_section_by_section([path_to_image])
print(results)
print("XXXX"*20)
return results
if __name__ == "__main__":
perform_disease_prediction_chexagent(path_to_image="")
```
### src/perform_disease_prediction_mi2.py
```python
from models.medimageinsightmodel import MedImageInsight
import os
import sys
current_dir = os.getcwd()
current_dir = current_dir + "/aiXperts/src/models"
sys.path.append(current_dir)
from PIL import Image
import base64
import io
def read_image(image_path):
with open(image_path, "rb") as f:
return f.read()
def load_model():
classifier = MedImageInsight(
model_dir=os.path.join(current_dir, "MedImageInsight/2024.09.27"),
vision_model_name="medimageinsigt-v1.0.0.pt",
language_model_name="language_model.pth"
)
classifier.load_model()
classifier.model.to(classifier.device)
classifier.model.eval()
return classifier
def perform_disease_prediction_mi2(path_to_image):
# Load model
classifier = load_model()
# Read image
print(path_to_image)
image = Image.open(path_to_image)
# Convert the image to bytes
buffered = io.BytesIO()
image.save(buffered, format="JPEG")
img_byte = buffered.getvalue()
# Encode to base64
img_base64 = base64.encodebytes(img_byte)
# If you need it as a string
img_base64_str = img_base64.decode("utf-8")
disease = [
"Enlarged Cardiomediastinum",
"Cardiomegaly",
"Lung Opacity","Lung Lesion",
"Edema","Consolidation","Pneumonia","Atelectasis","Pneumothorax","Pleural Effusion","Fracture","Support Devices"]
results = {}
for disease in disease:
labels = ["normal", disease, "unclear"]
result = classifier.predict([img_base64_str], labels)
disease_present = 1 if max(result[0], key=result[0].get)==disease else 0
results[disease] = disease_present
if all(value == 0 for value in results.values()):
results['No Finding'] = 1
else:
results['No Finding'] = 0
##TODO: Remove print statements
#print result keys with value 1
# print([key for key, value in results.items() if value == 1])
return results
if __name__ == "__main__":
perform_disease_prediction_mi2(path_to_image="../data/42142.jpg")
```
[42 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]