# Project export: LUNA This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: We built a tool that predicts skin disease by combining skin images with blood transcriptomics, integrating visual and molecular signals in a clinician-friendly web app for more precise diagnosis. - Devpost: https://devpost.com/software/luna-7fe8xm - GitHub: https://github.com/isha-arora-02/LUNA - Team: 3 GitHub contributor(s) — isha-arora-02 (22 commits), cprovido (11 commits), ahowley1 (1 commits) ## Devpost submission (written by the team) ### Overview About the Project Skin rashes and lesions are notoriously difficult to diagnose. Many conditions, including dermatitis, psoriasis, melanoma, and systemic lupus erythematosus (SLE), present with overlapping visual features, even for experienced clinicians. To improve diagnostic precision, we pursued a novel approach that pairs visual symptoms with molecular signals from the blood. Rather than relying on images alone, we built a multimodal AI system that integrates skin photographs with blood-based transcriptomic measurements to predict whether a patient has a skin-related disease. Proof-of-Concept Website: click here! How We Built It Image Understanding with Foundation Models We leveraged a powerful pretrained dermatology vision-language model, MONET, built on a CLIP ViT-L/14 architecture. MONET was trained on over 105,000 dermatological images paired with medical text descriptions, enabling it to: Recognize dermatologic concepts with dermatologist-level accuracy Provide interpretable visual representations Maintain transparency throughout the AI pipeline This provided a strong visual encoder capable of extracting meaningful disease-related features from skin images. Transcriptomics Integration To complement the image data, we incorporated blood gene expression profiles. Rather than using the full transcriptome, which is high-dimensional and often noisy, we: Performed multicohort meta-analysis across multiple skin disease datasets Identified differentially expressed genes associated with disease Restricted the transcriptomic input to biologically relevant genes Mathematically, instead of learning from: \( X \in \mathbb{R}^{n \times G} \) where \(G\) represents the full set of measured genes, we trained on: \( X_{\text{filtered}} \in \mathbb{R}^{n \times g}, \quad g \ll G \) This improved both model efficiency and biological signal quality. Multimodal Fusion We integrated: Visual embeddings derived from MONET Gene expression features from blood transcriptomics into a joint predictive model. We then deployed the system through a user-friendly web interface that allows clinicians to upload images, input transcriptomic data, and receive disease predictions. Challenges We Faced Transcriptomic Data Quality One of the primary challenges was identifying usable public gene expression datasets. Many microarray datasets were: Poorly normalized Inconsistent across samples Not directly comparable across studies Reprocessing raw data from scratch exceeded the time constraints of TreeHacks. To address this, we prioritized datasets that were already properly processed and validated, enabling rapid integration while maintaining data quality. Additionally, because our imaging and transcriptomic data were not paired at the patient level, we carefully designed the neural network architecture to effectively learn multimodal representations despite this limitation. What We Learned Each member ventured to learn a new data implementation approach in this project. We describe below the novelty of multimodal datasets in clinical applications as the use of mRNA-based gene expression data is biologically rich yet undervalued in diagnostics. Multimodal models capture complementary biological signals more effectively than single-modality approaches Data preprocessing quality is critical to downstream model performance Unpaired data samples still hold predictive power at the cost of describing biomedical features separately and distinctively. Model architecture is critical in extracting meaningful topological features. We derive inspiration from the architecture of a recently described EHR-Omics prediction model [Matarso Nat Mach Intell, 2024] for Vision-Omics. Omics remains highly valuable. We believe future diagnostics will enhance the predictive ability of diagnosing correct disease conditions. Impact and Vision Most existing skin disease classifiers focus primarily on melanoma or skin cancer and rely exclusively on images. Rare diseases and autoimmune conditions remain underrepresented and difficult to diagnose using single data modalities. This diagnostic challenge often leads to prolonged diagnostic timelines spanning several years for many patients. Our integrative approach: Combines visual phenotypes with molecular biomarkers Improves diagnostic precision in ambiguous cases Provides a scalable framework for future multimodal medical AI systems We envision this platform as a clinical decision-support tool that enables more accurate, faster diagnoses and advances precision medicine. What’s Next for LUNA Our prototype demonstrates the promise of multimodal learning, but substantial opportunities remain to expand both the dataset and modeling capabilities. First, we will incorporate transcriptomic datasets that were excluded due to poor normalization. Since raw expression files are mandatory uploads in public repositories, we can retrieve and properly renormalize these data to ensure consistency across cohorts. Second, we will expand beyond microarrays, which measure gene expression via fluorescent probes, by integrating RNA sequencing (RNA-seq) data. RNA-seq directly quantifies mRNA transcript abundance and is increasingly prevalent in public datasets, enabling improved biological resolution and statistical power. In parallel, we will continue curating larger and more diverse skin image datasets. From a modeling perspective, we plan to conduct systematic hyperparameter tuning and explore architectural enhancements to improve multimodal fusion. Finally, we aim to translate this platform into a formal research study and potential clinical product, rigorously evaluating whether integrating skin imaging with blood transcriptomics significantly outperforms single-modality diagnostic approaches. ## README (from the GitHub repository) # LUNA ### A multi-omics driven prediction model for skin lesions and rashes. LUNA is a multimodal AI system that improves skin disease detection by integrating dermatologic images with blood-based transcriptomic data. Many conditions — including dermatitis, psoriasis, melanoma, and systemic lupus erythematosus — share overlapping visual features, making diagnosis challenging even for specialists. To address this, LUNA leverages MONET, a dermatology-specific vision-language foundation model built on OpenAI’s CLIP ViT-L/14 architecture, to extract clinically meaningful visual embeddings from skin images. These visual representations are combined with embeddings derived from publicly available transcriptomic datasets, which we refined through multicohort meta-analysis to isolate biologically meaningful gene expression signals and enhance downstream predictive performance. By fusing visual phenotypes with molecular biomarkers, LUNA moves beyond single-modality skin classifiers toward a more biologically grounded diagnostic framework. Despite working with unpaired imaging and transcriptomic datasets, we designed a neural architecture capable of learning shared multimodal representations and deployed it through a web interface for clinical interaction. LUNA represents a step toward scalable, precision-driven dermatological decision-support systems. ### To run predictions: Run predict.py, providing file paths for: * A skin image (e.g., .jpg or .png) * A transcriptomic data matrix (.csv) The model will process both inputs and output a predicted diagnosis based on the integrated multimodal features. Alternatively, visit our (proof-of-concept) website to upload the requisite files and obtain a diagnosis: https://luna-scan-view.base44.app/ ## Detected evidence (automated analysis) Indexed codebase: 12 recognized source files, 90 KB. - Python (language) — detected in the code - PyTorch (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (36 of 36) ``` best_model.pt diffexp_genes.py gene_expr_data/01_correct_sample_dims.ipynb gene_expr_data/final_expr.csv gene_expr_data/final_pheno.csv gene_expr_data/mergedExpr_old.csv gene_expr_data/mergedExpr.csv gene_expr_data/mergedPheno_old.csv gene_expr_data/mergedPheno.csv get_one_example.ipynb hopt_train.py image_data/dataset_scin_labels.csv image_data/final_sorted_samples-images.csv image_data/final_sorted_samples-images2.csv image_data/final_sorted_samples-images3.csv image_data/final_sorted_samples-images4.csv image_data/final_sorted_samples-images5.csv image_data/final_sorted_samples.csv image_data/scin_assigned_target_labels.csv image_data/scin_cases.csv image_data/scin_image_uris.txt load_monet.py myenv_cpu.yaml myenv_gpu.yaml old_files/load_monet.py old_files/README.md old_files/train.py plot_hopts-results.py predict.py preprocess_img_data/01_loading_datasets.ipynb preprocess_img_data/02_label_image_assignment.ipynb preprocess_img_data/03_process_assigned_data.ipynb preprocess_img_data/04_normalize-pickle_images.ipynb README.md test_data/example_test_dermatitis.csv train.py ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Add files via upload - Add files via upload - Add files via upload - Add files via upload - edits made by chris - updated files - Update README.md - test data - best model! super cool - best fitting scripts - moved files - preserving version history - updated files - truly final sample manifesto - updated gene expr files wo duplicates - move image data files - Add files via upload - move files - Image and metdata preprocessing!!! - save label map ## Key source files (fetched from GitHub, selected and truncated for size) ### myenv_cpu.yaml ```yaml name: luna channels: - pytorch - conda-forge - defaults dependencies: - python - pip - setuptools - numpy - pandas - scipy - scikit-learn - pytorch - torchvision - torchaudio - cpuonly - tokenizers - matplotlib - seaborn - jupyterlab - ipywidgets - ftfy - regex - transformers - tokenizers - huggingface_hub - pillow - pip: # Hugging Face - accelerate - timm # ViT backbone deps for MONET - opencv-python - albumentations # RNA-seq - anndata - scanpy - pydeseq2 - pickle # Loss - pytorch-metric-learning # Experiment tracking - wandb - tensorboard - tqdm - einops - torchmetrics - lightning # ISIC dataset access - isic-cli variables: TOKENIZERS_PARALLELISM: "false" # suppresses HuggingFace tokenizer warnings ``` ### myenv_gpu.yaml ```yaml name: luna channels: - pytorch - conda-forge - defaults dependencies: - python - pip - setuptools - numpy - pandas - scipy - scikit-learn - pytorch - torchvision - torchaudio - pytorch-cuda - tokenizers - matplotlib - seaborn - jupyterlab - ipywidgets - ftfy - regex - transformers - tokenizers - huggingface_hub - pillow - pip: # Hugging Face - accelerate - timm # ViT backbone deps for MONET - opencv-python - albumentations - accelerate - timm # ViT backbone deps for MONET - Pillow - opencv-python - albumentations # RNA-seq - anndata - scanpy - pydeseq2 - pickle # Loss - pytorch-metric-learning # Experiment tracking - wandb - tensorboard - tqdm - einops - torchmetrics - lightning # ISIC dataset access - isic-cli # SCIN dataset access - tabulate - google-cloud-storage variables: TOKENIZERS_PARALLELISM: "false" # suppresses HuggingFace tokenizer warnings ``` ### diffexp_genes.py ```python import scanpy as sc import pandas as pd import anndata as ad import numpy as np import torch # from pydeseq2.dds import DeseqDataSet # from pydeseq2.default_inference import DefaultInference # from pydeseq2.ds import DeseqStats def csv_to_anndata(fpath_expr, fpath_pheno): """ Convert CSV of expression and phenotype data into Anndata files. Parameters: fpath_expr: file path to expression data fpath_pheno: file path to pheno/metadata Returns: adata: anndata object with expression and metadata """ expr = pd.read_csv(fpath_expr).T pheno = pd.read_csv(fpath_pheno).T adata = ad.AnnData(np.array(expr)) adata.obs = pheno adata.obs_names = list(expr.index) adata.var_names = list(expr.columns) return adata def updated_expr_deseq(fpath_expr, fpath_pheno): """ Run differential gene expression on the expression matrix to obtain updated expression matrix. Keep top 768 diff expressed genes. Parameters: fpath_expr: file path to expression data fpath_pheno: file path to pheno/metadata Returns: adata: anndata object with expression and metadata torch_X: X matrix in torch torch_y: y matrix in torch """ adata = csv_to_anndata(fpath_expr, fpath_pheno) sc.tl.rank_genes_groups(adata, groupby="disease_status", method="wilcoxon", use_raw=False) de_df = sc.get.rank_genes_groups_df(adata, group=None) top_genes = ( de_df.groupby("names")["scores"] .max() .nlargest(768) .index.tolist() ) adata_filtered = adata[:, top_genes].copy() torch_X = torch.tensor(adata_filtered.X, dtype=torch.float32) torch_y = torch.tensor( pd.Categorical(adata_filtered.obs["disease_status"]).codes, dtype=torch.long ) return adata_filtered, torch_X, torch_y # extra code # expr = pd.read_csv(fpath_expr).T # pheno = pd.read_csv(fpath_pheno).T # inference = DefaultInference(n_cpus=8) # dds = DeseqDataSet( # counts=expr, # metadata=pheno["disease_status"], # design="~disease_status", # inference=inference # ) # dds.deseq2() # dds.obs["case_control"] = pheno["class"] # stats = DeseqStats(dds, contrast=["case_control", 1, 0], inference=inference) # stats.summary() # de_results = stats.results_df ``` ### load_monet.py ```python from transformers import AutoProcessor, AutoModelForZeroShotImageClassification import torch from torch import nn import torch.nn.functional as F from pytorch_metric_learning.losses import SupConLoss from tqdm import tqdm def get_monet_model(): processor = AutoProcessor.from_pretrained("suinleelab/monet") model = AutoModelForZeroShotImageClassification.from_pretrained("suinleelab/monet") # weights = model.state_dict() # print(type(weights)) # print(weights.keys()) # freeze weights for param in model.parameters(): param.requires_grad = False # set in eval mode model.eval() return model, processor def get_img_embeddings(model, processor, images: list, device='cpu'): """ Obtain embeddings for each image using MONET model. Parameters: images: list of the image objects device: which device to run on Returns: embeddings: MONET embeddings of shape (batch_size, 768) """ processed_imgs = processor(images=images, return_tensors="pt").to(device) with torch.no_grad(): embeddings_tmp = model.vision_model(**processed_imgs) embeddings = embeddings_tmp.pooler_output embeddings = F.normalize(embeddings, dim=1) return embeddings def precompute_img_embeddings(images: list, monet_model, monet_processor, device, batch_size=64): """ Pre-compute all MONET embeddings once before training. Parameters: images: list of image objs monet_model: MONET model monet_processor: MONET processor device: which device to run on batch_size: number of samples per batch Returns: precomputed_embeddings: all the embeddings from the MONET model for all batches """ precomputed_embeddings = [] monet_model.eval() for i in tqdm(range(0, len(images), batch_size), desc="pre-computing MONET embeddings"): batch = images[i:i+batch_size] with torch.no_grad(): emb = get_img_embeddings(monet_model, monet_processor, batch, device=device) precomputed_embeddings.append(emb.cpu()) return torch.cat(precomputed_embeddings, dim=0) class transcriptomics_encoder(nn.Module): def __init__(self, num_genes=663, out_dim=256): super(transcriptomics_encoder, self).__init__() self.nnlayers = nn.Sequential( nn.Linear(num_genes, out_dim), nn.BatchNorm1d(out_dim) ) def forward(self, X): return self.nnlayers(X) class image_encoder(nn.Module): def __init__(self, first_layer_dim=1024, out_dim=256): super(image_encoder, self).__init__() self.nnlayers = nn.Sequential( nn.Linear(first_layer_dim, out_dim), nn.BatchNorm1d(out_dim) ) def forward(self, X): return self.nnlayers(X) class joint_model(nn.Module): def __init__(self, in_dim=256, num_classes=7): super(joint_model, self).__init__() self.joint_layers = nn.Sequential( nn.Linear(in_dim*2, 128), nn.BatchNorm1d(128), nn.GELU(), nn.Dropout(0.3), nn.Linear(128, num_classes) ) self.img_skip_connect = nn.Sequential( nn.Linear(in_dim, num_classes) ) self.omics_skip_connect = nn.Sequential( nn.Linear(in_dim, num_classes) ) self.prediction_head = nn.Linear(3*num_classes, num_classes, bias=False) def forward(self, img_embed, omics_embed): pred_img = self.img_skip_connect(img_embed) pred_omics = self.omics_skip_connect(omics_embed) pred_joint = self.joint_layers( torch.cat([img_embed, omics_embed], dim=1) ) final_pred = self.prediction_head( torch.cat([pred_img, pred_omics, pred_joint], dim=1) ) return final_pred def save_model(self, fpath): torch.save(self.state_dict(), fpath) def load_model(self, fpath, device): pretrained_dict = torch.load(fpath, map_location=device) model_dict = self.state_dict() pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict} model_dict.update(pretrained_dict) self.load_state_dict(model_dict) def compute_loss(final_pred, img_embed, omics_embed, y_val, lambda_val=0.7): """ Compute loss of model. Both cross entropy and SupCon. Parameters: final_pred: output of joint_model of shape (batch_size, num_classes) img_embed: output of image_encoder of shape (batch_size, 512) omics_embed: output of transcriptomics_encoder of shape (batch_size, 512) y_val: disease class truth values (as indices) of shape (batch_size, ) lambda_val: weight on SupCon loss relative to CE """ ce = nn.CrossEntropyLoss() supcon = SupConLoss(temperature=0.1) loss_ce = ce(final_pred, y_val.long()) img_z = F.normalize(img_embed, dim=1) omics_z = F.normalize(omics_embed, dim=1) loss_supcon = supcon( torch.cat([img_z, omics_z], dim=0), torch.cat([y_val, y_val], dim=0) ) return loss_ce + lambda_val * loss_supcon def predict(img_encoder, omics_encoder, joint, x_img_embedding=None, x_omics=None): """ Inference from image data, omics data, or both. Parameters: img_encoder: image encoder instance omics_encoder: transcriptomics encoder instance joint: joint_model instance x_img_embedding: get_img_embeddings() output of shape (B, 768) (or none for no image data) x_omics: gene expression matrix of shape (B, 768) (or none for no omics data) """ img_encoder.eval() omics_encoder.eval() joint.eval() with torch.no_grad(): if x_omics is None: img_embed = img_encoder(x_img_embedding) return joint.img_skip_connect(img_embed) if x_img_embeddi [truncated — 314 more characters] ``` ### plot_hopts-results.py ```python import os, glob import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns RUN_DIR = "/home/provido/provido/luna_final/LUNA-main/run_histories" OUTDIR = "/home/provido/provido/luna_final/LUNA-main/out_data" os.makedirs(OUTDIR, exist_ok=True) # -------------------------- # Load all epoch-history CSVs # -------------------------- paths = sorted(glob.glob(os.path.join(RUN_DIR, "*_epoch_history.csv"))) assert len(paths) > 0, f"No csvs found in {RUN_DIR}" runs = [] for p in paths: df = pd.read_csv(p) df["run_file"] = os.path.basename(p) # run id from filename prefix: run12_... df["run_id"] = df["run_file"].str.split("_").str[0] runs.append(df) all_epochs = pd.concat(runs, ignore_index=True) # -------------------------- # Per-run summary table # -------------------------- hp_cols = ["embed_dim","lr","weight_decay","dropout","lambda_val","supcon_temp","batch_size"] def summarize_run(d): # best val_acc epoch i_best_acc = d["val_acc"].idxmax() # min val_loss epoch i_min_loss = d["val_loss"].idxmin() out = {c: d.iloc[0][c] for c in hp_cols} out["run_id"] = d.iloc[0]["run_id"] out["run_file"] = d.iloc[0]["run_file"] out["best_val_acc"] = float(d.loc[i_best_acc, "val_acc"]) out["best_val_acc_epoch"] = int(d.loc[i_best_acc, "epoch"]) out["val_loss_at_best_acc"] = float(d.loc[i_best_acc, "val_loss"]) out["min_val_loss"] = float(d.loc[i_min_loss, "val_loss"]) out["min_val_loss_epoch"] = int(d.loc[i_min_loss, "epoch"]) out["val_acc_at_min_loss"] = float(d.loc[i_min_loss, "val_acc"]) out["last_val_acc"] = float(d.iloc[-1]["val_acc"]) out["last_val_loss"] = float(d.iloc[-1]["val_loss"]) return pd.Series(out) summary = all_epochs.groupby(["run_id","run_file"], as_index=False).apply( lambda g: summarize_run(g.reset_index(drop=True)) ).reset_index(drop=True) summary = summary.sort_values("best_val_acc", ascending=False) summary.to_csv(os.path.join(OUTDIR, "run_summary.csv"), index=False) print("Wrote:", os.path.join(OUTDIR, "run_summary.csv")) print(summary.head(10)[["run_id","best_val_acc","best_val_acc_epoch","min_val_loss","min_val_loss_epoch"] + hp_cols]) # -------------------------- # Plot 3: Hyperparam sensitivity (best_val_acc) # -------------------------- for col in ["embed_dim","batch_size","dropout","lambda_val","supcon_temp","weight_decay"]: plt.figure(figsize=(7,4)) sns.boxplot(data=summary, x=col, y="best_val_acc") sns.stripplot(data=summary, x=col, y="best_val_acc", color="black", alpha=0.35, size=3) plt.title(f"Best val_acc by {col}") plt.tight_layout() plt.savefig(os.path.join(OUTDIR, f"best_val_acc_by_{col}.png"), dpi=200) plt.close() # lr is continuous-ish: scatter with log x-axis plt.figure(figsize=(7,4)) sns.scatterplot(data=summary, x="lr", y="best_val_acc", hue="embed_dim", style="batch_size", s=70, alpha=0.9) plt.xscale("log") plt.title("Best val_acc vs lr") plt.tight_layout() plt.savefig(os.path.join(OUTDIR, "best_val_acc_vs_lr.png"), dpi=200) plt.close() # -------------------------- # Plot 4: Heatmap (lambda_val x lr) showing mean best_val_acc # -------------------------- # (bin lr into strings so heatmap axes are readable) tmp = summary.copy() tmp["lr_str"] = tmp["lr"].astype(str) pivot = tmp.pivot_table(index="lambda_val", columns="lr_str", values="best_val_acc", aggfunc="mean") plt.figure(figsize=(10,4)) sns.heatmap(pivot, cmap="viridis", annot=False) plt.title("Mean best_val_acc heatmap: lambda_val (rows) x lr (cols)") plt.tight_layout() plt.savefig(os.path.join(OUTDIR, "heat_lambda_lr_best_val_acc.png"), dpi=200) plt.close() print("Saved plots to:", OUTDIR) ### #Plot 5 ### # -------------------------- # Plot 5: Accuracy vs epoch for many runs # - x: epoch # - y: accuracy # - color: chosen hyperparam # - linestyle: chosen hyperparam # - train lines alpha=0.5, val lines alpha=1.0 # -------------------------- COLOR_PARAM = "lambda_val" # change to: "embed_dim", "batch_size", "lambda_val", ... LINESTYLE_PARAM = "supcon_temp" # change to: "embed_dim", "batch_size", "lambda_val", ... # Use top N runs to avoid an unreadable plot (increase if you want) N_RUNS = min(30, len(summary)) chosen_runs = summary.head(N_RUNS)["run_file"].tolist() dplot = all_epochs[all_epochs["run_file"].isin(chosen_runs)].copy() dplot = dplot.sort_values(["run_file", "epoch"]) # Build a stable color map for COLOR_PARAM values color_vals = sorted(dplot[COLOR_PARAM].unique().tolist()) palette = sns.color_palette("tab10", n_colors=max(10, len(color_vals))) color_map = {v: palette[i % len(palette)] for i, v in enumerate(color_vals)} # Build a stable linestyle map for LINESTYLE_PARAM values ls_cycle = ["solid", "dashed", "dotted", "dashdot"] ls_vals = sorted(dplot[LINESTYLE_PARAM].unique().tolist()) ls_map = {v: ls_cycle[i % len(ls_cycle)] for i, v in enumerate(ls_vals)} plt.figure(figsize=(10,7)) for run_file, g in dplot.groupby("run_file"): g = g.sort_values("epoch") cval = g.iloc[0][COLOR_PARAM] lval = g.iloc[0][LINESTYLE_PARAM] color = color_map[cval] ls = ls_map[lval] # Train acc (faded) plt.plot(g["epoch"], g["train_acc"], color=color, linestyle=ls, alpha=0.25, linewidth=2) # Val acc (strong) plt.plot(g["epoch"], g["val_acc"], color=color, linestyle=ls, alpha=1.0, linewidth=2) plt.xlabel("Epoch") plt.ylabel("Accuracy") plt.title(f"Train (alpha=0.25) vs Val (alpha=1.0) accuracy\nColor={COLOR_PARAM}, Linestyle={LINESTYLE_PARAM} (top {N_RUNS} runs)") # Legends: one for colors (COLOR_PARAM), one for linestyles (LINESTYLE_PARAM) from matplotlib.lines import Line2D color_handles = [Line2D([0], [0], color=color_map[v], lw=3, label=f"{COLOR_PARAM}={v}") for v in color_vals] ls_handles = [Line2D([0], [0], color="black", lw=3, linestyle=ls_map[v], label=f"{LINESTYLE_PARAM}={v}") for v in ls_vals] train_val_handles = [ Line2D([0],[0], color="gray", lw=3, alpha=0.25, label="train_acc"), [truncated — 659 more characters] ``` ### train.py ```python import torch from torch.utils.data import DataLoader, Dataset from torch.optim import AdamW from torch.optim.lr_scheduler import CosineAnnealingLR from torch.utils.data import WeightedRandomSampler from tqdm import tqdm import pickle import pandas as pd from sklearn.model_selection import train_test_split from load_monet import image_encoder, transcriptomics_encoder, joint_model, get_img_embeddings, compute_loss, predict, get_monet_model, precompute_img_embeddings class DatasetPrep(Dataset): def __init__(self, img_embeddings, gene_expression, labels): self.img_embeddings = img_embeddings self.gene_expression = gene_expression self.labels = labels def __len__(self): return len(self.labels) def __getitem__(self, idx): return self.img_embeddings[idx], self.gene_expression[idx], self.labels[idx] #CP defined more args class TrainValLUNA: def __init__(self, num_classes=7, img_dim=1024, omics_dim=663, embed_dim=512, lr=1e-4, weight_decay=0.01, dropout=0.3, lambda_val=0.7, supcon_temp=0.1): """ Parameters: num_classes: number of predicted classes img_dim: input image dimension omics_dim: input omics dimension out_dim: output dimension for encoder models embed_dim: embedding dimension in mid of training device: device to run on lr: learning rate """ #CP adds the init of these 2 args self.lambda_val = lambda_val self.supcon_temp = supcon_temp self.device = 'cuda' if torch.cuda.is_available() else 'cpu' self.num_classes = num_classes self.img_encoder = image_encoder( first_layer_dim=img_dim, out_dim=embed_dim ).to(self.device) self.omics_encoder = transcriptomics_encoder( num_genes=omics_dim, out_dim=embed_dim ).to(self.device) #CP added dropout self.joint_model = joint_model( in_dim=embed_dim, num_classes=num_classes, dropout=dropout).to(self.device) self.monet_model, self.monet_processor = get_monet_model() self.monet_model.to(self.device) trainable_params = ( list(self.img_encoder.parameters()) + list(self.omics_encoder.parameters()) + list(self.joint_model.parameters()) ) # CP added wd arg self.optimizer = AdamW(trainable_params, lr=lr, weight_decay=weight_decay) self.scheduler = None self.history = { 'train_loss': [], 'train_acc': [], 'val_loss': [], 'val_acc': [], } self.best_val_loss = float('inf') def precompute_monet_embeddings(self, images, batch_size): return precompute_img_embeddings(images, self.monet_model, self.monet_processor, self.device, batch_size) def train_model(self, train_loader, val_loader, num_epochs): """ Train the model over multiple epochs and print statistics at each iteration. Parameters: train_loader: dataloader function for train dataset val_loader: dataloader function for validation dataset num_epochs: number of epochs of training Returns: img_enc: encoder model for images omics_enc: encoder model for omics joint: joint model """ self.scheduler = CosineAnnealingLR(self.optimizer, T_max=num_epochs) for epoch in range(num_epochs): # Training phase self.img_encoder.train() self.omics_encoder.train() self.joint_model.train() train_loss = 0.0 train_correct = 0 train_total = 0 for img_embeddings, gene_expr, y_val in tqdm(train_loader, desc=f"Epoch {epoch}"): # get precomputed img embeddings img_embeddings = img_embeddings.to(self.device) # to device for expr and y labels gene_expr = gene_expr.to(self.device) y_val = y_val.to(self.device) # forward pass through models img_embed = self.img_encoder(img_embeddings) omics_embed = self.omics_encoder(gene_expr) final_pred = self.joint_model(img_embed, omics_embed) # calc loss # CP adds lambda/supcon loss = compute_loss(final_pred, img_embed, omics_embed, y_val, lambda_val=self.lambda_val, supcon_temp=self.supcon_temp) # backward pass through model self.optimizer.zero_grad() loss.backward() self.optimizer.step() # track eval/accuracy %/loss values train_loss += loss.item() _, predicted_class = torch.max(final_pred.squeeze(1), 1) train_total += y_val.size(0) train_correct += (predicted_class == y_val).sum().item() # run validation val_loss, val_acc = self.validate(val_loader) # update lr self.scheduler.step() # print stats so far train_loss /= len(train_loader) train_acc = 100 * train_correct / train_total print(f"Epoch {epoch}: Train Loss={train_loss:.4f}, Train Acc={train_acc:.2f}%, " f"Val Loss={val_loss:.4f}, Val Acc={val_acc:.2f}%") self.history['train_loss'].append(train_loss) self.history['train_acc'].append(train_acc) self.history['val_loss'].append(val_loss) self.history['val_acc'].append(val_acc) with open('train_val_acc_history.pkl', 'wb') as f: [truncated — 7649 more characters] ``` ### hopt_train.py ```python import torch, os from torch.utils.data import DataLoader, Dataset from torch.optim import AdamW from torch.optim.lr_scheduler import CosineAnnealingLR from torch.utils.data import WeightedRandomSampler from tqdm import tqdm import pickle, csv import pandas as pd import random, itertools from sklearn.model_selection import train_test_split from load_monet import image_encoder, transcriptomics_encoder, joint_model, get_img_embeddings, compute_loss, predict, get_monet_model, precompute_img_embeddings class DatasetPrep(Dataset): def __init__(self, img_embeddings, gene_expression, labels): self.img_embeddings = img_embeddings self.gene_expression = gene_expression self.labels = labels def __len__(self): return len(self.labels) def __getitem__(self, idx): return self.img_embeddings[idx], self.gene_expression[idx], self.labels[idx] #Chris changed parameter args class TrainValLUNA: def __init__(self, num_classes=7, img_dim=1024, omics_dim=663, embed_dim=512, lr=1e-4, weight_decay=0.01, dropout=0.3, lambda_val=0.7, supcon_temp=0.1): """ Parameters: num_classes: number of predicted classes img_dim: input image dimension omics_dim: input omics dimension out_dim: output dimension for encoder models embed_dim: embedding dimension in mid of training device: device to run on lr: learning rate """ #CP adds the init of these 2 args self.lambda_val = lambda_val self.supcon_temp = supcon_temp self.cfg = { "embed_dim": embed_dim, "dropout": dropout, "img_dim": img_dim, "omics_dim": omics_dim, "num_classes": num_classes } self.device = 'cuda' if torch.cuda.is_available() else 'cpu' self.num_classes = num_classes self.img_encoder = image_encoder( first_layer_dim=img_dim, out_dim=embed_dim ).to(self.device) self.omics_encoder = transcriptomics_encoder( num_genes=omics_dim, out_dim=embed_dim ).to(self.device) # Chris is adding dropout here self.joint_model = joint_model( in_dim=embed_dim, num_classes=num_classes, dropout=dropout ).to(self.device) self.monet_model, self.monet_processor = get_monet_model() self.monet_model.to(self.device) trainable_params = ( list(self.img_encoder.parameters()) + list(self.omics_encoder.parameters()) + list(self.joint_model.parameters()) ) #CP adjusts weight_decay self.optimizer = AdamW(trainable_params, lr=lr, weight_decay=weight_decay) self.scheduler = None self.history = { 'train_loss': [], 'train_acc': [], 'val_loss': [], 'val_acc': [], } self.best_val_loss = float('inf') self.best_val_acc = -1.0 def precompute_monet_embeddings(self, images, batch_size): return precompute_img_embeddings(images, self.monet_model, self.monet_processor, self.device, batch_size) def train_model(self, train_loader, val_loader, num_epochs, run_name=None): """ Train the model over multiple epochs and print statistics at each iteration. Parameters: train_loader: dataloader function for train dataset val_loader: dataloader function for validation dataset num_epochs: number of epochs of training Returns: img_enc: encoder model for images omics_enc: encoder model for omics joint: joint model """ self.scheduler = CosineAnnealingLR(self.optimizer, T_max=num_epochs) for epoch in range(num_epochs): # Training phase self.img_encoder.train() self.omics_encoder.train() self.joint_model.train() train_loss = 0.0 train_correct = 0 train_total = 0 for img_embeddings, gene_expr, y_val in tqdm(train_loader, desc=f"Epoch {epoch}"): # get precomputed img embeddings img_embeddings = img_embeddings.to(self.device) # to device for expr and y labels gene_expr = gene_expr.to(self.device) y_val = y_val.to(self.device) # forward pass through models img_embed = self.img_encoder(img_embeddings) omics_embed = self.omics_encoder(gene_expr) final_pred = self.joint_model(img_embed, omics_embed) # calc loss ## CP changes here loss = compute_loss(final_pred, img_embed, omics_embed, y_val, lambda_val=self.lambda_val, supcon_temp=self.supcon_temp) # backward pass through model self.optimizer.zero_grad() loss.backward() self.optimizer.step() # track eval/accuracy %/loss values train_loss += loss.item() _, predicted_class = torch.max(final_pred.squeeze(1), 1) train_total += y_val.size(0) train_correct += (predicted_class == y_val).sum().item() # run validation val_loss, val_acc = self.validate(val_loader) # save best-by-accuracy checkpoint savep="/home/provido/provido/luna_final/LUNA-main/checkpoints/" fname = "/home/provido/provido/luna_final/LUNA-main/checkpoints/best_model_by_acc.pt" if run_name is None else f"{savep}best_model_by_acc_{run_name}.pt" if val_acc > self.best_val_acc: self.best_va [truncated — 12281 more characters] ``` ### predict.py ```python import warnings import torch import pickle from PIL import Image import argparse from pathlib import Path import pandas as pd import numpy as np from load_monet import ( image_encoder, transcriptomics_encoder, joint_model, get_monet_model, get_img_embeddings, predict ) class LUNAPredictor: def __init__(self, checkpoint_path, label_mapping_path, device=None): """ Initialize the predictor with a trained model checkpoint. Parameters: checkpoint_path: path to the saved model checkpoint (.pt file) label_mapping_path: path to the label mapping pickle file device: device to run inference on (None = auto-detect) """ self.device = device if device else ('cuda' if torch.cuda.is_available() else 'cpu') print(f"Using device: {self.device}") with open(label_mapping_path, 'rb') as f: self.label_mapping = pickle.load(f) self.classes = self.label_mapping['classes'] print(f"Loaded {len(self.classes)} classes: {self.classes}") print("Loading MONET model...") self.monet_model, self.monet_processor = get_monet_model() self.monet_model.to(self.device) self.monet_model.eval() print(f"Loading checkpoint from {checkpoint_path}...") checkpoint = torch.load(checkpoint_path, map_location=self.device) SUMMARY_CSV = "out_data/run_summary.csv" summary = pd.read_csv(SUMMARY_CSV) best_row = summary.sort_values("best_val_acc", ascending=False).iloc[0] # cfg = checkpoint.get("cfg", None) # if cfg is None: # warnings.warn( # "Checkpoint is missing 'cfg'. Re-save your checkpoint during training with a cfg dict " # "(embed_dim, dropout, img_dim, omics_dim, num_classes)." # , RuntimeWarning) # # cfg = {} # cfg["embed_dim"] = int(best_row["embed_dim"]) # cfg["img_dim"] = int(1024) # cfg["omics_dim"] = int(663) # cfg["dropout"] = float(best_row["dropout"]) # print("Loaded model cfg:", cfg) # embed_dim = int(cfg["embed_dim"]) # img_dim = int(cfg.get("img_dim", 1024)) # omics_dim = int(cfg.get("omics_dim", 663)) # dropout = float(cfg.get("dropout", 0.3)) img_dim = 1024 omics_dim = 663 embed_dim = int(best_row["embed_dim"]) dropout = float(best_row["dropout"]) self.img_encoder = image_encoder(first_layer_dim=img_dim, out_dim=embed_dim).to(self.device) self.omics_encoder = transcriptomics_encoder(num_genes=omics_dim, out_dim=embed_dim).to(self.device) self.joint_model = joint_model(in_dim=embed_dim, num_classes=len(self.classes), dropout=dropout).to(self.device) self.img_encoder.load_state_dict(checkpoint["img_encoder"]) self.omics_encoder.load_state_dict(checkpoint["omics_encoder"]) self.joint_model.load_state_dict(checkpoint["joint_model"]) #cfg = checkpoint.get("cfg", None) #if cfg is None: # raise ValueError("Checkpoint missing 'cfg'. Re-save the checkpoint including model hyperparams.") # embed_dim = int(cfg["embed_dim"]) # img_dim = int(cfg.get("img_dim", 1024)) # omics_dim = int(cfg.get("omics_dim", 663)) # dropout = float(cfg.get("dropout", 0.3)) self.img_encoder = image_encoder(first_layer_dim=img_dim, out_dim=embed_dim).to(self.device) self.omics_encoder = transcriptomics_encoder(num_genes=omics_dim, out_dim=embed_dim).to(self.device) self.joint_model = joint_model(in_dim=embed_dim, num_classes=len(self.classes), dropout=dropout).to(self.device) self.img_encoder.load_state_dict(checkpoint["img_encoder"]) self.omics_encoder.load_state_dict(checkpoint["omics_encoder"]) self.joint_model.load_state_dict(checkpoint["joint_model"]) self.img_encoder.eval() self.omics_encoder.eval() self.joint_model.eval() print("Model loaded successfully!") def load_image(self, image_path): """ Load an image from a file path. Parameters: image_path: path to the image file Returns: PIL Image object """ img = Image.open(image_path).convert('RGB') return img def predict_from_image(self, image_input, return_probabilities=False): """ Make a prediction from an image (or list of images). Parameters: image_input: single image path (str), PIL Image, or list of either return_probabilities: if True, return class probabilities Returns: Dictionary containing predictions and optional probabilities """ if isinstance(image_input, (str, Path)): images = [self.load_image(image_input)] single_image = True elif isinstance(image_input, Image.Image): images = [image_input] single_image = True elif isinstance(image_input, list): images = [] for img in image_input: if isinstance(img, (str, Path)): images.append(self.load_image(img)) else: images.append(img) single_image = False else: raise ValueError("image_input must be a path, PIL Image, or list of these") with torch.no_grad(): img_embeddings = get_img_embeddings( self.monet_model, self.monet_processor, images, device=self.device ) predictions = predict( self.img_encoder, self.omics_encoder, self.joint_model, x_img_embedding=img_embeddings, [truncated — 16780 more characters] ``` [2 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]