# Project export: Hearti This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Simplifying congenital heart disease diagnosing, and treatment planning, and medical explanations for doctors and patients using computer vision, machine learning, VLMs, and more. - Devpost: https://devpost.com/software/hearti - GitHub: https://github.com/ethan-tam33/heartAI - Demo: https://heart-ai-seven.vercel.app/ - Video: https://www.youtube.com/embed/uNFEGP6hMRg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Most Impactful Hack (JBL Partybox 110 per team member)) - Team: 4 GitHub contributor(s) — Socrates Osorio Diaz (31 commits), Ethan Tam (26 commits), Steven Luo (1 commits), David Martinez (1 commits) ## Devpost submission (written by the team) ### Inspiration Congenital heart disease (CHD) is the most common birth defect in the United States, and thanks to advances in medicine, more children with CHD are surviving into adulthood. But with this improved survival comes a lifelong need for careful monitoring - as symptoms can change over time. Recognizing this challenge, we set out to build an AI-powered web application that helps both doctors and patients navigate the complexities of CHD. Our vision is simple: provide a tool that offers clear, reliable insights for global healthcare professionals and their patients while ensuring that patients receive a smooth transition into their medical journey. ### What it does Interactive 3D model of the input 2D heart. Instead of relying on flat 2D images, users can explore a fully interactive 3D representation of the heart. This provides doctors and patients with a clearer, more intuitive understanding of the condition. Multi-classification of CHD categories. The model identifies the most likely congenital heart disease (CHD) categories based on the uploaded scan, helping users understand potential diagnoses. Severity Classification. Users can also view how serious their CHD symptoms are. Model Confidence. We understand that AI may not always be correct, so a confidence score is included in the web application to guide a doctor’s conclusion with the AI findings. Feedback Form. For doctors, we have a space for them to accept or reject the model’s findings and explain their reasonings. This will help better fine-tune the model and ensure continuous improvement in its accuracy and reliability for diagnosing CHD conditions. AI Voice Chat. Guided by the previous datapoints, the AI voice chat is ready to discuss with the user any questions they have about the model's output and reasoning. Custom Heart Segmentation Model Currently, CMRI segmentation is performed manually to ensure accuracy. To automate this process, we developed a custom segmentation model trained on an NVIDIA A100 GPU using Brev Compute. While the model requires additional data and refinement for full accuracy, it successfully identified regions of interest despite the limited training dataset. Our architecture is specifically designed for cardiac imaging, labeling 3D volumes into eight major heart structures (four chambers and four arteries/veins). With further data and training, we aim to eliminate the need for manual segmentation in the pipeline. ### How we built it Surface Intensity and Segmented Heart Models Combining density and resonance information from the CMRI and the segmentation boundaries, we built an algorithm that can map and shade the volume of the heart based on the intensity of the MRI scan. This algorithm removes the background, carefully maps MRI data to volumes of the heart. This ensures that we do not impart any unwanted artifacts or information, an integral safety feature for this application. In addition to mapping MRI intensity, we also implemented an algorithm that generates smooth meshes using image processing techniques to provide an easy-to-read interpretation of the heart’s many shapes and volumes, without focusing too much on the color of the MRI data. We realized that all of this data can take quite a bit of memory to render, so we created optimized models that prioritize surfaces over volume, use sparse views to provide a lightweight model, and have the ability to select which specific major volumes of the heart you would like to view at a time. Embedding similarity using VespaAI We converted our data into vector embeddings paired with CHD labels and leveraged VespaAI to run precise, rapid queries. By using an angular metric, we account for both the direction and magnitude of our high-dimensional data. Web App Tech Stack For the frontend and backend, we used FastAPI, React.js, Next.js, Typescript, and TailwindCSS. ### Challenges we ran into While the dataset we found online was extremely useful and new, it was very limited (59 hearts only). To train a truly accurate model, we need a much larger sample size. ### Accomplishments we're proud of -We successfully built and rendered 3D representations of 2D heart CMRI scans. Besides looking extremely cool, it is both new and valuable for the ways in which we treat people with CHD. -Optimized the models to prioritize surface pixels, smooth representations, and segment-specific views to make them more accessible across devices and browsers. -We built a web-based AI agent that can not only predict a user’s CHD conditions but can also act as a source of medical information for decades. If inputted with enough patient data, it can even more effectively provide accurate information that doctors can approve. ### What we learned We had a lot of ideas for this project, so we needed to quickly define what was most relevant and useful for our target audience (not what sounded the coolest). We moved quickly from talking with each other frequently and truly as a team. Some technical highlights: learned how to process research/medical data into more intricate + interpretable visualizations that capture features of the heart while minimizing artifacts, learned about different metrics to assess how correlated 2 embeddings are, and learned how to learn quickly using VespaAI’s querying functionality. ### What's next While building Hearti, we considering incorporating AI-style transfer to turn the MRI data insights into realistic textures of real heart tissue. While exciting, this has a lot of challenges that have to be approached carefully because hallucinations can have consequences. Allowing Hearti to take advantage of a greater range of medical imaging techniques and scans. It would be great if we could upload a full-body or chest MRI and have Hearti extract the important regions and use our model to create stunning visuals of the heart. Incorporating time series data would be interesting because it would allow us to make use of our optimized models to create 4D reconstructions of the heart. We aim to expand our custom segmentation model’s capabilities by processing more heart scans directly and enhancing automation in cardiac imaging. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 45 recognized source files, 100525 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - PyTorch (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 131) ``` .gitignore api/.gitignore api/app/__init__.py api/app/deployment.xml api/app/embeddings.py api/app/feed_data.py api/app/main.py api/app/schema.json api/requirements.txt api/treehacks/bin/activate api/treehacks/bin/activate.csh api/treehacks/bin/activate.fish api/treehacks/bin/Activate.ps1 api/treehacks/bin/fastapi api/treehacks/bin/httpx api/treehacks/bin/normalizer api/treehacks/bin/pip api/treehacks/bin/pip3 api/treehacks/bin/pip3.11 api/treehacks/bin/py2pyi api/treehacks/bin/python api/treehacks/bin/python3 api/treehacks/bin/python3.11 api/treehacks/bin/replace_wildcards api/treehacks/bin/uvicorn api/treehacks/bin/vespa api/treehacks/pyvenv.cfg api2/.gitignore api2/create_knowledge_base.py api2/embeddings.py api2/main.py api2/requirements.txt api2/smart_diagnosis.py api2/smooth_heart_vis.py api2/treehacks/bin/activate api2/treehacks/bin/activate.csh api2/treehacks/bin/activate.fish api2/treehacks/bin/Activate.ps1 api2/treehacks/bin/fastapi api2/treehacks/bin/httpx api2/treehacks/bin/normalizer api2/treehacks/bin/pip api2/treehacks/bin/pip3 api2/treehacks/bin/pip3.11 api2/treehacks/bin/py2pyi api2/treehacks/bin/python api2/treehacks/bin/python3 api2/treehacks/bin/python3.11 api2/treehacks/bin/replace_wildcards api2/treehacks/bin/uvicorn api2/treehacks/bin/vespa api2/treehacks/pyvenv.cfg api2/vespa-app/Dockerfile api2/vespa-app/schemas/clinical_data.sd api2/vespa-app/security/clients.pem api2/vespa-app/services.xml api2/vespa-app/start-vespa.sh LICENSE ml/embeder.ipynb modeling/vis.py modeling/visualize.ipynb src/.env.local src/.gitignore src/deployment.xml src/heartaivespa/schemas/medical_records.sd src/heartaivespa/security/clients.pem src/heartaivespa/security/private-key.pem src/heartaivespa/services.xml src/main.py src/test_data.ipynb src/test_main.py src/treehacks/bin/activate src/treehacks/bin/activate.csh src/treehacks/bin/activate.fish src/treehacks/bin/Activate.ps1 src/treehacks/bin/f2py src/treehacks/bin/fastapi src/treehacks/bin/httpx src/treehacks/bin/huggingface-cli src/treehacks/bin/isympy src/treehacks/bin/normalizer src/treehacks/bin/pip src/treehacks/bin/pip3 src/treehacks/bin/pip3.11 src/treehacks/bin/py2pyi src/treehacks/bin/python src/treehacks/bin/python3 src/treehacks/bin/python3.11 src/treehacks/bin/replace_wildcards src/treehacks/bin/torchfrtrace src/treehacks/bin/torchrun src/treehacks/bin/tqdm src/treehacks/bin/transformers-cli src/treehacks/bin/uvicorn src/treehacks/bin/vespa src/treehacks/pyvenv.cfg src/treehacks/share/man/man1/isympy.1 src/typings.py src/vespa_test.ipynb web/.gitignore web/next.config.ts web/package.json web/postcss.config.mjs web/public/heart_visual.ply web/public/pat1_healthy_11yo_segmented.html web/public/pat1_healthy_11yo.html web/public/pat3_unhealthy_52yo_segmented.html web/public/pat3_unhealthy_52yo.html web/public/smooth_heart_2.html web/public/smooth_heart.html web/README.md web/src/app/about/page.tsx web/src/app/ai-agent/page.tsx web/src/app/components/ComparisonViewer.tsx web/src/app/components/ECGViewer.tsx web/src/app/components/ElevenLabsConvai.tsx web/src/app/components/FeedbackForm.tsx web/src/app/components/Heart.tsx web/src/app/components/ProcessingState.tsx web/src/app/components/ResultsDashboard.tsx [11 more files omitted for size] ``` ### Dependencies - api/requirements.txt: fastapi, pandas, pyvespa, sentence-transformers, uvicorn - api2/requirements.txt: fastapi, monai, nibabel, pandas, python-dotenv, sentence-transformers, uvicorn, vespa - web/package.json: @11labs/react@^0.0.7, @types/node@^20, @types/react@^19, @types/react-dom@^19, @types/three@^0.173.0, html2canvas@^1.4.1, jspdf@^2.5.2, next@15.1.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, react-dropzone@^14.3.5, tailwindcss@^3.4.1, typescript@^5.7.3 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/ethan-tam33/heartAI - Master fusion combines - added modeling and ml final versions - MASTER FUSION - fix heart - final ui stuff - IT WORKS - Fusion works - yay - before fusion - added data - merging - VESPA WORKSSS - VESPA WORKSSS - not working base64 logic - not working base64 logic - only nii files allowed - clean - clean - removed ecg ## Key source files (fetched from GitHub, selected and truncated for size) ### api/requirements.txt ``` fastapi uvicorn pandas sentence-transformers pyvespa ``` ### api2/requirements.txt ``` fastapi uvicorn pandas sentence-transformers vespa nibabel monai python-dotenv ``` ### web/package.json ``` { "name": "minimal-tailwind", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@11labs/react": "^0.0.7", "@types/three": "^0.173.0", "html2canvas": "^1.4.1", "jspdf": "^2.5.2", "next": "15.1.7", "react": "^19.0.0", "react-dom": "^19.0.0", "react-dropzone": "^14.3.5" }, "devDependencies": { "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5.7.3" } } ``` ### api2/vespa-app/Dockerfile ``` FROM vespaengine/vespa # Copy Vespa application package into container COPY . /app/ # Wait for services to start, deploy application, then keep container running CMD vespa status --wait 300 && \ cd /app && \ vespa deploy --wait 300 && \ tail -f /dev/null ``` ### src/main.py ```python # main.py from fastapi import FastAPI, HTTPException from pydantic import BaseModel from vespa.application import Vespa from vespa.package import ApplicationPackage, Schema, Document, Field from vespa.deployment import VespaCloud import os from typings import MedicalRecord from typing import List app = FastAPI() # IMPORTANT: Change the port so you’re not conflicting with the FastAPI port. vespa_app = Vespa(url="http://localhost", port=8080) tenant_name = "socrates" application = "heartaivespa" torch_embed_size = 128 distance_metric = "dotproduct" # Or "euclidean", but make it consistent below. schema = Schema( name="medical_records", document=Document( fields=[ Field(name="Pat", type="int", indexing=["summary"]), Field(name="Age", type="int", indexing=["summary"]), Field(name="Category", type="string", indexing=["index", "summary"]), Field( name="heart_embedding", type=f"tensor(x[{torch_embed_size}])", indexing=["index"], distance_metric=f"{distance_metric}", ), # ... rest of your boolean fields ... ] ) ) package = ApplicationPackage( name=application, schema=[schema] ) vespa_cloud = VespaCloud( tenant=tenant_name, application=application, application_package=package, key_content=os.getenv("VESPA_TEAM_API_KEY") ) # deploy! from vespa.deployment import VespaDocker vespa_container = VespaDocker() vespa_connection = vespa_container.deploy(application_package=package) @app.post("/insert_records/") async def insert_records(records: List[MedicalRecord]): print("Inserting records...") print(records) responses = [] try: # If you want to use vespa_app.syncio, do so here. with vespa_app.syncio() as sync_app: for record in records: print("data_id: ", str(record.Pat)) print("dict: ", record.dict) response = sync_app.feed_data_point( schema="medical_records", data_id=str(record.Pat), fields=record.dict() ) print(response.json()) print("byeee") responses.append(response.json()) # CRITICAL: Return at the end so FastAPI can finish the request return {"message": "Records inserted successfully", "responses": responses} except Exception as e: print(f"An error occurred: {e}") raise HTTPException(status_code=500, detail=str(e)) @app.post("/query/") async def query_vespa(embedding: List[float], top_k: int = 5): if len(embedding) != torch_embed_size: raise HTTPException( status_code=400, detail=f"Embedding size must be {torch_embed_size}" ) query_body = { "yql": "select * from sources medical_records " f"where {{targetHits:{top_k}}} nearestNeighbor(heart_embedding, query_embedding)", "input.query(query_embedding)": { "value": embedding } } response = vespa_app.query(body=query_body) return response.json() ``` ### api2/main.py ```python # main.py from fastapi import FastAPI, HTTPException, Query import pandas as pd import os from sentence_transformers import SentenceTransformer from vespa.application import Vespa import requests from dotenv import load_dotenv from embeddings import NIfTIToEmbedding from smart_diagnosis import sMaRTDiagnosis from create_knowledge_base import ingest_data_from_zip from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from typing import Any, Dict import base64 app = FastAPI(title="Vespa Embeddings/RAG FastAPI Demo") # Initialize Vespa client – assumes Vespa is running at localhost:8080 vespa_app = Vespa(url="http://localhost", port=8080) # vespa_app = Vespa(url="https://e7032d12.d1f1f075.z.vespa-app.cloud/", port=8080) # --------------------------- # CORS Middleware # --------------------------- app.add_middleware( CORSMiddleware, allow_origins=["*"], # In production, restrict this! allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # --------------------------- # Pydantic Models # --------------------------- class QueryRequest(BaseModel): query_text: str class UploadRequest(BaseModel): nii_path: str # base64-encoded .nii file content # --------------------------- # Existing Endpoints # --------------------------- @app.post("/search") def search_documents(req: QueryRequest): """ Given a query text, generate an embedding, and search for nearest docs in Vespa. (This was from the original snippet, for textual queries.) """ from app.embeddings import generate_embedding query_vec = generate_embedding(req.query_text) hits = vespa_app.query( body={ "yql": "select * from sources * where ([{\"targetNumHits\":10}]nearestNeighbor(embedding, query_embedding));", "hits": 10, "input.query_embedding": query_vec, "ranking.features.query(query_embedding)": query_vec, "ranking.profile": "default" }, schema="hvsmr" ) return hits.json @app.get("/healthcheck") def healthcheck(): return {"status": "ok"} # --------------------------- # New "/upload" Endpoint # --------------------------- @app.post("/upload") def upload_file(req: UploadRequest) -> Dict[str, Any]: """ Receive a base64-encoded NIfTI file, convert it to an embedding, perform a Vespa ANN search, gather relevant doc data, then call the sMaRTDiagnosis function to get an AI-based diagnosis + links. Return them to the client as JSON. """ base64_str = req.nii_path if not base64_str: raise HTTPException(status_code=400, detail="No NIfTI data received") # 1. Generate embedding from base64 embedder = NIfTIToEmbedding() try: embedding = embedder.embedding_from_base64(base64_str).flatten().tolist() except Exception as e: raise HTTPException(status_code=500, detail=f"Error generating embedding: {str(e)}") # 2. Query Vespa for nearest docs try: response = vespa_app.query( body={ "yql": "select * from sources * where ([{\"targetNumHits\":3}]nearestNeighbor(image_embedding, query_vec));", "hits": 3, "input.query_vec": embedding, "ranking.features.query(query_vec)": embedding, "ranking.profile": "default" }, schema="clinical_data" ) except Exception as e: raise HTTPException(status_code=500, detail=f"Vespa query error: {str(e)}") hits = response.json["root"].get("children", []) if not hits: # If we get no results, we still pass something to sMaRTDiagnosis # but let's at least let the user know. pass # 3. Gather doc data. We will just combine them into one string for sMaRTDiagnosis. # For example, we might join 'data' fields from the top 3 hits. # Or you can do something more advanced if you want. doc_strings = [] for h in hits: fields = h.get("fields", {}) doc_data = fields.get("data", "") doc_confidence = h.get("relevance", "") doc_strings.append(doc_data) combined_data = "\n".join(doc_strings) if doc_strings else "No data from Vespa" # 4. Pass to sMaRTDiagnosis to get textual diagnosis + relevant links diagnosis_text, diagnosis_links, first_diagnosis = sMaRTDiagnosis(combined_data) # 5. Return the final JSON to the client print(doc_confidence) return { "diagnosis_text": diagnosis_text, "links": diagnosis_links, "confidence": doc_confidence, "num_hits": len(hits), # "first_diagnosis": first_diagnosis } ``` ### api/app/main.py ```python # app/main.py from fastapi import FastAPI, Query from typing import List, Optional from vespa.query import Vespa from pydantic import BaseModel app = FastAPI() # Connect to Vespa vespa_app = Vespa(url="http://localhost", port=8080) class QueryRequest(BaseModel): query_text: str @app.post("/search") def search_documents(req: QueryRequest): """ Given a query text, generate an embedding, and search for nearest docs in Vespa. """ from app.embeddings import generate_embedding query_vec = generate_embedding(req.query_text) # We'll use Vespa's ANN search on 'embedding' field. hits = vespa_app.query( body={ "yql": "select * from sources * where ([{\"targetNumHits\":10}]nearestNeighbor(embedding, query_embedding));", "hits": 10, "input.query_embedding": query_vec, "ranking.features.query(query_embedding)": query_vec, "ranking.profile": "default" }, schema="hvsmr" ) return hits.json @app.get("/healthcheck") def healthcheck(): return {"status": "ok"} ``` ### web/src/app/page.tsx ```typescript // app/page.tsx export default function LandingPage() { return (
{/* Background video */} {/* Dark overlay + content */}

Hearti ❤️

Upload your Cardiovascular Magnetic Resonance scans and get instant analysis with AI-powered insights on a sleek, user-friendly interface.

Get Started
) } ``` ### web/src/app/layout.tsx ```typescript // // app/layout.tsx // import './globals.css' // import type { Metadata } from 'next' // import { Inter } from 'next/font/google' // import Link from 'next/link' // const inter = Inter({ subsets: ['latin'], variable: '--font-inter' }) // export const metadata: Metadata = { // title: 'ECG Analysis System', // description: 'Medical research platform for ECG classification', // } // export default function RootLayout({ // children, // }: { // children: React.ReactNode // }) { // return ( // // //
//
// // // // ECG Research // // // //
//
//
{children}
// // // ) // } // function HeartIcon(props: React.SVGProps) { // return ( // // // // ) // } import './globals.css' export default function RootLayout({ children }: { children: React.ReactNode }) { return ( // pages/_app.tsx {process.env.NODE_ENV === 'development' && (