HackStack

Project Info

Omnyla

Devpost

Inspiration

Cancer is personal for us. Several teammates watched loved ones get diagnosed at a late stage: some recovered, others did not. We want oncologists, already in short supply, to move faster so families like ours get help sooner. My own (Vatsal) 125-pound weight-loss journey also had me in clinics and ICUs from childhood on, where I saw the same problem everywhere: doctors spend more time hunting for data than using it. Conversations with relatives in oncology departments validated our hypothesis: Each patient can come with 25 + files across 10 formats (MRI, pathology slides, VCFs, PDFs). Genetics labs still copy-paste for 4 hours to match every DNA variant to drugs or trials. Oncologists burn 2 hours of prep for a 15-minute tumor-board slot. A wrong pick wastes $12 000 per patient, and 80 % of reports contain at least one error. 92 % of U.S. adults never reach a clinical trial that could save or extend their lives. So we set out to build an AI copilot that turns this data maze into one clear, real-time interface.

What it does

Omnyla is an AI copilot for cancer teams. Doctors drag in slides, scans, VCFs, & clinic notes, and Omnyla: Ingests every modality: vision, genomics, text. Triages automatically, surfacing key findings and live-matching variants against PharmGKB / OncoKB for ranked drugs and trials. Joins the tumor board meeting as a voice agent that listens, answers questions (“Which trials fit this EGFR deletion?”) and writes a plain-English summary of the most optimal treatment option chosen by the pathologists, radiologists, and oncologists so that patients can finally read a condensed report that they can understand (free from the medical jargon). Hours of prep drop to minutes, and teams leave the meeting with one data-backed plan.

How we built it

Frontend: Next.js + Tailwind + shadcn/ui on Vercel Speech & voice: Vapi streams real-time transcription using Groq and converses in a conversational voice format to keep the tumor board conversation grounded in the right data, given at the right time. LLM orchestration: Gemini for image analysis reasoning; Claude for report generation and meeting notes Vision: Microsoft BiomedCLIP to spot lesions and stains on MRI, CT, and pathology images Genomics search: TSV file-based PharmGKB clinical annotations for drug recommendation queries Agent workflow: Pipeline exposed as a Fetch.ai agent linking Pathology, Radiology, Genomics, and Meeting modules Security: Audio processing handled through Vapi's secure infrastructure

Challenges we ran into

Putting BioMedCLIP onto a single GPU without missing tiny lesions Wrapping a multi-step pipeline as one Fetch.ai agent without blowing context limits Streaming hundreds of PharmGKB matches in under a second so the voice never stalls Harmonizing ten file types and keeping them synchronized after every upload

Accomplishments we're proud of

End-to-end demo: upload → variant-drug matches → live copilot in the tumor board Cut prep time to < 90 seconds per patient in testing Two oncologists told us they would “use this tomorrow” because it feels like a teammate, not another portal

What we learned

One model per modality beats one-size-fits-all; agents are the key Plain language matters more than perfect jargon. Doctors share summaries with patients immediately Latency kills trust; every second reduced off the pipeline increases adoption

What's next

Integrate directly with hospital FHIR / EHR feeds so uploads disappear Feed real-world outcomes back into ranking logic to keep suggestions fresh Launch a patient portal that turns board notes into an interactive care roadmap Begin FDA Software-as-a-Medical-Device clearance so Omnyla can surface full treatment recommendations, not just insights

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • AnthropicIn code
  • CSSIn code
  • FastAPIIn code
  • FlaskIn code
  • HTMLIn code
  • Hugging FaceIn code
  • Next.jsIn code
  • OpenAIIn code
  • PythonIn code
  • PyTorchIn code
  • ReactIn code
  • Tailwind CSSIn code
  • TypeScriptIn code
  • DockerClaimed
  • JavaScriptClaimed
  • VercelClaimed

13 of 16 appear in the indexed code. 3 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

838 KB

Source files

142

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars