# Project export: Surgentic This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Confidence. Safety. Every surgery, every time. - Devpost: https://devpost.com/software/surgentic - GitHub: https://github.com/SeanZCai/treehacks-agentic-surgeon - Demo: https://github.com/SeanZCai/treehacks - Video: https://www.youtube.com/embed/HX4DsCQwfDI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — SeanZCai (8 commits), yifantong1010 (5 commits) ## Devpost submission (written by the team) ### Overview The Problem Medical professionals often work under grueling schedules that can lead to fatigue and performance decline. Overworked surgeons may inadvertently skip critical mental checklists meant to ensure patient safety. Despite the success of initiatives like the WHO Surgical Safety Checklist—which has been proven to reduce post-surgical mortality by 36.6% when properly implemented—traditional paper-based systems see a completion rate of just 37% during surgical signouts. However, studies show that digital checklist aids can push compliance close to 100%. This gap underscores an urgent need: leveraging digital technology not only to support surgical safety but also to empower medical residents with real-world practice and continuous feedback. Our Solution We built a multimodal surgery data collection and observability platform designed specifically for white-collar healthcare operations. Our platform captures everything that happens in the operating room—integrating vision and audio data—to ensure surgical practices are safe, compliant, and educational. Key features include: Comprehensive Data Integration: Our system collects and synthesizes multimodal data from surgeries, acting as an advanced digital assistant. It aids surgeons in real-time compliance, replacing the need for traditional medical scribes and live EHR systems. Comprehensive Data Integration: Our system collects and synthesizes multimodal data from surgeries, acting as an advanced digital assistant. It aids surgeons in real-time compliance, replacing the need for traditional medical scribes and live EHR systems. Educational Enhancement: Addressing the shortage of real-world surgical practice for medical residents, our platform acts as an autonomous end-to-end educational assistant. It provides live feedback during surgery and generates comprehensive post-operation reports, with both qualitative and quantitative assessments. Educational Enhancement: Addressing the shortage of real-world surgical practice for medical residents, our platform acts as an autonomous end-to-end educational assistant. It provides live feedback during surgery and generates comprehensive post-operation reports, with both qualitative and quantitative assessments. Foundation for AI-Driven Training: The anonymized, aggregated data forms the basis for training models on successful surgeries, creating opportunities for continuous learning and improvement. By ensuring that data is aggregated at scale without compromising individual anonymity, we face minimal resistance from stakeholders. Foundation for AI-Driven Training: The anonymized, aggregated data forms the basis for training models on successful surgeries, creating opportunities for continuous learning and improvement. By ensuring that data is aggregated at scale without compromising individual anonymity, we face minimal resistance from stakeholders. How We Built It Our development journey was rooted in the challenge of achieving seamless multimodal data interoperability. We conducted extensive trial and error, fine-tuning models for each type of data through rigorous A/B testing to perfect our project flow. Key technical highlights include: Data Fusion in the OR: We set out to capture a diverse range of data—visual, audio, and video—from the operating room, wanting our system to provide both real-time and post-operation analysis. Data Fusion in the OR: We set out to capture a diverse range of data—visual, audio, and video—from the operating room, wanting our system to provide both real-time and post-operation analysis. Conversational Voice Assistance: Leveraging Eleven Labs, we built an augmented voice agent that interacts with surgical staff. This agent is supported by advanced search tools like Perplexity Sonar, which provide contextual awareness and sophisticated reasoning. Conversational Voice Assistance: Leveraging Eleven Labs, we built an augmented voice agent that interacts with surgical staff. This agent is supported by advanced search tools like Perplexity Sonar, which provide contextual awareness and sophisticated reasoning. Real-Time Compliance and Reporting: Our platform not only monitors compliance before, during, and after surgery but also delivers real-time conversational feedback. Post-surgery, it generates detailed reports that offer: An overview of the procedure (both pre-surgery and during surgery) Scoring metrics for educational assessment. Actionable insights for improvement. Real-Time Compliance and Reporting: Our platform not only monitors compliance before, during, and after surgery but also delivers real-time conversational feedback. Post-surgery, it generates detailed reports that offer: An overview of the procedure (both pre-surgery and during surgery) Scoring metrics for educational assessment. Actionable insights for improvement. By integrating various AI tools with a deep understanding of the surgical environment, our platform enhances patient safety, streamlines surgical workflows, and transforms the educational landscape for future medical professionals. ### Challenges we ran into Integrating many different APIs into one project Infrastructure for collecting data for AI agents was limited and hard to find Designing a functioning front-end with almost zero front end experience Being forced to pivot and change our idea halfway through the hackathon ### Accomplishments we're proud of Worked with multiple modalities (video, audio, vitals, text, images) Integrated an end-to-end product for the whole surgical process, from pre-surgery, to during, to post-surgery. Successfully built a functional MVP within 36 hours! ### What we learned How to integrate multiple APIs efficiently The importance of UX design for user experience Team collaboration and time management in an environment where we are forced to iterate and fail over and over again in a short period of time. User Interviews Conducted user interviews with multiple doctors and learned that extending our medical alerts system to monitor patient vitals would be ideal. This is why we started working with Terra API and incorporated sample data in order to simulate monitoring vitals and queue alerts based on this data. Acquired one LOI from a doctor holding purchasing power in his hospital (note: this hospital is based outside of the US). ### What's next Inject compliance requirements from major hospital chains Run a pilot study Refine and fine-tune AI models for better latency. Incorporate real-life and real-time patient vitals into our solution. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 136 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (36 of 36) ``` .gitignore .prettierignore .prettierrc.json app/api/c/route.ts app/api/conversations/route.ts app/api/i/route.ts app/api/recordings/route.ts app/c/[slug]/page.tsx app/conversations/route.ts app/globals.css app/layout.tsx app/preview/page.tsx app/recording/page.tsx app/route.ts components/Checklist.tsx components/Message.tsx components/RecordingsGallery.tsx components/TextAnimation.tsx components/useTypingEffect.ts global.d.ts lib/compliance.ts lib/supabase.ts LICENSE next.config.ts package.json postcss.config.mjs README.md reference/src/App.css reference/src/App.tsx reference/src/LandingPage.css reference/src/LandingPage.tsx reference/src/main.tsx reference/src/vite-env.d.ts schema.tsx tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @11labs/react@^0.0.4, @neondatabase/serverless@^0.10.4, @supabase/supabase-js@^2.48.1, @types/node@^22.10.2, @types/react@^19.0.1, @types/react-dom@^19.0.2, dotenv@^16.4.7, framer-motion@^11.18.2, motion@^11.11.17, next@^15.1.0, postcss@^8, prettier@^3.4.2, prettier-plugin-organize-imports@^4.1.0, prettier-plugin-tailwindcss@^0.6.9, react@^19.0.0, react-dom@^19.0.0, react-feather@^2.0.10, sonner@^1.7.1, supabase@^2.12.1, tailwindcss@^3.4.1, tsx@^4.19.2, typescript@^5 ### Recent commits (newest first) - final - final - help im dying - dbhajnsda - fix - fix - Merge branch 'main' of https://github.com/SeanZCai/treehacks-agentic-surgeon - fix - fix - push - fresh - push; - ElevenLabs - Create LICENSE - Update README.md - Update README.md - Update README.md - Update README.md - Update page.tsx - Update page.tsx ## Key source files (fetched from GitHub, selected and truncated for size) Source contents could not be fetched from GitHub right now. [29 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]