# Project export: Surgentic This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Confidence. Safety. Every surgery, every time. - Devpost: https://devpost.com/software/surgentic - GitHub: https://github.com/SeanZCai/treehacks-agentic-surgeon - Demo: https://github.com/SeanZCai/treehacks - Video: https://www.youtube.com/embed/HX4DsCQwfDI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — SeanZCai (8 commits), yifantong1010 (5 commits) ## Devpost submission (written by the team) ### Overview The Problem Medical professionals often work under grueling schedules that can lead to fatigue and performance decline. Overworked surgeons may inadvertently skip critical mental checklists meant to ensure patient safety. Despite the success of initiatives like the WHO Surgical Safety Checklist—which has been proven to reduce post-surgical mortality by 36.6% when properly implemented—traditional paper-based systems see a completion rate of just 37% during surgical signouts. However, studies show that digital checklist aids can push compliance close to 100%. This gap underscores an urgent need: leveraging digital technology not only to support surgical safety but also to empower medical residents with real-world practice and continuous feedback. Our Solution We built a multimodal surgery data collection and observability platform designed specifically for white-collar healthcare operations. Our platform captures everything that happens in the operating room—integrating vision and audio data—to ensure surgical practices are safe, compliant, and educational. Key features include: Comprehensive Data Integration: Our system collects and synthesizes multimodal data from surgeries, acting as an advanced digital assistant. It aids surgeons in real-time compliance, replacing the need for traditional medical scribes and live EHR systems. Comprehensive Data Integration: Our system collects and synthesizes multimodal data from surgeries, acting as an advanced digital assistant. It aids surgeons in real-time compliance, replacing the need for traditional medical scribes and live EHR systems. Educational Enhancement: Addressing the shortage of real-world surgical practice for medical residents, our platform acts as an autonomous end-to-end educational assistant. It provides live feedback during surgery and generates comprehensive post-operation reports, with both qualitative and quantitative assessments. Educational Enhancement: Addressing the shortage of real-world surgical practice for medical residents, our platform acts as an autonomous end-to-end educational assistant. It provides live feedback during surgery and generates comprehensive post-operation reports, with both qualitative and quantitative assessments. Foundation for AI-Driven Training: The anonymized, aggregated data forms the basis for training models on successful surgeries, creating opportunities for continuous learning and improvement. By ensuring that data is aggregated at scale without compromising individual anonymity, we face minimal resistance from stakeholders. Foundation for AI-Driven Training: The anonymized, aggregated data forms the basis for training models on successful surgeries, creating opportunities for continuous learning and improvement. By ensuring that data is aggregated at scale without compromising individual anonymity, we face minimal resistance from stakeholders. How We Built It Our development journey was rooted in the challenge of achieving seamless multimodal data interoperability. We conducted extensive trial and error, fine-tuning models for each type of data through rigorous A/B testing to perfect our project flow. Key technical highlights include: Data Fusion in the OR: We set out to capture a diverse range of data—visual, audio, and video—from the operating room, wanting our system to provide both real-time and post-operation analysis. Data Fusion in the OR: We set out to capture a diverse range of data—visual, audio, and video—from the operating room, wanting our system to provide both real-time and post-operation analysis. Conversational Voice Assistance: Leveraging Eleven Labs, we built an augmented voice agent that interacts with surgical staff. This agent is supported by advanced search tools like Perplexity Sonar, which provide contextual awareness and sophisticated reasoning. Conversational Voice Assistance: Leveraging Eleven Labs, we built an augmented voice agent that interacts with surgical staff. This agent is supported by advanced search tools like Perplexity Sonar, which provide contextual awareness and sophisticated reasoning. Real-Time Compliance and Reporting: Our platform not only monitors compliance before, during, and after surgery but also delivers real-time conversational feedback. Post-surgery, it generates detailed reports that offer: An overview of the procedure (both pre-surgery and during surgery) Scoring metrics for educational assessment. Actionable insights for improvement. Real-Time Compliance and Reporting: Our platform not only monitors compliance before, during, and after surgery but also delivers real-time conversational feedback. Post-surgery, it generates detailed reports that offer: An overview of the procedure (both pre-surgery and during surgery) Scoring metrics for educational assessment. Actionable insights for improvement. By integrating various AI tools with a deep understanding of the surgical environment, our platform enhances patient safety, streamlines surgical workflows, and transforms the educational landscape for future medical professionals. ### Challenges we ran into Integrating many different APIs into one project Infrastructure for collecting data for AI agents was limited and hard to find Designing a functioning front-end with almost zero front end experience Being forced to pivot and change our idea halfway through the hackathon ### Accomplishments we're proud of Worked with multiple modalities (video, audio, vitals, text, images) Integrated an end-to-end product for the whole surgical process, from pre-surgery, to during, to post-surgery. Successfully built a functional MVP within 36 hours! ### What we learned How to integrate multiple APIs efficiently The importance of UX design for user experience Team collaboration and time management in an environment where we are forced to iterate and fail over and over again in a short period of time. User Interviews Conducted user interviews with multiple doctors and learned that extending our medical alerts system to monitor patient vitals would be ideal. This is why we started working with Terra API and incorporated sample data in order to simulate monitoring vitals and queue alerts based on this data. Acquired one LOI from a doctor holding purchasing power in his hospital (note: this hospital is based outside of the US). ### What's next Inject compliance requirements from major hospital chains Run a pilot study Refine and fine-tune AI models for better latency. Incorporate real-life and real-time patient vitals into our solution. ## README (from the GitHub repository) # Surgentic - AI-Powered Surgical Safety Assistant Surgentic is a cutting-edge AI assistant designed to enhance surgical safety and compliance. By combining real-time voice interaction, screen recording, and automated checklist management, Surgentic ensures confidence and safety in every surgical procedure. Surgentic Interface ## 🌟 Features - Real-time AI voice interaction during procedures - Automated surgical safety checklist management - Screen recording and session playback - Beautiful, modern interface with dark mode - Compliance report generation - Persistent conversation history - Real-time transcription - Responsive design optimized for medical environments ## 🚀 Tech Stack - **Framework**: [Next.js](https://nextjs.org/) 15 - **Styling**: [Tailwind CSS](https://tailwindcss.com/) with [shadcn/ui](https://ui.shadcn.com/) - **Database**: [Supabase](https://supabase.com) - **Animation**: [Framer Motion](https://www.framer.com/motion/) - **Voice AI**: [ElevenLabs](https://elevenlabs.io/) ## 📦 Installation 1. Clone the repository: ```bash git clone https://github.com/yourusername/surgentic cd surgentic ``` 2. Install dependencies: ```bash pnpm install ``` 3. Create a `.env` file in the root directory with the following variables: ```bash NEXT_PUBLIC_SUPABASE_URL="your-supabase-url" NEXT_PUBLIC_SUPABASE_ANON_KEY="your-supabase-anon-key" XI_API_KEY="your-elevenlabs-api-key" AGENT_ID="your-elevenlabs-agent-id" ``` 4. Run the development server: ```bash pnpm dev ``` 5. Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. ## 🏗️ Project Structure - `/app` - Next.js app directory containing routes and layouts - `/components` - Reusable React components - `/lib` - Utility functions and shared logic - `/public` - Static assets and recorded sessions ## 🔑 Environment Variables - `NEXT_PUBLIC_SUPABASE_URL`: Your Supabase project URL - `NEXT_PUBLIC_SUPABASE_ANON_KEY`: Your Supabase anonymous key - `XI_API_KEY`: ElevenLabs API key - `AGENT_ID`: ElevenLabs agent ID ## 🏥 Features in Detail ### Real-time Voice Interaction Surgentic provides seamless voice communication during surgical procedures, allowing hands-free interaction with the AI assistant. ### Surgical Safety Checklist An interactive checklist based on WHO Surgical Safety standards, ensuring all critical steps are followed and documented. ### Screen Recording Built-in screen recording capabilities allow for procedure documentation and later review, enhancing training and quality assurance. ### Compliance Reporting Automated generation of compliance reports based on checklist completion and procedure documentation. ## 📄 License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## 🙏 References - [Next.js Documentation](https://nextjs.org/docs) - [Tailwind CSS](https://tailwindcss.com/docs) - [Supabase Documentation](https://supabase.com/docs) - [ElevenLabs Documentation](https://elevenlabs.io/docs) - [WHO Surgical Safety Checklist](https://www.who.int/teams/integrated-health-services/patient-safety/research/safe-surgery) ## 👥 Contributing Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details on our code of conduct and the process for submitting pull requests. ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 136 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (36 of 36) ``` .gitignore .prettierignore .prettierrc.json app/api/c/route.ts app/api/conversations/route.ts app/api/i/route.ts app/api/recordings/route.ts app/c/[slug]/page.tsx app/conversations/route.ts app/globals.css app/layout.tsx app/preview/page.tsx app/recording/page.tsx app/route.ts components/Checklist.tsx components/Message.tsx components/RecordingsGallery.tsx components/TextAnimation.tsx components/useTypingEffect.ts global.d.ts lib/compliance.ts lib/supabase.ts LICENSE next.config.ts package.json postcss.config.mjs README.md reference/src/App.css reference/src/App.tsx reference/src/LandingPage.css reference/src/LandingPage.tsx reference/src/main.tsx reference/src/vite-env.d.ts schema.tsx tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @11labs/react@^0.0.4, @neondatabase/serverless@^0.10.4, @supabase/supabase-js@^2.48.1, @types/node@^22.10.2, @types/react@^19.0.1, @types/react-dom@^19.0.2, dotenv@^16.4.7, framer-motion@^11.18.2, motion@^11.11.17, next@^15.1.0, postcss@^8, prettier@^3.4.2, prettier-plugin-organize-imports@^4.1.0, prettier-plugin-tailwindcss@^0.6.9, react@^19.0.0, react-dom@^19.0.0, react-feather@^2.0.10, sonner@^1.7.1, supabase@^2.12.1, tailwindcss@^3.4.1, tsx@^4.19.2, typescript@^5 ### Recent commits (newest first) - final - final - help im dying - dbhajnsda - fix - fix - Merge branch 'main' of https://github.com/SeanZCai/treehacks-agentic-surgeon - fix - fix - push - fresh - push; - ElevenLabs - Create LICENSE - Update README.md - Update README.md - Update README.md - Update README.md - Update page.tsx - Update page.tsx ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "pulse", "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "schema": "tsx ./schema.tsx", "fmt": "prettier --write '**/*' --ignore-unknown" }, "dependencies": { "@11labs/react": "^0.0.4", "@neondatabase/serverless": "^0.10.4", "@supabase/supabase-js": "^2.48.1", "dotenv": "^16.4.7", "framer-motion": "^11.18.2", "motion": "^11.11.17", "next": "^15.1.0", "react": "^19.0.0", "react-dom": "^19.0.0", "react-feather": "^2.0.10", "sonner": "^1.7.1", "supabase": "^2.12.1" }, "devDependencies": { "@types/node": "^22.10.2", "@types/react": "^19.0.1", "@types/react-dom": "^19.0.2", "postcss": "^8", "prettier": "^3.4.2", "prettier-plugin-organize-imports": "^4.1.0", "prettier-plugin-tailwindcss": "^0.6.9", "tailwindcss": "^3.4.1", "tsx": "^4.19.2", "typescript": "^5" }, "packageManager": "pnpm@9.12.3+sha512.cce0f9de9c5a7c95bef944169cc5dfe8741abfb145078c0d508b868056848a87c81e626246cb60967cbd7fd29a6c062ef73ff840d96b3c86c40ac92cf4a813ee" } ``` ### app/route.ts ```typescript export const runtime = 'edge' export const dynamic = 'force-dynamic' export const fetchCache = 'force-no-store' import { NextResponse } from 'next/server' export async function GET(request: Request) { return NextResponse.redirect(new URL(`/c/${performance.now()}_${Math.random()}`, request.url)) } ``` ### app/layout.tsx ```typescript import type { Metadata } from 'next' import { Inter } from 'next/font/google' import { Toaster } from 'sonner' import './globals.css' const interFont = Inter({ subsets: ['latin'], display: 'swap', }) const config = { url: '', title: 'Surgentic', description: 'Medical AI Assistant', } const btoa = (str: string) => Buffer.from(str).toString('base64') const images = `https://neon.tech/docs/og?title=${btoa('Pulse')}&breadcrumb=${btoa(config.title)}` export const metadata: Metadata = { title: config.title, description: config.description, openGraph: { images, url: config.url, title: config.title, description: config.description, }, twitter: { images, title: config.title, card: 'summary_large_image', description: config.description, }, } export default function ({ children }: Readonly<{ children: React.ReactNode }>) { return ( {children} ) } ``` ### app/preview/page.tsx ```typescript export default function () {} ``` ### reference/src/main.tsx ```typescript import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render( , ); ``` ### app/conversations/route.ts ```typescript import { neon, neonConfig } from '@neondatabase/serverless' import { NextResponse } from 'next/server' neonConfig.poolQueryViaFetch = true export async function GET() { if (!process.env.DATABASE_URL) { console.error('No DATABASE_URL provided') return NextResponse.json({ error: 'Database URL not configured' }) } const sql = neon(process.env.DATABASE_URL) try { // Simple count query to check if we have any messages const count = await sql`SELECT COUNT(*) FROM messages` console.log('Total messages in database:', count) if (count[0].count === '0') { return NextResponse.json({ message: 'No conversations found - database is empty', totalMessages: 0 }) } const rows = await sql(` SELECT session_id, array_agg( json_build_object( 'created_at', created_at, 'role', role, 'content', content_transcript ) ORDER BY created_at ) as messages FROM messages GROUP BY session_id ORDER BY session_id; `) return NextResponse.json(rows) } catch (error) { console.error('Detailed error:', error) return NextResponse.json({ error: 'Failed to fetch conversations', details: error instanceof Error ? error.message : 'Unknown error' }, { status: 500 }) } } ``` ### app/recording/page.tsx ```typescript 'use client' import { useSearchParams } from 'next/navigation' import { useEffect, useState } from 'react' export default function RecordingPage() { const searchParams = useSearchParams() const recordingUrl = searchParams.get('url') const [error, setError] = useState(null) useEffect(() => { if (!recordingUrl) { setError('No recording URL provided') } }, [recordingUrl]) return (

Recorded Session

{error ? (
{error}
) : (
)}
) } ``` ### reference/src/App.tsx ```typescript import { useState, useEffect, useRef } from "react"; import { invoke } from "@tauri-apps/api/core"; import LandingPage from "./LandingPage"; import "./App.css"; // Define interfaces for our data interface ChecklistItem { question: string; checked: boolean; } function App() { const [showMainApp, setShowMainApp] = useState(false); const [showConfirmDialog, setShowConfirmDialog] = useState(false); const [checklistItems, setChecklistItems] = useState([]); const [transcription, setTranscription] = useState("Listening for audio..."); // NEW: State and ref for screenshare const [screenStream, setScreenStream] = useState(null); const screenshareVideoRef = useRef(null); // NEW: When the screenStream changes, attach it to the video element useEffect(() => { if (screenshareVideoRef.current && screenStream) { screenshareVideoRef.current.srcObject = screenStream; } }, [screenStream]); // Simulate loading data from a spreadsheet useEffect(() => { // This is mock data - we replaced this with an actual Google Sheets API call const mockData: ChecklistItem[] = [ { question: "Patient identity verified", checked: false }, { question: "Surgical site marked", checked: false }, { question: "Anesthesia safety check completed", checked: false }, { question: "Pulse oximeter on patient and functioning", checked: false }, { question: "Known allergies verified", checked: false }, { question: "Difficult airway/aspiration risk assessed", checked: false }, { question: "Blood loss risk assessed", checked: false }, { question: "Essential imaging displayed", checked: false }, { question: "Antibiotic prophylaxis given", checked: false }, { question: "All team members introduced", checked: false }, { question: "Critical steps reviewed", checked: false }, { question: "Sterility confirmed", checked: false }, { question: "Equipment concerns addressed", checked: false }, { question: "Patient positioning verified", checked: false }, { question: "Temperature management plan in place", checked: false }, { question: "VTE prophylaxis plan confirmed", checked: false }, { question: "Specimen labeling reviewed", checked: false }, { question: "Equipment counts complete", checked: false }, { question: "Key concerns for recovery discussed", checked: false }, { question: "Post-op destination confirmed", checked: false }, ]; setChecklistItems(mockData); }, []); const handleCheckboxChange = (index: number) => { setChecklistItems(items => items.map((item, i) => i === index ? { ...item, checked: !item.checked } : item ) ); }; const handleBackClick = () => { setShowConfirmDialog(true); }; const handleConfirmBack = (confirmed: boolean) => { setShowConfirmDialog(false); if (confirmed) { setShowMainApp(false); } }; // NEW: Function to start screenshare using the browser's getDisplayMedia API const startScreenshare = async () => { try { const stream = await navigator.mediaDevices.getDisplayMedia({ video: true }); setScreenStream(stream); } catch (err) { console.error("Error starting screenshare:", err); } }; // NEW: Function to stop screenshare and release media tracks const stopScreenshare = () => { if (screenStream) { screenStream.getTracks().forEach(track => track.stop()); setScreenStream(null); } }; if (!showMainApp) { return setShowMainApp(true)} />; } return (
{showConfirmDialog && (

Are you sure?

Do you want to return to the home page?

)}
{/* NEW: Screenshare Section */}
{screenStream ? (
) : ( )}

Live Transcription

{transcription}

Surgical Safety Checklist

{checklistItems.map((item, index) => (