# Project export: Surgical.AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: An AI agent tracking every surgical tool for a seamless operation. - Devpost: https://devpost.com/software/surgical-ai - GitHub: https://github.com/Yatsz/CalHacks2024 - Demo: https://www.figma.com/design/640tomBSFF8KzCJWXr7P1v/calhacks-2024?node-id=45-949&t=6fC01Izq2yXGWu81-1 - Video: https://www.youtube.com/embed/xxoDXIWdy30?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Daniel Kim (20 commits), jotalis (19 commits), Aaron Ang (8 commits) ## Devpost submission (written by the team) ### Overview ❓The problem at hand. How might we leverage an AI tool tracker to ensure surgical safety for every patient? 🌱 Where did this stem from? A surgical tool "the size of a dinner plate" has been discovered inside a woman's abdomen 18 months after undergoing a c-section while giving birth to her child – ABC news. There are 3,000 surgeries like this PER YEAR where surgical tools get left inside a patient's body. You may ask, aren’t nurses there to ensure this doesn’t happen? Unfortunately, traditional methods of manual tracking are unreliable. Even with 12 medical staff present in the surgery, a surgical tool as large as the size of a dinner plate was left in the woman’s body. This ongoing risk to patient safety is unacceptable, especially when lives are on the line. 📑 What does Surgical.AI do? Our AI Tool Tracker is named Surgi. Surgi helps medical staff… Track surgical tools before and during surgeries Find the last 5 seconds of a tool in search Hands-free communication during surgery 🩻 How does it work? We constantly track and segment/highlight medical tools during surgery via a real-time video stream and update the state of the objects (in place, out of place, or missing). We periodically save snapshots of the state to our database. Medical staff can query for a tool using their voice to view a video snippet of when it was last seen. 👩‍⚕️ How we build it UI/UX Design We used Figma and FigJam to lay out the user flow, using research from notable media stations like ABC news and NBC news to support our visual designs and design system with data. Frontend Next.js, ShadCN, Tailwind CSS, Deepgram Backend Gemini, Yolov11, Firebase, Websockets 🩺 ### Challenges we ran into A challenge we faced was figuring out what technologies we could use to accurately and efficiently identify tools that matched before and during surgery. We had to find technologies that could match tools in real-time despite the noise around them. We also had to read up on a lot of new resources we used for this product. With all these backend technical advancements, we had to communicate these changes quickly to the frontend to be able to carry out our visions seamlessly. The frontend had to adjust accordingly to the viable features we were able to display. 📈 ### Accomplishments we're proud of We’re proud to have developed a powerful and well-rounded product in just 36 hours! By leveraging the expertise of our front-end and back-end teams, we created a fully functional tool designed to tackle an urgent and critical issue: ensuring surgical tools are tracked to prevent errors and enhance patient safety. 🍵 ### What we learned We learned that many patients and surgeons are anxious about the risks associated with surgical errors, especially given that thousands occur each year. Surprisingly, there aren’t any comprehensive tools currently available that accurately track surgical instruments in real-time, which is quite concerning. We also discovered that we could create the tool we wished existed to ensure patient safety. Learning how to leverage Meta SAM2 for tool detection and integrating it with a voice agent AI was an exciting new skill we developed in the process. ⏭ ### What's next We strive to implement Surgical.AI into healthcare educational systems to help train surgeons and medical staff with our tool-detecting system. We also plan on using our advanced software in other fields that require accurate detection as well. For example, spotting shoplifters at stores. ## README (from the GitHub repository) first commit ## Detected evidence (automated analysis) Indexed codebase: 34 recognized source files, 72 KB. - CSS (language) — detected in the code - Firebase (technology) — detected in the code - Google Gemini (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (50 of 50) ``` .DS_Store .gitignore backend/.gitignore backend/download_firebase.py backend/Experiments/determine_mask.py backend/Experiments/gemini_testing.ipynb backend/Experiments/get_data.ipynb backend/Experiments/yolo_live.ipynb backend/Experiments/yolo_testing.ipynb backend/Experiments/yolo_training.ipynb backend/gemini_agent.py backend/openai_agent.py backend/requirements.txt backend/ws_server_save.py backend/ws_server.py frontend/.eslintrc.json frontend/.gitignore frontend/components.json frontend/next.config.mjs frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/src/app/globals.css frontend/src/app/layout.tsx frontend/src/app/page.tsx frontend/src/app/selection/page.tsx frontend/src/app/surgery/page.tsx frontend/src/components/sound-bar.tsx frontend/src/components/surgical-chat.tsx frontend/src/components/surgical-header.tsx frontend/src/components/surgical-replay.tsx frontend/src/components/surgical-tools.tsx frontend/src/components/surgical-video.tsx frontend/src/components/tool-context.tsx frontend/src/components/tool-status.tsx frontend/src/components/ui/button.tsx frontend/src/components/ui/text-generate-effect.tsx frontend/src/components/ui/typewriter-effect.tsx frontend/src/lib/firebase.ts frontend/src/lib/gemini.ts frontend/src/lib/utils.ts frontend/tailwind.config.ts frontend/tsconfig.json README.md temp_backend/client_start.sh temp_backend/hyperbolic_setup.sh temp_backend/process_stream.py temp_backend/web_rtc.py temp_backend/ws_client.py temp_backend/ws_server.py ``` ### Dependencies - backend/requirements.txt: firebase_admin@==6.5.0, langchain_core@==0.3.12, moviepy@==1.0.3, opencv_python@==4.10.0.82, Pillow@==11.0.0, protobuf@==5.28.2, python-dotenv@==1.0.1, ultralytics@==8.3.18, websockets@==13.1 - frontend/package.json: @deepgram/sdk@^3.8.1, @google/generative-ai@^0.21.0, @radix-ui/react-slot@^1.1.0, @types/node@^20, @types/react@^18, @types/react-dom@^18, class-variance-authority@^0.7.0, clsx@^2.1.1, cross-fetch@^4.0.0, dotenv@^16.4.5, eslint@^8, eslint-config-next@14.2.15, firebase@^10.14.1, framer-motion@^11.11.9, lucide-react@^0.453.0, next@14.2.15, openai@^4.68.1, postcss@^8, react@^18, react-dom@^18, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 ### Recent commits (newest first) - fix: lag spikes - initial hyperbolic scripts - chore: add files - handle LLM inconsistency for mising tools - output all three tools from LLM, choose best frame - chore: merge server - Merge branch 'main' of https://github.com/Yatsz/CalHacks2024 - chore: tweak gemini prompting - feat: add openai agent - modify requirements.txt - improve gemini prompting - Animation pop up - Merge remote-tracking branch 'origin/main' - Completed frontend - feat: add gemini stuff - Fixed firebase upload logic - Merge remote-tracking branch 'origin/main' - Update to work - send two state metadata via websocket - upload interval vids to firebase ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/requirements.txt ``` firebase_admin==6.5.0 langchain_core==0.3.12 moviepy==1.0.3 opencv_python==4.10.0.82 Pillow==11.0.0 protobuf==5.28.2 python-dotenv==1.0.1 ultralytics==8.3.18 websockets==13.1 ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@deepgram/sdk": "^3.8.1", "@google/generative-ai": "^0.21.0", "@radix-ui/react-slot": "^1.1.0", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "cross-fetch": "^4.0.0", "dotenv": "^16.4.5", "firebase": "^10.14.1", "framer-motion": "^11.11.9", "lucide-react": "^0.453.0", "next": "14.2.15", "openai": "^4.68.1", "react": "^18", "react-dom": "^18", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import localFont from "next/font/local"; import "./globals.css"; const geistSans = localFont({ src: "./fonts/GeistVF.woff", variable: "--font-geist-sans", weight: "100 900", }); const geistMono = localFont({ src: "./fonts/GeistMonoVF.woff", variable: "--font-geist-mono", weight: "100 900", }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### frontend/src/app/page.tsx ```typescript import Image from 'next/image' import Link from 'next/link' import { Button } from "@/components/ui/button" import Svg1 from '../../public/picOne.svg' import Svg2 from '../../public//picTwo.svg' import Svg3 from '../../public//picThree.svg' import Svg4 from '../../public//picFour.svg' import logo from '../../public/white_logo.svg' export default function LandingPage() { return (
white logo

An AI Tool Tracker for Your Surgical Safety

SVG 1 SVG 2 SVG 3 SVG 4
) } ``` ### frontend/src/app/selection/page.tsx ```typescript import Link from 'next/link' export default function SelectionPage() { return (

Selection Page

Make your selection here.

Proceed to Surgery Page
) } ``` ### frontend/src/app/surgery/page.tsx ```typescript 'use client' import Link from 'next/link' import SurgicalHeader from '@/components/surgical-header' import SurgicalChat from '@/components/surgical-chat' import SurgicalVideo from '@/components/surgical-video' import { ToolProvider } from '@/components/tool-context' export default function SurgeryPage() { return (
) } ``` ### temp_backend/client_start.sh ```shell #!/bin/bash ssh -L 1935:localhost:1935 ubuntu@host ffmpeg -f avfoundation -framerate 30 -i "0" \ -vcodec libx264 -preset ultrafast -tune zerolatency \ -maxrate 3000k -bufsize 6000k \ -pix_fmt yuv420p -g 30 -f flv rtmp://localhost/live/stream ``` ### temp_backend/ws_client.py ```python import asyncio import websockets async def receive_metadata(): uri = "ws://localhost:8765" async with websockets.connect(uri) as websocket: print("Connected to WebSocket server") while True: metadata = await websocket.recv() print(f"Received metadata: {metadata}") if __name__ == "__main__": asyncio.run(receive_metadata()) ``` ### temp_backend/hyperbolic_setup.sh ```shell sudo apt-get update sudo apt-get install linux-headers-$(uname -r) dkms sudo apt-get install ffmpeg sudo apt install nginx sudo apt install libnginx-mod-rtmp sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak sudo tee -a /etc/nginx/nginx.conf <<'EOF' rtmp { server { listen 1935; chunk_size 4096; application live { live on; record off; } } } EOF sudo nginx -t sudo service nginx restart ``` ### backend/download_firebase.py ```python import argparse from firebase_admin import credentials, initialize_app, storage def main(source_path: str, output_path: str = None): cred = credentials.Certificate( "calhacks2024-c1a62-firebase-adminsdk-9obo0-63385ce9b4.json" ) initialize_app(cred, {"storageBucket": "calhacks2024-c1a62.appspot.com"}) bucket = storage.bucket() blob = bucket.blob(source_path) if output_path is None: output_path = source_path blob.download_to_filename(output_path) if __name__ == "__main__": args = argparse.ArgumentParser() args.add_argument("path", type=str, help="Path of the file in Firebase") args.add_argument( "-o", "--output", type=str, default=None, help="Path to the output file (default: same as input)", ) args = args.parse_args() main(args.path, args.output) ``` [24 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]