# Project export: VIPER This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Surgical Intelligence - Devpost: https://devpost.com/software/viper-3dr107 - GitHub: https://github.com/kkaura28/treehacks-2026 - Demo: https://viper-three-theta.vercel.app/ - Video: https://www.youtube.com/embed/_VdADTQNmL4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 6 GitHub contributor(s) — kkaura28 (24 commits), rrnM12 (4 commits), Adam Dai (3 commits), Ishan Baliyan (2 commits), Sukeerth R (2 commits), Cursor (1 commits) ## Devpost submission (written by the team) ### Inspiration Medical errors kill an estimated 2.6–3 million people globally each year. Every surgery follows a protocol, yet compliance is formally tracked in only a small fraction of cases. Feedback is delayed, manual, and often limited to post-operative notes. We asked: What if the operating room had a flight data recorder and a live command center? What It Does VIPER transforms first-person surgical video into structured, clinically actionable intelligence. Using smart-glasses/video capture, the vision pipeline extracts instrument and anatomy segmentation, tracks instrument tip trajectories across frames, estimates full 6D instrument pose, recovers surgeon hand joint positions, and segments continuous motion into discrete surgical strokes. This spatial data then feeds into a reasoning layer that maps observed actions to structured surgical protocols, flags missing, out-of-order, or unsafe steps, and cross-references each deviation against published surgical literature. But VIPER isn't just a backend engine. All of this surfaces in a purpose-built analytics platform. Surgeons and administrators get a real-time OR command center that tracks live procedure progress, an interactive timeline view with synchronized video playback alongside structured step events, and a deviation explorer that presents evidence-backed cards with literature context for every flagged issue. The platform also includes a procedure graph visualization that renders the full protocol as a directed graph color-coded by execution status, a skills assessment dashboard that maps 6DoF motion metrics to validated surgical skill frameworks, and full FHIR/EHR export for standards-compliant reports ready for hospital systems. Beyond post-op review, there's a pre-op voice mode that lets surgeons get briefed about a patient through a conversational agent and surgeon-level analytics for tracking individual performance over time. How We Built It The system is composed of two tightly integrated pipelines. On the vision side, raw video first passes through SAM 3 for pixel-level segmentation, then optical flow tracks keypoints across frames while depth models provide spatial context. FoundationPose handles full 6D instrument pose estimation, and MediaPipe recovers surgeon hand joints. Together, these produce per-frame trajectories and motion signals, which are then segmented into discrete surgical strokes using velocity profiling, spectral analysis (SPARC smoothness via FFT), and high-frequency tremor decomposition. From there, bimanual coordination is quantified through cross-correlation of hand velocity vectors, and motion economy is derived from path length ratios in both 2D pixel space and 3D world coordinates. On the reasoning side, Gemini watches the procedure and maps observed actions to protocol steps. A comparator then diffs the observed sequence against the expected SOP graph to detect deviations, and Scite retrieves relevant research snippets to provide evidence-backed context for each flag. The final outputs are structured and exportable as FHIR-compliant surgical reports. We validated the full pipeline on controlled "banana surgery" experiments before moving to medical footage, using them to stress-test segmentation and tracking reliability under controlled conditions. The analytics platform is built with Next.js 14 and Supabase for Postgres-backed storage and real-time subscriptions. The UI uses Tailwind CSS with a custom dark theme designed for clinical environments, ReactFlow for interactive procedure graph visualization, and Recharts for radar and bar chart rendering. Real-time updates in the OR command center are powered by Supabase channel subscriptions on observed events and procedure run tables. Challenges We Ran Into Surgical video is inherently messy. Frequent occlusions from hands and instruments, rapid lighting changes, constant camera movement from the first-person POV, and long-duration procedures with shifting visual context all make stable tracking extremely difficult. Maintaining consistent segmentation and pose estimation across an entire case required careful orchestration of multiple models working in concert. A deeper challenge was translating low-level motion data into clinically meaningful feedback. Surgeons want to know whether a safety-critical step was skipped and what the literature says about it. Bridging raw computer vision outputs with protocol-aware reasoning in a way that produces actionable, interpretable reports was essential to making the system useful in practice. Accomplishments We're Proud Of We built a complete end-to-end system that takes raw smart-glasses video and produces FHIR-compliant surgical reports, integrating segmentation, 6DoF pose estimation, and stroke-level motion analysis into a single cohesive pipeline. On the platform side, we designed a real-time OR dashboard with structured protocol tracking and implemented deviation detection backed by live literature retrieval. Most importantly, we created an interface that surgeons can actually review in minutes rather than hours. VIPER turns surgery into structured data and makes that data usable. What We Learned The immediate value is improved compliance and reduced error. Hospitals can save an estimated $2.5M–$3.6M annually. But the long-term opportunity is even larger. Robotics companies need high-quality, timestamped, stroke-level surgical motion data to train autonomous systems. Just as dashcams became foundational training data for self-driving cars, structured surgical video at scale becomes the training substrate for surgical robotics. What's Next for VIPER We're expanding to more procedure types and tightening the deviation-to-citation pipeline to make reports even more actionable. From there, we plan to launch pilot integrations with hospital systems and build out longitudinal surgeon performance analytics for credentialing and training feedback. Long term, every surgery generates a structured, auditable, ML-ready record, improving patient safety today and enabling surgical autonomy tomorrow. Credentials for Website Email: surgeon@viper.com Password: viper26! ## README (from the GitHub repository) # treehacks-2026 Devpost https://devpost.com/software/viper-3dr107#updates VIPER is an AI platform that watches first-person surgical video (via smart glasses) and extracts detailed spatial data like instrument tracking, pose estimation, hand positions, and surgical stroke segmentation, then reasons over it to map actions to protocols and flag deviations backed by literature. Everything surfaces through an analytics platform with live OR tracking, interactive timelines, deviation cards, protocol graphs, skills dashboards, EHR export, and a pre-op voice briefing mode, so it's essentially an AI system that watches surgery happen and tells you, in real time and after the fact, what went right, what didn't, and why. ## Setup (including FoundationPose) This repo uses [FoundationPose](https://github.com/NVlabs/FoundationPose) (NVlabs 6D pose estimation) as a **Git submodule**. **First-time clone (get everything in one step):** ```bash git clone --recurse-submodules https://github.com/YOUR_USERNAME/treehacks-2026.git ``` **If you already cloned without submodules:** ```bash git submodule update --init --recursive ``` Then follow FoundationPose’s own setup (Docker or conda, weights, etc.) inside `FoundationPose/` — see `FoundationPose/readme.md`. **RTX 5090 / Blackwell (sm_120):** Use the `fp5090` conda env (Python 3.11 + PyTorch nightly cu128). From repo root: `conda activate fp5090`, then `cd FoundationPose && python run_demo.py`. The demo uses a Python fallback for rotation clustering by default to avoid a C++ extension crash on this setup. ## Detected evidence (automated analysis) Indexed codebase: 87 recognized source files, 485 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Hugging Face (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 2821) ``` .gitignore .gitmodules analysis/__init__.py analysis/scripts/generate_synth_data.py analysis/src/__init__.py analysis/src/analysis_config.py analysis/src/analysis_pipeline.py analysis/src/annotate_video_final.py analysis/src/annotate_video.py analysis/src/bounding_box.ipynb analysis/src/data_types.py analysis/src/depth_detection.py analysis/src/event_detector.py analysis/src/frame_loader.py analysis/src/motion_metrics.py analysis/src/output_writer.py analysis/src/sam3.py analysis/src/signal_processing.py analysis/src/summary_stats.py analysis/src/track_tool_final.py analysis/src/track_tool_tip_sam3_reg.py analysis/src/track_tool_tip_sam3.py analysis/src/track_tool_tip.py analysis/src/track_tool_tips_sam3_reg.py analysis/tutorials/analyze_video.ipynb analysis/tutorials/depth_image.ipynb analysis/tutorials/depth_video.ipynb analysis/tutorials/diff_video.ipynb analysis/tutorials/sam3_image.ipynb analysis/tutorials/sam3_video.ipynb analysis/tutorials/track_tool_tip.ipynb analysis/visualization/input_visuals.ipynb analysis/visualization/output_visuals.ipynb analysis/visualization/tracking_visuals.ipynb Analytics_UI/.gitignore Analytics_UI/next-env.d.ts Analytics_UI/next.config.ts Analytics_UI/package.json Analytics_UI/postcss.config.mjs Analytics_UI/public/data/centroids.json Analytics_UI/public/data/hand_trajectories.csv Analytics_UI/public/data/kinematics.json Analytics_UI/public/data/tracked_tips_blade.csv Analytics_UI/public/data/tracked_tips_tweezer.csv Analytics_UI/README.md Analytics_UI/scripts/compute_kinematics.py Analytics_UI/src/app/api/chat/route.ts Analytics_UI/src/app/briefings/[id]/page.tsx Analytics_UI/src/app/briefings/page.tsx Analytics_UI/src/app/command-center/page.tsx Analytics_UI/src/app/globals.css Analytics_UI/src/app/layout.tsx Analytics_UI/src/app/login/page.tsx Analytics_UI/src/app/page.tsx Analytics_UI/src/app/procedures/[id]/page.tsx Analytics_UI/src/app/sessions/[id]/page.tsx Analytics_UI/src/app/surgeons/[name]/page.tsx Analytics_UI/src/app/surgeons/page.tsx Analytics_UI/src/components/app-shell.tsx Analytics_UI/src/components/auth-guard.tsx Analytics_UI/src/components/badges.tsx Analytics_UI/src/components/deviations-tab.tsx Analytics_UI/src/components/graph-tab.tsx Analytics_UI/src/components/overview-tab.tsx Analytics_UI/src/components/report-tab.tsx Analytics_UI/src/components/skills-tab.tsx Analytics_UI/src/components/timeline-tab.tsx Analytics_UI/src/lib/patient-data.ts Analytics_UI/src/lib/supabase.ts Analytics_UI/src/lib/types.ts Analytics_UI/src/lib/utils.ts Analytics_UI/tsconfig.json DEVPOST.md FHIR_Generation/__init__.py FHIR_Generation/fhir_mapper.py FHIR_Generation/main.py FHIR_Generation/README.md FHIR_Generation/requirements.txt parametric_data/analyze_rgb_depth_masks.py parametric_data/apple_npz_to_depth_png.py parametric_data/calculate_intrinsics.py parametric_data/delete_odd_frames.py parametric_data/depth_to_redblue_vis.py parametric_data/downscale_rgb_to_depth.py parametric_data/export_single_mesh_obj.py parametric_data/gurt/apple/000000.npz parametric_data/gurt/apple/000001.npz parametric_data/gurt/apple/000002.npz parametric_data/gurt/apple/000003.npz parametric_data/gurt/apple/000004.npz parametric_data/gurt/apple/000005.npz parametric_data/gurt/apple/000006.npz parametric_data/gurt/apple/000007.npz parametric_data/gurt/apple/000008.npz parametric_data/gurt/apple/000009.npz parametric_data/gurt/apple/000010.npz parametric_data/gurt/apple/000011.npz parametric_data/gurt/apple/000012.npz parametric_data/gurt/apple/000013.npz parametric_data/gurt/apple/000014.npz parametric_data/gurt/apple/000015.npz parametric_data/gurt/apple/000016.npz parametric_data/gurt/apple/000017.npz parametric_data/gurt/apple/000018.npz parametric_data/gurt/apple/000019.npz parametric_data/gurt/apple/000020.npz parametric_data/gurt/apple/000021.npz parametric_data/gurt/apple/000022.npz parametric_data/gurt/apple/000023.npz parametric_data/gurt/apple/000024.npz parametric_data/gurt/apple/000025.npz parametric_data/gurt/apple/000026.npz parametric_data/gurt/apple/000027.npz parametric_data/gurt/apple/000028.npz parametric_data/gurt/apple/000029.npz parametric_data/gurt/apple/000030.npz parametric_data/gurt/apple/000031.npz parametric_data/gurt/apple/000032.npz parametric_data/gurt/apple/000033.npz parametric_data/gurt/apple/000034.npz [2701 more files omitted for size] ``` ### Dependencies - Analytics_UI/package.json: @elevenlabs/react@^0.14.0, @supabase/supabase-js@^2.95.3, @tailwindcss/postcss@^4.1.18, @types/node@^25.2.3, @types/react@^19.2.14, @types/react-dom@^19.2.3, @xyflow/react@^12.10.0, autoprefixer@^10.4.24, class-variance-authority@^0.7.1, clsx@^2.1.1, geist@^1.7.0, lucide-react@^0.564.0, next@^16.1.6, postcss@^8.5.6, react@^19.2.4, react-dom@^19.2.4, recharts@^3.7.0, tailwind-merge@^3.4.0, tailwindcss@^4.1.18, typescript@^5.9.3 - FHIR_Generation/requirements.txt: fastapi@>=0.109.0, pydantic@>=2.5.0, uvicorn@>=0.27.0 - ScitePipeline/requirements.txt: fastapi@>=0.109.0, google-genai@>=1.0.0, httpx@>=0.27.0, pydantic@>=2.5.0, pydantic-settings@>=2.1.0, python-dotenv@>=1.0.0, supabase@>=2.3.0, torch@>=2.1.0, transformers@>=4.36.0, uvicorn@>=0.27.0 ### Recent commits (newest first) - Update README.md - Update README.md - Merge pull request #8 from kkaura28/add-rtx-5090 - all combined stuff - Merge pull request #7 from kkaura28/dev-sukeerth - Merge branch 'main' into dev-sukeerth - Update timeline-tab.tsx - Create joints_overlay.mp4 - devpost - Point tracking done! - New Video - Hands - Kinetmatics - Tip vvideo - New video - Merge pull request #6 from kkaura28/dev-karn-2 - Update layout.tsx - Merge pull request #5 from kkaura28/dev-karn-2 - Timeline tab - Export fhir automatically ## Key source files (fetched from GitHub, selected and truncated for size) ### DEVPOST.md ```markdown ## Inspiration Medical errors kill an estimated 2.6–3 million people globally each year. Every surgery follows a protocol, yet compliance is formally tracked in only a small fraction of cases. Feedback is delayed, manual, and often limited to post-operative notes. We asked: **What if the operating room had a flight data recorder and a live command center?** ## What It Does VIPER transforms first-person surgical video into structured, clinically actionable intelligence. Using smart-glasses capture, the vision pipeline extracts instrument and anatomy segmentation, tracks instrument tip trajectories across frames, estimates full 6D instrument pose, recovers surgeon hand joint positions, and segments continuous motion into discrete surgical strokes. This spatial data then feeds into a reasoning layer that maps observed actions to structured surgical protocols, flags missing, out-of-order, or unsafe steps, and cross-references each deviation against published surgical literature. But VIPER isn't just a backend engine. all of this surfaces in a purpose-built analytics platform. Surgeons and administrators get a real-time OR command center that tracks live procedure progress, an interactive timeline view with synchronized video playback alongside structured step events, and a deviation explorer that presents evidence-backed cards with literature context for every flagged issue. The platform also includes a procedure graph visualization that renders the full protocol as a directed graph color-coded by execution status, a skills assessment dashboard that maps 6DoF motion metrics to validated surgical skill frameworks, and full FHIR/EHR export for standards-compliant reports ready for hospital systems. Beyond post-op review, there's a pre-op voice mode that lets surgeons get briefed about a patient through a conversational agent, surgeon-level analytics for tracking individual performance over time, and a mock command center for monitoring real-time surgeries as they happen. VIPER doesn't just tell you what happened. It shows you how it happened, whether it followed protocol, and why it matters. ## How We Built It The system is composed of two tightly integrated pipelines. On the vision side, raw video first passes through SAM 2 for pixel-level segmentation, then optical flow tracks keypoints across frames while depth models provide spatial context. FoundationPose handles full 6D instrument pose estimation, and MediaPipe recovers surgeon hand joints. Together, these produce per-frame trajectories and motion signals, which are then segmented into discrete surgical strokes using velocity profiling, spectral analysis (SPARC smoothness via FFT), and high-frequency tremor decomposition. From there, bimanual coordination is quantified through cross-correlation of hand velocity vectors, and motion economy is derived from path length ratios in both 2D pixel space and 3D world coordinates. The analytics platform is built with Next.js 14 and Supabase for P [truncated — 3381 more characters] ``` ### FHIR_Generation/requirements.txt ``` fastapi>=0.109.0 uvicorn>=0.27.0 pydantic>=2.5.0 ``` ### ScitePipeline/requirements.txt ``` fastapi>=0.109.0 uvicorn>=0.27.0 supabase>=2.3.0 httpx>=0.27.0 pydantic>=2.5.0 pydantic-settings>=2.1.0 python-dotenv>=1.0.0 transformers>=4.36.0 torch>=2.1.0 google-genai>=1.0.0 ``` ### Analytics_UI/package.json ``` {"name":"analytics-ui","version":"0.1.0","private":true,"scripts":{"dev":"next dev","build":"next build","start":"next start"},"dependencies":{"@elevenlabs/react":"^0.14.0","@supabase/supabase-js":"^2.95.3","@tailwindcss/postcss":"^4.1.18","@types/node":"^25.2.3","@types/react":"^19.2.14","@types/react-dom":"^19.2.3","@xyflow/react":"^12.10.0","autoprefixer":"^10.4.24","class-variance-authority":"^0.7.1","clsx":"^2.1.1","geist":"^1.7.0","lucide-react":"^0.564.0","next":"^16.1.6","postcss":"^8.5.6","react":"^19.2.4","react-dom":"^19.2.4","recharts":"^3.7.0","tailwind-merge":"^3.4.0","tailwindcss":"^4.1.18","typescript":"^5.9.3"}} ``` ### ScitePipeline/main.py ```python """ FastAPI orchestrator for the post-op compliance analysis pipeline. Endpoints: POST /mock — generate mock events for a demo run POST /analyze/{id} — run full analysis on a completed procedure run GET /report/{id} — retrieve a stored report """ from fastapi import FastAPI, HTTPException from contextlib import asynccontextmanager from config import get_supabase from mock_events import generate_mock_events from comparator import compare from adjudicator import adjudicate from report import generate_report @asynccontextmanager async def lifespan(app: FastAPI): # Startup: verify Supabase connection sb = get_supabase() yield app = FastAPI( title="Surgical Compliance Pipeline", description="Post-op analysis: graph comparison + OpenEvidence adjudication", version="0.1.0", lifespan=lifespan, ) # ── POST /mock ───────────────────────────────────────────── @app.post("/mock") def create_mock_run(): """Generate a mock procedure run with deliberate deviations.""" result = generate_mock_events() return { "status": "created", "procedure_run_id": result["procedure_run_id"], "event_count": result["event_count"], "deviations_baked_in": [ "critical_view_of_safety — MISSING", "clip_cystic_duct before clip_cystic_artery — OUT OF ORDER", "antibiotic_prophylaxis — MISSING", ], } # ── POST /analyze/{procedure_run_id} ────────────────────── @app.post("/analyze/{procedure_run_id}") async def analyze_procedure(procedure_run_id: str): """ Run the full post-op analysis pipeline: 1. Load observed events from Supabase 2. Compare against gold-standard graph 3. Adjudicate deviations via OpenEvidence 4. Generate and store compliance report """ sb = get_supabase() # Fetch the procedure run run_resp = ( sb.table("procedure_runs") .select("*") .eq("id", procedure_run_id) .execute() ) if not run_resp.data: raise HTTPException(404, f"Procedure run {procedure_run_id} not found") run = run_resp.data[0] procedure_id = run["procedure_id"] # Get procedure metadata proc_resp = ( sb.table("procedures") .select("*") .eq("id", procedure_id) .execute() ) if not proc_resp.data: raise HTTPException(404, f"Procedure {procedure_id} not found") procedure_name = proc_resp.data[0]["name"] # Count expected mandatory nodes nodes_resp = ( sb.table("nodes") .select("id") .eq("procedure_id", procedure_id) .eq("mandatory", True) .execute() ) total_expected = len(nodes_resp.data) # Count observed events events_resp = ( sb.table("observed_events") .select("id") .eq("procedure_run_id", procedure_run_id) .execute() ) total_observed = len(events_resp.data) # ── Step 1: Compare ──────────────────────────────────── raw_deviations = compare(procedure_id, procedure_run_id) # ── Step 2: Adjudicate via OpenEvidence ──────────────── adjudicated = await adjudicate(raw_deviations, procedure_name) # ── Step 3: Generate report ──────────────────────────── compliance_report = generate_report( procedure_run_id=procedure_run_id, procedure_id=procedure_id, procedure_name=procedure_name, adjudicated=adjudicated, total_expected=total_expected, total_observed=total_observed, ) # ── Step 4: Store in Supabase ────────────────────────── sb.table("deviation_reports").upsert({ "procedure_run_id": procedure_run_id, "compliance_score": compliance_report.compliance_score, "total_expected": compliance_report.total_expected, "total_observed": compliance_report.total_observed, "confirmed_count": compliance_report.confirmed_count, "mitigated_count": compliance_report.mitigated_count, "review_count": compliance_report.review_count, "raw_deviations": [d.model_dump() for d in raw_deviations], "adjudicated": [d.model_dump() for d in adjudicated], "report_text": compliance_report.report_text, }).execute() return compliance_report.model_dump() # ── GET /report/{procedure_run_id} ───────────────────────── @app.get("/report/{procedure_run_id}") def get_report(procedure_run_id: str): """Retrieve a stored compliance report.""" sb = get_supabase() resp = ( sb.table("deviation_reports") .select("*") .eq("procedure_run_id", procedure_run_id) .execute() ) if not resp.data: raise HTTPException(404, "Report not found. Run /analyze first.") return resp.data[0] # ── GET /report/{procedure_run_id}/text ──────────────────── @app.get("/report/{procedure_run_id}/text") def get_report_text(procedure_run_id: str): """Retrieve just the human-readable report text.""" sb = get_supabase() resp = ( sb.table("deviation_reports") .select("report_text") .eq("procedure_run_id", procedure_run_id) .execute() ) if not resp.data: raise HTTPException(404, "Report not found. Run /analyze first.") from fastapi.responses import PlainTextResponse return PlainTextResponse(resp.data[0]["report_text"]) # ── Run server ───────────────────────────────────────────── if __name__ == "__main__": import uvicorn uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=True) ``` ### FHIR_Generation/main.py ```python """ FHIR Generation service for the surgical compliance pipeline. Converts live-video-derived surgical data into FHIR R4 Bundles that can be submitted to hospital EHR/FHIR servers — eliminating the manual operative-report backlog. Endpoints: POST /fhir/generate — convert a ComplianceReport payload to FHIR POST /fhir/from-pipeline/{id} — fetch a stored report from Supabase → FHIR POST /fhir/from-video — end-to-end: video → analysis → FHIR Bundle """ from __future__ import annotations import sys import json import logging from pathlib import Path from datetime import datetime, timezone from fastapi import FastAPI, HTTPException, UploadFile, File, Form from fastapi.responses import JSONResponse from pydantic import BaseModel, Field # Allow importing ScitePipeline modules when running standalone _SCITE_DIR = str(Path(__file__).resolve().parent.parent / "ScitePipeline") if _SCITE_DIR not in sys.path: sys.path.insert(0, _SCITE_DIR) from fhir_mapper import build_fhir_bundle logger = logging.getLogger(__name__) app = FastAPI( title="FHIR Generation Service", description="Converts surgical video analysis into FHIR R4 Bundles for EHR integration", version="0.1.0", ) # ── Request / Response models ───────────────────────────── class DeviationPayload(BaseModel): node_id: str node_name: str phase: str = "" deviation_type: str verdict: str = "confirmed" evidence_summary: str = "" citations: list[str] = [] original_mandatory: bool = True original_safety_critical: bool = False class CompliancePayload(BaseModel): """Mirrors ComplianceReport from the ScitePipeline.""" procedure_run_id: str procedure_id: str procedure_name: str compliance_score: float = Field(ge=0, le=1) total_expected: int total_observed: int confirmed_count: int = 0 mitigated_count: int = 0 review_count: int = 0 confirmed_deviations: list[DeviationPayload] = [] mitigated_deviations: list[DeviationPayload] = [] review_deviations: list[DeviationPayload] = [] report_text: str = "" class ObservedEventPayload(BaseModel): node_id: str timestamp: str = "" confidence: float = 1.0 source: str = "gemini" class FHIRGenerateRequest(BaseModel): compliance_report: CompliancePayload observed_events: list[ObservedEventPayload] = [] video_url: str = "" patient_name: str = "Surgical Patient" surgeon_name: str = "Attending Surgeon" class FHIRFromVideoRequest(BaseModel): procedure_json_path: str = Field( ..., description="Path to procedure SOP JSON (e.g. SOP/data/abcess_data/incision_drainage_abscess.json)" ) video_path: str = Field(..., description="Path to surgical video file") patient_name: str = "Surgical Patient" surgeon_name: str = "Attending Surgeon" # ── POST /fhir/generate ────────────────────────────────── @app.post("/fhir/generate") def generate_fhir(req: FHIRGenerateRequest): """ Convert a compliance report + observed events into a FHIR R4 Bundle. Accepts the same data the ScitePipeline produces and returns a transaction Bundle ready for submission to any FHIR R4 server. """ report_dict = req.compliance_report.model_dump() events = [ev.model_dump() for ev in req.observed_events] bundle = build_fhir_bundle( compliance_report=report_dict, observed_events=events if events else None, video_url=req.video_url, patient_name=req.patient_name, surgeon_name=req.surgeon_name, ) return JSONResponse(content=bundle, media_type="application/fhir+json") # ── POST /fhir/from-pipeline/{procedure_run_id} ────────── @app.post("/fhir/from-pipeline/{procedure_run_id}") def fhir_from_pipeline(procedure_run_id: str, video_url: str = ""): """ Fetch a stored compliance report from Supabase and convert it to FHIR. Requires ScitePipeline's Supabase config (.env in ScitePipeline/). """ try: from config import get_supabase except Exception: raise HTTPException(500, "Supabase config not available. Ensure .env is set up in ScitePipeline/.") sb = get_supabase() # Fetch stored report resp = ( sb.table("deviation_reports") .select("*") .eq("procedure_run_id", procedure_run_id) .execute() ) if not resp.data: raise HTTPException(404, f"No report found for run {procedure_run_id}. Run /analyze first.") stored = resp.data[0] # Fetch procedure metadata run_resp = sb.table("procedure_runs").select("procedure_id").eq("id", procedure_run_id).execute() procedure_id = run_resp.data[0]["procedure_id"] if run_resp.data else "" proc_resp = sb.table("procedures").select("name").eq("id", procedure_id).execute() procedure_name = proc_resp.data[0]["name"] if proc_resp.data else "Unknown Procedure" # Fetch observed events for richer FHIR Observations events_resp = ( sb.table("observed_events") .select("*") .eq("procedure_run_id", procedure_run_id) .order("timestamp") .execute() ) observed_events = events_resp.data or [] # Build node lookup for step names node_lookup: dict[str, dict] = {} if procedure_id: nodes_resp = sb.table("nodes").select("id, name, phase").eq("procedure_id", procedure_id).execute() node_lookup = {n["id"]: n for n in (nodes_resp.data or [])} report_dict = { "procedure_run_id": procedure_run_id, "procedure_id": procedure_id, "procedure_name": procedure_name, "compliance_score": stored.get("compliance_score", 0), "total_expected": stored.get("total_expected", 0), "total_observed": stored.get("total_observed", 0), "confirmed_count": stored.get("confirmed_count", 0), "mitigated_count": stored.get("mitigated_count", 0), "review_count": stored.get("review_count", 0), "confirmed_deviat [truncated — 5799 more characters] ``` ### parametric_data/main.py ```python #!/usr/bin/env python3 """Run process_videos.py on a video, or FoundationPose run_demo. Usage: python main.py --option 1 /path/to/video.MOV | --option 2 /path/to/scene_dir""" import argparse import glob import os import shutil import subprocess import sys import tempfile import time def thin_rgb_to_match_depth(scene_dir: str, depth_npz_path: str) -> None: """If rgb has more frames than depth, keep evenly spaced rgb frames and renumber so counts and timing match.""" import numpy as np rgb_dir = os.path.join(scene_dir, "rgb") if not os.path.isdir(rgb_dir): return data = np.load(depth_npz_path) if "depth" not in data: data.close() return n_depth = data["depth"].shape[0] data.close() rgb_files = sorted(glob.glob(os.path.join(rgb_dir, "*.png"))) n_rgb = len(rgb_files) if n_rgb <= n_depth: return # Keep n_depth frames evenly spaced from 0..n_rgb-1; then renumber to 000000, 000001, ... if n_depth <= 0: return if n_depth == 1: keep_indices = [0] else: keep_indices = [int(round(i * (n_rgb - 1) / (n_depth - 1))) for i in range(n_depth)] tmp_dir = tempfile.mkdtemp(prefix="rgb_thin_") try: for i, src_idx in enumerate(keep_indices): src = rgb_files[src_idx] dst = os.path.join(tmp_dir, f"{i:06d}.png") shutil.copy2(src, dst) for f in rgb_files: os.remove(f) for i in range(n_depth): shutil.move(os.path.join(tmp_dir, f"{i:06d}.png"), os.path.join(rgb_dir, f"{i:06d}.png")) finally: shutil.rmtree(tmp_dir, ignore_errors=True) print(f"Thinned rgb from {n_rgb} to {n_depth} frames (temporal match with depth).", flush=True) def main(): parser = argparse.ArgumentParser(description="Parametric data pipeline or FoundationPose run_demo.") parser.add_argument("--option", type=int, choices=(1, 2, 3, 4), default=None, help="1 = full pipeline (video); 2 = FoundationPose (scene_dir); 3 = hand joints (video); 4 = Apple Depth Pro + depth PNGs (rgb folder)") parser.add_argument("path", type=str, nargs="?", default=None, help="Video path (option 1 or 3), scene directory (option 2), or rgb folder (option 4)") parser.add_argument("--long_way", action="store_true", help="(Option 2 only) Run FoundationPose on consecutive two-frame windows for the full sequence") parser.add_argument("--mesh", type=str, default=None, help="(Option 2 only) Path to mesh .obj (default: first .obj in scene_dir/mesh/)") parser.add_argument("--model-dir", type=str, default=None, help="(Option 1 only) Depth model: depth-anything/DA3-BASE for whole sequence at once (relative depth); default Nested (metric)") parser.add_argument("--chunk-size", type=int, default=None, metavar="N", help="(Option 1 only) Depth frames per batch; 0 = whole sequence at once. Omit to use process_videos default.") parser.add_argument("--process-res", type=int, default=None, metavar="R", help="(Option 1 only) Depth processing resolution. Lower = less VRAM.") parser.add_argument("--scale-factor", type=float, default=None, metavar="S", help="(Option 1 only) Resize input by this factor before depth (e.g. 0.5 = half res). For full-sequence-in-one-pass.") parser.add_argument("--sample-ratio", type=float, default=None, metavar="R", help="(Option 1 only) Fraction of frames to keep: 1.0 = every frame, 0.5 = every other frame. Omit for default (1.0).") parser.add_argument("--no-depth", action="store_true", help="(Option 1 only) Run full pipeline (including depth) for real intrinsics, then delete depth/ and depth.npz at the end.") args = parser.parse_args() if args.option is None: parser.error("--option is required (1, 2, or 3)") if args.option == 3: if not args.path: parser.error("path (video path) is required when --option 3") video_path = os.path.abspath(args.path) if not os.path.isfile(video_path): print(f"Error: Video file not found: {video_path}", file=sys.stderr) sys.exit(1) script_dir = os.path.dirname(os.path.abspath(__file__)) video_name = os.path.splitext(os.path.basename(video_path))[0] hand_dir = os.path.join(script_dir, video_name, "hand") os.makedirs(hand_dir, exist_ok=True) json_path = os.path.join(hand_dir, "hand_joints.json") overlay_path = os.path.join(hand_dir, "joints_overlay.mp4") video_to_hand = os.path.join(script_dir, "video_to_hand_joints.py") if not os.path.isfile(video_to_hand): print(f"Error: video_to_hand_joints.py not found at {video_to_hand}", file=sys.stderr) sys.exit(1) result = subprocess.run( [ sys.executable, video_to_hand, video_path, "--output", json_path, "--output-video", overlay_path, ], cwd=script_dir, ) sys.exit(result.returncode) if args.option == 4: if not args.path: parser.error("path (rgb folder) is required when --option 4") rgb_folder = os.path.abspath(args.path) if not os.path.isdir(rgb_folder): print(f"Error: RGB folder not found: {rgb_folder}", file=sys.stderr) sys.exit(1) parent_dir = os.path.dirname(rgb_folder) apple_dir = os.path.join(parent_dir, "apple") script_dir = os.path.dirname(os.path.abspath(__file__)) # Find ml-depth-pro: sibling of parametric_data (repo_root) or inside parametric_data repo_root = os.path.dirname(script_dir) ml_depth_pro_dir = os.path.join(repo_root, "ml-depth-pro") if not os.path.is [truncated — 11870 more characters] ``` ### Analytics_UI/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { GeistSans } from "geist/font/sans"; import { GeistMono } from "geist/font/mono"; import "./globals.css"; import { AppShell } from "@/components/app-shell"; export const metadata: Metadata = { // This is the title of the page title: "Viper", description: "Post-operative compliance analysis dashboard", }; export default function RootLayout({ children }: { children: React.ReactNode }) { return ( {children} ); } ``` ### Analytics_UI/src/app/page.tsx ```typescript "use client"; import { useEffect, useState } from "react"; import { supabase } from "@/lib/supabase"; import { cn, scoreColor } from "@/lib/utils"; interface ProcedureSummary { id: string; name: string; sessionCount: number; avgCompliance: number; totalDeviations: number; } export default function Home() { const [procedures, setProcedures] = useState([]); const [loading, setLoading] = useState(true); useEffect(() => { async function load() { const [procsRes, runsRes, reportsRes] = await Promise.all([ supabase.from("procedures").select("id, name"), supabase.from("procedure_runs").select("id, procedure_id"), supabase.from("deviation_reports").select("procedure_run_id, compliance_score, confirmed_count, mitigated_count, review_count"), ]); const procs = procsRes.data || []; const runs = runsRes.data || []; const reports = reportsRes.data || []; const reportMap = new Map(reports.map((r: any) => [r.procedure_run_id, r])); const summaries: ProcedureSummary[] = procs.map((p: any) => { const procRuns = runs.filter((r: any) => r.procedure_id === p.id); const procReports = procRuns.map((r: any) => reportMap.get(r.id)).filter(Boolean); const avgScore = procReports.length > 0 ? procReports.reduce((s: number, r: any) => s + (r.compliance_score || 0), 0) / procReports.length : 0; const totalDevs = procReports.reduce((s: number, r: any) => s + (r.confirmed_count || 0) + (r.mitigated_count || 0) + (r.review_count || 0), 0); return { id: p.id, name: p.name, sessionCount: procRuns.length, avgCompliance: avgScore, totalDeviations: totalDevs, }; }).filter((p: ProcedureSummary) => p.sessionCount > 0); setProcedures(summaries); setLoading(false); } load(); }, []); const totalSessions = procedures.reduce((s, p) => s + p.sessionCount, 0); const totalDevs = procedures.reduce((s, p) => s + p.totalDeviations, 0); const overallAvg = procedures.length > 0 ? procedures.reduce((s, p) => s + p.avgCompliance * p.sessionCount, 0) / totalSessions : 0; return (
{/* Hero */}

Procedures

Select a procedure type to view surgery sessions and compliance analysis

{!loading && (
Total Sessions
{totalSessions}
Avg Compliance
{Math.round(overallAvg * 100)}%
Total Deviations
{totalDevs}
)}
{/* Procedure Cards */} {loading ? (
{[...Array(4)].map((_, i) => (
))}
) : (
{procedures.map((p) => (

{p.name}

{p.sessionCount} session{p.sessionCount !== 1 ? "s" : ""} recorded
Avg Compliance
= 0.8 ? "bg-gradient-to-r from-green-500 to-emerald-400" : p.avgCompliance >= 0.5 ? "bg-gradient-to-r from-yellow-500 to-amber-400" : "bg-gradient-to-r from-red-500 to-rose-400" )} style={{ [truncated — 618 more characters] ``` ### Analytics_UI/src/app/login/page.tsx ```typescript "use client"; import { useState } from "react"; import { useRouter } from "next/navigation"; export default function Login() { const router = useRouter(); const [email, setEmail] = useState(""); const [password, setPassword] = useState(""); const [loading, setLoading] = useState(false); function handleLogin(e: React.FormEvent) { e.preventDefault(); setLoading(true); // Demo: accept any credentials setTimeout(() => { localStorage.setItem("viper_auth", "1"); router.push("/"); }, 800); } return (
{/* Background effects */}
{/* Logo */}
Viper
setEmail(e.target.value)} placeholder="surgeon@hospital.org" className="w-full px-4 py-2.5 bg-white/[0.03] border border-zinc-800/50 rounded-xl text-sm text-white placeholder-zinc-600 focus:outline-none focus:border-teal-500/50 focus:ring-1 focus:ring-teal-500/20 transition-all duration-200" required />
setPassword(e.target.value)} placeholder="••••••••" className="w-full px-4 py-2.5 bg-white/[0.03] border border-zinc-800/50 rounded-xl text-sm text-white placeholder-zinc-600 focus:outline-none focus:border-teal-500/50 focus:ring-1 focus:ring-teal-500/20 transition-all duration-200" required />

Hospital-specific access · HIPAA compliant

TreeHacks 2026 · Demo Mode

); } ``` [72 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]