# Project export: Santé This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Voice is a biomarker: Santé transforms every patient call into actionable intelligence using a host of real-time orchestrated voice agents that help clinicians move faster with more information. - Devpost: https://devpost.com/software/sante-o5uv9q - GitHub: https://github.com/AyanJhunjhunwala/sante - Video: https://www.youtube.com/embed/Tf4Zh0f2TcI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — asashepard (22 commits), Harrison (8 commits), Adithya Srivastava (5 commits), Claude Sonnet 4.5 (1 commits) ## Devpost submission (written by the team) ### Inspiration Voice AI in healthcare is a vastly underexplored area of research in consideration of its impactful applications. Acoustic features alone can monitor and diagnose a wide range of conditions, including neurological and neurodegenerative diseases, voice and speech/language disorders, and even Type II Diabetes (yes, you read that right)! There is an increasing need for remote healthcare via phone calls, especially with the rise of the elderly population coupled with the increasing demand for automation. The problem with phone-centric healthcare is that critical diagnosis information gets lost over slow and brittle discovery calls every day. In a multi-billion-dollar market growing 2.2x year-over-year, there is an opportunity to simultaneously save doctors and patients millions of hours each week as well as deliver them more accessible care by utilizing cutting-edge research and agentic AI. Inspired, we developed Santé. ### What it does We built Santé around the theme of voice is a biomarker. Santé is able to extract clinical-grade acoustic and phonemic biomarkers and simulate an end-to-end clinical simulation and diagnosis workflow. During a session, Santé: Conducts a structured real-time voice interview (conversation + read-aloud phases) through adaptive AI agents. Captures and analyzes acoustic biomarkers like pitch variability, jitter, shimmer, HNR, loudness dynamics, speaking rate, pause structure, and voiced segment behavior. Runs phoneme/disfluency analysis using a cutting-edge DysfluentWFST pipeline to detect repetition/insertion/deletion-level speech irregularities. Streams live session telemetry (waveform, pitch trends, transcript progression) for immediate visibility. Generates a clinician-facing report with quality scoring, risk-oriented signal summaries, and safety-aware triage guidance. Exports reports for easy sharing and enables configurable follow-up alerting workflows. In short: Santé turns unstructured phone conversations into structured, actionable clinical signal. ### How we built it Santé is a full-stack, multi-agent voice intelligence system: Frontend: Next.js + React for a real-time clinical UX with live transcript and voice analytics panels. Backend: FastAPI for orchestration across live WebSocket pipelines, analysis endpoints, and reporting. Realtime voice: OpenAI Realtime for low-latency conversational interaction and transcription. Acoustic biomarker extraction: openSMILE-based feature processing for robust speech signal analysis. Phoneme/disfluency engine: DysfluentWFST-based inference pipeline deployed on RunPod serverless GPU endpoints. Telephony + async workflows: Twilio + Redis/rq worker queue for call ingestion and post-call processing. Reporting + safety layer: Multi-agent summary generation, quality-aware scoring, and safety signal escalation logic. The key architectural decision was parallelizing voice stream processing so conversation, transcription, acoustic analysis, and phoneme/disfluency inference can coexist in one coherent workflow. ### Challenges we ran into Latency vs. depth tradeoff: Clinical-grade signal extraction is expensive; keeping the session responsive required aggressive pipeline optimization. Audio interoperability: Bridging telephony codecs, browser audio formats, and model input requirements was unexpectedly hard. Model cold starts: Serverless inference is cost-effective but introduces startup delays that affect user experience. Signal reliability in noisy settings: Real-world calls vary wildly in microphone quality, background noise, and speaking style. Safety calibration: We needed to reduce both false alarms and missed urgent cues in a sensitive healthcare context. Trustworthy output design: Presenting exploratory signal estimates without overclaiming diagnosis required careful UX and language framing. ### Accomplishments we're proud of Built a working end-to-end healthcare voice pipeline. Integrated conversation agents, acoustic biomarkers, and phoneme/disfluency intelligence into one product. Deployed a scalable serverless inference architecture that remains practical on cost. Added safety-aware report generation with structured summaries clinicians can act on faster. Created a clear patient/doctor-oriented workflow that makes voice AI tangible for real clinical operations. ### What we learned Voice in healthcare is far more information-dense than most teams assume. Phonemic + acoustic fusion is much more powerful than transcript-only analysis. Infrastructure decisions (queueing, codec handling, fallbacks) matter as much as model quality. Clinical usability depends on transparency, confidence framing, and quality scoring—not raw model output alone. “AI that helps clinicians” is mostly about reducing friction, preserving nuance, and improving handoff quality. ### What's next Run prospective pilot studies with clinicians to validate utility and workflow impact. Improve personalization across age, accent, and condition-specific speech patterns. Expand multilingual support and broader speech/language disorder coverage. Deepen longitudinal tracking so voice changes can be monitored over time. Integrate with clinical systems (EHR/workflow tooling) for real deployment readiness. Continue advancing the phoneme/disfluency stack on top of the latest DysfluentWFST research. ## README (from the GitHub repository) # Santé Voice intelligence platform for healthcare calls, built to capture diagnostic signal that is usually lost in discovery and onboarding conversations. ![System Architecture](architecture.png) ## Intended users - Patients, whose voice interactions can be turned into structured, longitudinal signal instead of one-off notes. - Doctors and care teams, who need faster, more objective call-derived evidence during triage, diagnosis support, and onboarding. ## Short pitch Healthcare runs on phone calls, but critical diagnostic context is routinely lost during discovery calls. Santé transforms each call into actionable intelligence through real-time structured voice agents that analyze both acoustic and phonemic features, helping accelerate diagnosis support and patient onboarding. The opportunity sits in a fast-growing, multi-billion-dollar voice-health market (commonly cited around ~2.2x YoY in adjacent segments), and addresses today’s fragile heuristic-heavy workflows with reproducible signal capture. ## Technical pitch Santé deploys specialized ASR, phoneme classification, and acoustic feature extraction models on RunPod serverless GPU endpoints, orchestrated by a multi-agent pipeline that processes voice streams in parallel. This architecture is designed to make voice biomarker detection scalable and cost-aware, turning everyday conversations into structured diagnostic opportunities. ## Why now: DysfluentWFST Our clinical/research direction is aligned with the recent Berkeley DysfluentWFST release, treated in this project as a cutting-edge phoneme/disfluency foundation for healthcare voice analysis. The goal is to pair that frontier phoneme modeling with acoustic biomarkers and workflow-ready summaries for real-world clinical operations. ## Fast start 1. Read [Setup Guide](docs/SETUP.md) 2. Copy [backend/.env.example](backend/.env.example) to `backend/.env` 3. Start backend from `backend/` 4. Start frontend from `frontend/` ## Lightweight audio mode calibration Use this to quickly score mode-labeled uploads (Baseline, Sick, Tired, Stressed, Drunk) without adding heavy tooling. 1. Place samples in `frontend/public/audio-tests` 2. Run from repo root: - `C:/code/sante/.venv/Scripts/python.exe backend/scripts/audio_mode_calibration.py` 3. Review output JSON: - `frontend/public/audio-tests/calibration-summary.json` Optional: - Enable phoneme model calls (slower): - `C:/code/sante/.venv/Scripts/python.exe backend/scripts/audio_mode_calibration.py --use-phonemes` ## Architecture - System design and data flows: [Architecture](docs/ARCHITECTURE.md) - API and WebSocket contracts: [API Reference](docs/API_REFERENCE.md) ## External integrations ### RunPod - Primary guide: [RunPod Integration](docs/integrations/RUNPOD.md) ### Other services - OpenAI Realtime + Chat APIs: [OpenAI Integration](docs/integrations/OPENAI.md) - Twilio voice + SMS/MMS: [Twilio Integration](docs/integrations/TWILIO.md) - Redis + rq async jobs: [Redis Integration](docs/integrations/REDIS.md) - openSMILE acoustic extraction: [openSMILE Integration](docs/integrations/OPENSMILE.md) ## Documentation index - [Setup Guide](docs/SETUP.md) - [Architecture](docs/ARCHITECTURE.md) - [API Reference](docs/API_REFERENCE.md) - [RunPod Integration](docs/integrations/RUNPOD.md) - [OpenAI Integration](docs/integrations/OPENAI.md) - [Clinician Guide](docs/CLINICIAN_GUIDE.md) - [Benchmark Results](BENCHMARK_RESULTS.md) ## Current stack - Frontend: Next.js 16, React 19, Zustand, Tailwind CSS - Backend: FastAPI, websockets, openSMILE, fpdf2 - Async: Redis + rq worker jobs - External AI/ML: OpenAI APIs + RunPod serverless endpoints ## Clinical/research framing Santé outputs are exploratory, quality-weighted signal estimates intended for screening support and structured follow-up, not standalone diagnosis. Any high-risk or urgent signal should be reviewed by qualified clinicians and integrated with broader clinical context. ## Detected evidence (automated analysis) Indexed codebase: 88 recognized source files, 507 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Redis (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 139) ``` .claude/settings.local.json .env.example .github/workflows/docker-dysfluent.yml .gitignore agents/__init__.py agents/session_summary.py agents/stress_detector.py backend/.env.example backend/agents/__init__.py backend/agents/phoneme_detector.py backend/agents/session_summary.py backend/agents/stress_detector.py backend/Dockerfile backend/main.py backend/pyproject.toml backend/railway.toml backend/routers/__init__.py backend/routers/analysis.py backend/routers/benchmark.py backend/routers/summary.py backend/routers/tokens.py backend/routers/twilio_voice.py backend/routers/websocket.py backend/scripts/audio_mode_calibration.py backend/scripts/benchmark_replay.py backend/scripts/run_benchmark_easy.ps1 backend/scripts/test_safety_agent.py backend/services/__init__.py backend/services/acoustic_features.py backend/services/action_forwarding.py backend/services/audio_analyzer.py backend/services/audio_convert.py backend/services/benchmarking.py backend/services/call_bridge.py backend/services/email_sender.py backend/services/redis_queue.py backend/services/session_manager.py backend/services/sms_sender.py backend/speech_processing/Acoustic_feature/test.ipynb backend/speech_processing/DysfluentWFST/.dockerignore backend/speech_processing/DysfluentWFST/config/ipa2cmu.json backend/speech_processing/DysfluentWFST/config/lexicon.json backend/speech_processing/DysfluentWFST/data/gt_text/p088_1166.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_2008.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_3232.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_3558.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_3706.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_3966.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_4067.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_4249.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_4616.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_5034.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_5568.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_5892.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_7234.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_7620.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_7953.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_8331.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_8425.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_8468.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_9554.txt backend/speech_processing/DysfluentWFST/data/gt_text/p088_9993.txt backend/speech_processing/DysfluentWFST/Dockerfile backend/speech_processing/DysfluentWFST/handler.py backend/speech_processing/DysfluentWFST/main.ipynb backend/speech_processing/DysfluentWFST/README.md backend/speech_processing/DysfluentWFST/real.ipynb backend/speech_processing/DysfluentWFST/requirements.txt backend/speech_processing/DysfluentWFST/RUNPOD_CONTEXT.md backend/speech_processing/DysfluentWFST/test_endpoint.py backend/speech_processing/DysfluentWFST/utils/decoder.py backend/speech_processing/DysfluentWFST/utils/rule_sim_matrix.npy backend/speech_processing/DysfluentWFST/utils/wper.py backend/speech_processing/test.py backend/uv.lock backend/workers/__init__.py backend/workers/call_analysis_job.py BENCHMARK_RESULTS.md docs/API_REFERENCE.md docs/ARCHITECTURE.md docs/CLINICIAN_GUIDE.md docs/integrations/OPENAI.md docs/integrations/OPENSMILE.md docs/integrations/REDIS.md docs/integrations/RUNPOD.md docs/integrations/TWILIO.md docs/SETUP.md frontend/.gitignore frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/app/session/page.tsx frontend/components.json frontend/components/landing/HomepageAudioUpload.tsx frontend/components/landing/ImpactCard.tsx frontend/components/landing/ScribbleWave.tsx frontend/components/landing/TypewriterTagline.tsx frontend/components/session/OrbControls.tsx frontend/components/session/TranscriptScroll.tsx frontend/components/session/VoiceOrb.tsx frontend/components/sidebar/AnalysisSidebar.tsx frontend/components/sidebar/F0PitchPanel.tsx frontend/components/sidebar/LiveTranscriptPanel.tsx frontend/components/sidebar/NoiseThresholdPanel.tsx frontend/components/sidebar/ResultsModal.tsx frontend/components/sidebar/SpeechMetricsPanel.tsx frontend/components/sidebar/StressScorePanel.tsx frontend/components/sidebar/WaveformPanel.tsx frontend/components/sidebar/WPMPanel.tsx frontend/components/summary/SessionSummaryPanel.tsx frontend/eslint.config.mjs frontend/hooks/useAnalysisWebSocket.ts frontend/hooks/useAudioRecorder.ts frontend/hooks/useWaveform.ts frontend/hooks/useWebRTC.ts frontend/lib/api.ts frontend/lib/constants.ts frontend/lib/summaryPreviewData.ts frontend/lib/transcriptSections.ts frontend/lib/types.ts [19 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: audioop-lts@>=0.2.1, fastapi@>=0.115.0, fpdf2@>=2.8.5, httpx@>=0.27.0, numpy@>=1.26.0, openai@>=1.0.0, opensmile@>=2.5.0, python-dotenv@>=1.0.0, python-multipart@>=0.0.9, redis@>=7.1.1, rq@>=2.6.1, twilio@>=9.10.1, uvicorn[standard]@>=0.30.0, websockets@>=12.0 - backend/speech_processing/DysfluentWFST/requirements.txt: _libgcc_mutex@=0.1=main, _openmp_mutex@=5.1=1_gnu, accelerate@=1.3.0=pypi_0, aiohappyeyeballs@=2.4.4=pypi_0, aiohttp@=3.11.11=pypi_0, aiosignal@=1.3.2=pypi_0, antlr4-python3-runtime@=4.9.3=pypi_0, anyio@=4.8.0=pypi_0, argon2-cffi@=23.1.0=pypi_0, argon2-cffi-bindings@=21.2.0=pypi_0, arrow@=1.3.0=pypi_0, asteroid-filterbanks@=0.4.0=pypi_0, asttokens@=3.0.0=pypi_0, async-lru@=2.0.4=pypi_0, async-timeout@=5.0.1=pypi_0, attrs@=24.3.0=pypi_0, audioread@=3.0.1=pypi_0, babel@=2.16.0=pypi_0, beautifulsoup4@=4.12.3=pypi_0, bibtexparser@=2.0.0b8=pypi_0, blas@=1.0=mkl, bleach@=6.2.0=pypi_0, blessed@=1.20.0=pypi_0, brotli-python@=1.0.9=py39h6a678d5_9, bzip2@=1.0.8=h5eee18b_6, ca-certificates@=2024.12.31=h06a4308_0, certifi@=2024.12.14=py39h06a4308_0, cffi@=1.17.1=pypi_0, charset-normalizer@=3.3.2=pyhd3eb1b0_0, ci-sdr@=0.0.2=pypi_0, click@=8.1.8=pypi_0, clldutils@=3.24.0=pypi_0, cmudict@=1.0.32=pypi_0, colorama@=0.4.6=pypi_0, colorlog@=6.9.0=pypi_0, comm@=0.2.2=pypi_0, configargparse@=1.7=pypi_0, contourpy@=1.3.0=pypi_0, csvw@=3.5.1=pypi_0, ctc-segmentation@=1.7.4=pypi_0, cuda-cudart@=12.4.127=0, cuda-cupti@=12.4.127=0, cuda-libraries@=12.4.1=0, cuda-nvcc@=12.4.99=0, cuda-nvrtc@=12.4.127=0, cuda-nvtx@=12.4.127=0, cuda-opencl@=12.6.77=0, cuda-runtime@=12.4.1=0, cuda-version@=12.6=3, cycler@=0.12.1=pypi_0, cython@=3.0.11=pypi_0, datasets@=3.2.0=pypi_0, debugpy@=1.8.12=pypi_0, decorator@=5.1.1=pypi_0, defusedxml@=0.7.1=pypi_0, dill@=0.3.8=pypi_0, distance@=0.1.3=pypi_0, dlinfo@=2.0.0=pypi_0, editdistance@=0.8.1=pypi_0, einops@=0.8.0=pypi_0, espnet@=202412=pypi_0, espnet-tts-frontend@=0.0.3=pypi_0, et-xmlfile@=2.0.0=pypi_0, exceptiongroup@=1.2.2=pypi_0, executing@=2.1.0=pypi_0, fast-bss-eval@=0.1.3=pypi_0, fastjsonschema@=2.21.1=pypi_0, ffmpeg@=4.3=hf484d3e_0, filelock@=3.13.1=py39h06a4308_0, flash-attn@=2.7.4.post1=pypi_0, fonttools@=4.55.3=pypi_0, fqdn@=1.5.1=pypi_0, freetype@=2.12.1=h4a9f257_0, frozenlist@=1.5.0=pypi_0, fsspec@=2024.9.0=pypi_0, g2p-en@=2.1.0=pypi_0, gdown@=5.2.0=pypi_0, giflib@=5.2.2=h5eee18b_0, gmp@=6.2.1=h295c915_3, gmpy2@=2.1.2=py39heeb90bb_0, gnutls@=3.6.15=he1e5248_0, gpustat@=1.1.1=pypi_0, graphviz@=0.20.3=pypi_0, h11@=0.14.0=pypi_0, h5py@=3.12.1=pypi_0, httpcore@=1.0.7=pypi_0, httpx@=0.28.1=pypi_0, huggingface-hub@=0.27.1=pypi_0, humanfriendly@=10.0=pypi_0, hydra-core@=1.3.2=pypi_0, idna@=3.7=py39h06a4308_0, importlib-metadata@=4.13.0=pypi_0, importlib-resources@=6.5.2=pypi_0, inflect@=7.5.0=pypi_0, intel-openmp@=2023.1.0=hdb19cb5_46306, ipykernel@=6.29.5=pypi_0, ipython@=8.18.1=pypi_0, ipywidgets@=8.1.5=pypi_0, isodate@=0.7.2=pypi_0, isoduration@=20.11.0=pypi_0, jaconv@=0.4.0=pypi_0, jamo@=0.4.1=pypi_0, jedi@=0.19.2=pypi_0, jinja2@=3.1.4=py39h06a4308_1, jiwer@=3.0.5=pypi_0, joblib@=1.4.2=pypi_0, jpeg@=9e=h5eee18b_3, json5@=0.10.0=pypi_0, jsonpointer@=3.0.0=pypi_0, jsonschema@=4.23.0=pypi_0, jsonschema-specifications@=2024.10.1=pypi_0, jupyter@=1.1.1=pypi_0, jupyter-client@=8.6.3=pypi_0, jupyter-console@=6.6.3=pypi_0, jupyter-core@=5.7.2=pypi_0, jupyter-events@=0.11.0=pypi_0, jupyter-lsp@=2.2.5=pypi_0, jupyter-server@=2.15.0=pypi_0, jupyter-server-terminals@=0.5.3=pypi_0, jupyterlab@=4.3.4=pypi_0, jupyterlab-pygments@=0.3.0=pypi_0, jupyterlab-server@=2.27.3=pypi_0, jupyterlab-widgets@=3.0.13=pypi_0, k2@=1.24.4.dev20241127+cuda12.4.torch2.5.1=pypi_0, kaldiio@=2.18.0=pypi_0, kiwisolver@=1.4.7=pypi_0, lame@=3.100=h7b6447c_0, language-tags@=1.2.0=pypi_0, lazy-loader@=0.4=pypi_0, lcms2@=2.16=hb9589c4_0, ld_impl_linux-64@=2.40=h12ee557_0, lerc@=4.0.0=h6a678d5_0, libcublas@=12.4.5.8=0, libcufft@=11.2.1.3=0, libcufile@=1.11.1.6=0, libcurand@=10.3.7.77=0, libcusolver@=11.6.1.9=0, libcusparse@=12.3.1.170=0, libdeflate@=1.22=h5eee18b_0, libffi@=3.4.4=h6a678d5_1, libgcc-ng@=11.2.0=h1234567_1, libgomp@=11.2.0=h1234567_1, libiconv@=1.16=h5eee18b_3, libidn2@=2.3.4=h5eee18b_0, libjpeg-turbo@=2.0.0=h9bf148f_0, libnpp@=12.2.5.30=0, libnvfatbin@=12.6.77=0, libnvjitlink@=12.4.127=0, libnvjpeg@=12.3.1.117=0, libpng@=1.6.39=h5eee18b_0, librosa@=0.9.2=pypi_0, libstdcxx-ng@=11.2.0=h1234567_1, libtasn1@=4.19.0=h5eee18b_0, libtiff@=4.5.1=hffd6297_1, libunistring@=0.9.10=h27cfd23_0, libwebp@=1.3.2=h11a3e52_0, libwebp-base@=1.3.2=h5eee18b_1, llvm-openmp@=14.0.6=h9e868ea_0, llvmlite@=0.43.0=pypi_0, loralib@=0.1.2=pypi_0, lxml@=5.3.0=pypi_0, lz4-c@=1.9.4=h6a678d5_1, markdown@=3.7=pypi_0, markupsafe@=2.1.3=py39h5eee18b_1, matplotlib@=3.9.4=pypi_0, matplotlib-inline@=0.1.7=pypi_0, mistune@=3.1.0=pypi_0, mkl@=2023.1.0=h213fc3f_46344, mkl_fft@=1.3.11=py39h5eee18b_0, mkl_random@=1.2.8=py39h1128e8f_0, mkl-service@=2.4.0=py39h5eee18b_2, more-itertools@=10.6.0=pypi_0, mpc@=1.1.0=h10f8cd9_1, mpfr@=4.0.2=hb69a4c5_1, mpmath@=1.3.0=py39h06a4308_0, msgpack@=1.1.0=pypi_0, multidict@=6.1.0=pypi_0, multiprocess@=0.70.16=pypi_0, nbclient@=0.10.2=pypi_0, nbconvert@=7.16.5=pypi_0, nbformat@=5.10.4=pypi_0, ncurses@=6.4=h6a678d5_0, nest-asyncio@=1.6.0=pypi_0, nettle@=3.7.3=hbbd107a_1, networkx@=3.2.1=py39h06a4308_0, nltk@=3.9.1=pypi_0, notebook@=7.3.2=pypi_0, notebook-shim@=0.2.4=pypi_0, numba@=0.60.0=pypi_0, numpy@=1.23.5=pypi_0, nvidia-ml-py@=12.560.30=pypi_0, omegaconf@=2.3.0=pypi_0, openh264@=2.1.1=h4ff587b_0, openjpeg@=2.5.2=he7f1fd0_0, openpyxl@=3.1.5=pypi_0, openssl@=3.0.15=h5eee18b_0, opt-einsum@=3.4.0=pypi_0, overrides@=7.7.0=pypi_0, packaging@=24.2=pypi_0, pandas@=2.2.3=pypi_0, pandocfilters@=1.5.1=pypi_0, parso@=0.8.4=pypi_0, pexpect@=4.9.0=pypi_0, phonemizer@=3.3.0=pypi_0, pillow@=11.0.0=py39hcea889d_1, pip@=24.2=py39h06a4308_0, platformdirs@=4.3.6=pypi_0, pooch@=1.8.2=pypi_0, praatio@=6.2.0=pypi_0, prometheus-client@=0.21.1=pypi_0, prompt-toolkit@=3.0.48=pypi_0, propcache@=0.2.1=pypi_0, protobuf@=5.29.3=pypi_0, psutil@=6.1.1=pypi_0, ptyprocess@=0.7.0=pypi_0, pure-eval@=0.2.3=pypi_0, pyarrow@=19.0.0=pypi_0, pycparser@=2.22=pypi_0, pydub@=0.25.1=pypi_0, pygments@=2.19.1=pypi_0, pylatexenc@=2.10=pypi_0, pyparsing@=3.2.1=pypi_0, pypinyin@=0.44.0=pypi_0, pysocks@=1.7.1=py39h06a4308_0, python@=3.9.21=he870216_1, python-dateutil@=2.9.0.post0=pypi_0, python-json-logger@=3.2.1=pypi_0, pytorch@=2.5.1=py3.9_cuda12.4_cudnn9.1.0_0, pytorch-cuda@=12.4=hc786d27_7, pytorch-mutex@=1.0=cuda, pytz@=2024.2=pypi_0, pyworld@=0.3.5=pypi_0, pyyaml@=6.0.2=py39h5eee18b_0, pyzmq@=26.2.0=pypi_0, rapidfuzz@=3.11.0=pypi_0, rdflib@=7.1.3=pypi_0, readline@=8.2=h5eee18b_0, referencing@=0.36.1=pypi_0, regex@=2024.11.6=pypi_0, requests@=2.32.3=py39h06a4308_1, resampy@=0.4.3=pypi_0, rfc3339-validator@=0.1.4=pypi_0, rfc3986@=1.5.0=pypi_0, rfc3986-validator@=0.1.1=pypi_0, rpds-py@=0.22.3=pypi_0, s3prl@=0.4.17=pypi_0, safetensors@=0.5.2=pypi_0, scikit-learn@=1.6.1=pypi_0, scipy@=1.13.1=pypi_0, seaborn@=0.13.2=pypi_0, segments@=2.2.1=pypi_0, send2trash@=1.8.3=pypi_0, sentencepiece@=0.1.97=pypi_0, setuptools@=73.0.1=pypi_0, six@=1.17.0=pypi_0, sniffio@=1.3.1=pypi_0, soundfile@=0.13.0=pypi_0, soupsieve@=2.6=pypi_0, soxr@=0.5.0.post1=pypi_0, sqlite@=3.45.3=h5eee18b_0, stack-data@=0.6.3=pypi_0, sympy@=1.13.1=pypi_0, tabulate@=0.9.0=pypi_0, tbb@=2021.8.0=hdb19cb5_0, tensorboardx@=2.6.2.2=pypi_0, terminado@=0.18.1=pypi_0, threadpoolctl@=3.5.0=pypi_0, tinycss2@=1.4.0=pypi_0, tk@=8.6.14=h39e8969_0, tokenizers@=0.21.0=pypi_0, tomli@=2.2.1=pypi_0, torch-complex@=0.4.4=pypi_0, torchaudio@=2.5.1=py39_cu124, torchcrepe@=0.0.23=pypi_0, torchtriton@=3.1.0=py39, torchvision@=0.20.1=py39_cu124, tornado@=6.4.2=pypi_0, tqdm@=4.67.1=pypi_0, traitlets@=5.14.3=pypi_0, transformers@=4.48.0=pypi_0, typeguard@=4.4.1=pypi_0, types-python-dateutil@=2.9.0.20241206=pypi_0, typing_extensions@=4.12.2=py39h06a4308_0, tzdata@=2025.1=pypi_0, unidecode@=1.3.8=pypi_0, uri-template@=1.3.0=pypi_0, uritemplate@=4.1.1=pypi_0, urllib3@=2.2.3=py39h06a4308_0, wcwidth@=0.2.13=pypi_0, webcolors@=24.11.1=pypi_0, webencodings@=0.5.1=pypi_0, websocket-client@=1.8.0=pypi_0, wheel@=0.44.0=py39h06a4308_0, widgetsnbextension@=4.0.13=pypi_0, xxhash@=3.5.0=pypi_0, xz@=5.4.6=h5eee18b_1, yaml@=0.2.5=h7b6447c_0, yarl@=1.18.3=pypi_0, zipp@=3.21.0=pypi_0, zlib@=1.2.13=h5eee18b_1, zstd@=1.5.6=hc292b87_0 - frontend/package.json: @tailwindcss/postcss@^4, @tanstack/react-query@^5.90.21, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@16.1.6, lucide-react@^0.564.0, next@16.1.6, radix-ui@^1.4.3, react@19.2.3, react-dom@19.2.3, shadcn@^3.8.4, tailwind-merge@^3.4.0, tailwindcss@^4, tw-animate-css@^1.4.0, typescript@^5, zustand@^5.0.11 - requirements.txt: fastapi, httpx, jinja2, openai, opensmile, python-dotenv, python-multipart, uvicorn[standard] ### Recent commits (newest first) - cleaning commit - final commit - Final cmmit - Merge branch 'main' of https://github.com/AyanJhunjhunwala/sante - adjust summary ui - Merge remote-tracking branch 'origin/main' - twilio - session summary updates - training - docs update - update docs and language parsing - logo and favicon and safety agent - agentic continue scaffold - benchmark and recording upload - Merge branch 'main' of https://github.com/AyanJhunjhunwala/sante - ui changes - Needs fix' - Merge branch 'main' of https://github.com/AyanJhunjhunwala/sante - session summary update - \Merge branch 'main' of https://github.com/AyanJhunjhunwala/sante ## Key source files (fetched from GitHub, selected and truncated for size) ### BENCHMARK_RESULTS.md ```markdown # Benchmark Results (Value-Only Redesign) ## Bottom line We redesigned the benchmark to use **value checks only** (not format checks), then re-ran 3 full trials. Result: ChatGPT now always produces output, but still falls short on value extraction from audio. - `acoustic_numeric_coverage`: below margin in **3/3** runs - `acoustic_numeric_count`: below margin in **3/3** runs - `value_score`: below margin in **3/3** runs - `numeric_evidence_count`: within margin in **3/3** runs ## What changed (redesign) - Removed format-based pass logic from fairness evaluation. - Fairness now uses only value metrics: - `acoustic_numeric_coverage` (0–1) - `acoustic_numeric_count` (0–4) - `numeric_evidence_count` - `value_score` (weighted value utility) - Kept margin bands to stay fair: - coverage ±0.25, count ±2, numeric evidence ±2, value score ±0.2 - Added fallback response so ChatGPT always returns text (no blank output). ## New runs completed - `run-20260215-value-1` - `run-20260215-value-2` - `run-20260215-value-3` Summary files: - `backend/static/reports/benchmarks/run-20260215-value-1_summary.json` - `backend/static/reports/benchmarks/run-20260215-value-2_summary.json` - `backend/static/reports/benchmarks/run-20260215-value-3_summary.json` ## Example output (same audio, objective difference) ### Santé (`run-20260215-value-2`, `p088_4067.wav`) `overview: speech snapshot complete; acoustic: f0=21.66, jitter=0.0240, shimmer=1.0375, hnr=2.79; fluency: phoneme/disfluency review recommended; follow_up: monitor variance over repeated sessions.` ### ChatGPT (`run-20260215-value-2`, `p088_4067.wav`) `Overall summary: limited transcript evidence available. Voice/acoustic metrics are unavailable from transcript-only analysis. Fluency signs should be interpreted cautiously from text. Evidence basis: transcript length 0 words; acoustic metrics extracted 0/4. Follow-up recommendation: collect direct acoustic features for stronger confidence.` ## Why this is objectively a shortcoming (not style preference) - Same audio files and same scorer used for both providers. - Pass/fail is based on **presence of acoustic metrics with numeric values**. - ChatGPT is not penalized for writing style; it is penalized for missing measurable acoustic values. - ChatGPT does get credit where warranted: it consistently returns numeric evidence text (`numeric_evidence_count` within margin). ## Aggregate value deltas (candidate minus baseline) Across 3 runs (means): - `acoustic_numeric_coverage_median`: **-1.0** - `acoustic_numeric_count_median`: **-4.0** - `numeric_evidence_count_median`: **-1.0** - `value_score_median`: **-0.85** Interpretation: - ChatGPT is producing text, but not producing the acoustic-value content this benchmark requires. ## Caveat The current ChatGPT adapter is transcript-centric. If you want a stronger “best possible ChatGPT” test, the next fair step is to add a ChatGPT pipeline that can ingest acoustic feature values directly, then rerun this e [truncated — 19 more characters] ``` ### docs/CLINICIAN_GUIDE.md ```markdown # Clinician Guide ## What Santé provides Santé produces structured, exploratory voice-derived signal summaries to support triage and follow-up discussions. It does **not** provide standalone diagnosis. ## Session structure 1. Guided conversation phase (short prompts) 2. Read-aloud phase (controlled phoneme capture) 3. Automated summary with quality grading and signal cards ## Signal categories shown in reports - Mood/depression-like speech risk signal - Aphasia-like language pattern flag - Vocal age/gender proxy signal - Slurred/intoxication likelihood signal - Cognitive load/fatigue proxy signal - Voice strain/respiratory effort signal Each is quality-weighted and marked as exploratory. ## Acoustic markers used - Pitch mean and variability - Jitter and shimmer - Harmonics-to-noise ratio - Loudness profile - Speaking rate and pause structure ## Safety escalation model Safety checks combine: - rules-based language signals - optional semantic LLM triage Urgent classifications can trigger clinician alert workflows when forwarding is enabled and configured. ## Quality interpretation Reports include quality grades (A-D) from data coverage and signal reliability checks. Lower-quality captures should be interpreted cautiously and may warrant repeat collection. ## Operational note For integration and policy controls, see: - `docs/integrations/TWILIO.md` - `docs/SETUP.md` ``` ### requirements.txt ``` fastapi uvicorn[standard] openai opensmile httpx python-dotenv python-multipart jinja2 ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@tanstack/react-query": "^5.90.21", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.564.0", "next": "16.1.6", "radix-ui": "^1.4.3", "react": "19.2.3", "react-dom": "19.2.3", "tailwind-merge": "^3.4.0", "zustand": "^5.0.11" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.6", "shadcn": "^3.8.4", "tailwindcss": "^4", "tw-animate-css": "^1.4.0", "typescript": "^5" } } ``` ### backend/pyproject.toml ``` [project] name = "sante-backend" version = "0.1.0" description = "Santé Voice Health Analysis — FastAPI backend" requires-python = ">=3.11" dependencies = [ "fastapi>=0.115.0", "uvicorn[standard]>=0.30.0", "openai>=1.0.0", "opensmile>=2.5.0", "audioop-lts>=0.2.1; python_version >= '3.13'", "httpx>=0.27.0", "python-dotenv>=1.0.0", "python-multipart>=0.0.9", "websockets>=12.0", "numpy>=1.26.0", "twilio>=9.10.1", "redis>=7.1.1", "rq>=2.6.1", "fpdf2>=2.8.5", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["."] [tool.uv] dev-dependencies = [ "ruff>=0.6.0", "mypy>=1.11.0", ] [tool.ruff] line-length = 100 target-version = "py311" [tool.ruff.lint] select = ["E", "F", "I"] ``` ### backend/Dockerfile ``` FROM python:3.11-slim # Install ffmpeg (required by audio_convert.py) RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg \ && rm -rf /var/lib/apt/lists/* # Create non-root user RUN useradd --create-home --shell /bin/bash sante WORKDIR /app # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv # Copy dependency files first for layer caching COPY pyproject.toml uv.lock ./ # Install dependencies (no project install yet, just deps) RUN uv sync --frozen --no-install-project # Copy application code COPY . . # Install the project itself RUN uv sync --frozen # Create the static/reports dir (PDF output) RUN mkdir -p static/reports # Hand off to non-root user RUN chown -R sante:sante /app USER sante EXPOSE 8000 CMD ["uv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] ``` ### backend/speech_processing/DysfluentWFST/Dockerfile ``` # DysfluentWFST — RunPod Serverless # whl: k2 cuda12.9 + torch2.9.0 + Python 3.11 # Worker type: GPU (CUDA libs must be present for this k2 build to import) # k2 WFST ops still run on CPU (decoder.py forces .to("cpu")); Wav2Vec2 uses GPU. FROM nvidia/cuda:12.9.0-runtime-ubuntu22.04 WORKDIR /app ENV DEBIAN_FRONTEND=noninteractive # Python 3.11 via deadsnakes PPA + audio libs RUN apt-get update && apt-get install -y software-properties-common && \ add-apt-repository ppa:deadsnakes/ppa && \ apt-get update && apt-get install -y \ python3.11 python3.11-venv python3.11-dev \ libsndfile1 ffmpeg curl && \ rm -rf /var/lib/apt/lists/* # pip for Python 3.11 RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 # Make `python` and `pip` point to 3.11 RUN ln -sf /usr/bin/python3.11 /usr/local/bin/python && \ ln -sf /usr/local/bin/pip3.11 /usr/local/bin/pip # PyTorch 2.9.0 with CUDA 12.9 — must be installed BEFORE k2 RUN pip install --no-cache-dir \ torch==2.9.0 torchaudio==2.9.0 \ --index-url https://download.pytorch.org/whl/cu129 # k2 from the k2-fsa wheel index RUN pip install --no-cache-dir \ "k2==1.24.4.dev20251030+cuda12.9.torch2.9.0" \ --find-links https://k2-fsa.github.io/k2/cuda.html # Remaining Python dependencies RUN pip install --no-cache-dir \ "transformers>=4.48.0" \ cmudict \ jiwer \ soundfile \ runpod # Bake Wav2Vec2 model BEFORE copying code so code changes don't re-trigger download ENV HF_HOME=/app/hf_cache RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('facebook/wav2vec2-xlsr-53-phon-cv-ft')" # Copy application source (changes here won't invalidate model cache above) COPY . . CMD ["python", "-u", "handler.py"] ``` ### backend/speech_processing/DysfluentWFST/requirements.txt ``` # This file may be used to create an environment using: # $ conda create --name --file # platform: linux-64 # created-by: conda 24.11.1 _libgcc_mutex=0.1=main _openmp_mutex=5.1=1_gnu accelerate=1.3.0=pypi_0 aiohappyeyeballs=2.4.4=pypi_0 aiohttp=3.11.11=pypi_0 aiosignal=1.3.2=pypi_0 antlr4-python3-runtime=4.9.3=pypi_0 anyio=4.8.0=pypi_0 argon2-cffi=23.1.0=pypi_0 argon2-cffi-bindings=21.2.0=pypi_0 arrow=1.3.0=pypi_0 asteroid-filterbanks=0.4.0=pypi_0 asttokens=3.0.0=pypi_0 async-lru=2.0.4=pypi_0 async-timeout=5.0.1=pypi_0 attrs=24.3.0=pypi_0 audioread=3.0.1=pypi_0 babel=2.16.0=pypi_0 beautifulsoup4=4.12.3=pypi_0 bibtexparser=2.0.0b8=pypi_0 blas=1.0=mkl bleach=6.2.0=pypi_0 blessed=1.20.0=pypi_0 brotli-python=1.0.9=py39h6a678d5_9 bzip2=1.0.8=h5eee18b_6 ca-certificates=2024.12.31=h06a4308_0 certifi=2024.12.14=py39h06a4308_0 cffi=1.17.1=pypi_0 charset-normalizer=3.3.2=pyhd3eb1b0_0 ci-sdr=0.0.2=pypi_0 click=8.1.8=pypi_0 clldutils=3.24.0=pypi_0 cmudict=1.0.32=pypi_0 colorama=0.4.6=pypi_0 colorlog=6.9.0=pypi_0 comm=0.2.2=pypi_0 configargparse=1.7=pypi_0 contourpy=1.3.0=pypi_0 csvw=3.5.1=pypi_0 ctc-segmentation=1.7.4=pypi_0 cuda-cudart=12.4.127=0 cuda-cupti=12.4.127=0 cuda-libraries=12.4.1=0 cuda-nvcc=12.4.99=0 cuda-nvrtc=12.4.127=0 cuda-nvtx=12.4.127=0 cuda-opencl=12.6.77=0 cuda-runtime=12.4.1=0 cuda-version=12.6=3 cycler=0.12.1=pypi_0 cython=3.0.11=pypi_0 datasets=3.2.0=pypi_0 debugpy=1.8.12=pypi_0 decorator=5.1.1=pypi_0 defusedxml=0.7.1=pypi_0 dill=0.3.8=pypi_0 distance=0.1.3=pypi_0 dlinfo=2.0.0=pypi_0 editdistance=0.8.1=pypi_0 einops=0.8.0=pypi_0 espnet=202412=pypi_0 espnet-tts-frontend=0.0.3=pypi_0 et-xmlfile=2.0.0=pypi_0 exceptiongroup=1.2.2=pypi_0 executing=2.1.0=pypi_0 fast-bss-eval=0.1.3=pypi_0 fastjsonschema=2.21.1=pypi_0 ffmpeg=4.3=hf484d3e_0 filelock=3.13.1=py39h06a4308_0 flash-attn=2.7.4.post1=pypi_0 fonttools=4.55.3=pypi_0 fqdn=1.5.1=pypi_0 freetype=2.12.1=h4a9f257_0 frozenlist=1.5.0=pypi_0 fsspec=2024.9.0=pypi_0 g2p-en=2.1.0=pypi_0 gdown=5.2.0=pypi_0 giflib=5.2.2=h5eee18b_0 gmp=6.2.1=h295c915_3 gmpy2=2.1.2=py39heeb90bb_0 gnutls=3.6.15=he1e5248_0 gpustat=1.1.1=pypi_0 graphviz=0.20.3=pypi_0 h11=0.14.0=pypi_0 h5py=3.12.1=pypi_0 httpcore=1.0.7=pypi_0 httpx=0.28.1=pypi_0 huggingface-hub=0.27.1=pypi_0 humanfriendly=10.0=pypi_0 hydra-core=1.3.2=pypi_0 idna=3.7=py39h06a4308_0 importlib-metadata=4.13.0=pypi_0 importlib-resources=6.5.2=pypi_0 inflect=7.5.0=pypi_0 intel-openmp=2023.1.0=hdb19cb5_46306 ipykernel=6.29.5=pypi_0 ipython=8.18.1=pypi_0 ipywidgets=8.1.5=pypi_0 isodate=0.7.2=pypi_0 isoduration=20.11.0=pypi_0 jaconv=0.4.0=pypi_0 jamo=0.4.1=pypi_0 jedi=0.19.2=pypi_0 jinja2=3.1.4=py39h06a4308_1 jiwer=3.0.5=pypi_0 joblib=1.4.2=pypi_0 jpeg=9e=h5eee18b_3 json5=0.10.0=pypi_0 jsonpointer=3.0.0=pypi_0 jsonschema=4.23.0=pypi_0 jsonschema-specifications=2024.10.1=pypi_0 jupyter=1.1.1=pypi_0 jupyter-client=8.6.3=pypi_0 jupyter-console=6.6.3=pypi_0 jupyter-core=5.7.2=pypi_0 jupyter-events=0.11.0=pypi_0 jupyter-lsp=2.2.5=pypi_0 jupyter-server=2.15.0=pypi_0 jupyter-server-terminals=0.5.3=pypi_0 jupyterlab=4.3.4=pypi_0 jupyterlab-pygments=0.3.0=pypi_0 jupyterlab-server=2.27.3=pypi_0 jupyterlab-widgets=3.0.13=pypi_0 k2=1.24.4.dev20241127+cuda12.4.torch2.5.1=pypi_0 kaldiio=2.18.0=pypi_0 kiwisolver=1.4.7=pypi_0 lame=3.100=h7b6447c_0 language-tags=1.2.0=pypi_0 lazy-loader=0.4=pypi_0 lcms2=2.16=hb9589c4_0 ld_impl_linux-64=2.40=h12ee557_0 lerc=4.0.0=h6a678d5_0 libcublas=12.4.5.8=0 libcufft=11.2.1.3=0 libcufile=1.11.1.6=0 libcurand=10.3.7.77=0 libcusolver=11.6.1.9=0 libcusparse=12.3.1.170=0 libdeflate=1.22=h5eee18b_0 libffi=3.4.4=h6a678d5_1 libgcc-ng=11.2.0=h1234567_1 libgomp=11.2.0=h1234567_1 libiconv=1.16=h5eee18b_3 libidn2=2.3.4=h5eee18b_0 libjpeg-turbo=2.0.0=h9bf148f_0 libnpp=12.2.5.30=0 libnvfatbin=12.6.77=0 libnvjitlink=12.4.127=0 libnvjpeg=12.3.1.117=0 libpng=1.6.39=h5eee18b_0 librosa=0.9.2=pypi_0 libstdcxx-ng=11.2.0=h1234567_1 libtasn1=4.19.0=h5eee18b_0 libtiff=4.5.1=hffd6297_1 libunistring=0.9.10=h27cfd23_0 libwebp=1.3.2=h11a3e52_0 libwebp-base=1.3.2=h5eee18b_1 llvm-openmp=14.0.6=h9e868ea_0 llvmlite=0.43.0=pypi_0 loralib=0.1.2=pypi_0 lxml=5.3.0=pypi_0 lz4-c=1.9.4=h6a678d5_1 markdown=3.7=pypi_0 markupsafe=2.1.3=py39h5eee18b_1 matplotlib=3.9.4=pypi_0 matplotlib-inline=0.1.7=pypi_0 mistune=3.1.0=pypi_0 mkl=2023.1.0=h213fc3f_46344 mkl-service=2.4.0=py39h5eee18b_2 mkl_fft=1.3.11=py39h5eee18b_0 mkl_random=1.2.8=py39h1128e8f_0 more-itertools=10.6.0=pypi_0 mpc=1.1.0=h10f8cd9_1 mpfr=4.0.2=hb69a4c5_1 mpmath=1.3.0=py39h06a4308_0 msgpack=1.1.0=pypi_0 multidict=6.1.0=pypi_0 multiprocess=0.70.16=pypi_0 nbclient=0.10.2=pypi_0 nbconvert=7.16.5=pypi_0 nbformat=5.10.4=pypi_0 ncurses=6.4=h6a678d5_0 nest-asyncio=1.6.0=pypi_0 nettle=3.7.3=hbbd107a_1 networkx=3.2.1=py39h06a4308_0 nltk=3.9.1=pypi_0 notebook=7.3.2=pypi_0 notebook-shim=0.2.4=pypi_0 numba=0.60.0=pypi_0 numpy=1.23.5=pypi_0 nvidia-ml-py=12.560.30=pypi_0 omegaconf=2.3.0=pypi_0 openh264=2.1.1=h4ff587b_0 openjpeg=2.5.2=he7f1fd0_0 openpyxl=3.1.5=pypi_0 openssl=3.0.15=h5eee18b_0 opt-einsum=3.4.0=pypi_0 overrides=7.7.0=pypi_0 packaging=24.2=pypi_0 pandas=2.2.3=pypi_0 pandocfilters=1.5.1=pypi_0 parso=0.8.4=pypi_0 pexpect=4.9.0=pypi_0 phonemizer=3.3.0=pypi_0 pillow=11.0.0=py39hcea889d_1 pip=24.2=py39h06a4308_0 platformdirs=4.3.6=pypi_0 pooch=1.8.2=pypi_0 praatio=6.2.0=pypi_0 prometheus-client=0.21.1=pypi_0 prompt-toolkit=3.0.48=pypi_0 propcache=0.2.1=pypi_0 protobuf=5.29.3=pypi_0 psutil=6.1.1=pypi_0 ptyprocess=0.7.0=pypi_0 pure-eval=0.2.3=pypi_0 pyarrow=19.0.0=pypi_0 pycparser=2.22=pypi_0 pydub=0.25.1=pypi_0 pygments=2.19.1=pypi_0 pylatexenc=2.10=pypi_0 pyparsing=3.2.1=pypi_0 pypinyin=0.44.0=pypi_0 pysocks=1.7.1=py39h06a4308_0 python=3.9.21=he870216_1 python-dateutil=2.9.0.post0=pypi_0 python-json-logger=3.2.1=pypi_0 pytorch=2.5.1=py3.9_cuda12.4_cudnn9.1.0_0 pytorch-cuda=12.4=hc786d27_7 pytorch-mutex=1.0=cuda pytz=2024.2=pypi_0 pyworld=0.3.5=pypi_0 pyyaml=6.0.2=py39h5eee18b_0 pyzmq=26.2.0=pypi_0 rapidfuzz=3.11.0=pypi_0 rdflib=7.1.3=pypi_0 readline=8.2 [truncated — 1583 more characters] ``` ### main.py ```python import os from pathlib import Path import httpx from dotenv import load_dotenv from fastapi import FastAPI, File, HTTPException, Request, UploadFile from fastapi.responses import HTMLResponse, JSONResponse from fastapi.staticfiles import StaticFiles from fastapi.templating import Jinja2Templates from pydantic import BaseModel from agents.stress_detector import analyze_stress from agents.session_summary import generate_dummy_chat_reply, generate_dummy_session_report # --------------------------------------------------------------------------- # Config # --------------------------------------------------------------------------- load_dotenv() OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "") # --------------------------------------------------------------------------- # Single conversation workflow prompt (english-only) # --------------------------------------------------------------------------- PROMPT_CONVERSATION_PHASE = """ You are Santé, a live voice conversation agent. English only. This is the CONVERSATION phase. Core behavior: - Behave like a real discussion partner, not a script reader. - Respond to what the user just said, then ask ONE natural follow-up question. - Keep language supportive, neutral, and non-diagnostic. - Never claim medical certainty. Do not provide diagnosis or treatment. CRITICAL output rules: - NEVER start your response with "Conversation:", "Read Aloud:", or any label/prefix. - NEVER include a "Read Aloud" or "Repeat Back" section. - Just speak naturally without any labels or formatting. - Give exactly ONE response per user turn. Do NOT say multiple things. - After you ask your question, STOP and WAIT for the user to answer. - If you're unsure whether the user finished speaking, wait silently. Length constraints: - Max 12 words per response. - One sentence only. Flow guidance: - Start with: "Hi, ready to begin?" - Ask one question at a time, adapting based on the user's prior answer. - Topics: how they're feeling, their day, recent activities, sleep, energy, mood. - Use brief acknowledgments, then continue with the next best follow-up. """.strip() PROMPT_READ_ALOUD_PHASE = """ You are Santé, a live voice conversation agent. English only. This is the READ ALOUD phase. Core behavior: - Provide exactly ONE short sentence for the user to repeat back to you. - The sentence should be clear, natural English for speech signal capture. - After the user repeats it, give a brief one-word acknowledgment ("Good", "Great", "Nice") then the next sentence. - Do NOT have a conversation. Do NOT ask personal questions. CRITICAL output rules: - NEVER start your response with "Read Aloud:", "Conversation:", or any label/prefix. - Just say the sentence directly without any labels or formatting. - Give exactly ONE sentence per turn. Do NOT give multiple sentences. - Output only the repeat sentence; do NOT add a second sentence, follow-up, or extra prompt. - Do NOT give the next repeat sentence until the user has spoken. - After giving a sentence, STOP and WAIT for the user to repeat it. - If you're unsure whether the user finished speaking, wait silently. Length constraints: - Each sentence: 5-10 words. - Vary sentence structure and phoneme coverage. Flow guidance: - Start with: "Thanks—now repeat after me: The sun is shining today." - Use everyday sentences that cover diverse sounds and phonemes. - Keep a calm, supportive tone. """.strip() SEGMENTS = { "conversation": PROMPT_CONVERSATION_PHASE, } # --------------------------------------------------------------------------- # App # --------------------------------------------------------------------------- app = FastAPI(title="Santé", description="Voice AI Health Platform") BASE_DIR = Path(__file__).resolve().parent app.mount("/static", StaticFiles(directory=BASE_DIR / "static"), name="static") templates = Jinja2Templates(directory=BASE_DIR / "templates") # --------------------------------------------------------------------------- # Routes # --------------------------------------------------------------------------- @app.get("/", response_class=HTMLResponse) async def index(request: Request): return templates.TemplateResponse("index.html", {"request": request}) @app.get("/token/{segment}") async def get_ephemeral_token(segment: str): """ Mint an ephemeral key for a specific analysis segment. Segments: conversation """ if not OPENAI_API_KEY or OPENAI_API_KEY == "your_openai_api_key_here": raise HTTPException( status_code=500, detail="OpenAI API key not configured. Set OPENAI_API_KEY in .env", ) if segment == "read-aloud-prompt": return JSONResponse({"instructions": PROMPT_READ_ALOUD_PHASE}) instructions = SEGMENTS.get(segment) if not instructions: raise HTTPException(status_code=400, detail=f"Unknown segment: {segment}") session_config = { "type": "realtime", "model": "gpt-realtime", "instructions": instructions, "audio": { "input": { "turn_detection": { "type": "server_vad", "threshold": 0.8, "prefix_padding_ms": 500, "silence_duration_ms": 800, "create_response": True, }, "transcription": { "model": "gpt-4o-mini-transcribe", }, }, "output": { "voice": "shimmer", }, }, } try: async with httpx.AsyncClient(timeout=30.0) as client: resp = await client.post( "https://api.openai.com/v1/realtime/client_secrets", headers={ "Authorization": f"Bearer {OPENAI_API_KEY}", "Content-Type": "application/json", }, json={"session": session_config}, ) except httpx.TimeoutException as exc: [truncated — 5013 more characters] ``` ### backend/main.py ```python """ Santé — Voice Health Analysis FastAPI backend: token minting, stress analysis upload, and real-time WS analysis. """ import logging import os from pathlib import Path from dotenv import load_dotenv from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from fastapi.staticfiles import StaticFiles from routers import analysis, benchmark, summary, tokens, twilio_voice, websocket BASE_DIR = Path(__file__).resolve().parent ROOT_DIR = BASE_DIR.parent # Load env files from both backend/.env and repository root/.env. # This keeps local dev stable regardless of whether uvicorn is launched from # the backend folder or the repo root. load_dotenv(BASE_DIR / ".env", override=False) load_dotenv(ROOT_DIR / ".env", override=False) logging.basicConfig( level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s", ) app = FastAPI(title="Santé", description="Voice AI Health Platform") # --------------------------------------------------------------------------- # CORS — dev origins always included; production Vercel URL(s) read from env. # On Railway set ALLOWED_ORIGINS to a comma-separated list, e.g.: # https://sante.vercel.app,https://sante-git-main.vercel.app # --------------------------------------------------------------------------- _default_origins = ["http://localhost:3000", "http://127.0.0.1:3000"] _extra_origins = [ o.strip() for o in os.getenv("ALLOWED_ORIGINS", "").split(",") if o.strip() ] app.add_middleware( CORSMiddleware, allow_origins=_default_origins + _extra_origins, allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # --------------------------------------------------------------------------- # Routers # --------------------------------------------------------------------------- app.include_router(tokens.router) app.include_router(analysis.router) app.include_router(summary.router) app.include_router(benchmark.router) app.include_router(websocket.router) app.include_router(twilio_voice.router) # Serve generated PDF reports at /static/reports/{call_sid}.pdf app.mount("/static", StaticFiles(directory="static"), name="static") @app.get("/health") async def health() -> dict: return {"status": "ok"} # --------------------------------------------------------------------------- # Run # --------------------------------------------------------------------------- if __name__ == "__main__": import uvicorn uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=True) ``` [77 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]