Project Info
Inspiration
Language barriers are one of the most persistent and high-impact challenges in clinical care, especially in underserved and volunteer-based healthcare settings. As a medical assistant at a clinic, I frequently work with patients who are not fluent in English. In many cases, the current solution is to use human interpreters over the phone or video call to translate during patient visits. While effective, this approach introduces significant cost and logistical overhead, particularly in volunteer or low-resource environments where interpreter services must be scheduled and paid per session. Over time, these costs accumulate and can become difficult to sustain, limiting access to consistent interpretation services when they are needed most. This project aims to reduce that burden by providing a real-time, low-cost alternative for medical interpretation, enabling clinicians and patients to communicate directly without sacrificing clarity or safety.
What it does
Clinical Interpretation is a real-time bilingual voice system that enables seamless communication between English- and Spanish-speaking patients and providers. Captures live speech from either party Transcribes audio using Deepgram streaming speech-to-text Detects language and translates speech using an LLM Converts translated text into natural speech using Deepgram TTS Delivers real-time bidirectional voice translation during the conversation The result is a fluid, interpreter-like experience that supports natural clinical dialogue without a human intermediary.
How we built it
We built a streaming voice pipeline using: Deepgram Nova / Flux (STT) for real-time multilingual transcription Claude / GPT-4o-mini for strict, deterministic medical translation Deepgram Aura TTS for natural speech output in English and Spanish Node.js for orchestrating real-time audio streams Speaker + microphone streaming libraries to handle live audio input/output
Challenges we ran into
Handling real-time audio streaming without buffer underruns or audio cutouts Managing backpressure between streaming STT and speaker output Ensuring the LLM behaves strictly as a translator (not a conversational agent) Designing reliable bidirectional switching between speaker roles Preventing latency and audio overlap in a continuous conversation loop
Accomplishments we're proud of
Built a fully functional real-time medical voice interpreter Achieved stable bidirectional English ↔ Spanish communication Successfully integrated STT, LLM translation, and TTS into a single low-latency system Implemented buffering logic to eliminate audio cutoffs and improve stream reliability Created a system that closely mirrors real-world clinical interpreter workflows
What we learned
Real-time voice systems fail more often due to streaming and buffering issues than model quality In clinical communication, clarity and reliability matter more than conversational intelligence Role separation (doctor vs patient) is more important than language detection alone LLMs must be tightly constrained to behave as deterministic translators in healthcare settings Building voice AI requires thinking in terms of event-driven audio pipelines, not request-response APIs
What's next
Add structured clinical documentation generation (SOAP notes and visit summaries) Expand language support beyond English and Spanish Improve role detection (automatic doctor/patient inference) Add support for interruption handling (“barge-in” during speech) Integrate with electronic health record (EHR) systems for clinical documentation Optimize latency for near-instant conversational response in real clinical environments
Inspiration
Language barriers are one of the most persistent and high-impact challenges in clinical care, especially in underserved and volunteer-based healthcare settings.
As a medical assistant at a clinic, I frequently work with patients who are not fluent in English. In many cases, the current solution is to use human interpreters over the phone or video call to translate during patient visits. While effective, this approach introduces significant cost and logistical overhead, particularly in volunteer or low-resource environments where interpreter services must be scheduled and paid per session.
Over time, these costs accumulate and can become difficult to sustain, limiting access to consistent interpretation services when they are needed most.
This project aims to reduce that burden by providing a real-time, low-cost alternative for medical interpretation, enabling clinicians and patients to communicate directly without sacrificing clarity or safety.
What it Does
Clinical Interpretation is a real-time bilingual voice system that enables seamless communication between English- and Spanish-speaking patients and providers.
The system:
- Captures live speech from either party
- Transcribes audio using Deepgram streaming speech-to-text
- Detects language and translates speech using an LLM
- Converts translated text into natural speech using Deepgram TTS
- Delivers real-time bidirectional voice translation during the conversation
The result is a fluid, interpreter-like experience that supports natural clinical dialogue without a human intermediary.
How We Built It
We built a streaming voice pipeline using:
- Deepgram Nova / Flux (STT) for real-time multilingual transcription
- GPT-4o-mini for deterministic medical translation
- Deepgram Aura TTS for natural speech output in English and Spanish
- Node.js for orchestrating real-time audio streams
- Speaker and microphone streaming libraries to handle live audio input/output
The system operates as a continuous pipeline:
Speech → Speech-to-Text → Translation → Text-to-Speech → Playback
Challenges We Ran Into
- Handling real-time audio streaming without buffer underruns or audio cutouts
- Managing backpressure between streaming STT and speaker output
- Ensuring the LLM behaves strictly as a translator rather than a conversational agent
- Designing reliable bidirectional switching between speaker roles
- Preventing latency and audio overlap in a continuous conversation loop
Accomplishments That We're Proud Of
- Built a fully functional real-time medical voice interpreter
- Achieved stable bidirectional English ↔ Spanish communication
- Successfully integrated STT, LLM translation, and TTS into a single low-latency system
- Implemented buffering logic to improve stream reliability and reduce audio cutoffs
- Created a system that closely mirrors real-world clinical interpreter workflows
What We Learned
- Real-time voice systems fail more often due to streaming and buffering issues than model quality
- In clinical communication, clarity and reliability matter more than conversational intelligence
- Role separation (doctor vs. patient) is more important than language detection alone
- LLMs must be tightly constrained to behave as deterministic translators in healthcare settings
- Building voice AI requires thinking in terms of event-driven audio pipelines rather than traditional request-response APIs
What's Next for Clinical Interpretation
- Generate structured clinical documentation such as SOAP notes, visit summaries, and translated after-visit instructions
- Expand language support beyond English and Spanish
- Improve role detection and speaker attribution during conversations
- Add interruption handling ("barge-in") for more natural dialogue
- Integrate directly with EHR systems to fit within existing clinical workflows
- Use patient context from the EHR (medications, diagnoses, allergies, labs) to improve translation accuracy
- Support multimodal inputs including PDFs, medical records, discharge paperwork, lab reports, imaging reports, and photographs
- Allow patients to upload test results and ask questions in their preferred language
- Provide translated explanations of medical documents to improve patient understanding and health literacy
- Optimize latency for near-instant conversational response in real clinical environments
Analysis
View
Metric
- 1
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- ExpressIn code
- HTMLIn code
- JavaScriptIn code
4 of 4 appear in the indexed code.
AI coding agents
- CursorCommits
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
52 KB
Source files
14
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
dean8bel/clinical-interpreter
18 files · 52 KB · @ fa3e623
Structure
Application logic
6 files · 33%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- JavaScript43%
- YAML38%
- Markdown10%
- CSS6%
- HTML3%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
package.json
npm · 6- @deepgram/sdk
- dotenv
- express
- mic
- speaker
- ws
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
Feature verification
Buffering logic to reduce audio cutoffs / manage backpressureVerified
Implemented buffering logic to improve stream reliability and reduce audio cutoffs; managing backpressure between STT and speaker output
Claimed on readmemedium confidencemain.js:21— A queue plus playNext() checks speaker.write() return value and waits for the 'drain' event before continuing, handling backpressurepublic/app.js:144— playPcmChunk schedules audio using a running playbackTime cursor to avoid overlap/cutout in browser playback
CLI bidirectional voice interpreter (mic/speaker)Verified
Node.js orchestrating real-time audio streams using speaker + microphone streaming libraries
Claimed on readmehigh confidencemain.js:86— Uses the mic package to stream microphone audio into the session and the speaker package with a queued playback loop for output
Deepgram Aura text-to-speech output in English and SpanishVerified
Converts translated text into natural speech using Deepgram TTS (Aura) in English and Spanish
Claimed on readmehigh confidencelib/config.js:5— TTS_MODEL maps en to aura-2-asteria-en and es to aura-2-diana-eslib/createInterpreterSession.js:33— setSpeakModel sends UpdateSpeak to switch the Deepgram TTS model based on detected user language
Language detection heuristic (regex-based, with Deepgram-provided language hints)Verified
Detects language and translates speech using an LLM
Claimed on readmemedium confidencelib/config.js:13— detectLanguage() first uses Deepgram's languages field, falling back to a Spanish/English keyword regex heuristic
LLM-based translation (GPT-4o-mini) constrained to translator roleVerified
Detects language and translates speech using an LLM (Claude/GPT-4o-mini), strictly as a translator not a conversational agent
Claimed on readmehigh confidencelib/config.js:52— SETTINGS.agent.think configures open_ai gpt-4o-mini with a system prompt restricting the model to translation only, no answers/summaries/advice
Real-time bidirectional voice interpreter pipeline (browser demo)Verified
Delivers real-time bidirectional voice translation during the conversation via a web app
Claimed on readmehigh confidenceserver.js:27— WebSocket server relays mic audio to the Deepgram agent session and streams synthesized audio back to the browserpublic/app.js:100— Browser captures mic audio, streams PCM over the websocket, and plays back returned PCM chunks in real time
Real-time streaming speech-to-text (Deepgram)Verified
Transcribes audio using Deepgram streaming speech-to-text (Nova/Flux)
Claimed on readmehigh confidencelib/config.js:46— SETTINGS.agent.listen.provider configured with deepgram nova-3 model, language multilib/createInterpreterSession.js:77— ConversationText messages from the Deepgram agent socket are captured and emitted as transcripts
Role separation / bidirectional speaker switching (doctor vs patient)Verified
Designing reliable bidirectional switching between speaker roles; role separation matters more than language detection alone
Claimed on readmemedium confidencelib/createInterpreterSession.js:48— nextUserLang/nextSpeakerLabel toggle between doctor and patient after each AgentAudioDone, and drive expected-speaker UI events
Fully functional low-latency STT+LLM+TTS integrationCode-supported
Successfully integrated STT, LLM translation, and TTS into a single low-latency system
Claimed on readmemedium confidencelib/createInterpreterSession.js:9— Single session wires Deepgram's unified voice-agent socket (STT+LLM+TTS in one connection) end to end, but actual latency/stability claims cannot be verified from static code alone
Automatic doctor/patient role inferenceClaimed only
Improve role detection and speaker attribution during conversations (automatic doctor/patient inference)
Claimed on readmehigh confidenceBarge-in / interruption handlingClaimed only
Add support for interruption handling (barge-in during speech)
Claimed on readmehigh confidenceEHR integrationClaimed only
Integrate with electronic health record (EHR) systems for clinical documentation and to use patient context for translation accuracy
Claimed on readmehigh confidenceExpanded language support beyond English/SpanishClaimed only
Expand language support beyond English and Spanish
Claimed on readmehigh confidenceMultimodal document upload and translated explanationsClaimed only
Support multimodal inputs (PDFs, lab reports, imaging, photos) and allow patients to upload test results and ask questions in their preferred language
Claimed on readmehigh confidenceSOAP notes / structured clinical documentation generationClaimed only
Add structured clinical documentation generation (SOAP notes and visit summaries)
Claimed on readmehigh confidence
An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.