# Project export: NeuraFlow This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: The AI glasses for dementia patients that you'll never forget. - Devpost: https://devpost.com/software/neuraflow-udrik6 - GitHub: https://github.com/stratsid/NeuraFlow - Video: https://www.youtube.com/embed/Aj_nUz0O4-E?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) โ€” stratsid (10 commits), GabrielIrazabal (7 commits), TejasviTyagi14 (1 commits) ## Devpost submission (written by the team) ### Inspiration As all of us have come across Alzheimer's or dementia so far, we understand how difficult it is to lead a life in that way. When we looked at our family members or friends that deal with these problems, we were determined to use our time at TreeHacks to tackle this - and allow it to truly enhance people's everyday lives. Could we build Meta Glasses or something similar that can truly change the lives of these individuals? We are leaving this Hackathon knowing we just did this. ### What it does The NeuroFlow glasses provide dementia patients with an AI Healthcare Assistant that can provide them with any information about things or people they want to remember. It measures the heart rate of the patient, and if it notices distress, then it will ask the dementia patient how it can help. It has full context awareness of what the user likes for them to store on their behalf. The dementia patient can then easily recall whatever they are struggling to remember. With our implementation, it can even read and respond in 100+ languages! ### How we built it The hardware component of the product are 2 ESP32s that provide camera data and heart rate information. This data is analyzed to determine if an individual's heart rate is abnormal - in such a case we expect the dementia patient to be in need of help. This is when we activate our classification model that determines who a person is, providing helpful supplementary information to allow people to work. In the event that the dementia patient says, "Flow", our agent turns on to ask the user what they need help with. Access to their entire day's worth of information (and the internet too!) is at their disposal. In the event that they need some calm nature music, they can get that too! ### Challenges we ran into Some challenges we ran into were primarily related to connection. Learning how to parse the data provided by the ESP32 consistently with the WiFi faltering and our own novelty with the subject was especially difficult, but over time, we loved every minute of the challenges, and we learned so much! We also were trying to implement a CNN at first, but classifying many images of us through the grainy camera would have been difficult. To improve our efficiency, we applied FaceNet instead using embeddings. ### Accomplishments we're proud of Within 36 hours, we were able to build this entire project from 0 to 1. We started with the three of us and 2 ESP32, and we are leaving the Packard building with a solid voice recognition model for dementia patients that can not only register when they are stressed, but also recognize people around them and provide them with information that they need stored. ### What we learned That we can be a lot more ambitious nowadays. Each of us learned from each other - from the hardware to AI architecture to voice recognition and response models. We managed git conflicts, learned how to work under intense time pressure, and even how to CAD a sweet design. ### What's next We see ourselves scaling up our compute more, so this tool can be more assistive to more people with alzheimer's and dementia. We genuinely believe in this tool, and we think we can be used by many people that not only have dementia but could use a healthcare assistant at any given time. It truly can enhance human lives at a very minimal cost (literally fractions of a cent). We can also pick up on more biomarkers to be even more confident about how to respond with the in-built agent. Memory agents to parse through our vast amounts of data to personalize the user experience will also be a crucial next step. ## README (from the GitHub repository) # NeuraFlow v2.0 A real-time biometric and vision analysis system with a modern React UI. ## Features - โค๏ธ **Real-time Heart Rate**: Visualization and spike detection via BLE chest strap. - ๐Ÿ“ธ **Live Computer Vision**: Face recognition and person identification via ESP32-CAM. - ๐Ÿ—ฃ๏ธ **Flow Voice**: Personalized greeting and interaction using OpenAI. - ๐Ÿ“Š **Live Dashboard**: Real-time graphing and system logs. ## ๐Ÿš€ Setup Guide for Collaborators ### 1. Prerequisites - **Python 3.9+** - **Node.js 18+** - **Hardware**: ESP32-CAM, BLE Heart Rate Monitor (Polar/Garmin/etc.) ### 2. Clone the Repository ```bash git clone cd NeuraFlow ``` ### 3. Backend Setup Create a virtual environment and install dependencies: ```bash # Create venv python3 -m venv venv # Activate venv # On Mac/Linux: source venv/bin/activate # On Windows: # .\venv\Scripts\activate # Install dependencies pip install -r requirements.txt ``` ### 4. Frontend Setup Install Node.js dependencies: ```bash cd frontend npm install cd .. ``` ### 5. Configuration (.env) Create a `.env` file in the root `NeuraFlow/` directory. You can copy the structure below: ```bash # NeuraFlow/.env # ESP32 Camera IP (Get this from your serial monitor) CAM_IP=192.168.1.X # BLE Heart Rate Monitor UUID (Run 'python scan_ble.py' to find yours) ADDR=YOUR-BLE-UUID-HERE # OpenAI API Key (For voice features) OPENAI_API_KEY=sk-your-key-here ``` ### 6. Running the System **Step 1: Start the Backend** Make sure your venv is activated. ```bash python3 server.py # Server will start on http://localhost:8000 ``` **Step 2: Start the Frontend** Open a new terminal window. ```bash cd frontend npm run dev # UI will open at http://localhost:5173 ``` ## ๐Ÿ›  Troubleshooting - **BLE Connection Failed**: Ensure Bluetooth is on and run `python3 scan_ble.py` to verify your device is visible and get the correct UUID. - **Camera not showing**: Check if the ESP32 is powered on and the IP in `.env` matches. - **"Module not found"**: Ensure you have activated the virtual environment (`source venv/bin/activate`) before running python scripts. ## Detected evidence (automated analysis) Indexed codebase: 43 recognized source files, 348 KB. - C (language) โ€” detected in the code - C++ (language) โ€” detected in the code - CSS (language) โ€” detected in the code - FastAPI (technology) โ€” detected in the code - HTML (language) โ€” detected in the code - JavaScript (language) โ€” detected in the code - OpenAI (technology) โ€” detected in the code - Python (language) โ€” detected in the code - PyTorch (technology) โ€” detected in the code - React (technology) โ€” detected in the code - Tailwind CSS (technology) โ€” detected in the code ## Codebase structure (from repository index) ### Files (62 of 62) ``` .gitignore .vscode/face_classifier/face_classifier.py bpm_log.csv data/bpm_log.csv data/flow_memory.json data/spike_events.csv data/voice_turns.jsonl esp32code/ESP32CAM_Code_ino_copy_20260214011644/app_httpd.cpp esp32code/ESP32CAM_Code_ino_copy_20260214011644/board_config.h esp32code/ESP32CAM_Code_ino_copy_20260214011644/camera_index.h esp32code/ESP32CAM_Code_ino_copy_20260214011644/camera_pins.h esp32code/ESP32CAM_Code_ino_copy_20260214011644/ci.yml esp32code/ESP32CAM_Code_ino_copy_20260214011644/ESP32CAM_Code_ino_copy_20260214011644.ino esp32code/ESP32CAM_Code_ino_copy_20260214011644/partitions.csv esp32code/heartrate_ino/heartrate/heartrate.ino flow_pipeline.py frontend/.gitignore frontend/eslint.config.js frontend/index.html frontend/package.json frontend/postcss.config.js frontend/README.md frontend/src/App.jsx frontend/src/components/BiometricsChart.jsx frontend/src/components/CameraFeed.jsx frontend/src/components/ChatHistoryPanel.jsx frontend/src/components/EnrollmentForm.jsx frontend/src/components/HeartPulse.jsx frontend/src/components/IntroScreen.jsx frontend/src/components/LiveLog.jsx frontend/src/components/Manage.jsx frontend/src/components/Navbar.jsx frontend/src/components/VoiceWave.jsx frontend/src/context/NeuraContext.jsx frontend/src/index.css frontend/src/main.jsx frontend/tailwind.config.js frontend/vite.config.js Legacy/monitor_and_log.py Legacy/realtime_watch.py Legacy/spike_watcher.py Legacy/talk.py model monitor_and_log.py nohup.out README README.md realtime_watch.py requirements.txt run.sh scan_ble.py server.py setup_files/face_net.py setup_files/predict_outside.py setup_files/read_bpm.py SETUP.md spike_events.csv spike_watcher.py talk.py TODO vision_model/label_map.json vision_model/svm_clf.joblib ``` ### Dependencies - frontend/package.json: @eslint/js@^9.39.1, @types/react@^19.2.7, @types/react-dom@^19.2.3, @vitejs/plugin-react@^5.1.1, autoprefixer@^10.4.24, clsx@^2.1.1, eslint@^9.39.1, eslint-plugin-react-hooks@^7.0.1, eslint-plugin-react-refresh@^0.4.24, framer-motion@^12.34.0, globals@^16.5.0, lucide-react@^0.564.0, postcss@^8.5.6, react@^19.2.0, react-dom@^19.2.0, recharts@^3.7.0, tailwind-merge@^3.4.0, tailwindcss@^3.4.19, vite@^7.3.1 - requirements.txt: bleach, bleak, facenet-pytorch, fastapi, joblib, numpy, openai, opencv-python, python-dotenv, python-multipart, requests, scikit-learn, sounddevice, torch, torchaudio, torchvision, uvicorn, vosk, watchdog, websockets ### Recent commits (newest first) - Frontend Finales - context model - Ui setup - Merge branch 'main' of https://github.com/stratsid/NeuraFlow - merge try - random push - chore: Install project dependencies and add a footage image. - Web - setup steps - NEURAFLOW FREAKING TUFF AH - final clean and run - NOW VOICE - final cleanup and moving towards voice - cleanup - adding requirements - Classification Works - flow added - Flow Audio Assistant - Live footage - deleting crappy data - face recognition works ## Key source files (fetched from GitHub, selected and truncated for size) ### SETUP.md ```markdown # Setup Run these commands from the repo root. ```bash # 1) Create and activate virtual environment python3 -m venv venv source venv/bin/activate # 2) Install dependencies pip install -r requirements.txt # 3) Download Vosk wake-word model and point repo-local "model" to it curl -L -o /tmp/vosk-model-small-en-us-0.15.zip https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip unzip -q /tmp/vosk-model-small-en-us-0.15.zip -d /tmp ln -s /tmp/vosk-model-small-en-us-0.15 model # 4) Environment variables export OPENAI_API_KEY="sk-..." export ADDR="ESP32_BLE_ADDRESS" # Optional certificate path if your environment needs it export SSL_CERT_FILE="$(python -c 'import certifi; print(certifi.where())')" # 5) Run python flow_pipeline.py ``` ## If You Already Have a Vosk Model Use this instead of creating the `model` symlink: ```bash export VOSK_MODEL_PATH="/full/path/to/vosk-model-small-en-us-0.15" ``` ``` ### requirements.txt ``` fastapi uvicorn python-multipart websockets python-dotenv bleak # AI / ML torch torchvision torchaudio facenet-pytorch scikit-learn joblib numpy opencv-python # Voice openai vosk sounddevice # Utils bleach watchdog requests ``` ### frontend/package.json ``` { "name": "frontend", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "clsx": "^2.1.1", "framer-motion": "^12.34.0", "lucide-react": "^0.564.0", "react": "^19.2.0", "react-dom": "^19.2.0", "recharts": "^3.7.0", "tailwind-merge": "^3.4.0" }, "devDependencies": { "@eslint/js": "^9.39.1", "@types/react": "^19.2.7", "@types/react-dom": "^19.2.3", "@vitejs/plugin-react": "^5.1.1", "autoprefixer": "^10.4.24", "eslint": "^9.39.1", "eslint-plugin-react-hooks": "^7.0.1", "eslint-plugin-react-refresh": "^0.4.24", "globals": "^16.5.0", "postcss": "^8.5.6", "tailwindcss": "^3.4.19", "vite": "^7.3.1" } } ``` ### server.py ```python import asyncio import json import logging import os import threading from contextlib import asynccontextmanager from pathlib import Path import uvicorn from fastapi import FastAPI, WebSocket, WebSocketDisconnect, UploadFile, File, Form from fastapi.middleware.cors import CORSMiddleware from fastapi.staticfiles import StaticFiles from dotenv import load_dotenv load_dotenv() # Import the existing pipeline logic import flow_pipeline # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger("NeuraFlowServer") def _is_placeholder(value: str) -> bool: v = (value or "").strip().lower() return v in {"", "sk-...", "sk-your-key-here", "your-openai-key"} # Global state for WebSockets class ConnectionManager: def __init__(self): self.active_connections: list[WebSocket] = [] async def connect(self, websocket: WebSocket): await websocket.accept() self.active_connections.append(websocket) def disconnect(self, websocket: WebSocket): self.active_connections.remove(websocket) async def broadcast(self, message: dict): for connection in self.active_connections: try: await connection.send_json(message) except Exception: pass manager = ConnectionManager() # Hook for flow_pipeline to send data # We'll patch this into flow_pipeline def pipeline_callback(event_type: str, data: dict): """ Called by flow_pipeline when new data is available. We'll schedule a broadcast on the main event loop. """ asyncio.run_coroutine_threadsafe( manager.broadcast({"type": event_type, "data": data}), app_loop # We need a reference to the main loop ) @asynccontextmanager async def lifespan(app: FastAPI): # Startup global app_loop app_loop = asyncio.get_running_loop() key = os.getenv("OPENAI_API_KEY", "") logger.info("OPENAI_API_KEY present: %s", (not _is_placeholder(key))) if _is_placeholder(key): logger.error("OPENAI_API_KEY missing/placeholder. Voice synthesis will fail.") vosk_path = os.getenv("VOSK_MODEL_PATH", str(Path(__file__).resolve().parent / "model")) logger.info("VOSK model path: %s (exists=%s)", vosk_path, Path(vosk_path).expanduser().exists()) # Inject callback into flow_pipeline flow_pipeline.set_broadcast_callback(pipeline_callback) # Start pipeline in a separate thread to avoid blocking FastAPI # flow_pipeline.main contains blocking calls (like audio playback) def run_pipeline(): # Create a new event loop for this thread loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.run_until_complete(flow_pipeline.main()) loop.close() pipeline_thread = threading.Thread(target=run_pipeline, daemon=True) pipeline_thread.start() yield # Shutdown # We can't easily cancel a thread, but for now we rely on daemon=True # and maybe setting a stop event if we had one. app = FastAPI(lifespan=lifespan) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Serves build artifacts in production # app.mount("/", StaticFiles(directory="frontend/dist", html=True), name="static") @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): await manager.connect(websocket) try: while True: text_data = await websocket.receive_text() try: msg = json.loads(text_data) if msg.get("action") == "trigger_greeting": threading.Thread(target=flow_pipeline.trigger_greeting, daemon=True).start() except Exception as e: logger.error(f"Error handling client message: {e}") except WebSocketDisconnect: manager.disconnect(websocket) @app.post("/api/enroll") async def enroll_person( name: str = Form(...), files: list[UploadFile] = File(...) ): temp_dir = Path("temp_uploads") temp_dir.mkdir(exist_ok=True) saved_paths = [] try: for file in files: file_path = temp_dir / file.filename with open(file_path, "wb") as buffer: content = await file.read() buffer.write(content) saved_paths.append(file_path) # Trigger training in a thread to not block handling # But for now, we'll do it synchronously or in a thread and return status # Since training is heavy, better to offload. def run_train(): flow_pipeline.train_new_person(name, saved_paths) # Cleanup for p in saved_paths: try: p.unlink() except: pass temp_dir.rmdir() # only if empty threading.Thread(target=run_train).start() return {"status": "success", "message": f"Enrolling {name}. Training started..."} except Exception as e: return {"status": "error", "message": str(e)} if __name__ == "__main__": uvicorn.run("server:app", host="0.0.0.0", port=8000, reload=False) ``` ### frontend/src/main.jsx ```javascript import React from 'react' import ReactDOM from 'react-dom/client' import App from './App.jsx' import './index.css' import { NeuraProvider } from './context/NeuraContext.jsx' ReactDOM.createRoot(document.getElementById('root')).render( , ) ``` ### frontend/src/App.jsx ```javascript import React, { useState, useEffect } from 'react'; import { useNeura } from './context/NeuraContext'; import { HeartPulse } from './components/HeartPulse'; import { LiveLog } from './components/LiveLog'; import { CameraFeed } from './components/CameraFeed'; import { ChatHistoryPanel } from './components/ChatHistoryPanel'; import { Manage } from './components/Manage'; import { Navbar } from './components/Navbar'; import { VoiceWave } from './components/VoiceWave'; import { IntroScreen } from './components/IntroScreen'; import { motion, AnimatePresence } from 'framer-motion'; function App() { const { isGreetingComplete } = useNeura(); const [showIntro, setShowIntro] = useState(true); const [activeTab, setActiveTab] = useState('dashboard'); const [minTimeElapsed, setMinTimeElapsed] = useState(false); // Ensure intro is visible for at least 3 seconds useEffect(() => { const timer = setTimeout(() => setMinTimeElapsed(true), 3000); return () => clearTimeout(timer); }, []); // Transition to dashboard when greeting is complete AND minimum time has passed useEffect(() => { if (isGreetingComplete && minTimeElapsed) { setShowIntro(false); } }, [isGreetingComplete, minTimeElapsed]); // Fallback: Ensure intro always closes after 10s if signal never comes useEffect(() => { const timer = setTimeout(() => { if (showIntro) { setShowIntro(false); } }, 10000); return () => clearTimeout(timer); }, [showIntro]); return (
{showIntro ? ( setShowIntro(false)} /> ) : ( {activeTab === 'dashboard' ? (
{new Date().toLocaleDateString('en-US', { weekday: 'long', month: 'long', day: 'numeric' })}
Live

Neuraflow

{/* Box 1: Heart Rate */}
{/* Box 2: Chat History */}
{/* Box 3: Activity History */}
{/* Box 4: Visual Feed */}
) : ( )} )}
); } export default App; ``` ### spike_watcher.py ```python import subprocess import sys import re import time BPM_THRESHOLD = 80.0 BPM_RE = re.compile(r"\bBPM:\s*([0-9]+(?:\.[0-9]+)?)\b") def main(): # Launch your existing script proc = subprocess.Popen( [sys.executable, "monitor_and_log.py"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1, # line-buffered universal_newlines=True, ) print("Started monitor_and_log.py, watching output...\n") try: for raw_line in proc.stdout: line = raw_line.rstrip("\n") # 1) DO NOT ignore anything: mirror every line to your terminal print(line, flush=True) # 2) Parse BPM lines and detect spikes m = BPM_RE.search(line) if m: bpm = float(m.group(1)) if bpm >= BPM_THRESHOLD: ts = time.strftime("%Y-%m-%d %H:%M:%S") print(f"[{ts}] >>> SPIKE DETECTED: {bpm:.1f} BPM (>= {BPM_THRESHOLD}) <<<", flush=True) except KeyboardInterrupt: print("\nStopping...", flush=True) finally: proc.terminate() try: proc.wait(timeout=2) except subprocess.TimeoutExpired: proc.kill() if __name__ == "__main__": main() ``` ### scan_ble.py ```python import asyncio from bleak import BleakScanner async def main(): print("Scanning for BLE devices... (this may take a few seconds)") try: devices = await BleakScanner.discover(timeout=5.0) if not devices: print("No BLE devices found.") return print("\nFound Devices:") print("-" * 40) found_chest_strap = False for d in devices: name = d.name or "Unknown" # Highlight likely heart rate monitors is_likely = any(x in name.lower() for x in ["heart", "hrm", "polar", "garmin", "coospo", "wahoo"]) prefix = "โค๏ธ " if is_likely else " " print(f"{prefix}{d.address} {name}") if is_likely: found_chest_strap = True print("-" * 40) if found_chest_strap: print("\nโœ… Found likely Heart Rate Monitor(s) marked with โค๏ธ") print("Copy the address (UUID) and I will add it to your configuration.") else: print("\nNo obvious Heart Rate Monitors found.") print("If your device is on, try moving it closer or wetting the sensors.") except Exception as e: print(f"\nโŒ ERROR: {e}") if "turned off" in str(e): print("๐Ÿ‘‰ Please turn on Bluetooth on your computer and try again.") else: print("Try checking your system permissions for Bluetooth.") if __name__ == "__main__": asyncio.run(main()) ``` ### run.sh ```shell #!/bin/bash # --- NeuraFlow Unified Startup Script --- set -euo pipefail ROOT_DIR="$(cd "$(dirname "$0")" && pwd)" cd "$ROOT_DIR" # Load env vars from .env when present so backend/frontend share a stable config. if [ -f ".env" ]; then set -a # shellcheck disable=SC1091 source ".env" set +a fi if [ ! -x "venv/bin/python" ]; then echo "โŒ Missing venv interpreter at venv/bin/python" echo "Create it with: python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt" exit 1 fi if [ -z "${OPENAI_API_KEY:-}" ] || [ "${OPENAI_API_KEY:-}" = "sk-..." ] || [ "${OPENAI_API_KEY:-}" = "sk-your-key-here" ]; then echo "โŒ OPENAI_API_KEY missing or placeholder. Add it to .env or export it before starting." exit 1 fi echo "๐Ÿš€ Starting NeuraFlow ecosystem..." # 1. Cleanup stale processes echo "๐Ÿงน Cleaning up existing processes on ports 8000 and 5173..." PIDS_8000="$(lsof -ti:8000 2>/dev/null || true)" if [ -n "$PIDS_8000" ]; then kill -9 $PIDS_8000 2>/dev/null || true fi PIDS_5173="$(lsof -ti:5173 2>/dev/null || true)" if [ -n "$PIDS_5173" ]; then kill -9 $PIDS_5173 2>/dev/null || true fi # Give it a second to breathe sleep 1 # 2. Start Backend in background echo "๐Ÿง  Starting Backend (FastAPI)..." venv/bin/python server.py & BACKEND_PID=$! # 3. Start Frontend in background echo "๐Ÿ–ฅ๏ธ Starting Frontend (Vite)..." cd frontend npm run dev & FRONTEND_PID=$! echo "------------------------------------------------" echo "โœ… Both processes are launching!" echo "๐Ÿ“ก Backend: http://localhost:8000" echo "๐ŸŽจ Frontend: http://localhost:5173" echo "------------------------------------------------" echo "Press Ctrl+C to stop both." # Function to handle shutdown cleanup() { echo -e "\n๐Ÿ›‘ Shutting down NeuraFlow..." kill $BACKEND_PID 2>/dev/null kill $FRONTEND_PID 2>/dev/null exit } # Trap SIGINT (Ctrl+C) trap cleanup SIGINT # Keep the script running wait ``` ### talk.py ```python import json import queue import base64 import io import wave import numpy as np import sounddevice as sd from vosk import Model, KaldiRecognizer from openai import OpenAI from openai.helpers import LocalAudioPlayer # ------------------------- # CONFIG # ------------------------- MODEL_PATH = "model" SAMPLE_RATE = 16000 WAKE_WORD = "flow" AUDIO_MODEL = "gpt-4o-mini-audio-preview" VOICE = "marin" AUDIO_FORMAT = "wav" # we will consistently use WAV client = OpenAI() player = LocalAudioPlayer() # ------------------------- # AUDIO PLAYBACK (WAV bytes) # ------------------------- def play_wav_bytes(wav_bytes: bytes): with wave.open(io.BytesIO(wav_bytes), "rb") as wf: sr = wf.getframerate() nchan = wf.getnchannels() sampwidth = wf.getsampwidth() frames = wf.readframes(wf.getnframes()) if sampwidth != 2: raise ValueError(f"Expected 16-bit WAV (sampwidth=2), got sampwidth={sampwidth}") audio = np.frombuffer(frames, dtype=np.int16).reshape(-1, nchan) sd.play(audio, sr) sd.wait() # ------------------------- # OPENAI INTRO VOICE (WAV) # ------------------------- def speak_intro(): intro_text = "Hey, I'm Flow, your dementia assistant." speech = client.audio.speech.create( model="gpt-4o-mini-tts", voice=VOICE, input=intro_text, ) player.play(speech) print("\n[BOOT] Starting Flow...") speak_intro() print("[SPEAKING] Intro played.") print("\nFlow is ready.") print(f"[LISTENING] Waiting for wake word '{WAKE_WORD}'...\n") # ------------------------- # RECORD COMMAND (local) # ------------------------- def record_wav(seconds: float) -> bytes: print(f"[LISTENING] Recording command for {seconds:.1f}s...") audio = sd.rec( int(seconds * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=1, dtype="int16", ) sd.wait() print("[NOT LISTENING]") buf = io.BytesIO() with wave.open(buf, "wb") as wf: wf.setnchannels(1) wf.setsampwidth(2) wf.setframerate(SAMPLE_RATE) wf.writeframes(audio.tobytes()) return buf.getvalue() # ------------------------- # ONE OPENAI CALL (audio-in โ†’ audio-out) # ------------------------- def one_call_audio_response(user_wav: bytes): b64 = base64.b64encode(user_wav).decode("utf-8") resp = client.chat.completions.create( model=AUDIO_MODEL, modalities=["text", "audio"], audio={"format": AUDIO_FORMAT, "voice": VOICE}, messages=[ { "role": "developer", "content": ( "You are Flow, a dementia assistant. " "Speak naturally, warm, calm. " "Keep responses very brief in 1 sentences and as helpful as possible." "Remind the users very calmly who is around them if asked." ), }, { "role": "user", "content": [ {"type": "input_audio", "input_audio": {"data": b64, "format": "wav"}} ], }, ], ) msg = resp.choices[0].message transcript = "" if getattr(msg, "audio", None) and getattr(msg.audio, "transcript", None): transcript = msg.audio.transcript out_audio_bytes = base64.b64decode(msg.audio.data) return transcript, out_audio_bytes # ------------------------- # WAKE WORD DETECTOR (Vosk) # ------------------------- model = Model(MODEL_PATH) wake_rec = KaldiRecognizer(model, SAMPLE_RATE) q = queue.Queue() def callback(indata, frames, time, status): q.put(bytes(indata)) stream = sd.RawInputStream( samplerate=SAMPLE_RATE, blocksize=8000, dtype="int16", channels=1, callback=callback, ) # ------------------------- # MAIN LOOP (ONE TURN) # ------------------------- with stream: while True: data = q.get() if wake_rec.AcceptWaveform(data): result = json.loads(wake_rec.Result()) text = result.get("text", "").lower() if WAKE_WORD in text: print("[WAKE] Heard wake word.\n") user_wav = record_wav(seconds=6.0) print("[THINKING] Calling OpenAI...") transcript, out_wav = one_call_audio_response(user_wav) print("[DONE]\n") if transcript: print("Flow (transcript):", transcript) print("[SPEAKING] Playing response...") play_wav_bytes(out_wav) print("[END] Flow session ended.") break ``` [31 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]