# Project export: WingNote This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: WingNote – Smart hardware meets AI-powered documentation to listen, structure, and simplify care for nurses and patients. - Devpost: https://devpost.com/software/wingnote - GitHub: https://github.com/bencullenn/wing-note - Demo: https://gamma.app/docs/WingNote-Smart-Hardware-for-Seamless-Nurse-Documentation-ggxl8whe2cff0lg - Video: https://www.youtube.com/embed/TX3a1ze3Uwk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Google Cloud: Best Use of Gemini (Google Pixel Watches x4)) - Team: 2 GitHub contributor(s) — Ben Cullen (34 commits), elsa-bismuth (12 commits) ## Devpost submission (written by the team) ### Inspiration Nurses spend over 4 hours per shift on documentation, leading to burnout, high turnover rates, and reduced patient care time. At the same time, most patients don’t understand their own medical notes, creating a gap in healthcare accessibility. We set out to build WingNote—a fully integrated hardware + AI-powered solution that automates nurse documentation while ensuring patients receive clear, personalized summaries in their preferred language and at their reading level. With a tap of an RFID badge, a nurse can start recording a patient interaction, and AI does the rest—transcribing, analyzing, structuring, and presenting the information in an EHR-ready format, while also generating a patient-friendly summary. ### What it does WingNote is a smart badge & AI-powered software system that automates clinical documentation for nurses and improves patient understanding. Hardware RFID-based patient identification – Scans patient wristbands to log patient ID. Power efficient audio & video syncing – Captures speech & patient behavior for every interaction throughout the day. Xiao ESP32-S3 smart badge – Microphone, camera, and Wi-Fi enables wireless transmission of real-time data. Custom 3D-printed housing – Encases ESP32, RFID reader, and patient wristbands for hospital use. AI-powered software Speech recognition (Deepgram API) – Converts nurse-patient dialogue into detailed, diatorized text. Vision analysis (Gemini 2.0 Flash) – Identifies helpful details about the patient and room to enhance notes. EHR structuring (Mistral AI) – Transforms raw conversation into Epic-compatible notes. Doctor’s review portal (Next.js/Vercel v0) – Physicians edit & approve AI-generated documentation. Patient summary app (Next.js/Vercel v0) – Provides multilingual, easy-to-read summaries of doctor's notes. Patient Q&A (Perplexity Sonar API & Postgres SQL) – Patients can receive reliable medication answers grounded in their doctor's notes and web data. WingNote fully automates the documentation pipeline, improving workflow efficiency and patient care. ### How we built it Hardware stack Xiao ESP32-S3 Sense – Microcontroller with Wi-Fi, camera, and microphone. RC522 RFID reader – Used for patient wristband scanning. Arduino Nano (I2C bridge) – Converts SPI to I2C to allow RFID to work over a retractable wire. Custom 3D-printed cases for: Smart badge (ESP32-S3 with mic & camera). RFID reader & Arduino Nano (compact with retractable cable). Patient wristband (RFID tag enclosed for easy scanning). Smart badge (ESP32-S3 with mic & camera). RFID reader & Arduino Nano (compact with retractable cable). Patient wristband (RFID tag enclosed for easy scanning). Software stack Deepgram API – Speech-to-text transcription with enhanced medical-grade accuracy and speaker identification. Gemini 2.0 Flash API – Patient behavior tracking. Mistral AI – Converts conversation data into structured EHR notes. Next.js/Vercel v0 – Doctor's portal → EHR-ready notes for review & approval. Patient’s portal → Multilingual, age-adapted health summaries. Doctor's portal → EHR-ready notes for review & approval. Patient’s portal → Multilingual, age-adapted health summaries. Perplexity API – Provides LLM responses grounded in web data Supabase - Postgres SQL database and cloud storage for notes and artifacts From hardware to AI-powered documentation, we built WingNote from the ground up. ### Challenges we ran into RFID integration with ESP32-S3 The RC522 RFID module requires 6 wires, but the retractable cable we used only had 4 wires. Solution: Introduced an Arduino Nano as an I2C bridge, converting: RFID Reader (SPI, 6-wire) → Arduino Nano (SPI to I2C) → ESP32-S3 (4-wire I2C). RFID integration with ESP32-S3 The RC522 RFID module requires 6 wires, but the retractable cable we used only had 4 wires. Solution: Introduced an Arduino Nano as an I2C bridge, converting: RFID Reader (SPI, 6-wire) → Arduino Nano (SPI to I2C) → ESP32-S3 (4-wire I2C). Dual serial communication on Xiao ESP32-S3 Initially, we tried using two UART ports simultaneously, but the ESP32-S3 couldn’t support dual UART for both the computer and the RFID unit. Solution: Switched RFID communication to I2C and kept UART free for debugging. Dual serial communication on Xiao ESP32-S3 Initially, we tried using two UART ports simultaneously, but the ESP32-S3 couldn’t support dual UART for both the computer and the RFID unit. Solution: Switched RFID communication to I2C and kept UART free for debugging. Power efficient data transmission over Wi-Fi Issue: ESP32-S3 couldn’t stream high-resolution video & audio simultaneously and it takes a lot of battery to do so continuously. Solution: Store audio and video locally on the device while filming. Transfer audio and video between appointments to save on battery and improve quality. Reliably transferring data between the hardware and software remains a significant challenge. Power efficient data transmission over Wi-Fi Issue: ESP32-S3 couldn’t stream high-resolution video & audio simultaneously and it takes a lot of battery to do so continuously. Solution: Store audio and video locally on the device while filming. Transfer audio and video between appointments to save on battery and improve quality. Reliably transferring data between the hardware and software remains a significant challenge. Overcoming these challenges led to a fully functional, AI-powered clinical documentation system. ### Accomplishments we're proud of Built a complete hardware + AI solution from scratch. Seamless RFID scanning + multimodal AI processing (speech + video). Fully functioning real-time documentation workflow (nurse → AI → doctor → patient). Deployed an AI-powered patient portal that enhances healthcare accessibility. Integrated Deepgram, Gemini 2.0, Perplexity, and Mistral AI to process medical documentation. WingNote is a fully operational, AI-enhanced documentation system that works end-to-end. ### What we learned Embedded systems development – Learned how to interface ESP32-S3, RFID modules, and I2C communication. AI workflow optimization – Successfully merged audio & vision AI into a seamless pipeline. Real-time AI integration – Implemented low-latency STT + vision analysis with structured data formatting. Hardware design & 3D printing – Designed & printed custom enclosures for smart badges, RFID readers, and wristbands. This project pushed us to solve real-world hardware & AI integration challenges. ### What's next ? Direct EHR integration (Epic, Cerner, etc.) Currently, doctors review AI-generated notes in our portal, but future versions will push data directly into EHRs. Direct EHR integration (Epic, Cerner, etc.) Currently, doctors review AI-generated notes in our portal, but future versions will push data directly into EHRs. Real-time transcription for nurse efficiency Currently, transcription happens after the visit, but we plan to add live transcription during conversations. Real-time transcription for nurse efficiency Currently, transcription happens after the visit, but we plan to add live transcription during conversations. Wider language support Expanding multilingual AI summaries & patient Q&A support for even more languages. Wider language support Expanding multilingual AI summaries & patient Q&A support for even more languages. WingNote is just getting started—our vision is to make documentation completely seamless. ## README (from the GitHub repository) # wing-note ## Detected evidence (automated analysis) Indexed codebase: 43 recognized source files, 236 KB. - C (language) — detected in the code - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel AI SDK (technology) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - Mistral AI (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (60 of 60) ``` .DS_Store .gitignore back-end/.python-version back-end/main.py back-end/pyproject.toml back-end/README.md back-end/test.py back-end/uv.lock doctor-portal/.gitignore doctor-portal/components.json doctor-portal/config.tsx doctor-portal/eslint.config.mjs doctor-portal/next.config.ts doctor-portal/package.json doctor-portal/postcss.config.mjs doctor-portal/README.md doctor-portal/src/app/globals.css doctor-portal/src/app/layout.tsx doctor-portal/src/app/page.tsx doctor-portal/src/app/visit/[id]/page.tsx doctor-portal/src/components/EHRForm.tsx doctor-portal/src/components/ui/button.tsx doctor-portal/src/components/ui/card.tsx doctor-portal/src/components/ui/input.tsx doctor-portal/src/components/ui/label.tsx doctor-portal/src/components/ui/spinner.tsx doctor-portal/src/components/ui/textarea.tsx doctor-portal/src/lib/utils.ts doctor-portal/tailwind.config.ts doctor-portal/tsconfig.json firmware/firmware.c package.json patient-portal/.gitignore patient-portal/components.json patient-portal/config.tsx patient-portal/eslint.config.mjs patient-portal/next.config.ts patient-portal/package.json patient-portal/postcss.config.mjs patient-portal/README.md patient-portal/src/app/appointment/[id]/page.tsx patient-portal/src/app/globals.css patient-portal/src/app/layout.tsx patient-portal/src/app/page.tsx patient-portal/src/app/profile/page.tsx patient-portal/src/components/AppointmentList.tsx patient-portal/src/components/BottomNav.tsx patient-portal/src/components/ChatComponent.tsx patient-portal/src/components/NotificationList.tsx patient-portal/src/components/ui/button.tsx patient-portal/src/components/ui/card.tsx patient-portal/src/components/ui/dialog.tsx patient-portal/src/components/ui/input.tsx patient-portal/src/components/ui/select.tsx patient-portal/src/components/ui/switch.tsx patient-portal/src/components/VisitDetails.tsx patient-portal/src/lib/utils.ts patient-portal/tailwind.config.ts patient-portal/tsconfig.json README.md ``` ### Dependencies - back-end/pyproject.toml: deepgram-sdk@>=3.10.0, fastapi[standard]@>=0.115.8, google-genai@>=1.2.0, google-generativeai@>=0.8.4, opencv-python@>=4.11.0.86, pillow@>=11.1.0, pyaudio@>=0.2.14, python-dotenv@>=1.0.1, requests@>=2.32.3, supabase@>=2.13.0, websockets@>=14.2 - doctor-portal/package.json: @eslint/eslintrc@^3, @radix-ui/react-label@^2.1.2, @radix-ui/react-slot@^1.1.2, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@15.1.7, lucide-react@^0.475.0, next@15.1.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, tailwind-merge@^3.0.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 - package.json: react-markdown@^9.0.3 - patient-portal/package.json: @ai-sdk/react@^1.1.16, @eslint/eslintrc@^3, @radix-ui/react-dialog@^1.1.6, @radix-ui/react-select@^2.1.6, @radix-ui/react-slot@^1.1.2, @radix-ui/react-switch@^1.1.3, @types/node@^20, @types/react@^19, @types/react-dom@^19, ai@^4.1.41, class-variance-authority@^0.7.1, clsx@^2.1.1, date-fns@^4.1.0, eslint@^9, eslint-config-next@15.1.7, lucide-react@^0.475.0, next@15.1.7, openai-edge@^1.2.2, postcss@^8, react@^19.0.0, react-dom@^19.0.0, react-markdown@^9.0.3, speech-recognition-polyfill@^0.0.5, tailwind-merge@^3.0.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 ### Recent commits (newest first) - Added firmware code. Cleaned up back-end server. - Badge endpoint works end to end - Fixed build issues with patient portal - Fixed issue blocking doctor portal build - Fixed 500 error - Merge branch 'elsa' of github.com:bencullenn/wing-note - fixed bugs and added language - Refactored backend to support new data format from pin - Enhanced UI for patient portal - Merge branch 'main' of https://github.com/bencullenn/wing-note into elsa - Enhanced UI for doctor portal - Merge branch 'main' of github.com:bencullenn/wing-note - Got patient portal visit to return a basic summary of the visit - Updated LLM prompt for AI chat - Merge branch 'elsa' of https://github.com/bencullenn/wing-note into elsa - working AI chat added - Added AI chat - Refactored audio video buffer endpoint and updated visit types on frontend - fixed dependencies - Merged streaming branch ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "dependencies": { "react-markdown": "^9.0.3" } } ``` ### back-end/pyproject.toml ``` [project] name = "back-end" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.11" dependencies = [ "deepgram-sdk>=3.10.0", "fastapi[standard]>=0.115.8", "google-genai>=1.2.0", "google-generativeai>=0.8.4", "opencv-python>=4.11.0.86", "pillow>=11.1.0", "pyaudio>=0.2.14", "python-dotenv>=1.0.1", "requests>=2.32.3", "supabase>=2.13.0", "websockets>=14.2", ] ``` ### doctor-portal/package.json ``` { "name": "doctor-portal", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@radix-ui/react-label": "^2.1.2", "@radix-ui/react-slot": "^1.1.2", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.475.0", "next": "15.1.7", "react": "^19.0.0", "react-dom": "^19.0.0", "tailwind-merge": "^3.0.1", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### patient-portal/package.json ``` { "name": "patient-portal", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@ai-sdk/react": "^1.1.16", "@radix-ui/react-dialog": "^1.1.6", "@radix-ui/react-select": "^2.1.6", "@radix-ui/react-slot": "^1.1.2", "@radix-ui/react-switch": "^1.1.3", "ai": "^4.1.41", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "date-fns": "^4.1.0", "lucide-react": "^0.475.0", "next": "15.1.7", "openai-edge": "^1.2.2", "react": "^19.0.0", "react-dom": "^19.0.0", "react-markdown": "^9.0.3", "speech-recognition-polyfill": "^0.0.5", "tailwind-merge": "^3.0.1", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### back-end/main.py ```python # Standard library imports import asyncio import base64 import json import os import struct import subprocess import wave from datetime import datetime from io import BytesIO from typing import Dict, List # Third party imports import cv2 import google.generativeai as genai import numpy as np import requests from deepgram import ( DeepgramClient, PrerecordedOptions, ) from dotenv import load_dotenv from fastapi import ( FastAPI, File, Form, UploadFile, HTTPException, Query, Depends, WebSocket, Path, ) from fastapi.middleware.cors import CORSMiddleware from google.generativeai import GenerativeModel from PIL import Image from supabase import create_client, Client from google.genai.types import HttpOptions, Part load_dotenv() app = FastAPI() url: str = os.environ.get("SUPABASE_URL") key: str = os.environ.get("SUPABASE_KEY") supabase: Client = create_client(url, key) deepgram_key = os.environ.get("DEEPGRAM_KEY") deepgram_client = DeepgramClient(deepgram_key) google_api_key = os.environ.get("GOOGLE_API_KEY") genai.configure(api_key=google_api_key) mistral_api_key = os.getenv("MISTRAL_API_KEY") doctor_id = 1 perplexity_api_key = os.environ.get("PERPLEXITY_API_KEY") app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:3000"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) audio_file_paths = [] video_file_paths = [] ### Audio and Video Streaming SAMPLE_RATE = 16000 CHANNELS = 1 SAMPLE_WIDTH = 2 # 16-bit audio """audio_buffer = BytesIO() # will hold a valid WAV-file stream video_buffer = BytesIO() # will hold raw video packets (e.g. h264)""" """wav_file = wave.open(audio_buffer, "wb") wav_file.setnchannels(CHANNELS) wav_file.setsampwidth(SAMPLE_WIDTH) wav_file.setframerate(SAMPLE_RATE)""" buffer_lock = asyncio.Lock() HEADER_FORMAT = "!B" # 1 byte for type, HEADER_SIZE = struct.calcsize(HEADER_FORMAT) @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): global wav_file, audio_buffer, video_buffer await websocket.accept() print("WebSocket connection established.") try: while True: header = await websocket.receive_bytes() # Read the header to see if it's 0x01 (audio) or 0x02 (video) packet_type = struct.unpack(HEADER_FORMAT, header) data = await websocket.receive_bytes() async with buffer_lock: if packet_type[0] == 0x01: wav_file.writeframes(data) else: video_buffer.write(data) except Exception as e: print(f"WebSocket connection error: {e}") finally: # On disconnect, close the open WAV file resource. async with buffer_lock: try: wav_file.close() except Exception: pass print("WebSocket connection closed.") @app.post("/upload-audio") async def upload_audio(file: UploadFile = File(...)): # Read in the audio file and save it to a file and add the file path to the list file_bytes = await file.read() if file_bytes is None: raise HTTPException(status_code=400, detail="Invalid audio file") # Convert bytes to int16 samples file_data = np.frombuffer(file_bytes, dtype=np.int16) # get the timestamp ts = datetime.now().strftime("%Y%m%d_%H%M%S") audio_file_path = "audio" + ts + ".m4a" # Create WAV file with proper audio settings with wave.open(audio_file_path, "w") as wav_file: wav_file.setnchannels(CHANNELS) # Using the global CHANNELS value (1 for mono) wav_file.setsampwidth( SAMPLE_WIDTH ) # Using the global SAMPLE_WIDTH (2 bytes per sample) wav_file.setframerate(SAMPLE_RATE) # Using the global SAMPLE_RATE (16000) wav_file.writeframes(file_data.tobytes()) audio_file_paths.append(audio_file_path) return {"message": "Audio processed successfully", "path": audio_file_path} @app.post("/upload-video") async def upload_video(file: UploadFile = File(...)): # Read in the video file and save it to a file and add the file path to the list file_bytes = await file.read() if file_bytes is None: raise HTTPException(status_code=400, detail="Invalid video file") # get the timestamp ts = datetime.now().strftime("%Y%m%d_%H%M%S") # Video settings width, height = 320, 240 frame_rate = 10 # FPS output_folder = f"frames_{ts}" video_file_path = f"video_{ts}.raw" # Ensure output folder exists if not os.path.exists(output_folder): os.makedirs(output_folder) # Process raw video data index = 0 offset = 0 frames = [] while offset < len(file_bytes): # Read frame size (4 bytes) frame_size = int.from_bytes(file_bytes[offset : offset + 4], byteorder="little") offset += 4 # Move past frame size metadata # Extract frame data frame_data = file_bytes[offset : offset + frame_size] offset += frame_size # Save extracted frame as JPEG frame_path = os.path.join(output_folder, f"frame_{index:03d}.jpg") with open(frame_path, "wb") as img_file: img_file.write(frame_data) frames.append(frame_path) index += 1 print(f"Extracted {index} frames.") # Convert frames to video fourcc = cv2.VideoWriter_fourcc(*"mp4v") # Changed to MP4 format video_writer = cv2.VideoWriter(video_file_path, fourcc, frame_rate, (width, height)) try: for frame_path in frames: frame = cv2.imread(frame_path) if frame is not None: # Check if frame was read successfully video_writer.write(frame) finally: video_writer.release() # Clean up frame files for frame_path in frames: try: os.remove(frame_path) except OSError: pass try: o [truncated — 25873 more characters] ``` ### patient-portal/src/app/page.tsx ```typescript import NotificationList from "@/components/NotificationList"; import AppointmentList from "@/components/AppointmentList"; import { BellIcon, CalendarIcon } from "lucide-react"; export default function Home() { return (

Notifications

Appointments

); } ``` ### doctor-portal/src/app/layout.tsx ```typescript import type React from "react" import type { Metadata } from "next" import { Inter } from "next/font/google" import "./globals.css" const inter = Inter({ subsets: ["latin"] }) export const metadata: Metadata = { title: "EHR System", description: "Simple EHR system for patient management", } export default function RootLayout({ children, }: { children: React.ReactNode }) { return (

Wingnote EHR System

Electronic Health Records Management

{children}
) } ``` ### patient-portal/src/app/layout.tsx ```typescript import type { Metadata } from 'next' import { Inter } from 'next/font/google' import './globals.css' import BottomNav from '@/components/BottomNav' const inter = Inter({ subsets: ['latin'] }) export const metadata: Metadata = { title: 'Patient Portal', description: 'A portal for patients to view their medical information', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return (

Wingnote Patient Portal

Your Health Information at Your Fingertips

{children}
) } ``` ### doctor-portal/src/app/page.tsx ```typescript "use client"; import Link from "next/link"; import { Button } from "@/components/ui/button"; import { useEffect, useState } from "react"; import { API_URL } from "../../config"; interface Visit { id: number; patient: { first_name: string, last_name: string }; created_at: string; approved: boolean; } export default function Home() { const [visits, setVisit] = useState([]); async function fetchVisits() { const response = await fetch(API_URL + "/visits"); const data = await response.json(); console.log("Data: ", data); setVisit(data); } useEffect(() => { fetchVisits(); }, []); function formatDate(dateString: string) { return new Intl.DateTimeFormat("en-US", { year: "numeric", month: "short", day: "numeric", hour: "2-digit", minute: "2-digit", }).format(new Date(dateString)); } return (

Visit List

Total Visits: {visits.length} • Pending Approval: {visits.filter(v => !v.approved).length}
    {visits.map((visit) => (
  • {visit.patient.first_name} {visit.patient.last_name.toUpperCase()}
    {formatDate(visit.created_at)}
    {visit.approved ? "Approved" : "To Approve"}
    • ))}
); } ``` ### patient-portal/src/app/profile/page.tsx ```typescript "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { ChatComponent } from "@/components/ChatComponent"; import { useEffect, useState } from "react"; import { API_URL, PATIENT_MRN } from "../../../config"; interface PatientProfile { first_name: string; last_name: string; age: number; gender: string; mrn: number; } export default function ProfilePage() { const [patient, setPatient] = useState(null); async function getPatientProfile() { const response = await fetch(API_URL + `/patient/${PATIENT_MRN}`); const data = await response.json(); console.log("Data: ", data); setPatient(data); } useEffect(() => { getPatientProfile(); }, []); return (
Profile Information

Name: {patient?.first_name} {patient?.last_name}

Age: {patient?.age}

Medical Record Number: {patient?.mrn}

); } ``` [32 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]