# Project export: Phone With Hands This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Phone With Hand helps Deaf and ASL users make phone calls by turning signs into spoken voice, then translating the caller’s speech, tone, and emotion back into animated ASL-style visuals. - Devpost: https://devpost.com/software/call-with-hands - GitHub: https://github.com/quanle3001/phonewithhands - Video: https://www.youtube.com/embed/4cIc0yk14gE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Quan Le (10 commits) ## Devpost submission (written by the team) ### Inspiration I have a cousin who cannot hear and has difficulty speaking, so communicating with her, where truly understanding what she means and how she feels, is always really difficult. When I moved to the States, I realized the problem is much worse for her community as so much of daily life runs through a phone call. Booking a doctor's appointment, calling a pharmacy, reaching customer service, sorting out an issue with a bank, where the answer is always "just give us a call." But for people who have disabilities with hearing or speaking, a phone call is exactly the thing they can't easily do. Even with relay services or texting, something essential gets lost since there's no quick way to speak in their own voice, in real time, expressing their emotion and tone that makes a conversation feel human. They're often forced to depend on someone else to make the call for them, or have to communicate with people that cannot understand their own "voice" or language. Phone With Hands started with a simple, personal question: what if my cousin could pick up the phone, sign naturally, and the person on the other end would simply hear her with her own words, tone, and all? ### What it does Phone With Hands lets people who use sign language make and take real phone calls — live, in both directions. Sign → Speech: You sign into the camera, and the app turns your signs into a natural sentence and speaks it aloud to the other person in a real, expressive voice with meaningful tone — so you're actually talking, not just sending text. Speech → Sign: When they reply, the app instantly shows you the meaning, tone, and key info of what they said, and renders the response as ASL gloss with a 3D signing avatar, so the conversation comes back visually rather than as plain text. A real call: Your phone is the actual handset (mic + speaker) while the app handles the camera, translation, and avatar. It also includes a guided doctor's-appointment scenario, a free testing ground, quick-phrase buttons, and live captions. In short: it gives people who sign a way to talk on the phone in their own voice and emotion, and to receive replies back visually through sign — independently and in real time. How I built it Phone With Hands is a Next.js (React + TypeScript) web app combining on-device computer vision, a lightweight classifier, an AI "brain," and a real-time audio bridge. Sign recognition (on-device): MediaPipe Hands extracts 21 hand landmarks from the webcam in the browser (video never leaves the device), fed into a custom KNN classifier I wrote. I built an in-browser trainer and recorded ~1,400 samples across 18 signs as a seed model. Customized handshapes: For sign → speech I deliberately trained simplified, distinct handshapes instead of motion-heavy ASL — single-frame recognition can't handle real ASL's movement/two hands, so custom shapes train fast and run reliably. (Authentic ASL is reserved for the avatar output.) The brain (ASI:1): ASI:1 (Fetch.ai) sits in the middle — turning choppy sign-gloss into natural, tone-aware sentences, and distilling caller speech into meaning, tone, and key info. Voice: ElevenLabs for expressive TTS and Scribe STT, with the browser's Web Speech API as a free fallback. 3D avatar: A Three.js / React-Three-Fiber Mixamo avatar signs replies from the gloss, with text shown alongside. Real phone handset: A custom Node WebSocket bridge + ngrok turns a phone into the call's mic/speaker (16 kHz PCM streaming) while the Mac runs the camera + avatar. AI tools: As a solo dev and fairly new hacker, I used Claude Code for heavy feature builds and Simular's Sai agent for orchestration, debugging, and infrastructure — tight-scoped tasks, type-checked, one commit per phase. Reliability by design: Every link has a fallback (rule-based AI, Web Speech, typed input, text under the avatar) so a live call never hard-fails. Challenges I ran into Animating real ASL on the 3D avatar: This was the hardest part. My first approach tried to retarget live MediaPipe hand-tracking onto the avatar's 3D skeleton, but the mapping kept dropping the arms off-screen ("invisible hands"), and two-handed signs broke because the left hand was barely tracked. I made a pragmatic call to drive the avatar with clean Mixamo animations instead — realistically, only HELLO is fully animated (a wave) right now, while every other reply falls back to ASL gloss + text. Single-frame recognition can't see motion: Real ASL depends on movement, location, and two hands, but my KNN classifier only sees one frame at a time. I worked around it with custom static handshapes and a tiered strategy to keep recognition fast and reliable. Solo two-handed capture for training model: I couldn't hold a key while signing with both hands, so I built a hands-free Enter-countdown auto-capture for the trainer. Real calls without Twilio: Turning a phone into a real handset meant building a WebSocket audio bridge and getting around iOS Safari's strict secure-origin requirement for mic access — solved with an ngrok tunnel and a single-origin proxy. Accomplishments that I am proud of A genuinely two-way ASL ↔ speech call that runs live. Not a one-direction gimmick — you sign and they hear you, they speak and you see it come back as sign. Both directions work in real time. A real phone handset, not a screen demo. I turned an actual iPhone into the call's mic and speaker over my own WebSocket bridge — proving the conversation happens on a real phone, without needing Twilio or any telephony account. Expressive, human-sounding speech. The user's voice comes through with real tone and emotion via ElevenLabs, instead of flat robotic text-to-speech — which is the whole point of letting someone talk expressively. A from-scratch, in-browser sign recognizer. I built my own trainer and KNN classifier on top of MediaPipe, trained ~1,400 samples across 18 signs, and bundled it as a seed model that loads instantly on any machine — all on-device, with no video ever leaving the camera. Shipping this much, solo, in a weekend. Combining computer vision, an AI language brain, voice, a 3D avatar, and live telephony into one working app as a single developer — by orchestrating AI tools effectively rather than cutting scope. Built to never break. Layered fallbacks at every step mean the app keeps working even when a mic, model, or API isn't available. What I learned How to actually orchestrate AI tools and not just prompt them: The biggest lesson was learning to use AI development tools efficiently: scoping tight, well-defined tasks, verifying every change with type-checks, committing per phase, and knowing when to let Claude Code build heavy features versus when to make a surgical fix myself. Used well, AI tools let one person ship the surface area of a whole team. Computer vision in the browser. I learned how hand-landmark detection works with MediaPipe, how to turn raw landmarks into normalized, position-invariant features, and why on-device inference matters for both latency and privacy. Training a light, practical model: Building my own KNN classifier taught me that the right-sized model often beats the fanciest one — a simple, explainable classifier I could train live in minutes was far more reliable for a demo than a heavy deep model, as long as I designed the input (clean, distinct handshapes) to play to its strengths. Real-time audio is hard: Streaming microphone audio between two devices taught me about sampling rates, PCM downsampling, voice-activity detection, and the strict security rules browsers enforce around microphone access. Design for failure: Working on accessibility software made it concrete that reliability beats novelty — every feature needs a graceful fallback, because the people who'd actually depend on this can't afford for it to break. -Setting up the technical tools and basics: I got hands-on with the unglamorous-but-essential plumbing: configuring a Next.js project and dev environment, managing API keys and environment variables securely (.env.local, server-side proxy routes so secrets never reach the client), understanding API key permissions and scopes, wiring up multiple third-party APIs (ElevenLabs, ASI:1), and standing up local servers + an ngrok tunnel for cross-device testing. ### What's next Real phone calls (PSTN): Integrate Twilio so the app can dial any real phone number — not just a paired device — and work as a true relay replacement for everyday calls. Authentic, motion-aware sign language: Move beyond single-frame custom handshapes to temporal models (LSTM / MediaPipe Holistic) that recognize real ASL with movement, location, and two hands. More and richer training data: Expand the vocabulary far beyond 18 signs with a larger, more diverse dataset — different signers, lighting, and camera angles — for robust real-world accuracy. A fully signing 3D avatar: Build out the avatar so it can sign complete responses (not just HELLO), bringing authentic ASL to the speech → sign direction. On-device and private: Move more of the pipeline on-device for fully private, offline-capable calls. Multi-language support: Extend to other sign languages and spoken languages so it works beyond English/ASL. ## README (from the GitHub repository) # 🤟 Phone With Hand > Berkeley AI Hackathon 2026 — Accessibility bridge for Deaf / ASL users on phone calls. --- ## Prerequisites - **Node.js 18+** (project was developed with Node 24) - **Chrome** (recommended — MediaPipe WASM + WebRTC works best there) - **Webcam** connected and accessible to the browser --- ## Install & Run ```bash # 1. Install dependencies npm install # 2. Start the dev server npm run dev # 3. Open in Chrome open http://localhost:3000 ``` That's it — no API keys, no backend, everything runs in the browser. --- ## Granting Camera Permission in Chrome 1. Open `http://localhost:3000/demo` 2. Chrome shows a camera permission prompt — click **Allow** 3. If you accidentally denied it: click the **camera icon** (🎥) in the address bar → select **Allow** → refresh --- ## Pages | URL | Description | |---|---| | `http://localhost:3000` | Home page — contacts + **Train signs** button | | `http://localhost:3000/train` | **Sign trainer** — teach the app custom ASL handshapes | | `http://localhost:3000/call/dr-smith` | Scripted demo call | | `http://localhost:3000/call/testing-call` | Testing Call — sign playground (pretrained gestures work out of the box) | | `http://localhost:3000/demo` | Redirects to the wired call route | --- ## Training your own signs (`/train`) The trainer is **browser-only** — your webcam frames and the trained model never leave the device. 1. Open `http://localhost:3000/train` and allow camera access. 2. Pick a trainable sign from the **Vocabulary** list (right). The 7 *pretrained* gestures are marked and need **no training** — MediaPipe recognises them directly. 3. Make the handshape and **hold the Record button** (or press **Space**) to grab ~30 frames. Vary angle/distance slightly for robustness. The per-label sample count updates live. 4. Watch **Live prediction** — it shows what the current model thinks your hand is and turns green when it matches the selected sign. 5. Use **Clear** (trash icon / "Clear …" button) to redo a label, or **Clear all** to start over. 6. **Export** downloads the model as JSON; **Import** loads one back. Trained signs immediately drive sign→speech in the call screens. ### Where the model is stored - **localStorage** key `pwh.signModel.v1` (survives refreshes/restarts). - **Export to file** for backup or sharing between machines (`Import` to restore). ### How recognition works (classifier-agnostic) `components/HandTracker.tsx` owns the camera + MediaPipe Hands pipeline and emits, per frame, the 21 hand landmarks **and** MediaPipe's pretrained gesture. Landmarks are normalized to be translation- and scale-invariant (`lib/landmarks.ts`: wrist-centered, scaled by hand size) before classification. All recognition goes through the **`SignClassifier`** interface (`lib/classifier/types.ts`) — `train()`, `addSample()`, `predict()`, `export()`/`import()`. The current implementation is a single-frame **KNN** (`lib/classifier/knn.ts`); swap it for an LSTM later **without** rewriting the trainer UI or the call pages — just satisfy the same interface. `lib/signStore.ts` holds the one app-wide instance + persistence. --- ## Key Files ``` app/ page.tsx Home page (contacts + Train signs button) train/page.tsx ← Sign trainer (KNN, capture, export/import) call/[id]/page.tsx In-call sign→speech experience globals.css Tailwind base components/ HandTracker.tsx ← CORE: webcam + MediaPipe Hands, emits landmarks CameraSignDetector.tsx In-call readout: pretrained + KNN over HandTracker GlossPanel.tsx Animated ASL gloss cards (Framer Motion) lib/ landmarks.ts Translation/scale-invariant landmark normalization classifier/types.ts SignClassifier interface (classifier-agnostic) classifier/knn.ts KNN implementation of SignClassifier signStore.ts App-wide model: localStorage + file import/export data/ signs.ts Vocabulary: pretrained + KNN labels, phrases, tones ``` ### How CameraSignDetector works 1. Calls `getUserMedia` for the webcam stream 2. Dynamically imports `@mediapipe/tasks-vision` (avoids SSR issues) 3. Loads the `GestureRecognizer` WASM from jsDelivr CDN + gesture model from Google Storage — no API key needed 4. Runs `recognizeForVideo()` in a `requestAnimationFrame` loop 5. Draws green skeleton connectors + red landmark dots on a `` overlay 6. Shows: **hands detected count**, **gesture name** (Open_Palm / Closed_Fist / Victory / etc.), **confidence %** --- ## Tech Stack | Layer | Tool | |---|---| | Framework | Next.js 15 (App Router) | | Language | TypeScript | | Styles | Tailwind CSS | | Animations | Framer Motion | | Hand tracking | @mediapipe/tasks-vision (GestureRecognizer, browser WASM) | --- ## What's Mocked (Next Iterations) | Feature | Status | |---|---| | Hand landmark detection | ✅ Live (MediaPipe) | | Gesture → ASL phrase mapping | 🔜 Next iteration | | ASL interpretation (LLM) | 🔜 Next iteration | | TTS voice output | 🔜 Next iteration | | Sign avatar animation | 🔜 Next iteration | | Two-device WebSocket link | 🔜 Next iteration | --- ## Ethics Note This is an accessibility *guide / prototype*, **NOT a certified ASL interpreter**. It does not replace human interpreters, Video Relay Services (VRS), or CART services. In high-stakes conversations (medical, legal, financial), use a certified human interpreter. ## Detected evidence (automated analysis) Indexed codebase: 39 recognized source files, 1227 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (52 of 52) ``` .gitignore app/api/ai/route.ts app/api/comprehend/route.ts app/api/tts/route.ts app/call/[id]/page.tsx app/call/live-phone/page.tsx app/capture/page.tsx app/demo/page.tsx app/globals.css app/handset/page.tsx app/layout.tsx app/page.tsx app/train/page.tsx components/CallErrorBoundary.tsx components/CameraSignDetector.tsx components/GlobalErrorBanner.tsx components/GlossPanel.tsx components/HandTracker.tsx components/SigningAvatar.tsx data/contacts.ts data/signs.ts HANDSET-SETUP.md lib/ai.ts lib/classifier/knn.ts lib/classifier/types.ts lib/handset.ts lib/landmarks.ts lib/recents.ts lib/ringtone.ts lib/signClips.ts lib/signRetarget.ts lib/signStore.ts lib/tts.ts next.config.ts package.json postcss.config.mjs public/avatar.fbx public/mediapipe/gesture_recognizer.task public/mediapipe/wasm/vision_wasm_internal.js public/mediapipe/wasm/vision_wasm_internal.wasm public/mediapipe/wasm/vision_wasm_module_internal.js public/mediapipe/wasm/vision_wasm_module_internal.wasm public/mediapipe/wasm/vision_wasm_nosimd_internal.js public/mediapipe/wasm/vision_wasm_nosimd_internal.wasm public/phw-sign-clips.json public/sign-model.json public/wave.fbx README.md server/handset-bridge.mjs start-handset.sh tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @mediapipe/tasks-vision@^0.10.14, @react-three/drei@^10.7.7, @react-three/fiber@^9.6.1, @types/node@^20.11.5, @types/react@^19.0.0, @types/react-dom@^19.0.0, @types/three@^0.169.0, @types/ws@^8.18.1, autoprefixer@^10.4.17, dotenv@^17.4.2, express@^5.2.1, framer-motion@^11.2.10, http-proxy@^1.18.1, lucide-react@^1.21.0, next@^15.2.4, ngrok@^5.0.0-beta.2, postcss@^8.4.35, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^3.4.1, three@^0.169.0, twilio@^6.0.2, typescript@^5.3.3, ws@^8.21.0 ### Recent commits (newest first) - fix: route speech to iPhone handset only (no Mac echo) - Phase 5: iPhone handset telephony bridge + integration - Phase 3: avatar plays Mixamo wave on HELLO (idle otherwise); keep retargeting pipeline + recorded clips as record - Phase 3 Stage B — in-browser sign capture: /capture page (PoseLandmarker + HandLandmarker via tasks-vision) records 14 signs to phw-sign-clips.json with 3-2-1 countdown, localStorage persist, download/import; adds 'Train avatar' nav button on home; bundles recorded clips (14/14 signs) for Stage C - Phase 3 Stage A — 3D signing avatar (idle): SigningAvatar component loads Mixamo character directly as .fbx (FBXLoader) and plays embedded idle clip (natural pose, correct hands), chest-up upper-body framing, simple lighting, placeholder fallback; THEY SPEAK panel side-by-side no-scroll layout (avatar + text); text fallback always renders; adds three / @react-three/fiber / @react-three/drei deps - Phase 2 — Speech to Comprehension: mic/STT + ASI:1 (meaning/tone/keyInfo/gloss) in THEY SPEAK panel, rule-based fallback, Simulate-speech test box; Dr. Smith Scripted<->Live toggle + teleprompter cue card; Scripted mode = interactive 6-turn sign-driven conversation (RICO scenario); Live mode = dual-gate advance (speech fuzzy-match AND signs) - Phase 1.7 + call UI polish: sentence combination & 5s auto-speak, equal-width panels, stable big camera (no shrink/jump), stats HUD overlay, fixed caption box, retrained model - feat(phase-1.5): KNN sign trainer + custom-sign recognition in calls - feat: Phase 1 sign-to-speech (ElevenLabs + ASI:1) + Testing Call - style: full-bleed Apple light-mode redesign; remove fake window chrome - feat: scaffold Next.js app with macOS-style demo UI and live hand detection ## Key source files (fetched from GitHub, selected and truncated for size) ### HANDSET-SETUP.md ```markdown # 📱 Handset Mode — iPhone as the phone, Mac as the translator A **free, no-telephony** bridge. You hold your **iPhone** like a phone: its **mic** captures the caller's voice and its **speaker** plays the app's spoken reply. The **Mac** runs the translator UI (webcam Sign→Speech + 3D avatar Speech→Sign). Audio is relayed in real time over a WebSocket. **No Twilio, no phone number, no PSTN.** ``` iPhone (Safari, /handset) ⇄ Bridge (ws, :5051) ⇄ Mac (/call/live-phone) mic → PCM → bridge ── ElevenLabs Scribe STT ──→ transcript → Mac speaker ← PCM ← bridge ←─ ElevenLabs TTS ←── "speak" ← Mac (also speaks locally) ``` All audio is **16-bit PCM, mono, 16000 Hz** in both directions. --- ## Prerequisites - `.env.local` already has `ELEVENLABS_API_KEY` and `ASI1_API_KEY` (do not commit it). - Node 24 (the bridge uses built-in `fetch`/`FormData`/`Blob`). - Deps installed: `ws`, `express`, `dotenv`. --- ## Run steps > Use the Node on PATH: `export PATH="$HOME/.nvm/versions/node/v24.17.0/bin:$PATH"` 1. **Dev server (Mac UI)** — port 3000 (probably already running): ```bash npm run dev ``` 2. **Handset bridge** — port 5051: ```bash npm run handset ``` It logs `handset bridge on http://localhost:5051 (WS path /ws)`. 3. **ngrok for the iPhone.** iOS Safari only grants mic access over **HTTPS**, and the iPhone must reach BOTH the page and the WebSocket. Pick one: **Option A — two tunnels (most reliable):** ```bash ngrok http 3000 # → https://APP.ngrok-free.app (serves /handset page + Mac UI) ngrok http 5051 # → https://BRIDGE.ngrok-free.app (the WebSocket bridge) ``` Then set the WS env so BOTH roles use the public bridge (next step). **Option B — single tunnel on the bridge:** if you serve the Mac UI only on `localhost:3000` and just need the phone to reach the bridge, tunnel **5051** and open the handset page from that same origin is NOT served by the bridge — so Option A is recommended. (The bridge only serves `/` health + `/ws`.) 4. **Set env vars**, then restart `npm run dev` so `NEXT_PUBLIC_*` is picked up (the bridge reads its vars at `npm run handset` start): ```bash # .env.local (additions) NEXT_PUBLIC_HANDSET_WS=wss://BRIDGE.ngrok-free.app/ws # public bridge WS (Option A) # ELEVENLABS_VOICE_ID=JBFqnCBsd6RMkjVDRZzb # optional (default George) # HANDSET_PORT=5051 # optional (bridge port) ``` - If you DON'T set `NEXT_PUBLIC_HANDSET_WS`, the client derives it from the page host: `ws://localhost:5051/ws` on localhost, or `wss://:5051/ws` on an https host. Behind ngrok that derived `:5051` host usually isn't reachable — so **set `NEXT_PUBLIC_HANDSET_WS` to the public bridge wss URL**. - The Mac (on `localhost:3000`) can use the same public WS, or omit the env and fall back to `ws://localhost:5051/ws`. The iPhone **must** use the public wss. 5. **On the Mac:** open **` [truncated — 2577 more characters] ``` ### package.json ``` { "name": "phone-with-hand", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint", "handset": "node server/handset-bridge.mjs" }, "dependencies": { "@mediapipe/tasks-vision": "^0.10.14", "@react-three/drei": "^10.7.7", "@react-three/fiber": "^9.6.1", "@types/three": "^0.169.0", "@types/ws": "^8.18.1", "dotenv": "^17.4.2", "express": "^5.2.1", "framer-motion": "^11.2.10", "http-proxy": "^1.18.1", "lucide-react": "^1.21.0", "next": "^15.2.4", "react": "^19.0.0", "react-dom": "^19.0.0", "three": "^0.169.0", "twilio": "^6.0.2", "ws": "^8.21.0" }, "devDependencies": { "@types/node": "^20.11.5", "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "autoprefixer": "^10.4.17", "ngrok": "^5.0.0-beta.2", "postcss": "^8.4.35", "tailwindcss": "^3.4.1", "typescript": "^5.3.3" }, "overrides": { "rollup": "npm:@rollup/wasm-node@4.62.2" } } ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import "./globals.css"; import GlobalErrorBanner from "@/components/GlobalErrorBanner"; export const metadata: Metadata = { title: "Phone With Hand", description: "An accessibility bridge helping Deaf / ASL users make phone calls independently.", icons: { icon: "data:image/svg+xml,🤟", }, }; export default function RootLayout({ children, }: { children: React.ReactNode; }) { return ( {children} ); } ``` ### app/page.tsx ```typescript "use client"; import { useEffect, useRef, useState } from "react"; import { useRouter } from "next/navigation"; import Link from "next/link"; import { motion, AnimatePresence, useReducedMotion } from "framer-motion"; import { Search, Video, Hand } from "lucide-react"; import { CONTACTS, type Contact } from "@/data/contacts"; import { getRecents, type RecentCall } from "@/lib/recents"; // ── Apple light-mode tokens ──────────────────────────────────────────────────── const T = { bg: "#F5F5F7", surface: "#FFFFFF", label: "#1D1D1F", secondLabel: "rgba(60,60,67,0.60)", tertLabel: "rgba(60,60,67,0.30)", separator: "rgba(60,60,67,0.12)", blue: "#007AFF", green: "#34C759", red: "#FF3B30", } as const; const SPRING = { type: "spring" as const, stiffness: 260, damping: 30, mass: 0.9 }; // ── Helpers ──────────────────────────────────────────────────────────────────── function fmtTime(ts: number): string { return new Date(ts).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" }); } function fmtDuration(s: number): string { if (s < 60) return `${s}s`; return `${Math.floor(s / 60)}m ${s % 60}s`; } function AvatarCircle({ contact }: { contact: Contact }) { const isEmoji = (contact.avatar.codePointAt(0) ?? 0) > 127; return (
{isEmoji ? ( {contact.avatar} ) : ( {contact.avatar} )}
); } // ── Contact row ──────────────────────────────────────────────────────────────── interface RowProps { contact: Contact; index: number; isLast: boolean; onCall: (c: Contact) => void; rm: boolean; } function ContactRow({ contact, index, isLast, onCall, rm }: RowProps) { return ( <> onCall(contact)} onKeyDown={(e) => { if (e.key === "Enter" || e.key === " ") onCall(contact); }} whileHover={{ backgroundColor: "rgba(0,0,0,0.028)" }} whileTap={{ scale: 0.99 }} className="flex items-center gap-3 px-4 py-3 cursor-pointer focus:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-[#007AFF] transition-colors" >

{contact.name}

{contact.subtitle}

{ e.stopPropagation(); onCall(contact); }} aria-label={`Video call ${contact.name}`} className="flex items-center justify-center flex-shrink-0 rounded-full focus:outline-none focus-visible:ring-2 focus-visible:ring-[#007AFF]" style={{ width: 34, height: 34, background: contact.callable ? `${T.green}1a` : "rgba(0,0,0,0.05)", border: `1px solid ${contact.callable ? T.green + "44" : "rgba(0,0,0,0.09)"}`, }} > {/* iOS-style inset separator — hidden after last row */} {!isLast && (
)} ); } // ── Recent row ───────────────────────────────────────────────────────────────── function RecentRow({ recent, isLast }: { recent: RecentCall; isLast: boolean }) { const completed = recent.outcome === "completed"; return ( <>
{completed ? "📞" : "📵"}

{recent.contactName}

{completed ? "Connected" : "Cancelled"} {recent.duration > 0 && ` · ${fmtDuration(recent.duration)}`}

{fmtTime(recent.timestamp)}

{!isLast && (
)} ); } // ── Page ─────────────────────────────────────────────────────────────────────── export default function HomePage() { const router = useRouter(); const rm = useReducedMotion(); const [tab, setTab] = useState<"contacts" | "recents">("contacts"); const [query, setQuery] = useState(""); const [toast, setToast] = useState(null); const [recents, setRecents] = [truncated — 7230 more characters] ``` ### app/demo/page.tsx ```typescript import { redirect } from "next/navigation"; // /demo kept for backwards compatibility — forwards to the wired call route. export default function DemoPage() { redirect("/call/dr-smith"); } ``` ### app/handset/page.tsx ```typescript "use client"; import { useEffect, useRef, useState } from "react"; import { Mic, MicOff, PhoneOff, Phone } from "lucide-react"; import { getHandset } from "@/lib/handset"; // ───────────────────────────────────────────────────────────────────────────── // /handset — the iPhone handset page. Open this in iOS Safari via the HTTPS ngrok // URL (iOS blocks mic on http). Tap "Start call" (a user gesture is required to // start the mic + unlock audio). Captures mic → bridge; plays the Mac's TTS reply. // NO camera, NO avatar here. // ───────────────────────────────────────────────────────────────────────────── const T = { bg: "#0B0B0F", card: "#16161C", label: "#FFFFFF", sub: "rgba(255,255,255,0.6)", green: "#34C759", red: "#FF3B30", blue: "#0A84FF", }; export default function HandsetPage() { const [started, setStarted] = useState(false); const [peer, setPeer] = useState(false); const [muted, setMuted] = useState(false); const [level, setLevel] = useState(0); const [notice, setNotice] = useState(null); const levelRef = useRef(0); // Smooth the meter a touch via rAF so it doesn't thrash React. useEffect(() => { let raf = 0; const tick = () => { setLevel((l) => l + (levelRef.current - l) * 0.3); raf = requestAnimationFrame(tick); }; raf = requestAnimationFrame(tick); return () => cancelAnimationFrame(raf); }, []); useEffect(() => () => { try { getHandset().close(); } catch { /* noop */ } }, []); async function startCall() { setNotice(null); const h = getHandset(); h.onPeer((role, connected) => { if (role === "mac") setPeer(connected); }); h.onLevel((lv) => { levelRef.current = lv; }); try { h.connect("phone"); await h.startPlayback(); // unlock audio output on this user gesture await h.startMic(); // start capturing the caller's voice setStarted(true); } catch (e) { console.warn("[handset] start failed:", e); setNotice("Microphone blocked. Use the HTTPS link and allow mic access, then tap Start again."); } } function toggleMute() { const h = getHandset(); const next = !muted; setMuted(next); h.setMuted(next); } function endCall() { try { getHandset().close(); } catch { /* noop */ } setStarted(false); setPeer(false); } const meterPct = Math.round(Math.min(1, level) * 100); return (
{/* Top status */}

Phone With Hand

Handset

{!started ? "Not started" : peer ? "Connected to Mac" : "Waiting for Mac…"}
{/* Center: big call indicator + level meter */}
{started ? "📞" : "🤙"}
{/* mic level meter */}
Mic level {started ? (muted ? "Muted" : `${meterPct}%`) : "—"}
{notice && (

{notice}

)}
{/* Bottom controls */}
{!started ? ( ) : (