# Project export: Aside.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: It's a wearable narrator AI that watches your day and announces it out loud in any personality you want. - Devpost: https://devpost.com/software/aside-ai - GitHub: https://github.com/Da0t/AsideAI - Demo: https://canva.link/4wl7x5y0fwp96ma - Video: https://www.youtube.com/embed/3NdMEDV5Ews?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Best Use of Deepgram) - Team: 5 GitHub contributor(s) — Dat Nguyen (15 commits), Claude Opus 4.8 (11 commits), ethannguyennn (5 commits), Sukhman Cheema (1 commits), Bryan Pham (1 commits) ## Devpost submission (written by the team) ### Inspiration Wearables today either surveil you (cameras, trackers) or quantify you (steps, sleep, heart rate). None of them are fun. We kept joking that everyone deserves "main-character energy" — a narrator who makes your boring Tuesday sound legendary. So we asked: what if a wearable didn't record your life or coach you through it, but performed it back to you in real time? Walk into a room and get announced like a champion. Trip over a chair and hear a goth mommy gently fuss over you. That idea was too funny not to build. ### What it does Aside is a clip-on camera + mic that watches the world around you and narrates it out loud in real time in a personality you choose. The same moment becomes wildly different depending on the vibe: Hype Man: "AYO they're BACK and they brought the FUEL, let's GOOO!" female therapist: "Look at you, my little raven returning with your potion of warmth." Epic Quest Narrator: "Lo! The hero returns from the Bean Mines, sacred elixir in hand." ### How we built it Three machines, three jobs: Eyes: Raspberry Pi (QNX): captures camera frames and streams them over Wi-Fi. Brain: Laptop (Python): the orchestrator. It sends each frame to Claude Haiku 4.5 vision, which acts as both eyes and brain — looking at the image and writing one short in-character line in a single call. Deepgram handles speech-to-text (ears) and text-to-speech (voice), and Redis stores memory + which personality is active. Voice: Phone (React Native / Expo): the personality switcher and live feed that plays the narration. ### Challenges we ran into Our first "eyes" didn't pan out. We planned to use a dedicated vision service; it fell through. We pivoted to sending frames straight to Claude which turned out simpler and better. QNX is unforgiving. MediaPipe (our planned on-device detector) doesn't build on QNX, and writing native C++ camera capture was slow. We found a pragmatic path using QNX's built-in viewfinder + screenshot tools to grab JPEGs from Python. ### Accomplishments we're proud of It actually works end-to-end, live: Pi camera → laptop → Claude → voice → phone, in ~1–2 seconds. The narration is genuinely funny and stays in character — and the voice changes with the personality. Swappable personalities on the fly, with the AI remembering and calling back to earlier moments. We turned a stack of "this doesn't work on QNX" problems into a clean, modular system and shipped real engineering (change-gating, color correction, a custom Pi↔laptop protocol) under hackathon time pressure. ### What we learned Split capture from compute. Letting the Pi just see and the laptop think sidestepped a mountain of embedded/cloud-SDK pain. Modern vision models can be eyes and brain collapsing "detect the scene" and "describe it" into one call removed an entire service. Fix the input, not the output. Color-correcting the camera beat any amount of "please don't say fluorescent" prompting. For hardware demos, the network is the hardest part not the AI. A lot about prompt-engineering personality and comedic timing getting an AI to be funny and know when to stay quiet. ### What's next Greatness ## README (from the GitHub repository) **Proper explanation here: https://aside-website.vercel.app/** image Winner-> image image # The Narrator > A Raspberry Pi wearable that watches a room in real time and narrates what's > happening out loud — in a personality you can swap on the fly. A hype man. A goth mommy who emotionally supports you. An epic quest narrator who turns your day into a saga. Same moment, wildly different narration. Walk into a room and your wearable announces you over entrance music; trip over a chair and the laugh track fires. A mobile app switches personalities and modes, builds custom ones, and plays the audio — ducking music under the voice so the narration always wins. --- ## The Pitch Wearables today either record you or coach you. **The Narrator does neither — it performs your life back to you in real time.** A camera + mic clipped to your chest perceives the room, and a swappable AI personality narrates the scene out loud with its own voice, music, and sound effects. The magic is *interpretation*, not transcription. The same moment — someone walking in with coffee — becomes: - **Hype man:** "AYO they're BACK and they brought the FUEL, let's GOOO!" - **Goth mommy:** "Look at you, my little raven — returning with your potion of warmth. You're doing so beautifully today." - **Epic quest narrator:** "Lo! The hero returns from the Bean Mines, sacred elixir in hand. The morning campaign may at last begin." Added User Pearsons Comp anaylsis Simple Problem Statment with solution AI integratiton slide deck with images or not a lot of words ## Three-Component Architecture ``` ┌────────────────────────────┐ frame ─┐ audio ─┐ TFLite events ─┐ │ firmware/ Raspberry Pi │ │ │ (entrance/wave/ │ │ (QNX, C++) │ │ │ fall → cue) │ │ QSF camera + mic + TFLite │ ▼ ▼ ▼ └────────────────────────────┘ ════════ LAN (Wi-Fi) ════════════════ ┌─────────────────────────────────────────┐ │ backend/ (laptop, Python) │ │ frame → Claude Haiku (vision = eyes + │ │ brain, one call → SHORT line) │ │ + Deepgram STT + active personality │ │ (Redis) → Deepgram TTS (voice) │ │ Redis = memory + state │ │ Sentry = reliability │ └────────────────┬────────────────────────┘ voice audio │ + cue signals ▼ ┌──────────────────┐ │ frontend/ │ │ React Native / │ │ Expo │ │ switch + play + │ │ duck music/SFX │ └──────────────────┘ ``` **Deployment:** the Pi (QNX, C++) captures + runs on-device triggers; a **laptop** (Python) runs the orchestrator and calls the cloud. They talk over the LAN. - **`firmware/`** (Raspberry Pi, **QNX, C++**) — captures camera frames via **QSF** + mic audio, runs **TensorFlow Lite** on-device for fast event detection (entrance, wave, fall), and ships frames / audio / event signals to the laptop backend **over the LAN**. - **`backend/`** (Python, on a **laptop** on the same LAN) — the orchestrator. Sends the camera **frame directly to Claude Haiku 4.5 (vision)** — one call returns the in-character line, so Claude is both eyes and brain. Pulls speech from Deepgram STT and the active personality from Redis, builds the prompt, and sends the line to Deepgram TTS for voice audio. Holds memory + state in Redis, monitored by Sentry. (Running the orchestrator on the laptop keeps the cloud SDKs off QNX.) - **`frontend/`** (React Native / Expo) — personality + mode switcher, custom personality builder, audio output with a manager that ducks/cuts music under narration (voice has priority), and manual cue buttons (entrance theme, laugh track). See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full pipeline. ## Service → Job | Service / piece | Job in the system | | ---------------- | --------------------------------------- | | **Claude Haiku** | Vision **+** narration — eyes + brain (frame + prompt → short in-character line, one call) | | **Deepgram** | STT + TTS (ears + voice) | | **TFLite / QSF** | On-device event triggers (entrance/wave/fall → instant cues) — runs on the Pi (QNX) | | **Redis** | Memory + state (callbacks, active personality/mode) — on the laptop | | **Sentry** | Reliability (monitors the laptop orchestrator) | | **Midjourney** | Async illustrated "journal" (off the live path) | > **Overshoot** was the original "eyes" but is no longer used — Claude vision > replaced it and folded scene understanding into the same call as the narration. > **MediaPipe** was the planned on-device trigger but doesn't build on QNX — > **TensorFlow Lite via QSF** replaced it (proven by QNX's `ai-camera-app`). ## Build Order (short version) Build the live loop first, make it fast, then layer personality and polish on top. 1. **Firmware camera → laptop → Claude vision** — Pi sends a frame over the LAN; laptop sends it to Claude Haiku → a text scene. Prove we can see. 2. **Full core loop** — frame → laptop → Claude (line) → Deepgram → audio. Measure latency. 3. **Personality system** — the swappable bundles. 4. **Music/SFX + frontend** — cue buttons, ducking audio manager. 5. **Redis memory** — narration history (callbacks) + shared state. 6. **TFLite triggers** — on-device event detection on the Pi → auto-fire cues (cuttable; fork QNX's `ai-camera-app`). 7. **Journal (Midjourney)** — async, last, cuttable. Full sequencing and what's cuttable: [docs/BUILD_ORDER.md](docs/BUILD_ORDER.md). ## Getting Started ```bash cp .env.example .env # fill in your keys ``` Then follow the README in each component: - [firmware/README.md](firmware/README.md) - [backend/README.md](backend/README.md) - [frontend/README.md](frontend/README.md) ## Running It End-to-End (live demo) The working laptop + phone + Pi demo. All three devices must be on the **same Wi-Fi** — a phone hotspot is most reliable (campus/corporate Wi-Fi often blocks device-to-device traffic and may not let the Pi join). **One-time setup** - **Laptop:** `cp .env.example .env` and fill in `ANTHROPIC_API_KEY`, `DEEPGRAM_API_KEY`, and `REDIS_URL` (`redis://localhost:6379`). Then `brew install redis && brew services start redis` and `pip install opencv-python sounddevice certifi websockets redis pillow`. - **Frontend:** `cd frontend && npm install`. - **Pi (QNX):** copy `firmware/pi_frame_sender.py` onto the Pi and `sudo apk add libjpeg-turbo-utils` (provides `cjpeg`). **Each run** 1. **Get the laptop's LAN IP** — everything points at this: ```bash ipconfig getifaddr en0 # e.g. 172.20.10.14 → use as ``` 2. **Laptop — backend + app** (one command): ```bash ./start.sh --no-webcam # Pi-only camera; drop --no-webcam to fall back to the laptop cam ``` Runs the backend (logs → `backend.log`) and Expo (prints a QR). Add `--play` to also hear narration on the laptop (otherwise the phone is the speaker). 3. **Pi — camera sender** (SSH in, then r [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 85 recognized source files, 315 KB. - C++ (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (97 of 97) ``` .env.example .gitignore assets/sounds/epic-quest-narrator/README.md assets/sounds/goth-mommy/README.md assets/sounds/hype-man/README.md assets/sounds/manifest.json assets/sounds/README.md backend/__init__.py backend/_http.py backend/audio_out.py backend/clients/__init__.py backend/clients/claude.py backend/clients/deepgram_stt.py backend/clients/deepgram_tts.py backend/clients/redis_client.py backend/clients/sentry_client.py backend/config.py backend/frame_source.py backend/journal/__init__.py backend/journal/midjourney.py backend/main.py backend/mic.py backend/personalities.py backend/phone_ws.py backend/prompt_builder.py backend/protocol.py backend/README.md backend/requirements.txt backend/test_receiver.py backend/tests/__init__.py backend/tests/test_firmware_hardware.py backend/tests/test_prompt_builder.py backend/tests/test_protocol.py backend/watcher.py docs/ARCHITECTURE.md docs/BUILD_ORDER.md docs/DEPLOY.md docs/DESIGN.md docs/PERSONALITIES.md docs/PROTOCOL.md firmware/include/firmware.hpp firmware/Makefile firmware/pi_frame_sender.py firmware/README.md firmware/src/camera_capture.cpp firmware/src/event_detector.cpp firmware/src/main.cpp firmware/src/mic_capture.cpp firmware/src/net_client.cpp frontend/app.json frontend/App.tsx frontend/babel.config.js frontend/package.json frontend/README.md frontend/src/api/backend.ts frontend/src/audio/AudioManager.ts frontend/src/audio/soundCues.ts frontend/src/components/Button.tsx frontend/src/components/Chip.tsx frontend/src/components/CueButton.tsx frontend/src/components/CueIcon.tsx frontend/src/components/GoLiveButton.tsx frontend/src/components/IconButton.tsx frontend/src/components/InfoSheet.tsx frontend/src/components/Input.tsx frontend/src/components/Logo.tsx frontend/src/components/ModeCard.tsx frontend/src/components/ModeSwitcher.tsx frontend/src/components/NowPlaying.tsx frontend/src/components/PersonalityCard.tsx frontend/src/components/SegmentedControl.tsx frontend/src/components/SettingsSheet.tsx frontend/src/components/Sheet.tsx frontend/src/components/Switch.tsx frontend/src/components/TabBar.tsx frontend/src/components/Textarea.tsx frontend/src/components/usePress.ts frontend/src/components/VoicePicker.tsx frontend/src/components/Waveform.tsx frontend/src/config.ts frontend/src/data/personalities.ts frontend/src/navigation/types.ts frontend/src/screens/BuilderScreen.tsx frontend/src/screens/HomeScreen.tsx frontend/src/screens/LiveScreen.tsx frontend/src/screens/VisionScreen.tsx frontend/src/theme/personalities.ts frontend/src/theme/ThemeContext.tsx frontend/src/theme/tokens.ts frontend/tsconfig.json personalities/_template.json personalities/epic-quest-narrator.json personalities/goth-mommy.json personalities/hype-man.json README.md start.sh TECH_STACK.md ``` ### Dependencies - backend/requirements.txt: certifi, sounddevice, websockets - frontend/package.json: @babel/core@^7.26.0, @expo-google-fonts/bricolage-grotesque@*, @expo-google-fonts/hanken-grotesk@*, @expo-google-fonts/space-mono@*, @react-navigation/native@^7.1.0, @react-navigation/native-stack@^7.3.0, @types/react@~19.1.10, babel-preset-expo@~54.0.10, expo@~54.0.0, expo-asset@~12.0.13, expo-av@~16.0.8, expo-font@~14.0.12, expo-haptics@~15.0.8, expo-linear-gradient@~15.0.8, expo-splash-screen@~31.0.13, expo-status-bar@~3.0.9, lucide-react-native@^1.21.0, react@19.1.0, react-native@0.81.5, react-native-safe-area-context@~5.6.0, react-native-screens@~4.16.0, react-native-svg@15.12.1, typescript@~5.9.2 ### Recent commits (newest first) - sxs - Update README.md - imgs - Update README.md - Add project explanation link to README - Merge remote-tracking branch 'origin/main' - Auto white-balance the camera feed (fix fluorescent fixation) - Add TECH_STACK.md — plain-English stack, journey, and system design - Update README.md - the destroyer - Enhance README with setup and usage instructions - Merge remote-tracking branch 'origin/main' - works but color issue - Enhance README with setup and usage instructions - muh butt - Merge branch 'main' of https://github.com/Da0t/AsideAI - Refine Aside UI: per-narrator settings, Vision screen, conversation tracker - Update README with new content sections - we cooked - Change-gated realtime narration + goth-mommy ASMR companion ## Key source files (fetched from GitHub, selected and truncated for size) ### TECH_STACK.md ```markdown # Tech Stack — How "Aside" Works This is the plain-English story of what we built, how it evolved, and the tools behind it. If you read nothing else: **a Raspberry Pi running QNX is the eyes, a laptop is the brain, and your phone is the voice.** --- ## What the product does You clip a camera + mic to your chest. It watches the room and **narrates your life out loud in a personality you can swap** — a hype man, a doting "goth mommy," or an epic-quest narrator. Walk into a room and it announces you; the same moment sounds completely different depending on the personality. --- ## The journey: from "Pi + QNX" to today We started with one idea — *a Raspberry Pi running QNX that sees and narrates* — and hit real-world walls. Here's how the design changed at each wall: | # | The wall we hit | What we switched to | Why | |---|-----------------|---------------------|-----| | 1 | **Overshoot.ai** was going to be the "eyes" (scene understanding) — it didn't pan out | **Claude Haiku 4.5 vision** | We send the camera photo straight to Claude. One API call returns the narration — Claude is **both the eyes and the brain**. Fewer moving parts. | | 2 | **MediaPipe** (for detecting "a person walked in") **doesn't build on QNX** | A cheap **motion-detector on the laptop** (and TensorFlow Lite on the Pi as the future native path) | We don't need fancy on-device ML to know "something changed." A simple image-difference check does it for free. | | 3 | Running the whole AI **on the Pi (or a Jetson)** was painful — QNX + Python + cloud SDKs fight each other | **Split it up:** the Pi only *captures*, a **laptop** runs the AI and calls the cloud | Best of both worlds — QNX-native camera capture, plus a laptop that talks to the cloud easily. They talk over Wi-Fi. | | 4 | Writing the camera capture in **C++ with QNX's framework** was slow to get going | A small **Python sender** that uses QNX's built-in camera tools | It grabs a photo using QNX's `viewfinder` + `screenshot`, compresses it to JPEG, and ships it. The C++ version stays as the "do it properly later" path. | | 5 | Narrating **on a timer** made it talk constantly (annoying + expensive) | **Narrate only when the scene changes** ("change-gated") | A cheap motion check runs 5×/second; the expensive Claude call only fires when something actually happens. | | 6 | The Pi camera added a **green/"fluorescent" tint**, so it kept describing the lighting | **Auto white-balance** the image before Claude sees it | The photo gets color-corrected on the laptop, so narration describes the real scene, not the camera's tint. | --- ## System design (the big picture) ``` RASPBERRY PI (QNX) LAPTOP (Python "brain") PHONE (Expo app) ───────────────── ─────────────────────── ──────────────── 📷 camera 🧠 Claude Haiku 4.5 (eyes+brain) 📱 AI Vision screen → JPEG photo ──TCP────▶ photo → 1 short line (live camera f [truncated — 3732 more characters] ``` ### docs/DEPLOY.md ```markdown # Deploy & Run How to run The Narrator — talk to it on a laptop **today**, then the near-turnkey Pi flow once the firmware is built. The laptop runs the orchestrator (Claude vision + Deepgram + playback). The Pi streams the camera (and, optionally, audio). The two talk over the LAN. --- ## A. Talk to it on the laptop (no Pi needed) Plug the **Blue Yeti Snowball into the laptop** (it's auto-selected if present). ```bash cp .env.example .env # add ANTHROPIC_API_KEY + DEEPGRAM_API_KEY python3 -m pip install sounddevice certifi # mic + TLS certs (one-time) # Push-to-talk: press Enter, speak, it talks back out loud python -m backend.main --talk python -m backend.main --talk --image room.jpg # also give it eyes python -m backend.main --talk --personality goth-mommy # pick a character # Just hear it narrate an image (no mic): python -m backend.main --mock --image room.jpg --play # Watch it narrate the laptop WEBCAM live, out loud — stand-in for the Pi camera python3 -m pip install opencv-python python -m backend.main --webcam python -m backend.main --webcam --personality hype-man --interval 4 ``` That's the full magic loop — mic → Deepgram STT → Claude vision → Deepgram TTS → out your speakers — with zero hardware beyond the mic. `--webcam` is the live-video stand-in until the Pi camera is wired (first run asks for camera permission). --- ## B. Run with the Pi (near-turnkey) The one unavoidable step is a **one-time firmware build on QNX** — it can't be cross-built/tested without the board + SDP toolchain. After that it's "copy binary, point at the laptop, run." ### 1. Laptop (the orchestrator) — start it and leave it running ```bash ipconfig getifaddr en0 # the laptop's LAN IP, e.g. 192.168.1.50 python -m backend.main --serve --play ``` It binds `FIRMWARE_LISTEN` (default `0.0.0.0:8765`) and waits. Until the Pi connects it falls back to a webcam/placeholder frame, so you can sanity-check it first. `--play` makes it speak out loud; drop it if the phone handles audio. ### 2. Pi firmware (one-time build, then deploy) Build on the QNX Pi (or cross-compile with the SDP). **Fork [`qnx/projects/ai-camera-app`](https://gitlab.com/qnx/projects/ai-camera-app)** for the QSF camera + TFLite parts, then add this repo's [`firmware/src/net_client.cpp`](../firmware/src/net_client.cpp) to ship frames. ```bash source /qnxsdp-env.sh # puts qcc/q++ on PATH cd firmware && make # see firmware/Makefile + README # point it at the laptop: export BACKEND_HOST=192.168.1.50 BACKEND_PORT=8765 scp the-narrator-firmware qnxuser@:/tmp/ ``` ### 3. Run on the Pi ```bash ssh qnxuser@ BACKEND_HOST=192.168.1.50 BACKEND_PORT=8765 /tmp/the-narrator-firmware ``` The Pi connects over the LAN → frames flow → the laptop narrates out loud. The phone (when built) connects to the laptop's WebSocket to switch personalities and play audio with music ducking. --- ## What to verify on the Pi (t [truncated — 1006 more characters] ``` ### frontend/package.json ``` { "name": "aside", "version": "0.1.0", "private": true, "description": "Aside — React Native / Expo companion app for the AI narrator wearable.", "main": "node_modules/expo/AppEntry.js", "scripts": { "start": "expo start", "android": "expo start --android", "ios": "expo start --ios", "web": "expo start --web" }, "dependencies": { "@expo-google-fonts/bricolage-grotesque": "*", "@expo-google-fonts/hanken-grotesk": "*", "@expo-google-fonts/space-mono": "*", "@react-navigation/native": "^7.1.0", "@react-navigation/native-stack": "^7.3.0", "babel-preset-expo": "~54.0.10", "expo": "~54.0.0", "expo-asset": "~12.0.13", "expo-av": "~16.0.8", "expo-font": "~14.0.12", "expo-haptics": "~15.0.8", "expo-linear-gradient": "~15.0.8", "expo-splash-screen": "~31.0.13", "expo-status-bar": "~3.0.9", "lucide-react-native": "^1.21.0", "react": "19.1.0", "react-native": "0.81.5", "react-native-safe-area-context": "~5.6.0", "react-native-screens": "~4.16.0", "react-native-svg": "15.12.1" }, "devDependencies": { "@babel/core": "^7.26.0", "@types/react": "~19.1.10", "typescript": "~5.9.2" } } ``` ### backend/requirements.txt ``` # backend/ dependencies (Python) — runs on the LAPTOP orchestrator. # # The CORE backend is STDLIB-ONLY: config, the wire protocol, the live loop, the # Pi TCP server, and the Claude/Deepgram REST calls (via urllib) all work with no # installs. `python -m backend.main --mock` runs with nothing installed and no keys. # # Install these for the live/full experience: websockets # phone channel (laptop <-> phone WebSocket). Without it the # phone channel is disabled but the loop still runs. certifi # CA bundle for the HTTPS calls (fixes macOS-Python's empty # trust store -> CERTIFICATE_VERIFY_FAILED). Used if present. sounddevice # laptop mic capture for `--talk` (bundles PortAudio on macOS; # auto-selects a Blue Yeti / Snowball if plugged in) # Optional conveniences (each degrades gracefully if absent): # opencv-python # required for `--webcam` (live laptop-camera narration, a # Pi-camera stand-in); also enables the webcam fallback in --serve # redis # shared state across backend + frontend (else in-memory) # sentry-sdk # reliability monitoring (else no-op) # python-dotenv # .env loading (else a tiny built-in parser is used) # # Cloud calls use stdlib urllib — the anthropic / deepgram SDKs are NOT required. ``` ### frontend/App.tsx ```typescript import { View, ActivityIndicator, StyleSheet } from 'react-native'; import { StatusBar } from 'expo-status-bar'; import { useFonts } from 'expo-font'; import { BricolageGrotesque_700Bold, BricolageGrotesque_800ExtraBold, } from '@expo-google-fonts/bricolage-grotesque'; import { HankenGrotesk_400Regular, HankenGrotesk_500Medium, HankenGrotesk_600SemiBold, HankenGrotesk_700Bold, HankenGrotesk_800ExtraBold, } from '@expo-google-fonts/hanken-grotesk'; import { SpaceMono_700Bold } from '@expo-google-fonts/space-mono'; import { SafeAreaProvider } from 'react-native-safe-area-context'; import { NavigationContainer, DarkTheme } from '@react-navigation/native'; import { createNativeStackNavigator } from '@react-navigation/native-stack'; import { ThemeProvider } from './src/theme/ThemeContext'; import HomeScreen from './src/screens/HomeScreen'; import BuilderScreen from './src/screens/BuilderScreen'; import LiveScreen from './src/screens/LiveScreen'; import VisionScreen from './src/screens/VisionScreen'; import type { RootStackParamList } from './src/navigation/types'; import { colors } from './src/theme/tokens'; const Stack = createNativeStackNavigator(); const navTheme = { ...DarkTheme, colors: { ...DarkTheme.colors, background: colors.bgApp, card: colors.bgRaised, text: colors.textPrimary, border: '#1a0a10', primary: colors.brand, }, }; export default function App() { const [fontsLoaded] = useFonts({ BricolageGrotesque_700Bold, BricolageGrotesque_800ExtraBold, HankenGrotesk_400Regular, HankenGrotesk_500Medium, HankenGrotesk_600SemiBold, HankenGrotesk_700Bold, HankenGrotesk_800ExtraBold, SpaceMono_700Bold, }); if (!fontsLoaded) { return ( ); } return ( ); } const styles = StyleSheet.create({ loading: { flex: 1, backgroundColor: colors.bgApp, alignItems: 'center', justifyContent: 'center', }, }); ``` ### backend/main.py ```python """Backend orchestration entry point. Runs on a LAPTOP on the same LAN as the QNX Pi. Ingests frames/audio/events from the Pi over TCP, runs the live narration loop (Claude vision -> Deepgram TTS), and serves the phone over WebSocket. The clients/ modules do the talking; this is the conductor. python -m backend.main --mock # offline smoke loop (no Pi, no keys) python -m backend.main --mock --iterations 6 python -m backend.main --serve # bind the Pi TCP socket + phone WS Mock mode is stdlib-only: no API keys, no pip installs. It proves the loop and shows all three personalities. Live mode lights up Claude/Deepgram/Redis/Sentry as their keys/deps become available. """ import argparse import logging import socketserver import threading import time from .config import config from . import protocol, prompt_builder, personalities, frame_source, audio_out, mic from .watcher import FrameWatcher from .clients import claude, deepgram_tts, deepgram_stt, redis_client, sentry_client log = logging.getLogger("narrator") # Audio the Pi ships over the LAN: s16le mono @ 16 kHz (matches Deepgram STT). The # Hub keeps a bounded rolling window so STT always sees the most recent few seconds. PI_AUDIO_RATE = 16000 PI_AUDIO_KEEP_SEC = 10 def _is_skip(line: str) -> bool: """True if the model chose to stay silent this round (returned SKIP). Conservative: only an empty reply or a 1–2 word reply led by SKIP counts, so a real line like "Skipping rope, the kid..." is NOT mistaken for a skip. """ s = (line or "").strip() if not s: return True tokens = s.upper().strip('".!()').split() return len(tokens) <= 2 and bool(tokens) and tokens[0] == "SKIP" def _estimate_speech_seconds(line: str) -> float: """Rough spoken duration of a line — used to avoid talking over it when the phone (not the laptop) is the speaker.""" return max(2.0, len(line.split()) * 0.38) # ── Phone channel (laptop -> phone). Real impl is WebSocket; Noop when absent ── class NoopPhone: def broadcast(self, msg: dict) -> None: log.debug("phone <- %s", msg.get("type")) def broadcast_voice(self, audio: bytes, meta: dict) -> None: log.debug("phone <- voice (%d bytes)", len(audio)) def broadcast_frame(self, jpeg: bytes, meta: dict) -> None: log.debug("phone <- frame (%d bytes)", len(jpeg)) # ── Shared state + the live loop ───────────────────────────────────────────── class Hub: """Holds the latest perception state and runs one narration cycle.""" def __init__(self) -> None: self.bundles = personalities.load_all() if redis_client.get_active_personality() is None: redis_client.set_active_personality(personalities.default_slug(self.bundles)) if redis_client.get_mode() is None: redis_client.set_mode("chatty") self._lock = threading.Lock() self.latest_frame = None self.audio_buf = bytearray() self.pending_event = None self.phone = NoopPhone() self.play_audio = False # play narration out loud on this laptop (afplay) self.running = True # phone can pause/resume narration (set_running) self.spoke_this_session = False # force an establishing line once per run self.watcher = None # set by the live loop; lets Pi events wake narration self.frame_source = None # "pi" | "local" — origin of the last frame served # --- inbound from the Pi (called by the TCP handler) --- def on_frame(self, jpeg: bytes) -> None: with self._lock: self.latest_frame = jpeg def on_audio(self, pcm: bytes) -> None: with self._lock: self.audio_buf.extend(pcm) cap = PI_AUDIO_RATE * 2 * PI_AUDIO_KEEP_SEC # keep only the last ~10s if len(self.audio_buf) > cap: del self.audio_buf[: len(self.audio_buf) - cap] def on_event(self, ev: dict) -> None: with self._lock: self.pending_event = ev cue = ev.get("cue") if cue: # fire the cue to the phone instantly (the cinematic-entrance beat) self.phone.broadcast({"type": "cue", "name": cue}) log.info("Pi event: kind=%s cue=%s narrate_now=%s", ev.get("kind"), cue or "-", ev.get("narrate_now")) # "narrate now" interrupts a quiet wait so the line lands on the moment. if ev.get("narrate_now") and self.watcher is not None: self.watcher.wake() # --- accessors --- def active_bundle(self) -> dict: slug = redis_client.get_active_personality() or personalities.default_slug(self.bundles) return self.bundles.get(slug) or next(iter(self.bundles.values())) def take_frame(self, fallback) -> bytes: """Latest Pi frame if one has arrived, else the local fallback (laptop webcam, or None in Pi-only mode). Records which source supplied it for logging.""" with self._lock: frame = self.latest_frame if frame is not None: self.frame_source = "pi" return frame self.frame_source = "local" return fallback() def clear_frame(self) -> None: """Drop the last Pi frame (on disconnect) so we don't narrate a stale view.""" with self._lock: self.latest_frame = None def take_event(self): with self._lock: ev, self.pending_event = self.pending_event, None return ev def take_audio(self, seconds: float) -> bytes: """Most recent `seconds` of audio the Pi has streamed (b'' if none yet).""" n = int(PI_AUDIO_RATE * seconds) * 2 with self._lock: return bytes(self.audio_buf[-n:]) if self.audio_buf else b"" # --- one narration cycle (timed) --- def narrate_once(self, frame: bytes, bundle: dict = None, decide: bool = False, reason: str = None) -> dict: """Look at the [truncated — 22196 more characters] ``` ### start.sh ```shell #!/usr/bin/env bash # # Start the whole thing with one command: the laptop backend + the Aside phone app. # # ./start.sh # webcam + phone, narrates continuously # ./start.sh --mic # also listen to the mic (ambient audio) # ./start.sh --mic --play # also speak out loud on the laptop # # Any extra args are passed through to `python -m backend.main --serve`. # Scan the Expo QR with Expo Go on a phone joined to the SAME Wi-Fi. # set -uo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$ROOT" # --- detect this laptop's LAN IP (macOS Wi-Fi en0, then en1; Linux fallback) --- IP="$(ipconfig getifaddr en0 2>/dev/null \ || ipconfig getifaddr en1 2>/dev/null \ || hostname -I 2>/dev/null | awk '{print $1}' \ || echo localhost)" [ -z "$IP" ] && IP="localhost" WS_PORT="${PHONE_WS_PORT:-8780}" export EXPO_PUBLIC_BACKEND_WS="ws://${IP}:${WS_PORT}" echo "──────────────────────────────────────────────────────────" echo " Aside — starting backend + app" echo " Laptop IP : ${IP}" echo " Phone WS : ${EXPO_PUBLIC_BACKEND_WS}" echo " Backend : python -m backend.main --serve $*" echo " Logs : backend.log (tail -f backend.log to watch narration)" echo "──────────────────────────────────────────────────────────" if [ "$IP" = "localhost" ]; then echo " ⚠ No LAN IP found — a physical phone won't reach the backend." echo " Turn on Wi-Fi, or use the iOS simulator." fi # --- start the backend (laptop orchestrator) in the background --- # Logs go to a file so they don't scribble over Expo's QR/Metro UI. python -m backend.main --serve "$@" > backend.log 2>&1 & BACKEND_PID=$! cleanup() { echo echo "stopping backend (pid ${BACKEND_PID})…" kill "$BACKEND_PID" 2>/dev/null || true } trap cleanup EXIT INT TERM # give the backend a moment; if it died immediately, surface why sleep 2 if ! kill -0 "$BACKEND_PID" 2>/dev/null; then echo "✗ backend exited on startup — last lines of backend.log:" tail -n 20 backend.log exit 1 fi echo "✓ backend running (pid ${BACKEND_PID}). Starting the app…" echo # --- start the Expo app in the foreground (Ctrl-C stops everything) --- cd frontend exec npx expo start ``` ### frontend/babel.config.js ```javascript module.exports = function (api) { api.cache(true); return { presets: ['babel-preset-expo'], }; }; ``` ### backend/__init__.py ```python """The Narrator — laptop backend (orchestrator) package. Runs on a laptop on the same LAN as the QNX Pi. Ingests frames/audio/events from the Pi, calls Claude vision + Deepgram, serves the phone. Run it with: python -m backend.main --mock # offline smoke loop (no keys, no installs) python -m backend.main --serve # bind the Pi TCP socket + phone WS, run live """ ``` ### backend/personalities.py ```python """Load personality bundles from personalities/*.json. A bundle is {name, slug, claude_system_prompt, deepgram_voice, sound_cues, ...}. See docs/PERSONALITIES.md. The `_template.json` and any `_`-prefixed file are skipped. """ import glob import json import os _DIR = os.path.join(os.path.dirname(__file__), os.pardir, "personalities") def load_all() -> dict: """Return {slug: bundle} for every personality file.""" out = {} for path in sorted(glob.glob(os.path.join(_DIR, "*.json"))): stem = os.path.splitext(os.path.basename(path))[0] if stem.startswith("_"): continue with open(path, "r", encoding="utf-8") as f: bundle = json.load(f) out[bundle.get("slug", stem)] = bundle return out def default_slug(bundles: dict) -> str: """Pick a stable default active personality.""" if "hype-man" in bundles: return "hype-man" return next(iter(bundles), "") ``` [68 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]