# Project export: Glass This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: See the world, hear the room, talk as you walk. A hands free AI accessibility companion for Meta Ray-Ban glasses. - Devpost: https://devpost.com/software/glass-hkrwfy - GitHub: https://github.com/MrFibonacc1/calHacksAI - Video: https://www.youtube.com/embed/UXpjK-GNQB8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Claude Opus 4.8 (1M context) (1 commits), Huzaifa Rehan (1 commits) ## Devpost submission (written by the team) ### Overview IF THE VIDEO IS CORRUPTED, PLEASE WATCH THIS VID HERE. THANK YOU :) BACKUP: https://youtu.be/z4mf9RSSE5w note: main demo vid got corrupted this is backup vid ### Inspiration Recently i saw that meta ray ban with display released a SDK a couple weeks ago that allowed developers to code on them. Even though its currently in beta access i wanted to give it a shot. Its been a while since ive a development in the health industries with ai mainly in the glasses realm and with blindess/deaf. I have family members who have hearing/seeing disabilities so i thought it qwould be a good use case for them to use ### What it does Glass turns Meta Ray Ban Display glasses into a hands free assistant for people who are blind, low vision, Deaf, or hard of hearing. For blind and low vision users it describes what is in front of you, reads signs and menus and mail out loud, names objects, and gives spoken walking directions, all through the glasses speaker and the screen in the lens. For Deaf and hard of hearing users it shows live captions of the people around you, labeled by who is speaking, with a small tone tag so you get whether something was a question, excited, or urgent instead of just the bare words. It reads fingerspelling into captions, coaches you through how to sign common words back, and can even learn a small set of signs you teach it and read them aloud. And when you just want company, you say "explore mode" and talk to it like a friend as you walk, asking about whatever you see, until you say "normal mode" to stop. The whole thing is voice first, so you barely touch the phone. ### How we built it The glasses are the eyes, ears, mouth, and screen. The iPhone is the brain and does the heavy lifting. Camera frames and your voice come into the phone, and the answers go back out to the glasses speaker and the lens. We leaned on Anthropic's vision and language models to understand scenes, read text, hold a real conversation, and run a little voice agent that can operate the whole app and even flip settings when you ask. Deepgram gives us a natural sounding voice and fast, speaker separated live captions, with Apple's on device speech as a backup so it never goes quiet. We kept the always on safety and awareness loop on the phone itself using Apple Vision and a bundled segmentation model, so it works offline and keeps the camera private. The sign features run on device too, on hand pose landmarks, so no video of someone signing ever leaves the phone. And we wired Sentry through every fragile path, because a blind user cannot see a frozen screen and a Deaf user cannot hear a failed caption. ### Challenges we ran into Sign language humbled us fast. We started out assuming we could send video to a model and get a clean translation back, and the research said plainly that this does not work yet, even for the best systems out there. So we scoped down to something honest: reading fingerspelling, coaching common signs, and a teach it yourself recognizer, and we put that honesty right in the app instead of pretending otherwise. Audio was the other beast. The microphone and the speaker fight over the same hardware, so captions, the voice assistant, and the spoken replies kept stepping on each other until we built one coordinator to hand the audio back and forth cleanly. We also learned the lens can only show text and images served from a web link, not files sitting on the phone, which sent us digging into exactly what it could and could not render. And the glasses microphone is locked to Meta, so the phone has to do all the listening. ### Accomplishments we're proud of It actually works from end to end, on real hardware. You speak, the glasses see, and the answer comes back in your ear and on the lens. We are proud that it bends instead of breaking, that nothing private leaves the phone unless you ask it to, and that we stayed honest about sign language rather than overselling it. The little tone tag on captions is a small thing we love, because it hands back something captions usually strip away. And explore mode, just wandering around chatting about the world, felt like the future for a second. ### What we learned We learned that the hard part of accessibility is rarely the model, it is the plumbing: the audio routing, the latency, the privacy, the honesty. We learned how often the disability community has been burned by tech that overclaims, and that the kindest thing you can do is be clear about what your tool can and cannot do. We learned to build on the device first and reach for the cloud only when it truly earns its place. And we learned a lot about how blind and Deaf people actually move through the world, which quietly reshaped almost every decision we made. ### What's next We want to move the AI off an embedded key and behind a proper backend so it is safe to ship. We want real distance sensing so navigation can warn about steps and drop offs, not just describe them. We want to grow the sign vocabulary alongside Deaf signers, because they should be leading this, not us. We want sound alerts so a Deaf user knows when an alarm goes off or their name is called. And most of all we want to get it into the hands of the people we built it for and let their feedback steer the rest. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 66 recognized source files, 481 KB. - FastAPI (technology) — detected in the code - Python (language) — detected in the code - Swift (language) — detected in the code - Redis (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (82 of 82) ``` .gitignore ACCESSIBILITY_ML_RESEARCH.md ASL_IMAGE_CREDITS.md BRAILLE_SETUP.md CLAUDE.md deepgram-voice-agent/agent-prompt.txt deepgram-voice-agent/agent-settings.json deepgram-voice-agent/functions.json DISPLAY_SETUP.md glasse.xcodeproj/project.pbxproj glasse.xcodeproj/project.xcworkspace/contents.xcworkspacedata glasse.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved glasse.xcodeproj/xcuserdata/huzaifa.rehan.xcuserdatad/xcschemes/xcschememanagement.plist glasse/AccessibilityAgent.swift glasse/AgentBuilderClient.swift glasse/AgentBuilderView.swift glasse/AgentsListView.swift glasse/AgentStore.swift glasse/AnthropicVisionClient.swift glasse/Assets.xcassets/AccentColor.colorset/Contents.json glasse/Assets.xcassets/AppIcon.appiconset/Contents.json glasse/Assets.xcassets/Contents.json glasse/AudioCoordinator.swift glasse/BrailleReader.swift glasse/CapabilityNode.swift glasse/Conductor.swift glasse/ContentView.swift glasse/Deepgram.swift glasse/DeepgramAgentFunctions.swift glasse/DeviceSessionManager.swift glasse/FingerspellGuide.swift glasse/glasseApp.swift glasse/GlassesDisplay.swift glasse/GlassesMock.swift glasse/MemoryClient.swift glasse/MemoryStore.swift glasse/MonitorView.swift glasse/NavigateView.swift glasse/NavigationManager.swift glasse/ObjectDetector.swift glasse/OnboardingView.swift glasse/OutputSink.swift glasse/PinchToTalk.swift glasse/POVView.swift glasse/RemoteMemoryStore.swift glasse/SignAssembler.swift glasse/SignAssets.swift glasse/SignClassifier.swift glasse/SignMotionMatcher.swift glasse/SignReader.swift glasse/SignTemplateStore.swift glasse/SignView.swift glasse/SignVocabReader.swift glasse/SignVocabView.swift glasse/SignWriter.swift glasse/SignWriterView.swift glasse/SpeechCaptioner.swift glasse/SpeechVocabulary.swift glasse/StreamSessionViewModel.swift glasse/Telemetry.swift glasse/TestView.swift glasse/Theme.swift glasse/Tone+Tint.swift glasse/ToneClassifier.swift glasse/VoiceCommandClient.swift glasse/VoiceCommander.swift glasse/WakeWord.swift glasse/WakeWordListener.swift glasse/WearablesViewModel.swift Info.plist.template memory-service/.env.example memory-service/.gitignore memory-service/app.py memory-service/docker-compose.yml memory-service/Dockerfile memory-service/memory.py memory-service/README.md memory-service/requirements.txt PROJECT_STATUS.md REDIS_SETUP.md SENTRY.md SIGN_MODE.md ``` ### Dependencies - memory-service/requirements.txt: fastapi@==0.115.6, httpx@==0.28.1, pydantic@==2.10.4, uvicorn[standard]@==0.34.0 ### Recent commits (newest first) - glasse — Claude-powered accessibility companion for Meta Ray-Ban Display glasses ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # glasse — Agent Guidelines glasse is a SwiftUI iOS accessibility app for Meta Ray-Ban Display smart glasses, for blind, low-vision, deaf, and hard-of-hearing users. Claude is the "brain" (a tool-use conductor named "Bob"); the glasses are the camera / mic / speaker / lens; the iPhone runs everything. ## Read before starting work These docs hold context that is **not** obvious from the code. Read the relevant ones before making changes: - **[PROJECT_STATUS.md](PROJECT_STATUS.md)** — what's built, what's queued, and the hackathon prize tracks (Anthropic / Deepgram / Sentry). **Start here.** - **[ACCESSIBILITY_ML_RESEARCH.md](ACCESSIBILITY_ML_RESEARCH.md)** — on-device ML options per disability + roadmap. - **[SIGN_MODE.md](SIGN_MODE.md)** — fingerspelling → captions design + plan. - **[DISPLAY_SETUP.md](DISPLAY_SETUP.md)** — how the in-lens display was enabled. - **[SENTRY.md](SENTRY.md)** — telemetry / observability wiring + activation steps. ## Conventions - **Secrets:** API keys live in `glasse/Secrets.swift` (gitignored). Never commit keys or paste them where they could be committed. - **Large model:** `glasse/ISANet.mlmodel` is gitignored (download separately). - **Dependency:** the app uses the `meta-wearables-dat-ios` Swift Package (`MWDATCore` / `MWDATCamera` / `MWDATDisplay` / `MWDATMockDevice`). - **Concurrency:** Swift 5 language mode with `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor` — keep CPU-heavy work (image encode/resize, ML inference) off the main actor. - Core ML inference runs on a real device only (not the iOS Simulator). ``` ### SENTRY.md ```markdown # Sentry — observability & error monitoring glasse is wired for Sentry through one wrapper (`Telemetry.swift`). The wrapper is `#if canImport(Sentry)`-gated, so the app builds and runs **with or without** the Sentry package — it's a no-op until you add the package *and* set a DSN. ## Activate in 3 steps ### 1. Add the Sentry package (Xcode) - **File → Add Package Dependencies…** - URL: `https://github.com/getsentry/sentry-cocoa` - Dependency rule: **Up to Next Major Version** (8.x) - Add Package → check the **Sentry** product → add it to the **glasse** target. ### 2. Paste your DSN - In Sentry: create/select a project (platform **Apple → iOS**) → **Settings → Client Keys (DSN)** → copy the DSN (`https://…@o…ingest.sentry.io/…`). - Put it in `Secrets.swift`: ```swift static let sentryDSN = "https://…@o…ingest.sentry.io/…" ``` (Secrets.swift is gitignored — the DSN is not a secret credential, but keep it out of git anyway.) ### 3. Verify - Run the app → tap the **🐞** (Test screen) → **Observability (Sentry)** card → **Send test event** / **Send test error**. - Check **Sentry → Issues** (event arrives in ~1 min) and **Performance** for the `conductor.run` transaction after a voice command. ## What's instrumented (the reliability story) | Where | Signal | |---|---| | Launch (`glasseApp`) | `Telemetry.start()`; captures `Wearables.configure()` failures. | | **Claude conductor** (`startConductor`) | `conductor.run` **performance transaction**; tag `agent.kind`; a **breadcrumb per tool call** (`dispatchTool`); captures conductor errors. | | **Vision** (`AnthropicVisionClient`) | `vision.converse` span; captures **timeouts** and **non-200s** with status + model. | | **Deepgram TTS** (`Speaker`) | tag `tts.engine` (deepgram/apple) + breadcrumb on **silent fallback** to Apple. | | **Deepgram STT** (`SpeechCaptioner`) | tag `stt.engine`; **event** when Deepgram fails to start and we fall back. | | **create_agent** | event when Claude's agent draft fails to decode. | **Privacy:** screenshots + view hierarchy are disabled (no camera frames leave the device), and we send operation names / timings / error metadata — never raw transcripts or images. ## Framing for the submission - **Reliability from day one:** glasse is an accessibility tool — a blind user can't see a frozen screen, a Deaf user can't hear a failed caption — so we monitor errors and time every Claude / Deepgram round-trip. - **Observability of graceful degradation:** the app *degrades* silently (Deepgram→Apple, conductor→simple parser). Sentry makes those invisible fallbacks **visible**, so you know when the premium path is failing in the field. - **Uses the Sentry API** via the official `sentry-cocoa` SDK on a free account. ``` ### memory-service/requirements.txt ``` # glasse memory-service — thin proxy to managed Redis Agent Memory (Iris). # Iris does the embeddings + vector search + extraction server-side, so the proxy # only needs an HTTP client. fastapi==0.115.6 uvicorn[standard]==0.34.0 httpx==0.28.1 pydantic==2.10.4 ``` ### memory-service/Dockerfile ``` FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY memory.py app.py ./ EXPOSE 8080 # IRIS_HOST / IRIS_STORE_ID / IRIS_API_KEY injected at deploy time (never baked in). CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"] ``` ### memory-service/docker-compose.yml ```yaml # Local run of the proxy against managed Iris (no local Redis needed — Iris is the # store, in Redis Cloud). Put IRIS_* in a local .env. For production, deploy this # container on an approved cloud (AWS/GCP/Azure) with creds from its secret manager. services: memory: build: . ports: ["8080:8080"] environment: IRIS_HOST: ${IRIS_HOST} IRIS_STORE_ID: ${IRIS_STORE_ID} IRIS_API_KEY: ${IRIS_API_KEY} MEMORY_NAMESPACE: ${MEMORY_NAMESPACE:-glasse-dev} ``` ### memory-service/app.py ```python """ glasse memory-service — HTTP API the glasse app calls (through your backend) to give Bob per-user, cross-session memory. Endpoints GET /healthz liveness + index info POST /memory/search retrieve relevant memories for a user (call BEFORE Claude) POST /memory/learn extract durable prefs from a turn + store (call AFTER Claude) POST /memory/remember store one explicit memory ("remember that I…") The mobile app must NOT hold Redis/Anthropic creds — it talks to THIS service, which holds them via env vars. Front it with your auth so `owner_id` is a trusted user id, and run it on an approved cloud (AWS/GCP/Azure). See README.md. """ from __future__ import annotations from contextlib import asynccontextmanager from fastapi import FastAPI from pydantic import BaseModel, Field import memory @asynccontextmanager async def lifespan(_: FastAPI): memory.ensure_index() # idempotent: create the vector index on boot yield app = FastAPI(title="glasse memory-service", version="1.0.0", lifespan=lifespan) class SearchIn(BaseModel): owner_id: str = Field(..., description="stable per-user id from your auth") query: str namespace: str = memory.NAMESPACE_DEFAULT limit: int = 5 class LearnIn(BaseModel): owner_id: str user_text: str assistant_text: str = "" namespace: str = memory.NAMESPACE_DEFAULT class RememberIn(BaseModel): owner_id: str text: str type: str = "preference" namespace: str = memory.NAMESPACE_DEFAULT @app.get("/healthz") def healthz(): return {"ok": True, **memory.ensure_index()} @app.post("/memory/search") def search(body: SearchIn): mems = memory.search_memories(body.owner_id, body.query, namespace=body.namespace, limit=body.limit) return {"memories": mems} @app.post("/memory/learn") def learn(body: LearnIn): return memory.learn_from_turn(body.owner_id, body.user_text, body.assistant_text, namespace=body.namespace) @app.post("/memory/remember") def remember(body: RememberIn): return memory.remember(body.owner_id, body.text, mtype=body.type, namespace=body.namespace) ``` ### glasse/Tone+Tint.swift ```swift // // Tone+Tint.swift // glasse // // SwiftUI display tint for `Tone`, kept out of `ToneClassifier.swift` so the // classification logic there stays Foundation-only and unit-testable with `swiftc`. // import SwiftUI extension Tone { /// Pill color. Chosen for contrast on the captions card and for the lens. var tint: Color { switch self { case .neutral: return .secondary case .positive: return .green case .negative: return .indigo case .question: return .teal case .excited: return .orange case .urgent: return .red } } } ``` ### glasse/glasseApp.swift ```swift // // glasseApp.swift // glasse // import SwiftUI import MWDATCore @main struct glasseApp: App { init() { Telemetry.start() // reliability/observability from launch (no-op without a DSN) // Configure the Meta Wearables Device Access Toolkit once at launch. do { try Wearables.configure() } catch { print("[glasse] Wearables.configure() failed: \(error)") Telemetry.capture(error, ["phase": "wearables.configure"]) } } var body: some Scene { WindowGroup { RootView() } } } /// Shows the onboarding intro on first launch, then the main screen. struct RootView: View { @AppStorage("hasOnboarded") private var hasOnboarded = false var body: some View { if hasOnboarded { ContentView() } else { OnboardingView { hasOnboarded = true } } } } ``` ### glasse/OutputSink.swift ```swift // // OutputSink.swift // glasse // // Modality-specific delivery of a result. The text is always shown on the // phone; the sink adds the agent's chosen output channel on top: speech, // on-screen (with a VoiceOver announcement), or the in-lens display. // import Foundation import UIKit /// Posts a VoiceOver announcement so blind users hear screen-only results. @MainActor func announce(_ text: String) { guard !text.isEmpty else { return } UIAccessibility.post(notification: .announcement, argument: text) } @MainActor protocol OutputSink { func deliver(_ text: String) } /// Speaks the text aloud (routes to the glasses' open-ear speaker over Bluetooth). @MainActor struct SpeechSink: OutputSink { let speaker: Speaker func deliver(_ text: String) { speaker.speak(text) } } /// Text is shown large on the phone screen and announced to VoiceOver. @MainActor struct ScreenSink: OutputSink { func deliver(_ text: String) { announce(text) } } ``` ### glasse/SpeechVocabulary.swift ```swift // // SpeechVocabulary.swift // glasse // // Words and phrases the app expects to hear, used to BIAS speech-to-text toward // them so it recognizes the terms that actually matter more reliably: // • Deepgram Nova-3 "keyterm" prompting (streaming STT), and // • Apple's `SFSpeechRecognitionRequest.contextualStrings` (on-device fallback). // // Focused on the wake word and the app's distinctive command vocabulary — // especially rarer terms ("fingerspelling", "braille") that generic recognizers // often mis-hear. Kept small on purpose: over-boosting hurts general accuracy. // // Pure (Foundation only) so it stays unit-testable like the other speech helpers. // import Foundation enum SpeechVocabulary { /// Terms to boost. The wake word first, then the command vocabulary. static let terms: [String] = [ "glasses", // wake word "captions", "live captions", "fingerspelling", "sign language", "braille", "describe", "what's in front of me", "read text", "identify objects", "navigate", "directions", "text to speech", "translate", ] } ``` [56 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]