# Project export: JARVIS This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: JARVIS ⚡Wearable, multi-modal assistant that reads the room and answers in context — live. - Devpost: https://devpost.com/software/jarvis-8sg0xh - GitHub: not linked - Video: https://www.youtube.com/embed/fBwfDFL8Vbw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: contributor stats unavailable ## Devpost submission (written by the team) ### Inspiration We wanted to see if we could recreate state-of-the-art smart-glasses behavior ourselves in a weekend: real-time, multimodal, on consumer hardware. No hand-waving, just running code. We wanted to see if we could recreate state-of-the-art smart-glasses behavior ourselves in a weekend: real-time, multimodal, on consumer hardware. No hand-waving, just running code. A last-minute approval for Snap glasses shifted our plans and sparked the goal: replicate the feel of Meta Ray-Bans using off-the-shelf tools, then push latency down with Groq until it felt instant. A last-minute approval for Snap glasses shifted our plans and sparked the goal: replicate the feel of Meta Ray-Bans using off-the-shelf tools, then push latency down with Groq until it felt instant. The sponsor tech buffet — Studio Lens, Groq, MCP — made this the perfect lab to stress-test a full multimodal stack: vision ticks, tiny rolling memory, wake-word loops, and tool calls that actually do something. The sponsor tech buffet — Studio Lens, Groq, MCP — made this the perfect lab to stress-test a full multimodal stack: vision ticks, tiny rolling memory, wake-word loops, and tool calls that actually do something. Curiosity first, everything else second: by building the pipeline end-to-end, we proved to ourselves how far modern tooling can go when you keep timestamps tight and outputs structured — and only then noticed how useful it is in the real world. ### What it does Say “hey jarvis” to start a session; get glanceable on-glasses text and a spoken reply without touching your phone. Say “hey jarvis” to start a session; get glanceable on-glasses text and a spoken reply without touching your phone. Every few seconds, Jarvis fuses a fresh visual frame with live speech to surface quick emotion/context hints and a concise, conversational response. Every few seconds, Jarvis fuses a fresh visual frame with live speech to surface quick emotion/context hints and a concise, conversational response. A tiny rolling memory (last 5 moments) keeps the thread coherent, enabling continuous conversation and simple moment bookmarking for follow-ups. A tiny rolling memory (last 5 moments) keeps the thread coherent, enabling continuous conversation and simple moment bookmarking for follow-ups. Uses surroundings awareness to tailor suggestions and trigger lightweight actions (e.g., mark a moment, prep a follow-up). Privacy defaults to ephemeral processing with opt-in saves. Uses surroundings awareness to tailor suggestions and trigger lightweight actions (e.g., mark a moment, prep a follow-up). Privacy defaults to ephemeral processing with opt-in saves. ### How we built it 🛠️ How we built it (end-to-end) Wake phrase “hey jarvis” opens a session and primes capture. Wake phrase “hey jarvis” opens a session and primes capture. Every 3 seconds the glasses capture a frame, encode it as base64 JPEG with a timestamp, and ship it to our Python FastAPI endpoint. Every 3 seconds the glasses capture a frame, encode it as base64 JPEG with a timestamp, and ship it to our Python FastAPI endpoint. A VLM analyzes each frame and returns compact JSON per tick. A VLM analyzes each frame and returns compact JSON per tick. We keep a small memory using a stack/queue of size 5; on each new frame the oldest is popped. Frame JSON is merged with Groq-based analysis into a single, stable JSON state. We keep a small memory using a stack/queue of size 5; on each new frame the oldest is popped. Frame JSON is merged with Groq-based analysis into a single, stable JSON state. That state is sent to our final API, which generates the conversational, context-aware response and streams it back; the loop continues until the termination prompt. That state is sent to our final API, which generates the conversational, context-aware response and streams it back; the loop continues until the termination prompt. ### Challenges we ran into Getting comfortable with Studio Lens data flows and tools: wiring vision events, structured outputs, and guardrails so downstream code could trust the schema. Getting comfortable with Studio Lens data flows and tools: wiring vision events, structured outputs, and guardrails so downstream code could trust the schema. On-glasses UX: crisp, readable text sizing; aligning activation and termination prompts so they feel instant instead of laggy. On-glasses UX: crisp, readable text sizing; aligning activation and termination prompts so they feel instant instead of laggy. Surroundings detection: initial runs frequently reported no recognizable environment; we tuned capture cadence and fallback logic to improve reliability. Surroundings detection: initial runs frequently reported no recognizable environment; we tuned capture cadence and fallback logic to improve reliability. Continuous conversation: early builds ended after one full loop; we reworked the session state so it keeps listening and responding seamlessly after each exchange. Continuous conversation: early builds ended after one full loop; we reworked the session state so it keeps listening and responding seamlessly after each exchange. ### Accomplishments we're proud of In 36 hours, we assembled a working wearable assistant that feels close to the latest smart-glasses experiences using available sponsor tech. Approved last-minute for Snap glasses, pivoted fast, and still shipped an end-to-end demo. Built something technically ambitious that’s also genuinely fun to use. ### What we learned Decent proficiency with Lens Studio and reading docs. Decent proficiency with Lens Studio and reading docs. How to stitch VLM vision, speech-to-text/text-to-speech, and fast reasoning into one steady flow. How to stitch VLM vision, speech-to-text/text-to-speech, and fast reasoning into one steady flow. The unglamorous truths: latency budgets, tiny memories, and session management make or break wearables. The unglamorous truths: latency budgets, tiny memories, and session management make or break wearables. ### What's next Scale up with MCP actions: navigation by voice, add calendar tasks, even place a call, all from “hey jarvis.” Scale up with MCP actions: navigation by voice, add calendar tasks, even place a call, all from “hey jarvis.” On-device models, better speaker diarization, and personal calibration for more stable cues. On-device models, better speaker diarization, and personal calibration for more stable cues. Longer sessions: thermal/battery tuning and a richer but still glanceable HUD. Longer sessions: thermal/battery tuning and a richer but still glanceable HUD. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) No repository was indexed for this project. Claimed technologies below could not be checked against code. - Python (language) — claimed on Devpost, not found in the code - TypeScript (language) — claimed on Devpost, not found in the code ## Codebase structure No repository index available. ## Key source files No repository index available; no source files included.