Project Info
Inspiration
We wanted to see if we could recreate state-of-the-art smart-glasses behavior ourselves in a weekend: real-time, multimodal, on consumer hardware. No hand-waving, just running code. We wanted to see if we could recreate state-of-the-art smart-glasses behavior ourselves in a weekend: real-time, multimodal, on consumer hardware. No hand-waving, just running code. A last-minute approval for Snap glasses shifted our plans and sparked the goal: replicate the feel of Meta Ray-Bans using off-the-shelf tools, then push latency down with Groq until it felt instant. A last-minute approval for Snap glasses shifted our plans and sparked the goal: replicate the feel of Meta Ray-Bans using off-the-shelf tools, then push latency down with Groq until it felt instant. The sponsor tech buffet — Studio Lens, Groq, MCP — made this the perfect lab to stress-test a full multimodal stack: vision ticks, tiny rolling memory, wake-word loops, and tool calls that actually do something. The sponsor tech buffet — Studio Lens, Groq, MCP — made this the perfect lab to stress-test a full multimodal stack: vision ticks, tiny rolling memory, wake-word loops, and tool calls that actually do something. Curiosity first, everything else second: by building the pipeline end-to-end, we proved to ourselves how far modern tooling can go when you keep timestamps tight and outputs structured — and only then noticed how useful it is in the real world.
What it does
Say “hey jarvis” to start a session; get glanceable on-glasses text and a spoken reply without touching your phone. Say “hey jarvis” to start a session; get glanceable on-glasses text and a spoken reply without touching your phone. Every few seconds, Jarvis fuses a fresh visual frame with live speech to surface quick emotion/context hints and a concise, conversational response. Every few seconds, Jarvis fuses a fresh visual frame with live speech to surface quick emotion/context hints and a concise, conversational response. A tiny rolling memory (last 5 moments) keeps the thread coherent, enabling continuous conversation and simple moment bookmarking for follow-ups. A tiny rolling memory (last 5 moments) keeps the thread coherent, enabling continuous conversation and simple moment bookmarking for follow-ups. Uses surroundings awareness to tailor suggestions and trigger lightweight actions (e.g., mark a moment, prep a follow-up). Privacy defaults to ephemeral processing with opt-in saves. Uses surroundings awareness to tailor suggestions and trigger lightweight actions (e.g., mark a moment, prep a follow-up). Privacy defaults to ephemeral processing with opt-in saves.
How we built it
🛠️ How we built it (end-to-end) Wake phrase “hey jarvis” opens a session and primes capture. Wake phrase “hey jarvis” opens a session and primes capture. Every 3 seconds the glasses capture a frame, encode it as base64 JPEG with a timestamp, and ship it to our Python FastAPI endpoint. Every 3 seconds the glasses capture a frame, encode it as base64 JPEG with a timestamp, and ship it to our Python FastAPI endpoint. A VLM analyzes each frame and returns compact JSON per tick. A VLM analyzes each frame and returns compact JSON per tick. We keep a small memory using a stack/queue of size 5; on each new frame the oldest is popped. Frame JSON is merged with Groq-based analysis into a single, stable JSON state. We keep a small memory using a stack/queue of size 5; on each new frame the oldest is popped. Frame JSON is merged with Groq-based analysis into a single, stable JSON state. That state is sent to our final API, which generates the conversational, context-aware response and streams it back; the loop continues until the termination prompt. That state is sent to our final API, which generates the conversational, context-aware response and streams it back; the loop continues until the termination prompt.
Challenges we ran into
Getting comfortable with Studio Lens data flows and tools: wiring vision events, structured outputs, and guardrails so downstream code could trust the schema. Getting comfortable with Studio Lens data flows and tools: wiring vision events, structured outputs, and guardrails so downstream code could trust the schema. On-glasses UX: crisp, readable text sizing; aligning activation and termination prompts so they feel instant instead of laggy. On-glasses UX: crisp, readable text sizing; aligning activation and termination prompts so they feel instant instead of laggy. Surroundings detection: initial runs frequently reported no recognizable environment; we tuned capture cadence and fallback logic to improve reliability. Surroundings detection: initial runs frequently reported no recognizable environment; we tuned capture cadence and fallback logic to improve reliability. Continuous conversation: early builds ended after one full loop; we reworked the session state so it keeps listening and responding seamlessly after each exchange. Continuous conversation: early builds ended after one full loop; we reworked the session state so it keeps listening and responding seamlessly after each exchange.
Accomplishments we're proud of
In 36 hours, we assembled a working wearable assistant that feels close to the latest smart-glasses experiences using available sponsor tech. Approved last-minute for Snap glasses, pivoted fast, and still shipped an end-to-end demo. Built something technically ambitious that’s also genuinely fun to use.
What we learned
Decent proficiency with Lens Studio and reading docs. Decent proficiency with Lens Studio and reading docs. How to stitch VLM vision, speech-to-text/text-to-speech, and fast reasoning into one steady flow. How to stitch VLM vision, speech-to-text/text-to-speech, and fast reasoning into one steady flow. The unglamorous truths: latency budgets, tiny memories, and session management make or break wearables. The unglamorous truths: latency budgets, tiny memories, and session management make or break wearables.
What's next
Scale up with MCP actions: navigation by voice, add calendar tasks, even place a call, all from “hey jarvis.” Scale up with MCP actions: navigation by voice, add calendar tasks, even place a call, all from “hey jarvis.” On-device models, better speaker diarization, and personal calibration for more stable cues. On-device models, better speaker diarization, and personal calibration for more stable cues. Longer sessions: thermal/battery tuning and a richer but still glanceable HUD. Longer sessions: thermal/battery tuning and a richer but still glanceable HUD.
This project did not link a GitHub repository.
Analysis
No indexed repository for this project, so there are no commit stats to show.
Technology
- PythonUnchecked
- TypeScriptUnchecked
No repository was indexed for this project, so these Devpost claims have not been checked against code.
AI coding agents
No repository was indexed, so agent usage could not be checked.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
No repository was indexed, so there is no codebase to measure.
This project did not link a GitHub repository, so there is nothing to diagram.
This project did not link a GitHub repository, so its feature claims have not been checked against code.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.