Project Info

Winner

Most Wacky Hack (presented by Wordware)

Jarvis

Inspiration

Cluely kinda sucks, and Iron Man is the coolest movie of 2009.

What it does

Yeah, so we hacked the Meta Ray-Bans display to let superintelligence assist you 24/7. It guarantees you the best line for your sales pitch, the smoothest rizz for your date, and the answer to every combinatorics question Citadel could ever throw at you. Oh, and we also made it cross-reference any face you come across with a giant database that probably has you in it too, just for fun. This isn’t Cluely; this is Jarvis.

Challenges we ran into

We were really excited to put facial recognition in our glasses, but we ran into tons of issues trying to find a provider. Absolutely no provider supported APIs, so we had to build our own scrapers to dynamically extract information from sites using Selenium. We burned through around $70 testing different sites for face searching. A few didn’t work, and one even banned us for bot usage, which violated their ToS. So yeah, that was $30 down the drain, unfortunately.

Accomplishments we're proud of

We were the first ones to actually make something like this work. Cluely talked about doing something similar, but we beat them to it. Unlike Cluely, we don’t overprice and underdeliver. We just built it, hacked it, and made it real.

What we learned

Nothing’s impossible, seriously. Meta has put so many restrictions in place to prevent people from hacking their glasses, but we scrapped it together at a hackathon anyway.

🕶️ Jarvis — AI Networking Assistant for Meta Glasses

“Because LinkedIn is for amateurs.”
Jarvis transforms Meta Glasses into a real-time networking assistant — recognizing faces, recalling names, and generating AI-powered conversation cues during in-person interactions.

🚀 Overview

Cluely uses live speech transcription, facial recognition, and conversational AI to enhance real-world interactions.
It helps you remember people, recall context, and sound sharp — instantly.

Core Capabilities

Facial Recognition – Identify and recall people in real time
Live Transcription – Multi-speaker diarization with low latency
AI Conversation Hints – Context-aware, adaptive dialogue prompts
Voice Commands – Say “banana” to trigger recognition
On-Glasses UI – Optimized interface for Meta Glasses streaming

🧠 Tech Stack

Frontend

React 19 + Vite – Fast, modular UI
WebRTC APIs – Camera and mic access for live recognition
CSS3 (Glassmorphism) – Lightweight visual effects

Backend

FastAPI + SQLModel + PostgreSQL – Async Python stack
Docker Compose – Unified deployment
Face Recognition API – Custom image-matching service

AI & APIs

Deepgram Nova-3 – Real-time speech-to-text
OpenRouter (Grok-4-Fast) – Conversation intelligence model

⚙️ Setup

Prerequisites

Node.js 18+ and npm/pnpm
Python 3.10+ with UV
Deepgram + OpenRouter API keys
Docker (optional)

Frontend

npm install
npm run dev

Backend

cd backend
uv sync --dev
uv run uvicorn app.main:app --reload

Access the app at:

Frontend → http://localhost:5173
Backend → http://localhost:8000

📂 Project Structure

.
├── src/           # React frontend
│   ├── App.jsx
│   ├── main.jsx
│   └── styles/
├── backend/       # FastAPI service
│   ├── app/
│   ├── Dockerfile
│   └── pyproject.toml
└── public/        # Static assets

🔑 Environment Variables

Variable	Description
`VITE_DEEPGRAM_API_KEY`	Deepgram speech-to-text
`VITE_OPENROUTER_API_KEY`	OpenRouter AI models
`DATABASE_URL`	PostgreSQL connection string
`SECRET_KEY`	Backend secret key
`ENVIRONMENT`	`local`, `staging`, or `production`

🧩 Usage

Enable Camera → Allow facial recognition access
Start Transcription → Begin real-time analysis
Say “banana” → Trigger face identification
View Suggestions → Watch AI conversation prompts appear

🧪 Development Commands

Frontend

npm run build
npm run preview
npm run lint

Backend

make run
make check
make test

🧾 License

This project was developed for Calhacks 12.0.
© 2025

Analysis

Compare with all teams

View

Metric

ts6ki
7
AndresNinou
6

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
HTMLIn code
JavaScriptIn code
PythonIn code
ReactIn code

5 of 5 appear in the indexed code.

AI coding agents

Claude CodeConfig

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

257 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

5 Verified2 Claimed only

AI-generated conversation suggestions / conversational cuesVerified
AI Conversation Hints: context-aware adaptive dialogue prompts; guarantees the best line for your sales pitch or rizz for your date
Claimed on readmehigh confidence
- src/App.jsx:179 — getConversationSuggestion posts recent transcript segments and person context to OpenRouter's chat completions API using model x-ai/grok-4-fast, with a system prompt instructing concise conversational suggestions
- src/App.jsx:333 — A periodic interval and an on-new-segment effect call debouncedAnalysis to keep suggestions flowing during an active conversation
Custom scrapers built with browser automation (described as Selenium) due to lack of face-search APIsVerified
No provider supported APIs, so we had to build our own scrapers to dynamically extract information from sites using Selenium
Claimed on Devpostmedium confidence
- backend/app/browser_automation.py:9 — Uses patchright.async_api (a Playwright-based stealth automation library), not literally Selenium, to drive a real browser against lenso.ai
- backend/app/lenso_automation.py:120 — Extensive DOM-scraping logic (consent modals, result cards, modal extraction) matches the claimed 'custom scraper' effort, though the library is Patchright rather than Selenium as literally stated
Facial recognition / cross-referencing faces against a databaseVerified
Cross-reference any face you come across with a giant database; Facial Recognition: identify and recall people in real time
Claimed on Devposthigh confidence
- src/App.jsx:112 — Captures a photo from the camera and POSTs it to /api/v1/automation/upload-image-file
- backend/app/api/routes/automation.py:119 — Endpoint receives the uploaded image and calls upload_image_to_lenso
- backend/app/lenso_automation.py:27 — Uploads the image to lenso.ai (a real reverse-image/face search service) and scrapes result image URLs from the search results page
Live speech transcription with speaker diarizationVerified
Multi-speaker diarization with low latency, live transcription via Deepgram Nova-3
Claimed on readmehigh confidence
- src/App.jsx:425 — Opens a WebSocket to wss://api.deepgram.com/v1/listen with model=nova-3&diarize=true and streams MediaRecorder audio chunks to it
- src/App.jsx:452 — onmessage handler parses Deepgram responses, extracts per-word speaker IDs, and renders transcript segments labeled by speaker
Voice command trigger ("say banana") for face recognitionVerified
Voice Commands: Say banana to trigger recognition
Claimed on readmehigh confidence
- src/App.jsx:50 — detectBananaKeyword uses regex /\bbanana\b/i
- src/App.jsx:488 — On detecting the keyword in a final transcript, calls getPersonInfo() which captures a photo and triggers recognition
On-glasses UI / Meta Ray-Ban glasses hardware integrationClaimed only
Hacked the Meta Ray-Bans display; On-Glasses UI optimized interface for Meta Glasses streaming
Claimed on Devposthigh confidence
OpenCV-based computer vision processingClaimed only
Built with: opencv
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.

Project Info

Jarvis

Inspiration

What it does

Challenges we ran into

Accomplishments we're proud of

What we learned

🕶️ Jarvis — AI Networking Assistant for Meta Glasses

🚀 Overview

Core Capabilities

🧠 Tech Stack

Frontend

Backend

AI & APIs

⚙️ Setup

Prerequisites

Frontend

Backend

📂 Project Structure

🔑 Environment Variables

🧩 Usage

🧪 Development Commands

Frontend

Backend

🧾 License

Analysis

Technology

AI coding agents

Codebase size

Structure

Interface

API & routing

Application logic

Languages

Dependencies

Feature verification