# Project export: Mira Mira on Da Wall This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: MIRA is an AI powered smart mirror that helps you decide what to wear and what to buy. "Mira Mira On The Wall..." - Devpost: https://devpost.com/software/mira-3xqlos - GitHub: https://github.com/23jmo/mirrorless - Video: https://www.youtube.com/embed/ki_rh4TUoOQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([Visa] The Generative Edge: Future of Commerce ($10,000 total)) - Team: 4 GitHub contributor(s) — Johnathan Mo (80 commits), Claude Opus 4.6 (78 commits), Vivian Zhou (12 commits), LouisY06 (9 commits) ## Devpost submission (written by the team) ### Inspiration “Mirror, mirror on the wall… who’s the best dressed of them all?” We realized that we already ask our mirrors questions every day. Does this look good? Is this too formal? Is this giving the right vibe? The mirror never answers. That gap inspired MIRA. We believed that if smart TVs and smart homes are becoming standard, smart mirrors are next. Instead of building another shopping app, we wanted to transform something you already use daily into something interactive. MIRA was born from the idea that your mirror could be your stylist, your honest friend, and your fashion guide all at once. ### What it does MIRA is an AI powered smart mirror that helps you decide what to wear and what to buy. You stand in front of it and see clothes visually layered onto your reflection. You can get real time styling feedback, personalized recommendations based on your shopping history, emails, and calendar, and direct links to purchase items. At the center is Mira, an expressive assistant with personality who talks to you, gives feedback, and makes the experience feel natural and conversational instead of transactional. ### How we built it We combined hardware, computer vision, and multiple AI systems into one integrated experience. On the hardware side, we assembled a two way mirror with a vertically mounted monitor behind it and an embedded webcam for real time capture. On the software side: Frontend built with React and a mirror optimized interface Gesture detection using MediaPipe Real time 2D clothing visualization over live video OpenAI powered agent for reasoning and styling ElevenLabs for voice synthesis Google OAuth for secure email scraping MCP server with Poke integration for shopping data Search integrations using tools like Perplexity Sonar and Serp APIs Mobile onboarding app deployed on Vercel We engineered a custom real time animated assistant by dynamically blending emotional states to create a more expressive presence. ### Challenges we ran into One major challenge was clothing visualization. We initially explored full 3D garment meshing and rendering, but it proved too complex for our timeframe. We pivoted to a 2D overlay system that delivered strong user value while remaining feasible. Another challenge was scraping reliability. Extracting useful signals from emails and calendars—without hallucinating or misinterpreting context—required tightening our pipelines and improving streaming for accuracy and consistency. On the fashion side, the sheer volume of public images made precision critical. We had to give our OpenAI agent very clear instructions on exactly which items to identify, and build structured labels for each category to power accurate, personalized recommendations. Creating emotional presence was also difficult. We needed the assistant to feel expressive and empathetic, not robotic. ### Accomplishments we're proud of Successfully assembling the physical smart mirror hardware Getting real time clothing visualization working Integrating voice input and output seamlessly Building an expressive animated assistant that enables lip sync Creating an end to end scraping and recommendation pipeline and feed it to our magic pipe for users to interact with. We are especially proud that the experience feels natural. It does not feel like using an app. It feels like interacting with your reflection. ### What we learned We learned the importance of prioritization. Not every technically impressive feature belongs in an MVP. We learned that empathy in AI comes from more than intelligence. Voice, timing, and visual expression matter just as much as reasoning. We also learned how complex it is to integrate multiple APIs and systems into one seamless experience. Real time interaction requires careful coordination across hardware, frontend, backend, and AI layers. ### What's next for Mira Explore full 3D garment rendering and body meshing Improve scraping accuracy and personalization depth (potentially gathering information from diverse sources, such as social media) Develop a more advanced multi agent reasoning system Expand the onboarding phone app for users to manage all information input and output of the smart mirror Refine the mirror specific user interface and gestures and have a personalized interface /avatar for each user who remembers their exact personality type and preferences. Deploy and test with real users in household settings Our long term vision is simple. Smart mirrors will become common in homes. When that happens, MIRA will already be there, ready to style you. ## README (from the GitHub repository) # Mirrorless An AI-powered smart mirror that gives personalized outfit recommendations overlaid on your body in real-time. Users onboard via phone (Google OAuth), their purchase history is scraped from Gmail, and AI stylist "Mira" delivers styling advice through a two-way mirror display. ## How It Works 1. **Scan** the QR code on the mirror with your phone 2. **Sign in** with Google, take a selfie, and fill out a quick style questionnaire 3. **Step up** to the mirror when it's your turn 4. **Talk to Mira** — she roasts your current outfit, searches for new pieces, and overlays clothing on your body in real-time 5. **React with gestures** — thumbs up/down to like or skip, swipe to browse outfits 6. **Get your picks** saved to your phone when the session ends ## Architecture ``` Phone (Next.js) ──┐ ├── Socket.io ──▶ Backend (FastAPI + Python) Mirror (Next.js) ──┘ │ ├── Claude API (Mira agent) ├── Deepgram (STT) ├── ElevenLabs (TTS) ├── Serper.dev (Google Shopping) ├── Gemini (flat lay generation) └── Neon Postgres (database) ``` - **Frontend**: Next.js app serving the mirror display (full-screen kiosk), phone UI (onboarding + dashboard), and admin dashboard. Deployed on Vercel. - **Backend**: Python FastAPI with Socket.io for real-time communication. Hosts the Mira agent orchestrator, Gmail scraping, and an MCP server for external AI integrations. Deployed on Render. - **Database**: Neon Postgres with dual-mode connections (asyncpg pool in production, Neon HTTP locally). ## Tech Stack | Layer | Technology | |-------|-----------| | Frontend | Next.js, TypeScript, Tailwind CSS, shadcn/ui | | Backend | FastAPI, Python 3.11+, Socket.io | | AI Agent | Claude (Anthropic API), custom event-driven orchestrator | | Voice | Deepgram streaming STT, ElevenLabs streaming TTS | | Avatar | Pre-recorded MP4 emotion loops (26 emotions) | | Body Tracking | MediaPipe BlazePose + MediaPipe Hands | | Clothing Data | Serper.dev Google Shopping API | | Image Processing | Gemini (flat lays), rembg (background removal) | | Database | Neon Postgres | | Deployment | Vercel (frontend), Render (backend) | ## Setup ### Prerequisites - Node.js 18+ - Python 3.11+ - A Neon Postgres database ### Frontend ```bash cd frontend npm install npm run dev ``` ### Backend ```bash cd backend pip install -r requirements.txt uvicorn main:app --reload ``` ### Environment Variables **Frontend** (`.env.local`): - `NEXT_PUBLIC_SOCKET_URL` — Backend WebSocket URL - `NEXT_PUBLIC_GOOGLE_CLIENT_ID` — Google OAuth client ID - `NEXT_PUBLIC_PHONE_URL` — Phone onboarding URL (for QR code) **Backend** (`.env`): - `DATABASE_URL` — Neon Postgres connection string - `ANTHROPIC_API_KEY` — Claude API key - `SERPER_API_KEY` — Serper.dev API key - `DEEPGRAM_API_KEY` — Deepgram STT key - `ELEVENLABS_API_KEY` — ElevenLabs TTS key - `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` — Google OAuth ## Mirror Kiosk Flow The mirror runs as a full-screen kiosk with four states: 1. **Attract** — QR code and branding, waiting for users to scan 2. **Waiting** — Shows "Up next: [name]" with a 2-minute auto-skip timeout 3. **Session** — Active AI stylist session with voice, gestures, and clothing overlay 4. **Recap** — Session summary with liked items and stats ## Project Structure ``` frontend/ # Next.js app src/ app/ mirror/ # Mirror display (kiosk) phone/ # Phone onboarding admin/ # Admin dashboard hooks/ # Camera, STT, gestures, pose detection, avatar components/mirror/ # ClothingCanvas, ProductCarousel, SpeechDisplay, etc. lib/ # API client, TTS, emotion parser, socket backend/ # FastAPI server agent/ # Mira orchestrator, prompts, tools routers/ # REST endpoints (auth, queue, TTS, admin) mcp_server/ # MCP server for external AI integrations scraper/ # Gmail scraping services/ # Serper, Gemini, background removal models/ # Pydantic models, DB schemas migrations/ # Raw SQL migrations ``` ## Testing ```bash # Backend cd backend pytest # Frontend cd frontend npm test ``` ## License Private repository. ## Detected evidence (automated analysis) Indexed codebase: 825 recognized source files, 28259 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 929) ``` .claude/commands/poke.md .claude/settings.json .entire/.gitignore .entire/settings.json .gitignore backend/.dockerignore backend/.env.example backend/agent/__init__.py backend/agent/memory.py backend/agent/orchestrator.py backend/agent/prompts.py backend/agent/test_harness.py backend/agent/tools.py backend/Dockerfile backend/Dockerfile.mcp backend/judges/__init__.py backend/judges/db.py backend/judges/routes.py backend/judges/scraper.py backend/main.py backend/mcp_server/__init__.py backend/mcp_server/requirements.txt backend/mcp_server/server.py backend/mcp_server/tests/__init__.py backend/mcp_server/tests/test_integration.py backend/mcp_server/tests/test_sessions.py backend/mcp/__init__.py backend/migrations/001_initial_schema.sql backend/migrations/002_enrich_purchases.sql backend/migrations/003_add_last_scraped_at.sql backend/migrations/004_add_summary_reaction.sql backend/migrations/005_create_judges.sql backend/migrations/006_create_demo_user.sql backend/migrations/007_add_is_fashion.sql backend/migrations/008_create_calendar_events.sql backend/migrations/009_add_selfie_column.sql backend/migrations/010_phone_constraints.sql backend/models/__init__.py backend/models/database.py backend/models/schemas.py backend/pyproject.toml backend/requirements.txt backend/routers/__init__.py backend/routers/admin.py backend/routers/auth.py backend/routers/queue.py backend/routers/sessions.py backend/routers/tts.py backend/routers/users.py backend/scrape_debug.py backend/scraper/__init__.py backend/scraper/brand_scanner.py backend/scraper/calendar_fetch.py backend/scraper/db.py backend/scraper/gmail_auth.py backend/scraper/gmail_fetch.py backend/scraper/pipeline.py backend/scraper/profile_builder.py backend/scraper/purchase_parser.py backend/scraper/routes.py backend/scraper/socket_events.py backend/services/__init__.py backend/services/auth.py backend/services/background_removal.py backend/services/gemini_flatlay.py backend/services/serper_cache.py backend/services/serper_search.py backend/services/user_data_service.py backend/test_env/bin/activate backend/test_env/bin/activate.csh backend/test_env/bin/activate.fish backend/test_env/bin/Activate.ps1 backend/test_env/bin/pip backend/test_env/bin/pip3 backend/test_env/bin/pip3.11 backend/test_env/bin/python backend/test_env/bin/python3 backend/test_env/bin/python3.11 backend/test_env/lib/python3.11/site-packages/_distutils_hack/__init__.py backend/test_env/lib/python3.11/site-packages/_distutils_hack/override.py backend/test_env/lib/python3.11/site-packages/distutils-precedence.pth backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/entry_points.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/INSTALLER backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/AUTHORS.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/LICENSE.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/cachecontrol/LICENSE.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/certifi/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/dependency_groups/LICENSE.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/distlib/LICENSE.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/distro/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/idna/LICENSE.md backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/msgpack/COPYING backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/packaging/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/packaging/LICENSE.APACHE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/packaging/LICENSE.BSD backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/pkg_resources/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/platformdirs/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/pygments/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/pyproject_hooks/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/requests/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/resolvelib/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/rich/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/tomli_w/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/tomli/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/truststore/LICENSE backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/licenses/src/pip/_vendor/urllib3/LICENSE.txt backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/METADATA backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/RECORD backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/REQUESTED backend/test_env/lib/python3.11/site-packages/pip-26.0.1.dist-info/WHEEL backend/test_env/lib/python3.11/site-packages/pip/__init__.py backend/test_env/lib/python3.11/site-packages/pip/__main__.py backend/test_env/lib/python3.11/site-packages/pip/__pip-runner__.py backend/test_env/lib/python3.11/site-packages/pip/_internal/__init__.py backend/test_env/lib/python3.11/site-packages/pip/_internal/build_env.py backend/test_env/lib/python3.11/site-packages/pip/_internal/cache.py backend/test_env/lib/python3.11/site-packages/pip/_internal/cli/__init__.py backend/test_env/lib/python3.11/site-packages/pip/_internal/cli/autocompletion.py backend/test_env/lib/python3.11/site-packages/pip/_internal/cli/base_command.py backend/test_env/lib/python3.11/site-packages/pip/_internal/cli/cmdoptions.py [809 more files omitted for size] ``` ### Dependencies - backend/mcp_server/requirements.txt: asyncpg@==0.30.0, fastmcp@==2.14.5, python-dotenv@>=1.1.0, uvicorn[standard]@>=0.35 - backend/requirements.txt: anthropic@==0.42.0, asyncpg@==0.30.0, beautifulsoup4@==4.12.3, deepgram-sdk@==3.11.0, fastapi@==0.115.6, fastmcp@==2.14.5, google-api-python-client@==2.159.0, google-auth@==2.37.0, google-auth-oauthlib@==1.2.1, httpx@==0.28.1, onnxruntime@==1.21.1, phonenumbers@==8.13.53, pydantic@>=2.11.7, pytest@==8.3.4, pytest-asyncio@==0.24.0, python-dotenv@>=1.1.0, python-socketio@==5.11.4, rembg@==2.0.69, serpapi@==0.1.5, tabulate@==0.9.0, uvicorn[standard]@>=0.35 - frontend/package.json: @mediapipe/tasks-vision@^0.10.18, @neondatabase/serverless@^1.0.2, @react-three/drei@^10.7.7, @react-three/fiber@^9.5.0, @tailwindcss/postcss@^4.1.18, @testing-library/jest-dom@^6.9.1, @testing-library/react@^16.3.2, @types/node@^22.0.0, @types/react@^19.0.0, @types/react-dom@^19.0.0, @types/three@^0.182.0, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9.0.0, eslint-config-next@^15.1.0, google-auth-library@^10.5.0, jsdom@^28.0.0, libphonenumber-js@^1.12.36, lucide-react@^0.564.0, next@^15.1.0, postcss@^8.5.6, qrcode.react@^4.2.0, radix-ui@^1.4.3, react@^19.0.0, react-dom@^19.0.0, shadcn@^3.8.4, socket.io-client@^4.8.0, tailwind-merge@^3.4.0, tailwindcss@^4.1.18, three@^0.182.0, tw-animate-css@^1.4.0, typescript@5.9.3, vitest@^4.0.18 ### Recent commits (newest first) - chore: commit all local changes - Add Neon HTTP fallback to MCP server and fix UUID casts for session queries - Fix tool_use blocks getting invalid text field in conversation history - Add persistent QR code to mirror display - Add Dockerfiles for Railway deployment - Add phone number authentication for Poke MCP integration - Fix assistant message token overflow from tool input mutation - Swap avatar and end session button positions on mirror - Compact phone UI for portrait kiosk display - Update QR code to point to Poke onboarding link - Add project README with architecture, setup, and usage docs - Move submitOnboarding to Vercel API route - Add Next.js API routes for phone auth + queue (Vercel serverless) - Add end_session tool and mirror End Session button - Lazy-load rembg/onnxruntime to fix server startup timeout on Render - Fix all dependency conflicts for fastmcp 2.14.5 compatibility - Fix dependency conflict: pydantic>=2.11.7 required by fastmcp 2.14.5 - Add onnxruntime dependency required by rembg - Add confidence threshold to STT config, gesture improvements, liked items tray - Pin fastmcp==2.14.5 to fix http_app() AttributeError on Render ## Key source files (fetched from GitHub, selected and truncated for size) ### MERGE_RESOLUTION_SPEC.md ```markdown # Merge Resolution Spec: Kiosk Mode + Video Avatar ## Context Local `main` (13 commits ahead) has kiosk mode, onboarding flow, queue system, and admin dashboard. Origin `main` (8 commits) has Vivian's MiraVideoAvatar (replacing Orb), rembg service, emotion loop videos, and test pipeline. ## Decisions ### Avatar System: **Video Avatar wins** - Use `MiraVideoAvatar` component + `useMiraVideoAvatar` hook (Vivian's) - Remove all Orb references (`@/components/ui/orb`, dynamic import, `useOrbAvatar`) - 26 emotion loop MP4s with idle/talking variants stay - No real-time audio reactivity needed — loop switching is sufficient ### Emotion Detection: **Keep fallback** - Keep Vivian's `detectEmotionFromText()` in `emotion-parser.ts` - Use `parseEmotionTag()` first, then `detectEmotionFromText()` as fallback when no `[emotion:X]` tag present - This feeds the video avatar's richer emotion set (13 emotions vs Orb's 4) ### Kiosk Flow: **Local branch wins entirely** - Keep attract → waiting → session state machine - QR code attract screen, "Up next" waiting screen with Start Session button - Mirror button triggers session start (not phone, not auto-start) - `session_force_end` handler stays - `handleSkipUser` callback stays - `WaitingCountdown` component stays - Vivian's standalone "Start Session" button overlay is removed (superseded by kiosk waiting state) ### Avatar Positioning: **Fixed position** - Video avatar stays in a fixed corner position, no context-aware movement - Remove the `orbStyle`/`avatarStyle` useMemo that changes position based on state ### Backend: **Keep both sides** - Admin router + queue socket handlers (local) - rembg background removal service (Vivian) - Nano Banana test pipeline (Vivian) - Wire everything together in main.py ## Files Affected - `frontend/src/app/mirror/page.tsx` — main conflict file - `frontend/src/lib/emotion-parser.ts` — keep Vivian's expanded version - `backend/main.py` — merge both sides' additions - New files from origin (no conflicts): `mira-video-avatar.tsx`, `useMiraVideoAvatar.ts`, demo pages, avatar loops ``` ### SPEC.md ```markdown # Mirrorless — Full Project Specification Mirrorless is a hackathon project (36h, 4+ people in pairs) — an AI-powered smart mirror that provides hyper-personalized clothing recommendations. Users onboard via phone, get their purchase history and style analyzed, then step up to a physical mirror where AI stylist "Mira" gives them outfit recommendations overlaid on their body in real-time. An MCP server exposes user taste data to Poke and other services. ## Architecture Overview ``` ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │ Phone UI │────▶│ Next.js on │────▶│ Python FastAPI │ │ (onboarding │◀────│ Vercel │◀────│ on Render │ │ + dashboard)│ │ (frontend) │ │ (backend) │ └─────────────┘ └──────┬───────┘ └───────┬────────┘ │ │ Socket.io ┌──────┴──────┐ │ │ │ ┌──────▼───────┐ Neon Postgres Claude API │ Mirror Display│ (DB) (Haiku 4.5) │ (full-screen │ │ │ Chrome on TV)│ ┌─────┴──────┐ └──────────────┘ │ SerpAPI │ │ │ Deepgram │ MediaPipe │ HeyGen │ (pose + hands) └────────────┘ ``` ## Tech Stack - **Frontend**: Next.js (mirror display + phone UI + onboarding) - **Backend**: Python FastAPI - **Database**: Neon Postgres - **Real-time**: Socket.io - **STT**: Deepgram streaming - **TTS/Avatar**: HeyGen LiveAvatar API - **Body tracking**: MediaPipe (BlazePose + Hand landmarks) - **Clothing sourcing**: SerpAPI / Google Shopping API - **AI**: Claude API (Haiku 4.5 via Anthropic API with OAuth + beta headers) - **Auth**: Google OAuth (Gmail + Calendar scopes) - **Deploy**: Vercel (frontend) + Render (Python backend) ## Core Components ### 1. Onboarding (Phone → Mobile Web) - QR code → Next.js mobile page - Google OAuth (scopes: Gmail read, Calendar read, profile) - Collect name, phone number - Link to Poke signup: https://poke.com/treehacks - After onboarding, phone becomes session dashboard ### 2. Data Scraping Pipeline **Fast pass (~10-15s, all in parallel)**: - Recent 5-10 receipt emails from major retailers - Brand frequency scan from last 100 email subject lines - Google profile photo for initial style assessment **Background deep scrape (async)**: - Full inbox: all receipts, newsletter subscriptions, shipping notifications - Calendar events for lifestyle context (gym, travel, meetings) - Results stream to agent context as they complete **MVP cut**: Skip Google Photos analysis ### 3. AI Agent — Mira **Architecture**: Custom event-driven orchestrator (NOT Agents SDK) - Central event loop receives: voice transcripts, gesture events, pose data, scraping updates - B [truncated — 4942 more characters] ``` ### backend/pyproject.toml ``` [tool.pytest.ini_options] asyncio_mode = "auto" asyncio_default_fixture_loop_scope = "session" ``` ### backend/Dockerfile ``` FROM python:3.11-slim RUN apt-get update && \ apt-get install -y --no-install-recommends libgl1-mesa-glx libglib2.0-0 && \ rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . EXPOSE ${PORT:-8000} CMD uvicorn main:socket_app --host 0.0.0.0 --port ${PORT:-8000} ``` ### backend/requirements.txt ``` fastapi==0.115.6 uvicorn[standard]>=0.35 python-socketio==5.11.4 anthropic==0.42.0 httpx==0.28.1 asyncpg==0.30.0 python-dotenv>=1.1.0 google-auth==2.37.0 google-auth-oauthlib==1.2.1 google-api-python-client==2.159.0 serpapi==0.1.5 deepgram-sdk==3.11.0 beautifulsoup4==4.12.3 pydantic>=2.11.7 pytest==8.3.4 pytest-asyncio==0.24.0 tabulate==0.9.0 rembg==2.0.69 onnxruntime==1.21.1 fastmcp==2.14.5 phonenumbers==8.13.53 ``` ### frontend/package.json ``` { "name": "mirrorless-frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint", "test": "vitest run", "test:watch": "vitest" }, "dependencies": { "@mediapipe/tasks-vision": "^0.10.18", "@neondatabase/serverless": "^1.0.2", "@react-three/drei": "^10.7.7", "@react-three/fiber": "^9.5.0", "@types/three": "^0.182.0", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "google-auth-library": "^10.5.0", "libphonenumber-js": "^1.12.36", "lucide-react": "^0.564.0", "next": "^15.1.0", "qrcode.react": "^4.2.0", "radix-ui": "^1.4.3", "react": "^19.0.0", "react-dom": "^19.0.0", "socket.io-client": "^4.8.0", "tailwind-merge": "^3.4.0", "three": "^0.182.0" }, "devDependencies": { "@tailwindcss/postcss": "^4.1.18", "@testing-library/jest-dom": "^6.9.1", "@testing-library/react": "^16.3.2", "@types/node": "^22.0.0", "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "eslint": "^9.0.0", "eslint-config-next": "^15.1.0", "jsdom": "^28.0.0", "postcss": "^8.5.6", "shadcn": "^3.8.4", "tailwindcss": "^4.1.18", "tw-animate-css": "^1.4.0", "typescript": "5.9.3", "vitest": "^4.0.18" } } ``` ### backend/mcp_server/requirements.txt ``` fastmcp==2.14.5 asyncpg==0.30.0 python-dotenv>=1.1.0 uvicorn[standard]>=0.35 ``` ### backend/main.py ```python import asyncio import json import os import uuid from uuid import UUID from anthropic import AsyncAnthropic from dotenv import load_dotenv from fastapi import FastAPI, HTTPException from starlette.middleware.cors import CORSMiddleware import socketio from routers import auth, queue, users, tts, admin, sessions from scraper.routes import router as scraper_router from judges.routes import router as judges_router from agent.orchestrator import MiraOrchestrator, generate_outfit_recommendations, update_outfit_reaction, _outfits_to_display_payloads from models.database import get_neon_client from models.schemas import OnboardingQuestionnaireResponse, OutfitReactionUpdate from services.user_data_service import save_onboarding_data from services.serper_search import build_brand_queries, fetch_clothing_batch from services.gemini_flatlay import generate_flat_lays_batch load_dotenv() sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins="*") # Map socket IDs to user IDs for disconnect cleanup _sid_to_user: dict[str, str] = {} # Create FastAPI app app = FastAPI(title="Mirrorless API", version="0.1.0") # CORS on FastAPI only — Socket.io handles its own CORS via cors_allowed_origins app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) app.include_router(auth.router) app.include_router(queue.router) app.include_router(users.router) app.include_router(scraper_router) app.include_router(judges_router) app.include_router(tts.router) app.include_router(admin.router) app.include_router(sessions.router) # Make sio and Mira accessible to routes app.state.sio = sio mira = MiraOrchestrator(socket_io=sio) app.state.mira = mira @app.get("/health") async def health(): return {"status": "ok"} # --- REST API endpoints for recommendations --- @app.post("/api/sessions/{session_id}/recommendations") async def create_outfit_recommendations(session_id: str): """ Generate recommendations for active session. 1. Verify session exists and is active 2. Get user_id from session 3. Call generate_outfit_recommendations() 4. Handle new user case (return needs_onboarding) 5. Return results """ db = await get_neon_client() try: # Get session info print(f"[API] Recommendations requested for session: {session_id}") session_query = "SELECT * FROM sessions WHERE id = $1::uuid AND status = 'active'" session_rows = await db.execute(session_query, [session_id]) if not session_rows: raise HTTPException(status_code=404, detail="Active session not found") session = session_rows[0] user_id = str(session["user_id"]) # Generate recommendations result = await generate_outfit_recommendations(user_id, session_id, db) # Also emit to mirror display via socket so ClothingCanvas picks it up if result.get("status") == "success" and result.get("data"): payloads = _outfits_to_display_payloads(result["data"].get("outfits", [])) for payload in payloads: await sio.emit("tool_result", payload, room=user_id) return result finally: await db.close() @app.patch("/api/outfits/{outfit_id}/reaction") async def update_outfit_reaction_endpoint( outfit_id: str, body: OutfitReactionUpdate ): """ Record user reaction (liked/disliked/skipped). """ db = await get_neon_client() try: result = await update_outfit_reaction(db, outfit_id, body.reaction) return result finally: await db.close() @app.post("/api/users/{user_id}/onboarding") async def complete_onboarding( user_id: str, questionnaire: OnboardingQuestionnaireResponse ): """ Save onboarding questionnaire to style_profiles table. Enables recommendations for new users without purchase history. """ db = await get_neon_client() try: await save_onboarding_data(db, user_id, questionnaire.dict()) return {"status": "success", "message": "Onboarding completed"} except Exception as e: raise HTTPException(status_code=500, detail=f"Failed to save onboarding: {e}") finally: await db.close() # --- Test endpoint: recommendation pipeline --- @app.post("/api/test/recommend") async def test_recommend(body: dict): """ Test endpoint combining Serper search → Claude Haiku outfit curation → Nano Banana flat lays → transparent overlay images. Expects: { brands: string[], gender: string, style_notes: string } Returns: { outfits: [{ outfit_name, voice, items: [{ id, category, imageUrl, name }] }] } """ brands = body.get("brands", ["Nike", "Zara", "H&M"]) gender = body.get("gender", "mens") style_notes = body.get("style_notes", "casual") serper_key = os.getenv("SERPER_API_KEY") anthropic_key = os.getenv("ANTHROPIC_API_KEY") if not serper_key: raise HTTPException(status_code=500, detail="SERPER_API_KEY not configured") if not anthropic_key: raise HTTPException(status_code=500, detail="ANTHROPIC_API_KEY not configured") # Step 1: Serper search for tops + bottoms print(f"[test/recommend] Searching for {brands} ({gender}, {style_notes})") brand_queries = build_brand_queries(brands[:5], gender) tops_items, bottoms_items = await asyncio.gather( fetch_clothing_batch(brand_queries["tops"], serper_key, num_results_per_query=3), fetch_clothing_batch(brand_queries["bottoms"], serper_key, num_results_per_query=3), ) # Limit to reasonable count before sending to Claude tops_items = tops_items[:15] bottoms_items = bottoms_items[:15] print(f"[test/recommend] Found {len(tops_items)} tops, {len(bottoms_items)} bottoms") if not tops_items and not bottoms_items: return {"outfits": []} # Step 2: Claude Haiku picks 2 outfits with Mira commentary tops_list = "\n [truncated — 11728 more characters] ``` ### jenny/src/app.js ```javascript // Main application class MiraStyleAssistant { constructor() { this.analyzeBtn = document.getElementById('analyze-btn'); this.talkBtn = document.getElementById('talk-btn'); this.speechBubble = document.getElementById('speech-bubble'); this.miraText = document.getElementById('mira-text'); this.userText = document.getElementById('user-text'); this.voiceIndicator = document.getElementById('voice-indicator'); this.isProcessing = false; this.hasInteracted = false; this.init(); } async init() { console.log('Initializing Mira Style Assistant...'); // Initialize avatar videos await miraAvatar.init(); console.log('Avatar OK'); // Initialize webcam const webcamOk = await webcam.init(); if (!webcamOk) { this.showMessage("couldn't access your camera - check permissions?"); } else { console.log('Webcam OK'); } // Initialize Gemini if (!initGemini()) { this.showMessage("need a Gemini API key in config.js to work~"); return; } console.log('Gemini OK'); // Initialize ElevenLabs (optional, will fall back to browser TTS) if (initElevenLabs()) { console.log('ElevenLabs OK'); } else { console.log('Using browser TTS (add ELEVENLABS_API_KEY for better voice)'); } // Set up button listeners if (this.analyzeBtn) { this.analyzeBtn.addEventListener('click', () => this.handleAnalyze()); } if (this.talkBtn) { this.talkBtn.addEventListener('click', () => this.handleTalk()); } // Set up speech recognition callbacks if (speechInput.supported) { speechInput.onStart = () => { this.voiceIndicator?.classList.add('active'); miraAvatar.idle(); // Show idle when user is talking }; speechInput.onInterim = (text) => { if (this.userText) { this.userText.textContent = text + '...'; this.userText.parentElement.style.display = 'block'; } }; speechInput.onResult = (text) => { if (this.userText) { this.userText.textContent = text; } this.processUserSpeech(text); }; speechInput.onEnd = () => { this.voiceIndicator?.classList.remove('active'); if (this.talkBtn) { this.talkBtn.textContent = '🎤 Talk to Mira'; this.talkBtn.classList.remove('listening'); } }; console.log('Speech recognition OK'); } else { console.warn('Speech recognition not supported'); if (this.talkBtn) { this.talkBtn.style.display = 'none'; } } // Keyboard shortcuts document.addEventListener('keydown', (e) => { if (e.target.tagName === 'INPUT') return; if (e.code === 'Space' && !this.isProcessing) { e.preventDefault(); this.handleAnalyze(); } else if (e.code === 'KeyT' && !this.isProcessing) { e.preventDefault(); this.handleTalk(); } }); this.showMessage("hey! press space to show your outfit, or T to talk to me~"); console.log('Mira Style Assistant ready!'); } handleAnalyze() { this.unlockAudio(); this.analyzeOutfit(); } handleTalk() { this.unlockAudio(); if (speechInput.isListening) { speechInput.stop(); } else { if (this.talkBtn) { this.talkBtn.textContent = '🔴 Listening...'; this.talkBtn.classList.add('listening'); } speechInput.start(); } } unlockAudio() { if (!this.hasInteracted) { this.hasInteracted = true; const warmup = new SpeechSynthesisUtterance(''); window.speechSynthesis.speak(warmup); } } showMessage(text) { if (this.miraText) { this.miraText.textContent = text; } } async processUserSpeech(userText) { if (this.isProcessing) return; this.isProcessing = true; miraAvatar.thinking(); this.showMessage("hmm let me think about that..."); try { // Capture current frame for context const imageData = webcam.captureFrame(); // Send to Gemini with user's question console.log('Processing:', userText); const response = await gemini.analyzeOutfit(imageData, userText); console.log('Response:', response); await this.deliverResponse(response); } catch (err) { console.error('Error:', err); miraAvatar.concerned(); this.showMessage("sorry, something went wrong - " + err.message); } finally { this.isProcessing = false; } } async analyzeOutfit() { if (this.isProcessing || !gemini) return; this.isProcessing = true; if (this.analyzeBtn) { this.analyzeBtn.disabled = true; this.analyzeBtn.textContent = 'Thinking...'; } miraAvatar.thinking(); this.showMessage("hmm let me see what you're wearing..."); try { const imageData = webcam.captureFrame(); if (!imageData) { throw new Error('Could not capture image'); } console.log('Analyzing outfit...'); const response = await gemini.analyzeOutfit(imageData); console.log('Response:', response); await this.deliverResponse(response); } catch (err) { console.error('Analysis error:', err); miraAvatar.concerned(); this.showMessage("oops - " + err.message); } finally { this.isProcessing = false; if (this.analyzeBtn) { this.analyzeBtn.disabled = false; this.analyzeBtn.textContent = 'Ask Mira'; } } } async deliverResponse(text) { this.showMessage(text); // Check for scripted response match const scripted = findScriptedResponse(text); if (scripted) { console.log('Using scripted response:', scripted.phrase); // Play scripted video with baked-in audio await miraAvatar.playScriptedVideo(scripted.video [truncated — 1205 more characters] ``` ### backend/mcp_server/server.py ```python """Poke MCP server for Mirrorless session data. Exposes two tools for the Poke AI agent: - get_past_sessions: Retrieve a user's past mirror sessions with liked items - save_session: Save a summary for a completed session Connects to Neon Postgres via asyncpg (production) or Neon HTTP API (local dev fallback). """ from __future__ import annotations import asyncio import json import logging import os import sys import uuid from pathlib import Path import asyncpg from dotenv import load_dotenv # Add backend/ to sys.path to import from models.database _backend_dir = str(Path(__file__).parent.parent) if _backend_dir not in sys.path: sys.path.insert(0, _backend_dir) from models.database import NeonHTTPClient load_dotenv() log = logging.getLogger("mcp") logging.basicConfig( level=logging.INFO, format="[%(asctime)s] %(name)s %(levelname)s: %(message)s", datefmt="%H:%M:%S", ) DATABASE_URL = os.getenv("DATABASE_URL", "") # --------------------------------------------------------------------------- # Database client (lazy singleton — asyncpg pool or Neon HTTP fallback) # --------------------------------------------------------------------------- _db_client: asyncpg.Pool | NeonHTTPClient | None = None _db_mode: str | None = None # "asyncpg" or "http" async def _get_db_client() -> asyncpg.Pool | NeonHTTPClient: """Return (or create) the database client. Tries asyncpg first (production). If port 5432 is unreachable (local dev), falls back to Neon's serverless HTTP API. """ global _db_client, _db_mode if _db_client is not None: return _db_client # Try asyncpg first (production) log.info("Attempting asyncpg connection (port 5432)...") try: _db_client = await asyncio.wait_for( asyncpg.create_pool( DATABASE_URL, min_size=1, max_size=5, ssl="require", ), timeout=3.0, ) _db_mode = "asyncpg" log.info("Database connection: asyncpg (production mode)") return _db_client except Exception as e: log.warning( "asyncpg connection failed (%s: %s), falling back to Neon HTTP", type(e).__name__, e, ) # Fallback to Neon HTTP (local dev) _db_client = NeonHTTPClient(DATABASE_URL) _db_mode = "http" log.info("Database connection: Neon HTTP (local dev mode)") return _db_client # --------------------------------------------------------------------------- # Query adapters (normalize asyncpg pool vs NeonHTTPClient) # --------------------------------------------------------------------------- async def _fetch(client, query: str, *params): """Execute query and return all rows.""" if isinstance(client, NeonHTTPClient): return await client.execute(query, list(params)) else: async with client.acquire() as conn: return await conn.fetch(query, *params) async def _fetchrow(client, query: str, *params): """Execute query and return first row.""" if isinstance(client, NeonHTTPClient): rows = await client.execute(query, list(params)) return rows[0] if rows else None else: async with client.acquire() as conn: return await conn.fetchrow(query, *params) async def _execute(client, query: str, *params): """Execute query without returning rows.""" if isinstance(client, NeonHTTPClient): await client.execute(query, list(params)) else: async with client.acquire() as conn: await conn.execute(query, *params) # --------------------------------------------------------------------------- # Core logic (testable with mock pool or NeonHTTPClient) # --------------------------------------------------------------------------- async def _get_past_sessions(client, phone: str, limit: int = 10) -> dict: """Look up a user by phone and return their past sessions with outfits.""" try: # Find user by phone log.info("Looking up user by phone=%s", phone) user = await _fetchrow( client, "SELECT id, phone FROM users WHERE phone = $1", phone ) if user is None: log.warning("User not found for phone=%s", phone) return {"ok": False, "error": f"User with phone {phone} not found"} user_id = user["id"] log.info("Found user_id=%s for phone=%s", user_id, phone) # Fetch sessions ordered by most recent sessions = await _fetch( client, "SELECT id, started_at, ended_at, status " "FROM sessions WHERE user_id = $1 " "ORDER BY started_at DESC LIMIT $2", user_id, limit, ) log.info("Found %d sessions for user_id=%s", len(sessions), user_id) if not sessions: return {"ok": True, "sessions": []} result_sessions = [] for session in sessions: session_id = session["id"] # Fetch outfits for this session (liked + summary only) outfits = await _fetch( client, "SELECT id, reaction, outfit_data, clothing_items " "FROM session_outfits WHERE session_id = $1 " "AND reaction IN ('liked', 'summary')", session_id, ) liked_items = [] summary = None for o in outfits: if o["reaction"] == "liked": # Resolve clothing_items UUIDs to full product details item_ids = o["clothing_items"] or [] if item_ids: rows = await _fetch( client, "SELECT id, name, brand, price, image_url, buy_url, category " "FROM clothing_items WHERE id = ANY($1::uuid[])", item_ids, [truncated — 8359 more characters] ``` [770 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]