# Project export: Jarvis This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Know everything - Devpost: https://devpost.com/software/jarvis-0xt8ro - GitHub: https://github.com/dgne58/jarvis - Demo: https://youtube.com/shorts/VQDHUFmHNZA - Video: https://www.youtube.com/embed/wGPBxhYchgQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Most Wacky Hack (presented by Wordware)) - Team: 2 GitHub contributor(s) — ts6ki (7 commits), AndresNinou (6 commits) ## Devpost submission (written by the team) ### Inspiration Cluely kinda sucks, and Iron Man is the coolest movie of 2009. ### What it does Yeah, so we hacked the Meta Ray-Bans display to let superintelligence assist you 24/7. It guarantees you the best line for your sales pitch, the smoothest rizz for your date, and the answer to every combinatorics question Citadel could ever throw at you. Oh, and we also made it cross-reference any face you come across with a giant database that probably has you in it too, just for fun. This isn’t Cluely; this is Jarvis. ### Challenges we ran into We were really excited to put facial recognition in our glasses, but we ran into tons of issues trying to find a provider. Absolutely no provider supported APIs, so we had to build our own scrapers to dynamically extract information from sites using Selenium. We burned through around $70 testing different sites for face searching. A few didn’t work, and one even banned us for bot usage, which violated their ToS. So yeah, that was $30 down the drain, unfortunately. ### Accomplishments we're proud of We were the first ones to actually make something like this work. Cluely talked about doing something similar, but we beat them to it. Unlike Cluely, we don’t overprice and underdeliver. We just built it, hacked it, and made it real. ### What we learned Nothing’s impossible, seriously. Meta has put so many restrictions in place to prevent people from hacking their glasses, but we scrapped it together at a hackathon anyway. ## README (from the GitHub repository) # 🕶️ Jarvis — AI Networking Assistant for Meta Glasses **“Because LinkedIn is for amateurs.”** Jarvis transforms Meta Glasses into a real-time networking assistant — recognizing faces, recalling names, and generating AI-powered conversation cues during in-person interactions. ## 🚀 Overview **Cluely** uses live speech transcription, facial recognition, and conversational AI to enhance real-world interactions. It helps you *remember people, recall context, and sound sharp — instantly.* ### Core Capabilities - **Facial Recognition** – Identify and recall people in real time - **Live Transcription** – Multi-speaker diarization with low latency - **AI Conversation Hints** – Context-aware, adaptive dialogue prompts - **Voice Commands** – Say “banana” to trigger recognition - **On-Glasses UI** – Optimized interface for Meta Glasses streaming ## 🧠 Tech Stack ### Frontend - **React 19 + Vite** – Fast, modular UI - **WebRTC APIs** – Camera and mic access for live recognition - **CSS3 (Glassmorphism)** – Lightweight visual effects ### Backend - **FastAPI + SQLModel + PostgreSQL** – Async Python stack - **Docker Compose** – Unified deployment - **Face Recognition API** – Custom image-matching service ### AI & APIs - **Deepgram Nova-3** – Real-time speech-to-text - **OpenRouter (Grok-4-Fast)** – Conversation intelligence model ## ⚙️ Setup ### Prerequisites - Node.js 18+ and npm/pnpm - Python 3.10+ with [UV](https://docs.astral.sh/uv) - Deepgram + OpenRouter API keys - Docker (optional) ### Frontend ```bash npm install npm run dev ``` ### Backend ```bash cd backend uv sync --dev uv run uvicorn app.main:app --reload ``` Access the app at: - Frontend → `http://localhost:5173` - Backend → `http://localhost:8000` ## 📂 Project Structure ``` . ├── src/ # React frontend │ ├── App.jsx │ ├── main.jsx │ └── styles/ ├── backend/ # FastAPI service │ ├── app/ │ ├── Dockerfile │ └── pyproject.toml └── public/ # Static assets ``` ## 🔑 Environment Variables | Variable | Description | | --| -| | `VITE_DEEPGRAM_API_KEY` | Deepgram speech-to-text | | `VITE_OPENROUTER_API_KEY` | OpenRouter AI models | | `DATABASE_URL` | PostgreSQL connection string | | `SECRET_KEY` | Backend secret key | | `ENVIRONMENT` | `local`, `staging`, or `production` | ## 🧩 Usage 1. **Enable Camera** → Allow facial recognition access 2. **Start Transcription** → Begin real-time analysis 3. **Say “banana”** → Trigger face identification 4. **View Suggestions** → Watch AI conversation prompts appear ## 🧪 Development Commands ### Frontend ```bash npm run build npm run preview npm run lint ``` ### Backend ```bash make run make check make test ``` ## 🧾 License This project was developed for Calhacks 12.0. © 2025 ## Detected evidence (automated analysis) Indexed codebase: 50 recognized source files, 257 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (61 of 61) ``` .claude/settings.local.json .gitignore backend/.dockerignore backend/.env.example backend/.gitignore backend/.pre-commit-config.yaml backend/.python-version backend/.testmondata backend/app/__init__.py backend/app/api/__init__.py backend/app/api/deps.py backend/app/api/main.py backend/app/api/routes/__init__.py backend/app/api/routes/automation.py backend/app/api/routes/pages.py backend/app/api/routes/private.py backend/app/api/routes/utils.py backend/app/browser_automation.py backend/app/core/__init__.py backend/app/core/_logging.py backend/app/core/config.py backend/app/core/db.py backend/app/core/log_config.py backend/app/crud.py backend/app/lenso_automation.py backend/app/main.py backend/app/models.py backend/app/tests/__init__.py backend/app/tests/api/__init__.py backend/app/tests/api/routes/__init__.py backend/app/tests/api/routes/test_pages.py backend/app/tests/api/routes/test_private.py backend/app/tests/conftest.py backend/app/tests/crud/__init__.py backend/app/tests/scripts/__init__.py backend/app/tests/utils/__init__.py backend/app/tests/utils/page.py backend/app/tests/utils/utils.py backend/app/utils.py backend/docker-compose.override.yml backend/docker-compose.yml backend/Dockerfile backend/docs/automation_api.md backend/Makefile backend/manual_test.py backend/pyproject.toml backend/README.md backend/test_automation.py backend/uv.lock docs/jarvis-tech-stack-recommendations.md docs/jarvis-workflow-diagram.md docs/pull-request-design.md eslint.config.js index.html package.json README.md src/App.css src/App.jsx src/index.css src/main.jsx vite.config.js ``` ### Dependencies - backend/pyproject.toml: aiosqlite@<1.0.0,>=0.19.0, bcrypt@==4.0.1, emails@<1.0,>=0.6, fastapi[standard]@<1.0.0,>=0.114.2, httpx@<1.0.0,>=0.25.1, jinja2@<4.0.0,>=3.1.4, loguru@<1.0.0,>=0.7.2, patchright@>=1.40.0,<2.0.0, playwright@<2.0.0,>=1.40.0, psycopg[binary]@<4.0.0,>=3.1.13, psycopg2-binary@>=2.9.10, pydantic@>2.0, pydantic-settings@<3.0.0,>=2.2.1, pyrefly@>=0.22.1, python-dotenv@(>=1.1.0,<2.0.0), python-multipart@<1.0.0,>=0.0.7, sentry-sdk[fastapi]@<2.0.0,>=1.40.6, sqlmodel@<1.0.0,>=0.0.21, tenacity@<9.0.0,>=8.2.3, webdriver-manager@>=4.0.2 - package.json: @eslint/js@^9.22.0, @types/react@^19.0.10, @types/react-dom@^19.0.4, @vitejs/plugin-react@^4.3.4, eslint@^9.22.0, eslint-plugin-react-hooks@^5.2.0, eslint-plugin-react-refresh@^0.4.19, globals@^16.0.0, react@^19.0.0, react-dom@^19.0.0, vite@^6.3.1 ### Recent commits (newest first) - readme update - readme update - upload backend - fix face recognition API - updated ui - aspect ratio 4:3 - Merge branch 'main' of https://github.com/AndresNinou/echo - ui changes - real FACE API - camera test - faster responses - fix bugs initial concept - initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/pull-request-design.md ```markdown # Pull Request Design: JARVIS AI Agent Implementation **Author:** Chief Systems Architect **Date:** 2025-10-25 **Related Issue(s):** JARVIS AI Agent for Smart Glasses ## 0. Implementation Approach Justification * **Chosen Approach:** Frontend Implementation (React/NextJS) * **Justification:** * This implementation is primarily UI-focused with moderate complexity logic that can be handled effectively in React * The real-time nature of the application benefits from client-side processing for immediate feedback * The existing transcription functionality is already implemented in React, maintaining consistency * Face recognition and LLM interactions can be handled through API calls from the frontend * The liquid glassmorphism UI design requires extensive frontend styling work * While some logic could be moved to a backend, the current requirements don't justify the additional complexity of a separate backend service --- ## 1. Problem Statement * **Description:** Transform the existing real-time transcription application into JARVIS, an AI agent that provides intelligent conversation assistance through smart glasses. The system needs to detect a "banana" keyword to trigger face recognition, identify people, analyze conversations in real-time, and provide suggestions on what to say. * **Success Criteria (High-Level):** * The app detects when someone says "banana" and triggers a mock face recognition API call * Person information is displayed when recognized (name, job, CV-style details) * Conversation analysis starts immediately when transcription begins * Every 3 seconds, conversation history is sent to OpenRouter's deepseek model for analysis * AI suggestions on what to say are displayed in a separate panel in real-time * The UI features a cool liquid glassmorphism transparent dark design * **Business Context & User Impact:** This creates an intelligent assistant that helps users navigate conversations more effectively, providing real-time guidance and contextual information about people they're speaking with. ## 2. Solution Overview * **High-Level Description:** Enhance the existing transcription app with JARVIS AI capabilities. The app will continuously transcribe audio with speaker diarization, detect the "banana" keyword to trigger face recognition, maintain conversation history, and periodically send this data to an LLM for analysis. The AI's suggestions will be displayed in a sleek glassmorphism UI. * **Key Architectural Decisions (Frontend):** * Use React hooks for state management of conversation history, person data, and AI suggestions * Implement a timer-based system to send conversation data to OpenRouter every 3 seconds * Create a modular component structure with separate panels for transcription, person info, and AI suggestions * Apply glassmorphism design patterns with CSS for the liquid transparent dark aesthetic * ** [truncated — 15665 more characters] ``` ### backend/docs/automation_api.md ```markdown # Browser Automation API Documentation This document describes the browser automation API endpoints for using Patchright to automate image uploads to Lenso.ai. ## Overview The browser automation API provides endpoints for: - Uploading images to Lenso.ai for reverse image search - Managing persistent browser sessions for multiple operations - Extracting search results and image URLs ## Base URL ``` http://localhost:8000/api/v1/automation ``` ## Endpoints ### 1. Upload Image by File Path Upload an image to Lenso.ai using a local file path. **Endpoint:** `POST /upload-image` **Request Body:** ```json { "image_path": "/path/to/image.jpg", "headless": true, "wait_time": 10 } ``` **Parameters:** - `image_path` (string, required): Path to the image file - `headless` (boolean, optional): Run browser in headless mode (default: true) - `wait_time` (integer, optional): Seconds to wait for results (default: 10) **Response:** ```json { "success": true, "image_urls": [ "https://api.lenso.ai/proxy/...", "https://api.lenso.ai/proxy/...", ... ], "count": 8, "message": "Successfully found 8 results" } ``` ### 2. Upload Image File Upload an image file directly to Lenso.ai. **Endpoint:** `POST /upload-image-file` **Request:** `multipart/form-data` - `file` (file, required): Image file to upload - `headless` (boolean, optional): Run browser in headless mode (default: true) - `wait_time` (integer, optional): Seconds to wait for results (default: 10) **Response:** Same as upload by path ### 3. Create Browser Session Create a persistent browser session for multiple operations. **Endpoint:** `POST /create-session` **Request Body:** ```json { "headless": true, "user_data_dir": "/path/to/profile" } ``` **Parameters:** - `headless` (boolean, optional): Run browser in headless mode (default: true) - `user_data_dir` (string, optional): Directory for browser profile **Response:** ```json { "session_id": "uuid-string", "status": "created" } ``` ### 4. Upload Image with Session Upload an image using an existing browser session. **Endpoint:** `POST /session/{session_id}/upload-image` **Request:** `multipart/form-data` - `file` (file, required): Image file to upload - `wait_time` (integer, optional): Seconds to wait for results (default: 10) **Response:** Same as upload by path, with additional metadata ### 5. Close Browser Session Close a browser session and cleanup resources. **Endpoint:** `DELETE /session/{session_id}` **Response:** ```json { "session_id": "uuid-string", "status": "closed" } ``` ### 6. List Active Sessions List all active browser sessions. **Endpoint:** `GET /sessions` **Response:** ```json { "active_sessions": ["uuid-string-1", "uuid-string-2"] } ``` ## Usage Examples ### Python Example ```python import requests # Upload image by path response = requests.post( "http://localhost:8000/api/v1/automation/upload-image", json={ "image_path": "/path/to/image.jpg", [truncated — 2188 more characters] ``` ### package.json ``` { "name": "react-transcription-app", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "react": "^19.0.0", "react-dom": "^19.0.0" }, "devDependencies": { "@eslint/js": "^9.22.0", "@types/react": "^19.0.10", "@types/react-dom": "^19.0.4", "@vitejs/plugin-react": "^4.3.4", "eslint": "^9.22.0", "eslint-plugin-react-hooks": "^5.2.0", "eslint-plugin-react-refresh": "^0.4.19", "globals": "^16.0.0", "vite": "^6.3.1" } } ``` ### backend/docker-compose.yml ```yaml services: backend: # For pure local development use build: build: context: . # For CI/CD pipelines, comment out build: above and uncomment image: line below # image: '${DOCKER_IMAGE_BACKEND?Variable not set}:${TAG-latest}' restart: always env_file: - .env environment: - ENVIRONMENT=${ENVIRONMENT} - BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS} - SECRET_KEY=${SECRET_KEY?Variable not set} - SENTRY_DSN=${SENTRY_DSN} - PYTHONUNBUFFERED=1 deploy: resources: limits: memory: 512M healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/utils/health-check/"] interval: 10s timeout: 5s retries: 5 ports: - "8000:8000" # Notice: No explicit depends_on as Prisma migrations run separately # Backend will retry connecting until Postgres is ready # Database service required for the sample DATABASE_URL in .env.example db: image: postgres:15 volumes: - postgres-data:/var/lib/postgresql/data environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres - POSTGRES_DB=app ports: - "5432:5432" healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s timeout: 5s retries: 5 volumes: postgres-data: ``` ### backend/Dockerfile ``` FROM python:3.10 ENV PYTHONUNBUFFERED=1 # Set a longer timeout for UV HTTP operations (default is 30s) ENV UV_HTTP_TIMEOUT=120 WORKDIR /app/ # Install uv # Ref: https://docs.astral.sh/uv/guides/integration/docker/#installing-uv COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /uvx /bin/ # Place executables in the environment at the front of the path # Ref: https://docs.astral.sh/uv/guides/integration/docker/#using-the-environment ENV PATH="/app/.venv/bin:$PATH" # Compile bytecode # Ref: https://docs.astral.sh/uv/guides/integration/docker/#compiling-bytecode ENV UV_COMPILE_BYTECODE=1 # uv Cache # Ref: https://docs.astral.sh/uv/guides/integration/docker/#caching ENV UV_LINK_MODE=copy # Copy dependency specification files first for better layer caching COPY ./pyproject.toml ./uv.lock ./Makefile /app/ # Install dependencies with retry mechanism for network resilience # Ref: https://docs.astral.sh/uv/guides/integration/docker/#intermediate-layers RUN --mount=type=cache,target=/root/.cache/uv \ for i in 1 2 3; do \ echo "Attempt $i: Installing dependencies..." && \ uv sync --frozen --no-install-project && break || \ echo "Attempt $i failed, retrying in 5 seconds..." && \ sleep 5; \ done ENV PYTHONPATH=/app # Copy application code after dependencies are installed COPY ./app /app/app # Install the project (development install) RUN --mount=type=cache,target=/root/.cache/uv \ for i in 1 2 3; do \ echo "Attempt $i: Installing project..." && \ uv sync && break || \ echo "Attempt $i failed, retrying in 5 seconds..." && \ sleep 5; \ done # Production configuration settings with defaults that can be overridden at runtime # - LOG_LEVEL controls uvicorn's log verbosity ENV LOG_LEVEL=info # Use shell form to allow environment variable expansion # The exec-form (JSON array) doesn't expand ${VARS} # Use default values with the :- syntax in case ENV vars aren't set CMD bash -c "uvicorn app.main:app --host 0.0.0.0 --port 8000 --log-level ${LOG_LEVEL:-info}" ``` ### backend/pyproject.toml ``` [project] name = "app" version = "0.1.0" description = "" requires-python = ">=3.10,<4.0" dependencies = [ "fastapi[standard]<1.0.0,>=0.114.2", "python-multipart<1.0.0,>=0.0.7", "tenacity<9.0.0,>=8.2.3", "pydantic>2.0", "emails<1.0,>=0.6", "jinja2<4.0.0,>=3.1.4", "httpx<1.0.0,>=0.25.1", "psycopg[binary]<4.0.0,>=3.1.13", "sqlmodel<1.0.0,>=0.0.21", "aiosqlite<1.0.0,>=0.19.0", "bcrypt==4.0.1", "pydantic-settings<3.0.0,>=2.2.1", "sentry-sdk[fastapi]<2.0.0,>=1.40.6", "python-dotenv (>=1.1.0,<2.0.0)", # Structured logging with Loguru "loguru<1.0.0,>=0.7.2", "psycopg2-binary>=2.9.10", "pyrefly>=0.22.1", # Browser automation with Playwright "playwright<2.0.0,>=1.40.0", # Undetected browser automation with Patchright "patchright>=1.40.0,<2.0.0", "webdriver-manager>=4.0.2", ] [tool.uv] dev-dependencies = [ "pytest<8.0.0,>=7.4.3", "pytest-asyncio<1.0.0,>=0.23.5", "ruff<1.0.0,>=0.2.2", "pre-commit<4.0.0,>=3.6.2", "coverage<8.0.0,>=7.4.3", "pytest-testmon>=2.1.3", "sqlacodegen>=3.0.0", "pytest-cov>=6.1.1", ] [tool.pyrefly.errors] import-error = true [tool.poetry.group.dev.dependencies] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.ruff] # General settings line-length = 100 target-version = "py310" respect-gitignore = true exclude = [ ".git", ".ruff_cache", ".venv", "build", "dist", "__pypackages__", "__pycache__", ] [tool.ruff.lint] # More comprehensive selection of rules select = [ "E", # pycodestyle errors "F", # pyflakes "B", # flake8-bugbear "I", # isort "C4", # flake8-comprehensions "UP", # pyupgrade (modernize Python code) "N", # pep8-naming "FA", # flake8-fastapi - FastAPI specific rules "RUF", # Ruff-specific rules "TCH", # Type checking "TID", # Import tidying "SIM", # Code simplification "ERA", # Eradicate commented-out code detection "S", # Security checks (bandit) "D", # Docstring quality (pydocstyle) "ARG", # Unused arguments check "PTH", # Pathlib vs os.path check "A", # Builtins shadowing prevention "PERF",# Performance anti-patterns "C90", # McCabe complexity ] # Ignores that complement pyright ignore = [ # Let pyright handle type checking "ANN001", # Missing type annotation for function argument "ANN002", # Missing type annotation for *args "ANN003", # Missing type annotation for **kwargs "ANN201", # Missing return type annotation for public function "ANN202", # Missing return type annotation for private function "ANN204", # Missing return type annotation for special method "ANN205", # Missing return type annotation for staticmethod "ANN206", # Missing return type annotation for classmethod # Common FastAPI patterns "FBT001", # Boolean positional arg in function definition (common in FastAPI query params) "FBT002", # Boolean default value in function definition # Project-specific needs "N815", # Camel case variables (snake_case conflict in Prisma models) "RUF012", # Mutable class attributes (common in Pydantic models) # Other specific ignores "E501", # Line too long (handled by formatter) # Docstring related ignores "D203", # 1 blank line required before class docstring "D213", # Multi-line docstring summary should start at the second line "D107", # undocumented-public-init "D102", # undocumented-public-method "D205", # Multi-line docstring summary should start at the second line # Additional docstring formatting rules (common in your codebase) "D200", # One-line docstring should fit on one line "D212", # Multi-line docstring summary should start at first line "D415", # First line should end with punctuation # Additional minor issues "D105", # undocumented-magic-method "D202", # blank-line-after-function ] # Allow autofix for most enabled rules fixable = ["ALL"] unfixable = [ "ERA001", # Don't auto-remove commented-out code ] # Per-file overrides for special cases [tool.ruff.lint.per-file-ignores] "__init__.py" = ["F401", "F403"] # Allow unused imports and * imports in __init__ files "app/tests/**/*.py" = ["E501", "S101", "S311", "S105"] # Allow long lines, assert statements, non-crypto random, and test passwords in tests "app/api/**/*.py" = ["B008"] # Function calls in argument defaults (common in FastAPI) "app/models/**/*.py" = ["N815"] # Allow camelCase in model files (for Prisma compatibility) "app/models.py" = ["ALL"] # Ignore all linting issues in autogenerated models.py file # Module import settings [tool.ruff.lint.isort] known-first-party = ["app"] section-order = ["future", "standard-library", "third-party", "first-party", "local-folder"] combine-as-imports = true # Formatting options [tool.ruff.format] quote-style = "double" indent-style = "space" line-ending = "auto" docstring-code-format = true docstring-code-line-length = "dynamic" # Add docstring configuration [tool.ruff.lint.pydocstyle] convention = "google" # Use Google-style docstrings to match your requirements # -- Complexity guardrail ------------------------------------------------------- [tool.ruff.lint.mccabe] max-complexity = 15 # >15 tends to stump LLM refactors; adjust as needed [tool.codespell] skip = "*.svg,Gemfile.lock" write-changes = true # Pyright configuration [tool.pyright] includePackages = ["app"] typeCheckingMode = "strict" reportMissingImports = true reportMissingTypeStubs = false pythonVersion = "3.10" [tool.pyrefly] python-version = "3.10.0" ``` ### src/main.jsx ```javascript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import App from './App.jsx' createRoot(document.getElementById('root')).render( , ) ``` ### backend/app/main.py ```python """Main FastAPI application entry point. Configures and initializes the FastAPI application with routes, middleware, and error handling. Responsible for setting up the application context, including logging, Sentry integration, and database sessions. Initializes and configures the FastAPI application with middleware, error handlers, and routers. Entry point for the backend service. Uses SQLModel metadata to create tables directly instead of Alembic migrations. """ from contextlib import asynccontextmanager from fastapi import FastAPI from fastapi.routing import APIRoute from starlette.middleware.cors import CORSMiddleware from app.api.main import api_router from app.core.config import settings from app.core.log_config import logger def custom_generate_unique_id(route: APIRoute) -> str: """Generate a unique operation ID for OpenAPI documentation. Creates a consistent, readable ID based on the route's tag and name. Used for better OpenAPI documentation and client generation. Args: route: FastAPI route object Returns: Unique operation ID string combining tag and route name """ return f"{route.tags[0]}-{route.name}" @asynccontextmanager async def lifespan(_app: FastAPI): """Application lifespan context. This context manager runs tasks before the application starts, and after it shuts down. """ # Pre-startup initialization task try: logger.info("FastAPI application starting up") # Warm-up database connection to fail fast if unreachable from sqlalchemy import text from app.core.db import engine async with engine.connect() as conn: await conn.execute(text("SELECT 1")) yield finally: # Shutdown tasks logger.info("FastAPI application shutting down") # Cleanup on shutdown is handled in the finally block above def get_application() -> FastAPI: """Create and configure the FastAPI application. Sets up the FastAPI instance with title, version, CORS middleware, and API routers. This factory pattern allows for easier testing and application configuration. Returns: Configured FastAPI application instance """ app = FastAPI( title=settings.PROJECT_NAME, version=settings.VERSION, description=settings.DESCRIPTION, openapi_url=f"{settings.API_V1_STR}/openapi.json", docs_url="/docs", generate_unique_id_function=custom_generate_unique_id, lifespan=lifespan, ) # Set all CORS enabled origins if settings.all_cors_origins: app.add_middleware( CORSMiddleware, allow_origins=settings.all_cors_origins, allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) app.include_router(api_router, prefix=settings.API_V1_STR) return app # Sentry integration is now handled in app.core._logging module # This ensures Sentry is initialized before any log messages are sent logger.info(f"Application initialized in {settings.ENVIRONMENT} environment") app = get_application() ``` ### src/App.jsx ```javascript import { useRef, useState, useEffect, useCallback, useMemo } from 'react'; import './App.css'; // Color palette for different speakers - moved outside component to be truly constant const SPEAKER_COLORS = [ '#3498db', // Blue '#e74c3c', // Red '#2ecc71', // Green '#f39c12', // Orange '#9b59b6', // Purple '#1abc9c', // Turquoise '#34495e', // Dark Gray '#e67e22', // Carrot ]; function App() { // Existing transcription state const [transcriptSegments, setTranscriptSegments] = useState([]); const [isTranscribing, setIsTranscribing] = useState(false); const [activeSpeakers, setActiveSpeakers] = useState(new Set()); const [rawResponses, setRawResponses] = useState([]); const [showRawData, setShowRawData] = useState(false); // New JARVIS state const [personInfo, setPersonInfo] = useState(null); const [isRecognizing, setIsRecognizing] = useState(false); const [recognitionComplete, setRecognitionComplete] = useState(false); const [aiSuggestions, setAiSuggestions] = useState([]); const [isAnalyzing, setIsAnalyzing] = useState(false); const [conversationHistory, setConversationHistory] = useState({ segments: [], lastAnalyzed: 0 }); const [error, setError] = useState(null); const [cameraPermission, setCameraPermission] = useState('pending'); const [capturedImage, setCapturedImage] = useState(null); // Refs const socketRef = useRef(null); const mediaRecorderRef = useRef(null); const streamRef = useRef(null); const analysisIntervalRef = useRef(null); const transcriptBoxRef = useRef(null); const videoRef = useRef(null); const canvasRef = useRef(null); const cameraStreamRef = useRef(null); const rawDataRef = useRef(null); // Keyword detection for "banana" const detectBananaKeyword = useCallback((text) => { const bananaRegex = /\bbanana\b/i; return bananaRegex.test(text); }, []); // Memoize speaker colors to prevent recalculation const speakerColorMap = useMemo(() => { const colorMap = {}; activeSpeakers.forEach(speakerId => { colorMap[speakerId] = SPEAKER_COLORS[speakerId % SPEAKER_COLORS.length]; }); return colorMap; }, [activeSpeakers]); // Capture photo from camera const capturePhoto = useCallback(() => { if (videoRef.current && canvasRef.current) { const video = videoRef.current; const canvas = canvasRef.current; const context = canvas.getContext('2d'); // Set canvas dimensions to match video canvas.width = video.videoWidth; canvas.height = video.videoHeight; // Draw current video frame to canvas context.drawImage(video, 0, 0, canvas.width, canvas.height); // Convert canvas to image data URL const imageDataUrl = canvas.toDataURL('image/jpeg', 0.9); setCapturedImage(imageDataUrl); return imageDataUrl; } return null; }, []); // Real face recognition API with captured image const getPersonInfo = useCallback(async (imageDataUrl = null) => { setIsRecognizing(true); setRecognitionComplete(false); setError(null); try { // If no image provided, capture one const photoData = imageDataUrl || capturePhoto(); if (!photoData) { throw new Error('Failed to capture photo'); } // Convert data URL to blob for API upload const response = await fetch(photoData); const blob = await response.blob(); // Create form data for API request using correct field names const formData = new FormData(); formData.append('file', blob, 'captured-image.jpg'); // Use 'file' as field name formData.append('headless', 'false'); // Add headless parameter formData.append('wait_time', '150'); // Add wait_time parameter // Call real face recognition API with the correct endpoint and multipart format const apiResponse = await fetch('http://localhost:8000/api/v1/automation/upload-image-file', { method: 'POST', // Don't set Content-Type header when sending FormData - browser sets it automatically with boundary body: formData // Send the FormData with the actual captured image }); if (!apiResponse.ok) { throw new Error(`API request failed: ${apiResponse.status}`); } const apiData = await apiResponse.json(); if (!apiData.success) { throw new Error('Face recognition failed'); } // Create person info from API response const personInfo = { id: 'recognized_person', name: 'Recognized Person', job: 'Unknown', company: 'Face Recognition API', bio: 'Person identified through face recognition system', interests: ['Face Recognition', 'AI', 'Computer Vision'], lastMet: 'Just now', notes: `Found ${apiData.count} potential matches in database`, capturedImage: photoData, // Store the captured image apiResponse: apiData // Store the full API response for debugging }; setPersonInfo(personInfo); setRecognitionComplete(true); return personInfo; } catch (err) { setError('Failed to recognize person. Please try again.'); console.error('Face recognition error:', err); } finally { setIsRecognizing(false); } }, [capturePhoto]); // Cache for API responses to reduce redundant calls const apiCacheRef = useRef(new Map()); // OpenRouter API integration with caching and optimization const getConversationSuggestion = useCallback(async (history) => { // Create a cache key based on the last 3 segments and last 2 suggestions const cacheKey = JSON.stringify({ segments: history.segments.slice(-3).map(s => ({ text: s.text, speaker: s.speaker })), suggestions: aiSuggestions.slice(-2).map(s => s.text) }); // Check cache first if (apiCacheRef.current.has(cacheKey)) { const cached = apiCacheR [truncated — 23884 more characters] ``` ### backend/app/api/main.py ```python """ API router configuration and setup. Configures the main API router with all route modules and middleware. Provides customization for FastAPI route generation and documentation. """ from fastapi import APIRouter # APIRoute is not used in this module # Removed login, users routes as per microservice architecture from app.api.routes import automation, pages, private, utils from app.core.config import settings # No prefix here since main.py already adds the /api/v1 prefix api_router = APIRouter() # Authentication handled by external system api_router.include_router(utils.router) # Pages router already has its own prefix and tags, so we don't add them again api_router.include_router(pages.router) # Register private router only in local environment if settings.ENVIRONMENT == "local": # Private router already has its own prefix and tags, so we don't add them again api_router.include_router(private.router) # Automation router for browser automation functionality api_router.include_router(automation.router) ``` [38 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]