# Project export: I-IMO This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Intelligent IRL Meeting Organizer - A conversational AI agent designed with the busiest people in mind. - Devpost: https://devpost.com/software/i-imo - GitHub: https://github.com/satvikprasad/i-imo/ - Video: https://www.youtube.com/embed/fTmvGxSzU3I?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([MLH] Best Use of DigitalOcean Gradientβ„’ AIOpt) - Team: 4 GitHub contributor(s) β€” Satvik Prasad (38 commits), Duy Pham (32 commits), Artem Kim (9 commits), coderabbitai[bot] (1 commits) ## Devpost submission (written by the team) ### Inspiration Ever had an incredibly busy day and can't remember all the people you've met, or all the tasks you've got planned for the future? We're humans! Our memory is imperfect and that's difficult, but it doesn't need to be: so we built I-IMO! I-IMO is a conversational AI agent designed with the busiest people in mind. I-IMO scans both visual input (webcam) and aural input (OMI devkit2) throughout your day to organize and summarize profiles for everyone you meet, while maintaining task lists and tracking important events so you never forget anything important from your meetings. N.B: We originally wanted to do this app on any smart glasses, but unfortunately due to supply constraints we couldn't get one. In the future, the plan is to adapt our project to run off of in-built webcams in devices like the Omi Glass. What is I-IMO? I-IMO is a personal conversational intelligence assistant that is an always-on second-in-command: Captures conversations through real-time audio transcription Recognizes faces via webcam to identify meeting participants Builds personal profiles automatically from conversation context Extracts action items and creates task lists from discussions Maintains conversation history with semantic search capabilities Never lets you forget important details about the people you meet Generates context to infer insightful predictions/aids for upcoming events in your calendar. Our Tech Stack React + Typescript + Vite, FastAPI + Custom YOLO11n model for face detection, Tailwind + Shad/CN UI, Convex Backend, Express.js Backend for NLP, Groq-hosted gpt-whisper for audio transcribing, Digital Ocean AI Gradient for GPU intense task, ChromaDB for vectorised queries and semantic search, OpenAI Text Embedding. Architecture Challenges We Ran Into connection issues 😑, streaming raw audio data from Omi (with low latency), high accuracy face classification that remains consistent (tagging id and tag name with it), deduplication of misspellings of names and mistranscriptions, nature of audio streaming: 5 second chunks clipping certain transcribed words, syncing vectorised ChromaDB with traditional Convex backend, minimising non-deterministic nature of profile summaries and task predictions, Important Lessons Learnt how to integrate a lot of different technologies and choose/route between the best models for unique workloads, Designed ways to sync between multiple backends and frameworks, especially with fundamental infra differences between vectorised semantic-search db's and traditional backends, Balanced the benefits lightweight, fast inference models with the lower completion accuracy, Developed a lot of ad-hoc teamwork skills while working on the same project with time-pressure (minimising merge-conflicts, documenting pull-requests using tools like CodeRabbit, keeping commit tree clean...) What's Next Deep integration with glasses (Meta Raybans, Omi Glass, Snapchat Spectacles), Potentially link 3rd party MCP providers, Connect with social media platforms for deeper context / automated actions based on the user's day, Time-sensitive summaries, Mobile, watch, and native versions of the dashboard. ## README (from the GitHub repository) # I-IMO - Intelligent IRL Meeting Organizer ## If you are applovin judges - please check out this section
Applovin Click Here The readme for applovin's challange is located at `applovin` folder. You can also [click here](https://github.com/satvikprasad/i-imo/tree/master/applovin)
## Inspiration I usually a yapper and talk with a lot of people, but at the end of the day, you look back yourself and can't remember all of them or sometimes you just remember the faces but not what their infomation. Maybe I'm a bad person or I'm just having a bad memory ;D. So we built I-IMO! A conversational AI agent designed with the busiest people in mind. I-IMO scans both visual input (webcam) and oral input (OMI devkit2) throughout your day to organize and summarize profiles for everyone you meet, while maintaining task lists and tracking important events so you never forget anything important from your meetings. - We want to do this app on any smart glasses, but we could not get one. Snapchat spectacles are all reserved and Omi glasses - turn out they are giving out omi devkit (voice only) device. ## What It Does I-IMO is your personal meeting intelligence assistant that: - Captures conversations through real-time audio transcription - Recognizes faces via webcam to identify meeting participants - Builds personal profiles automatically from conversation context - Extracts action items and creates task lists from discussions - Maintains conversation history with semantic search capabilities - Never lets you forget important details about the people you meet ## How We Built It - React + Typescript + Vite - FastAPI + Custom YOLO model for face detection - Tailwind + Shad/CN UI - Convex Backend - Express Backend that process voice - Groq for audio transcribing - Digital Ocean AI Gradient for GPU intense task - ChromaDB for VectorDB and semantic search - OpenAI Text Embedding ## Architecture image ## Challenges we ran into - connection issues - high accuracy face classification (tagging id and tag name with it). - 5 seconds chunk audio cut of certains words transcripting. ## What we learned - We use a lot of technologies and learn how to use it together. - Implements muitple backend with low latency. ## Whats next - Deep intergration with glasses. - Link local profile with linkedin / github and more social medias. - When you wake up, remind you the whole day ahead. ## Detected evidence (automated analysis) Indexed codebase: 97 recognized source files, 616 KB. - CSS (language) β€” detected in the code - Express (technology) β€” detected in the code - FastAPI (technology) β€” detected in the code - HTML (language) β€” detected in the code - JavaScript (language) β€” detected in the code - OpenAI (technology) β€” detected in the code - Python (language) β€” detected in the code - React (technology) β€” detected in the code - Tailwind CSS (technology) β€” detected in the code - TypeScript (language) β€” detected in the code - AI coding agent: Cursor β€” evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (117 of 117) ``` .DS_Store .gitignore applovin/__init__.py applovin/assembler.py applovin/inputs.py applovin/main.py applovin/montior.py applovin/optimized.py applovin/README.md applovin/requirements.txt backend/.DS_Store backend/.gitignore backend/main.ts face_recognition/main.py face_recognition/requirements.txt frontend/.cursor/rules/convex_rules.mdc frontend/.gitattributes frontend/.github/workflows/gitleaks.yml frontend/.github/workflows/lint.yml frontend/.gitignore frontend/biome.json frontend/components.json frontend/convex/_generated/api.d.ts frontend/convex/_generated/api.js frontend/convex/_generated/dataModel.d.ts frontend/convex/_generated/server.d.ts frontend/convex/_generated/server.js frontend/convex/http.ts frontend/convex/images.ts frontend/convex/profile.ts frontend/convex/README.md frontend/convex/schema.ts frontend/convex/task.ts frontend/convex/tsconfig.json frontend/example.env.local frontend/index.html frontend/LICENSE.txt frontend/package.json frontend/README.md frontend/src/components/CameraComponent.tsx frontend/src/components/ui/accordion.tsx frontend/src/components/ui/alert-dialog.tsx frontend/src/components/ui/alert.tsx frontend/src/components/ui/aspect-ratio.tsx frontend/src/components/ui/avatar.tsx frontend/src/components/ui/badge.test.tsx frontend/src/components/ui/badge.tsx frontend/src/components/ui/breadcrumb.tsx frontend/src/components/ui/button-group.tsx frontend/src/components/ui/button.test.tsx frontend/src/components/ui/button.tsx frontend/src/components/ui/calendar.tsx frontend/src/components/ui/card.tsx frontend/src/components/ui/carousel.tsx frontend/src/components/ui/chart.tsx frontend/src/components/ui/checkbox.tsx frontend/src/components/ui/collapsible.tsx frontend/src/components/ui/command.tsx frontend/src/components/ui/context-menu.tsx frontend/src/components/ui/dialog.tsx frontend/src/components/ui/drawer.tsx frontend/src/components/ui/dropdown-menu.tsx frontend/src/components/ui/empty.tsx frontend/src/components/ui/field.tsx frontend/src/components/ui/form.tsx frontend/src/components/ui/hover-card.tsx frontend/src/components/ui/input-group.tsx frontend/src/components/ui/input-otp.tsx frontend/src/components/ui/input.test.tsx frontend/src/components/ui/input.tsx frontend/src/components/ui/item.tsx frontend/src/components/ui/kbd.test.tsx frontend/src/components/ui/kbd.tsx frontend/src/components/ui/label.tsx frontend/src/components/ui/menubar.tsx frontend/src/components/ui/navigation-menu.tsx frontend/src/components/ui/pagination.tsx frontend/src/components/ui/popover.tsx frontend/src/components/ui/progress.tsx frontend/src/components/ui/radio-group.tsx frontend/src/components/ui/resizable.tsx frontend/src/components/ui/scroll-area.tsx frontend/src/components/ui/select.tsx frontend/src/components/ui/separator.tsx frontend/src/components/ui/sheet.tsx frontend/src/components/ui/sidebar.tsx frontend/src/components/ui/skeleton.test.tsx frontend/src/components/ui/skeleton.tsx frontend/src/components/ui/slider.tsx frontend/src/components/ui/sonner.tsx frontend/src/components/ui/spinner.test.tsx frontend/src/components/ui/spinner.tsx frontend/src/components/ui/switch.tsx frontend/src/components/ui/table.tsx frontend/src/components/ui/tabs.tsx frontend/src/components/ui/textarea.tsx frontend/src/components/ui/toggle-group.tsx frontend/src/components/ui/toggle.tsx frontend/src/components/ui/tooltip.tsx frontend/src/index.css frontend/src/lib/utils.test.ts frontend/src/lib/utils.ts frontend/src/main.tsx frontend/src/pages/index.tsx frontend/src/pages/pages-sc/index.tsx frontend/src/vite-env.d.ts frontend/TEST_SETUP.md frontend/TESTING_SUMMARY.md frontend/tsconfig.app.json frontend/tsconfig.json frontend/tsconfig.node.json frontend/vite.config.ts frontend/vitest.config.ts package.json Procfile README.md tsconfig.json ``` ### Dependencies - applovin/requirements.txt: duckdb@>=1.1.1, pandas@>=2.2.0, psutil - face_recognition/requirements.txt: albucore@==0.0.24, albumentations@==2.0.8, annotated-doc@==0.0.3, annotated-types@==0.7.0, anyio@==4.11.0, certifi@==2025.10.5, charset-normalizer@==3.4.4, click@==8.3.0, coloredlogs@==15.0.1, contourpy@==1.3.3, cycler@==0.12.1, Cython@==3.1.6, easydict@==1.13, fastapi@==0.120.0, flatbuffers@==25.9.23, fonttools@==4.60.1, h11@==0.16.0, humanfriendly@==10.0, idna@==3.11, imageio@==2.37.0, insightface@==0.7.3, joblib@==1.5.2, kiwisolver@==1.4.9, lazy_loader@==0.4, matplotlib@==3.10.7, ml_dtypes@==0.5.3, mpmath@==1.3.0, networkx@==3.5, numpy@==2.2.6, onnx@==1.19.1, onnxruntime@==1.23.2, opencv-python@==4.12.0.88, opencv-python-headless@==4.12.0.88, packaging@==25.0, pillow@==12.0.0, prettytable@==3.16.0, protobuf@==6.33.0, pydantic@==2.12.3, pydantic_core@==2.41.4, pyparsing@==3.2.5, python-dateutil@==2.9.0.post0, PyYAML@==6.0.3, requests@==2.32.5, scikit-image@==0.25.2, scikit-learn@==1.7.2, scipy@==1.16.2, simsimd@==6.5.3, six@==1.17.0, sniffio@==1.3.1, starlette@==0.48.0, stringzilla@==4.2.1, sympy@==1.14.0, threadpoolctl@==3.6.0, tifffile@==2025.10.16, tqdm@==4.67.1, typing_extensions@==4.15.0, typing-inspection@==0.4.2, urllib3@==2.5.0, uvicorn@==0.38.0, wcwidth@==0.2.14 - frontend/package.json: @auth0/auth0-react@^2.7.0, @biomejs/biome@2.2.6, @chroma-core/default-embed@^0.1.8, @heroicons/react@^2.2.0, @hookform/resolvers@^5.2.2, @radix-ui/react-accordion@^1.2.12, @radix-ui/react-alert-dialog@^1.1.15, @radix-ui/react-aspect-ratio@^1.1.7, @radix-ui/react-avatar@^1.1.10, @radix-ui/react-checkbox@^1.3.3, @radix-ui/react-collapsible@^1.1.12, @radix-ui/react-context-menu@^2.2.16, @radix-ui/react-dialog@^1.1.15, @radix-ui/react-dropdown-menu@^2.1.16, @radix-ui/react-hover-card@^1.1.15, @radix-ui/react-label@^2.1.7, @radix-ui/react-menubar@^1.1.16, @radix-ui/react-navigation-menu@^1.2.14, @radix-ui/react-popover@^1.1.15, @radix-ui/react-progress@^1.1.7, @radix-ui/react-radio-group@^1.3.8, @radix-ui/react-scroll-area@^1.2.10, @radix-ui/react-select@^2.2.6, @radix-ui/react-separator@^1.1.7, @radix-ui/react-slider@^1.3.6, @radix-ui/react-slot@^1.2.3, @radix-ui/react-switch@^1.2.6, @radix-ui/react-tabs@^1.1.13, @radix-ui/react-toggle@^1.1.10, @radix-ui/react-toggle-group@^1.1.11, @radix-ui/react-tooltip@^1.2.8, @tailwindcss/vite@^4.0.14, @types/node@^22.13.10, @types/react@^19.0.10, @types/react-dom@^19.0.4, @vitejs/plugin-react@^5.0.0, chromadb@^3.0.17, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@^1.1.1, convex@^1.27.0, date-fns@^4.1.0, embla-carousel-react@^8.6.0, fastest-levenshtein@^1.0.16, globals@^15.15.0, input-otp@^1.4.2, lucide-react@^0.545.0, next-themes@^0.4.6, npm-run-all@^4.1.5, prettier@^3.5.3, react@^19.0.0, react-day-picker@^9.11.1, react-dom@^19.0.0, react-hook-form@^7.65.0, react-resizable-panels@^3.0.6, react-router@^7.9.4, react-router-dom@^7.9.4, react-webcam@^7.2.0, recharts@2.15.4, sonner@^2.0.7, tailwind-merge@^3.3.1, tailwindcss@^4.0.14, tw-animate-css@^1.4.0, typescript@~5.7.2, vaul@^1.1.2, vite@^6.2.0, vite-plugin-pages@^0.33.1, vitest@^4.0.3, zod@^4.1.12 - package.json: @chroma-core/default-embed@^0.1.8, @chroma-core/openai@^0.1.7, @heroicons/react@^2.2.0, @types/cors@^2.8.19, @types/express@^5.0.4, @types/node@^22.13.10, chromadb@^3.0.17, cors@^2.8.5, dotenv@^17.2.3, express@^5.1.0, openai@^6.7.0, typescript@^5.9.3 ### Recent commits (newest first) - Merge branch 'master' of github.com:satvikprasad/imo-calhacks-12.0 - Reduced audio chunk window length - Update README.md - Merge branch 'master' of https://github.com/satvikprasad/imo-calhacks - add monitor script - update url - Update README.md - Update README.md - Merge branch 'master' of github.com:satvikprasad/imo-calhacks-12.0 - typo - Merge branch 'master' of https://github.com/satvikprasad/imo-calhacks - Merge branch 'master' of https://github.com/satvikprasad/imo-calhacks - Update images.ts - Removed pnpm - Merge branch 'master' of https://github.com/satvikprasad/imo-calhacks - Update README.md - Merge pull request #15 from satvikprasad/calendar - calendar + images loading - Merge branch 'master' of github.com:satvikprasad/imo-calhacks-12.0 - Added regenerate tasks button ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/TESTING_SUMMARY.md ```markdown # Testing Summary - Comprehensive Test Suite Added ## What Was Added A complete testing infrastructure for the React + TypeScript frontend application, including configuration, setup files, and over **200 comprehensive test cases**. ## Files Created ### Configuration & Setup (2 files) 1. **vitest.config.ts** - Vitest configuration with React, jsdom, coverage settings 2. **src/test/setup.ts** - Global test setup with mocks and matchers ### Test Files (8 files, 200+ tests) | Test File | Test Count | Coverage | |-----------|------------|----------| | `src/lib/utils.test.ts` | 50+ | `cn()` utility - class merging, Tailwind merge, edge cases | | `src/hooks/use-mobile.test.ts` | 40+ | `useIsMobile` hook - responsive behavior, lifecycle, memory | | `src/components/ui/button.test.tsx` | 50+ | Button variants, sizes, states, accessibility, events | | `src/components/ui/input.test.tsx` | 60+ | Input types, validation, interaction, accessibility | | `src/components/ui/badge.test.tsx` | 30+ | Badge variants, asChild, styling, props | | `src/components/ui/spinner.test.tsx` | 15+ | Spinner rendering, ARIA, styling | | `src/components/ui/skeleton.test.tsx` | 20+ | Skeleton animation, use cases, accessibility | | `src/components/ui/kbd.test.tsx` | 40+ | Kbd & KbdGroup, keyboard shortcuts, styling | ### Documentation (2 files) 1. **TEST_SETUP.md** - Comprehensive guide for setup, running tests, patterns 2. **TESTING_SUMMARY.md** - This file ## Test Coverage Areas βœ… **Utility Functions** - Pure function testing - Edge case handling - Tailwind CSS class merging - Performance consistency βœ… **React Hooks** - Custom hook behavior - State management - Lifecycle and cleanup - Memory leak prevention - Concurrent usage βœ… **UI Components** - Rendering and structure - Props and variants - Styling and class merging - Component composition (asChild) βœ… **User Interactions** - Click events - Keyboard input - Focus management - Form submission - Tab navigation βœ… **Accessibility** - ARIA roles and labels - Screen reader support - Keyboard navigation - Semantic HTML - Focus management βœ… **Edge Cases** - Empty/null/undefined values - Very long inputs - Special characters & Unicode - Extreme screen sizes - Rapid state changes ## Installation Required ```bash cd frontend pnpm add -D vitest @vitest/ui @vitest/coverage-v8 jsdom \ @testing-library/react @testing-library/jest-dom @testing-library/user-event ``` ## Package.json Updates Needed Add these scripts to `frontend/package.json`: ```json { "scripts": { "test": "vitest", "test:ui": "vitest --ui", "test:run": "vitest run", "test:coverage": "vitest run --coverage" } } ``` ## Running Tests ```bash # Watch mode (default) pnpm test # Run once pnpm test:run # UI mode pnpm test:ui # With coverage pnpm test:coverage ``` ## Key Testing Technologies - **Vitest** - Fast Vite-native test runner - **React Testing Library** - Component testing utilities - **@testing-library/user-event** - Realistic [truncated β€” 1662 more characters] ``` ### frontend/TEST_SETUP.md ```markdown # Test Suite Setup and Documentation This document describes the comprehensive test suite added to the frontend project. ## Overview A complete testing infrastructure has been added using **Vitest** and **React Testing Library**. The test suite includes over 200+ test cases covering utilities, hooks, and UI components. ## Installation Install the required testing dependencies: ```bash pnpm add -D vitest @vitest/ui @vitest/coverage-v8 jsdom @testing-library/react @testing-library/jest-dom @testing-library/user-event ``` ## Configuration Files ### `vitest.config.ts` Main Vitest configuration with: - React plugin integration - jsdom environment for DOM testing - Path aliases matching the project structure - Coverage configuration - Test setup file ### `src/test/setup.ts` Global test setup including: - React Testing Library matchers - Cleanup after each test - Mock implementations for window.matchMedia - Mock implementations for IntersectionObserver - Mock implementations for ResizeObserver ## Test Scripts Add these scripts to `package.json`: ```json { "scripts": { "test": "vitest", "test:ui": "vitest --ui", "test:run": "vitest run", "test:coverage": "vitest run --coverage" } } ``` ### Running Tests ```bash # Run tests in watch mode pnpm test # Run tests once pnpm test:run # Run tests with UI pnpm test:ui # Run tests with coverage report pnpm test:coverage ``` ## Test Files Created ### 1. **src/lib/utils.test.ts** (50+ tests) Comprehensive testing of the `cn()` utility function including: - Basic functionality (class merging, empty values, conditionals) - Array handling (nested arrays, conditionals in arrays) - Tailwind CSS merge functionality (conflicting classes, variants, responsive) - Edge cases (no arguments, many arguments, duplicates, whitespace) - Real-world component scenarios - Performance and consistency tests ### 2. **src/hooks/use-mobile.test.ts** (40+ tests) Thorough testing of the `useIsMobile` hook: - Initialization at different screen widths - Responsive behavior on window resize - Edge cases (small/large screens, zero/negative widths) - Cleanup and memory management - Concurrent usage with multiple instances - matchMedia query correctness - Boolean coercion behavior ### 3. **src/components/ui/spinner.test.tsx** (15+ tests) Testing the Spinner component: - Rendering and ARIA roles - Styling (animation, size, custom classes) - Accessibility (screen readers, ARIA labels) - Props forwarding - Edge cases ### 4. **src/components/ui/skeleton.test.tsx** (20+ tests) Testing the Skeleton component: - Rendering and structure - Styling (animation, background, custom classes) - Use cases (text, circular, card skeletons) - Props forwarding - Accessibility - Edge cases ### 5. **src/components/ui/badge.test.tsx** (30+ tests) Testing the Badge component: - Rendering and children - Variants (default, secondary, destructive, outline) - asChild prop functionality - Styling and class merging - Icon integration [truncated β€” 5337 more characters] ``` ### package.json ``` { "name": "imo", "version": "1.0.0", "description": "", "license": "ISC", "author": "", "type": "commonjs", "main": "index.js", "scripts": { "build": "tsc" }, "devDependencies": { "@types/cors": "^2.8.19", "@types/express": "^5.0.4", "@types/node": "^22.13.10", "typescript": "^5.9.3" }, "dependencies": { "@chroma-core/default-embed": "^0.1.8", "@chroma-core/openai": "^0.1.7", "@heroicons/react": "^2.2.0", "chromadb": "^3.0.17", "cors": "^2.8.5", "dotenv": "^17.2.3", "express": "^5.1.0", "openai": "^6.7.0" } } ``` ### applovin/requirements.txt ``` duckdb>=1.1.1 pandas>=2.2.0 psutil ``` ### face_recognition/requirements.txt ``` albucore==0.0.24 albumentations==2.0.8 annotated-doc==0.0.3 annotated-types==0.7.0 anyio==4.11.0 certifi==2025.10.5 charset-normalizer==3.4.4 click==8.3.0 coloredlogs==15.0.1 contourpy==1.3.3 cycler==0.12.1 Cython==3.1.6 easydict==1.13 fastapi==0.120.0 flatbuffers==25.9.23 fonttools==4.60.1 h11==0.16.0 humanfriendly==10.0 idna==3.11 imageio==2.37.0 insightface==0.7.3 joblib==1.5.2 kiwisolver==1.4.9 lazy_loader==0.4 matplotlib==3.10.7 ml_dtypes==0.5.3 mpmath==1.3.0 networkx==3.5 numpy==2.2.6 onnx==1.19.1 onnxruntime==1.23.2 opencv-python==4.12.0.88 opencv-python-headless==4.12.0.88 packaging==25.0 pillow==12.0.0 prettytable==3.16.0 protobuf==6.33.0 pydantic==2.12.3 pydantic_core==2.41.4 pyparsing==3.2.5 python-dateutil==2.9.0.post0 PyYAML==6.0.3 requests==2.32.5 scikit-image==0.25.2 scikit-learn==1.7.2 scipy==1.16.2 simsimd==6.5.3 six==1.17.0 sniffio==1.3.1 starlette==0.48.0 stringzilla==4.2.1 sympy==1.14.0 threadpoolctl==3.6.0 tifffile==2025.10.16 tqdm==4.67.1 typing-inspection==0.4.2 typing_extensions==4.15.0 urllib3==2.5.0 uvicorn==0.38.0 wcwidth==0.2.14 ``` ### frontend/package.json ``` { "name": "fullstack-react-convex", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "npm-run-all --parallel dev:frontend dev:backend", "dev:frontend": "vite --open", "dev:backend": "convex dev", "predev": "convex dev --until-success", "build": "tsc -b && vite build", "lint": "tsc && eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0", "preview": "vite preview" }, "dependencies": { "@auth0/auth0-react": "^2.7.0", "@heroicons/react": "^2.2.0", "@chroma-core/default-embed": "^0.1.8", "@hookform/resolvers": "^5.2.2", "@radix-ui/react-accordion": "^1.2.12", "@radix-ui/react-alert-dialog": "^1.1.15", "@radix-ui/react-aspect-ratio": "^1.1.7", "@radix-ui/react-avatar": "^1.1.10", "@radix-ui/react-checkbox": "^1.3.3", "@radix-ui/react-collapsible": "^1.1.12", "@radix-ui/react-context-menu": "^2.2.16", "@radix-ui/react-dialog": "^1.1.15", "@radix-ui/react-dropdown-menu": "^2.1.16", "@radix-ui/react-hover-card": "^1.1.15", "@radix-ui/react-label": "^2.1.7", "@radix-ui/react-menubar": "^1.1.16", "@radix-ui/react-navigation-menu": "^1.2.14", "@radix-ui/react-popover": "^1.1.15", "@radix-ui/react-progress": "^1.1.7", "@radix-ui/react-radio-group": "^1.3.8", "@radix-ui/react-scroll-area": "^1.2.10", "@radix-ui/react-select": "^2.2.6", "@radix-ui/react-separator": "^1.1.7", "@radix-ui/react-slider": "^1.3.6", "@radix-ui/react-slot": "^1.2.3", "@radix-ui/react-switch": "^1.2.6", "@radix-ui/react-tabs": "^1.1.13", "@radix-ui/react-toggle": "^1.1.10", "@radix-ui/react-toggle-group": "^1.1.11", "@radix-ui/react-tooltip": "^1.2.8", "chromadb": "^3.0.17", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "^1.1.1", "convex": "^1.27.0", "date-fns": "^4.1.0", "embla-carousel-react": "^8.6.0", "fastest-levenshtein": "^1.0.16", "input-otp": "^1.4.2", "lucide-react": "^0.545.0", "next-themes": "^0.4.6", "react": "^19.0.0", "react-day-picker": "^9.11.1", "react-dom": "^19.0.0", "react-hook-form": "^7.65.0", "react-resizable-panels": "^3.0.6", "react-router": "^7.9.4", "react-router-dom": "^7.9.4", "react-webcam": "^7.2.0", "recharts": "2.15.4", "sonner": "^2.0.7", "tailwind-merge": "^3.3.1", "vaul": "^1.1.2", "vitest": "^4.0.3", "zod": "^4.1.12" }, "devDependencies": { "@biomejs/biome": "2.2.6", "@tailwindcss/vite": "^4.0.14", "@types/node": "^22.13.10", "@types/react": "^19.0.10", "@types/react-dom": "^19.0.4", "@vitejs/plugin-react": "^5.0.0", "globals": "^15.15.0", "npm-run-all": "^4.1.5", "prettier": "^3.5.3", "tailwindcss": "^4.0.14", "tw-animate-css": "^1.4.0", "typescript": "~5.7.2", "vite": "^6.2.0", "vite-plugin-pages": "^0.33.1" }, "pnpm": { "onlyBuiltDependencies": [ "@tailwindcss/oxide", "browser-tabs-lock", "esbuild", "onnxruntime-node", "protobufjs", "sharp" ] } } ``` ### applovin/main.py ```python #!/usr/bin/env python3 """ DuckDB Baseline Benchmark Demo ------------------------------ Reads data from a given folder (CSV or Parquet), adds derived day/minute columns, executes JSON queries, and reports timings. Usage: python main.py --data-dir ./data --out-dir ./out """ import duckdb import time from pathlib import Path import csv import argparse from assembler import assemble_sql from inputs import queries # from judges import queries # ------------------- # Configuration # ------------------- DB_PATH = Path("tmp/baseline.duckdb") TABLE_NAME = "events" # ------------------- # Load Data # ------------------- def load_data(con, data_dir: Path): csv_files = list(data_dir.glob("events_part_*.csv")) if csv_files: print(f"🟩 Loading {len(csv_files)} CSV parts from {data_dir} ...") con.execute(f""" CREATE OR REPLACE VIEW {TABLE_NAME} AS WITH raw AS ( SELECT * FROM read_csv( '{data_dir}/events_part_*.csv', AUTO_DETECT = FALSE, HEADER = TRUE, union_by_name = TRUE, COLUMNS = {{ 'ts': 'VARCHAR', 'type': 'VARCHAR', 'auction_id': 'VARCHAR', 'advertiser_id': 'VARCHAR', 'publisher_id': 'VARCHAR', 'bid_price': 'VARCHAR', 'user_id': 'VARCHAR', 'total_price': 'VARCHAR', 'country': 'VARCHAR' }} ) ), casted AS ( SELECT to_timestamp(TRY_CAST(ts AS DOUBLE) / 1000.0) AS ts, type, auction_id, TRY_CAST(advertiser_id AS INTEGER) AS advertiser_id, TRY_CAST(publisher_id AS INTEGER) AS publisher_id, NULLIF(bid_price, '')::DOUBLE AS bid_price, TRY_CAST(user_id AS BIGINT) AS user_id, NULLIF(total_price, '')::DOUBLE AS total_price, country FROM raw ) SELECT ts, DATE_TRUNC('week', ts) AS week, DATE(ts) AS day, DATE_TRUNC('hour', ts) AS hour, STRFTIME(ts, '%Y-%m-%d %H:%M') AS minute, type, auction_id, advertiser_id, publisher_id, bid_price, user_id, total_price, country FROM casted; """) print(f"🟩 Loading complete") else: raise FileNotFoundError(f"No events_part_*.csv found in {data_dir}") # ------------------- # Run Queries # ------------------- def run(queries, data_dir: Path, out_dir: Path): # Ensure directories exist DB_PATH.parent.mkdir(parents=True, exist_ok=True) out_dir.mkdir(parents=True, exist_ok=True) con = duckdb.connect(DB_PATH) load_data(con, data_dir) out_dir.mkdir(parents=True, exist_ok=True) results = [] for i, q in enumerate(queries, 1): sql = assemble_sql(q) print(f"\n🟦 Query {i}:\n{q}\n") t0 = time.time() res = con.execute(sql) cols = [d[0] for d in res.description] rows = res.fetchall() dt = time.time() - t0 print(f"βœ… Rows: {len(rows)} | Time: {dt:.3f}s") out_path = out_dir / f"q{i}.csv" with out_path.open("w", newline="") as f: w = csv.writer(f) w.writerow(cols) w.writerows(rows) results.append({"query": i, "rows": len(rows), "time": dt}) con.close() print("\nSummary:") for r in results: print(f"Q{r['query']}: {r['time']:.3f}s ({r['rows']} rows)") print(f"Total time: {sum(r['time'] for r in results):.3f}s") # ------------------- # Main Entry Point # ------------------- if __name__ == "__main__": parser = argparse.ArgumentParser( description="DuckDB Baseline Benchmark Demo β€” runs benchmark queries on input CSV data." ) parser.add_argument( "--data-dir", type=Path, required=True, help="The folder where the input CSV is provided" ) parser.add_argument( "--out-dir", type=Path, required=True, help="Where to output query results-full" ) args = parser.parse_args() run(queries, args.data_dir, args.out_dir) ``` ### face_recognition/main.py ```python # app.py import base64 import io import json import os import urllib.request from contextlib import asynccontextmanager from typing import List, Optional import numpy as np from fastapi.middleware.cors import CORSMiddleware from fastapi import FastAPI, HTTPException import uvicorn from pydantic import BaseModel from PIL import Image from insightface.app import FaceAnalysis # ----------------------------- # Config # ----------------------------- THRESHOLD = 0.65 MODEL_NAME = "buffalo_l" PROVIDERS = ["CPUExecutionProvider"] # or ["CUDAExecutionProvider"] IMAGE_PATH = "output.jpg" # always overwritten ONLY_CLOSEST_FACE = True class ConvexClient: def __init__(self, base_url, create_path, get_all_path, auth_header=None): self.base_url = base_url.rstrip("/") self.create_url = self.base_url + create_path self.get_all_url = self.base_url + get_all_path self.auth_header = auth_header def get_all_embeddings(self) -> List[List[float]]: # Uses GET req = urllib.request.Request(self.get_all_url, method="GET") if self.auth_header: req.add_header("Authorization", self.auth_header) with urllib.request.urlopen(req) as resp: return json.loads(resp.read().decode("utf-8")) def create_person(self, emb: np.ndarray, label: str, image_bytes: bytes, thumb_b64: Optional[str]) -> dict: """ POST JSON to Convex: { "emb": [float...], "label": "name", "image": "", "thumbnail": "" # face crop (optional) } """ image_b64 = base64.b64encode(image_bytes).decode("utf-8") payload = { "emb": np.asarray(emb, np.float32).tolist(), "label": label, "image": image_b64, "thumbnail": thumb_b64, } req = urllib.request.Request(self.create_url, method="POST") req.add_header("Content-Type", "application/json") if self.auth_header: req.add_header("Authorization", self.auth_header) with urllib.request.urlopen(req, data=json.dumps(payload).encode("utf-8")) as resp: response = json.loads(resp.read().decode("utf-8")) # You returned the whole response in your version; keep that behavior return response # ----------------------------- # Helpers # ----------------------------- def read_image_bytes_to_bgr(image_bytes: bytes) -> np.ndarray: img = Image.open(io.BytesIO(image_bytes)).convert("RGB") return np.array(img)[:, :, ::-1] # RGB β†’ BGR def unit_norm(v: np.ndarray) -> np.ndarray: v = v.astype(np.float32) n = np.linalg.norm(v) return v / n if n else v def best_face_index(faces) -> int: best, best_key = -1, (-1.0, -1.0) for i, f in enumerate(faces): score = float(getattr(f, "det_score", 0.0)) x1, y1, x2, y2 = f.bbox.astype(int) area = float(max(0, x2 - x1) * max(0, y2 - y1)) key = (score, area) if key > best_key: best, best_key = i, key return best def clamp(val: int, lo: int, hi: int) -> int: return max(lo, min(hi, val)) def face_thumb_b64_from_bgr(bgr: np.ndarray, bbox_np: np.ndarray, max_side: int = 160, jpeg_quality: int = 85, expand: float = 0.2) -> str: """ Create a base64 JPEG thumbnail (face crop) from BGR image and bbox. - Expands bbox by `expand` (e.g., 0.2 = 20%) - Keeps aspect ratio, scales so max(H, W) = max_side (if larger) - Returns base64 string (no data URL prefix) """ H, W = bgr.shape[:2] x1, y1, x2, y2 = [float(v) for v in bbox_np] # Expand bbox by given percentage w = x2 - x1 h = y2 - y1 pad_x = w * expand / 2 pad_y = h * expand / 2 x1 -= pad_x x2 += pad_x y1 -= pad_y y2 += pad_y # Clamp bbox inside image x1 = max(0, int(round(x1))) y1 = max(0, int(round(y1))) x2 = min(W, int(round(x2))) y2 = min(H, int(round(y2))) # Fallback if invalid box if x2 <= x1 or y2 <= y1: crop_rgb = bgr[:, :, ::-1] else: crop_bgr = bgr[y1:y2, x1:x2] crop_rgb = crop_bgr[:, :, ::-1] # convert to RGB pil_img = Image.fromarray(crop_rgb) w, h = pil_img.size scale = min(1.0, max_side / float(max(w, h))) if max(w, h) > 0 else 1.0 if scale < 1.0: pil_img = pil_img.resize((int(round(w * scale)), int(round(h * scale))), Image.LANCZOS) buf = io.BytesIO() pil_img.save(buf, format="JPEG", quality=jpeg_quality, optimize=True) thumb_b64 = base64.b64encode(buf.getvalue()).decode("utf-8") return thumb_b64 # ----------------------------- # FastAPI app with lifespan # ----------------------------- class ImageModel(BaseModel): image: str # data URL (e.g. "data:image/jpeg;base64,...") class DetectModel(BaseModel): name: str threshold: Optional[float] = None @asynccontextmanager async def lifespan(app: FastAPI): # ---- Startup ---- global insight, db insight = FaceAnalysis(name=MODEL_NAME, providers=PROVIDERS) insight.prepare(ctx_id=-1) db = ConvexClient( base_url="https://stoic-cat-389.convex.site", create_path="/upload", # your JSON POST endpoint get_all_path="/getEmbeddings", # your GET endpoint returning [[...], ...] auth_header=None, ) print("βœ… InsightFace and Convex client initialized") yield print("πŸ›‘ Shutting down") app = FastAPI(title="Face Upsert (single image: output.jpg)", lifespan=lifespan) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # 1) Save the last photo as output.jpg @app.post("/upload/") async def upload_image(img: ImageModel): try: _, encoded = img.image.split(",", 1) binary_data = base64.b64decode(encoded) except Exception: raise HTTPException(status_code=400, detail="Invalid base64 image data [truncated β€” 1695 more characters] ``` ### backend/main.ts ```typescript import express, { Request, Response } from "express"; import OpenAI, { toFile } from "openai"; import dotenv from "dotenv"; import { CloudClient as ChromaClient } from "chromadb"; import { OpenAIEmbeddingFunction } from "@chroma-core/openai"; import cors from "cors"; // Source env dotenv.config(); const client = new OpenAI({ apiKey: process.env.GROQ_API_KEY, baseURL: "https://api.groq.com/openai/v1", }); const chromaClient = new ChromaClient({ apiKey: process.env.CHROMA_API_KEY, tenant: process.env.CHROMA_TENANT, database: "omi", }); const app = express(); app.use(cors()); app.get("/", async (req: Request, res: Response) => { res.send(200); }); app.use( express.raw({ type: "application/octet-stream", limit: "50mb", }) ); interface TranscriptionSession { chunks: Buffer[]; transcripts: string[]; lastActivity: number; } interface ParsedNameResponse { isSpeakingToPerson: boolean; name: string | null; confidence: "HIGH" | "MEDIUM" | "LOW"; evidence: string; } const sessions = new Map(); function pcmToWav(pcm: Buffer, sampleRate: number) { const header = Buffer.alloc(44); // RIFF header header.write("RIFF", 0); header.writeUInt32LE(36 + pcm.length, 4); header.write("WAVE", 8); // fmt chunk header.write("fmt ", 12); header.writeUInt32LE(16, 16); // PCM header.writeUInt16LE(1, 20); // PCM format header.writeUInt16LE(1, 22); // Mono header.writeUInt32LE(sampleRate, 24); // Sample rate header.writeUInt32LE(sampleRate * 2, 28); // Byte rate (sampleRate * channels * bitsPerSample/8) header.writeUInt16LE(2, 32); // Block align header.writeUInt16LE(16, 34); // 16-bit // data chunk header.write("data", 36); header.writeUInt32LE(pcm.length, 40); return Buffer.concat([header, pcm]); } app.post("/omi/audio", async (req: Request, res: Response) => { const { sample_rate: sampleRate, uid } = req.query; const sr = Number(sampleRate); if (!uid || typeof uid != "string") { return res.status(400).json({ error: "Missing uid" }); } if (!sampleRate || Number.isNaN(sr) || sr <= 0) { res.status(400).json({ error: "Invalid sample_rate", }); return; } const octetData: Buffer = req.body; if (octetData instanceof Buffer) { try { if (!sessions.has(uid)) { sessions.set(uid, { chunks: [], transcripts: [], lastActivity: Date.now(), }); } const session = sessions.get(uid)!; session.chunks.push(octetData); session.lastActivity = Date.now(); if (session.chunks.length > 3) { session.chunks.shift(); } const combinedPcm = Buffer.concat(session.chunks); const wavFile = pcmToWav( combinedPcm, parseInt(sampleRate as string) ); const file = await toFile(wavFile, "audio.wav", { type: "audio/wav", }); const transcription = await client.audio.transcriptions.create({ file: file, model: "whisper-large-v3-turbo", response_format: "json", language: "en", temperature: 0.0, }); const completion = await client.chat.completions.create({ messages: [ { role: "system", content: `Analyze conversations to detect if the user is speaking to a real person and extract that person\'s name. Return ONLY valid JSON: { "isSpeakingToPerson": true/false, "name": [detected name]/null, "confidence": "HIGH"/"MEDIUM"/"LOW", "evidence": "brief reason" }. DON'T format it as a code block, just raw text. Prioritise information at the end of the conversation rather than the beginning.`, }, { role: "user", content: `Conversation:\n${transcription.text}`, }, ], model: "llama-3.3-70b-versatile", }); const collection = await chromaClient.getOrCreateCollection({ name: `transcriptions`, embeddingFunction: new OpenAIEmbeddingFunction({ modelName: "text-embedding-3-small", apiKey: process.env.OPENAI_API_KEY, }), }); await collection.upsert({ documents: [transcription.text], ids: [Date.now().toString()], }); const result = JSON.parse( completion.choices[0].message.content ?? "{}" ) as ParsedNameResponse; console.log( `${result.name}: ${result.confidence} [${transcription.text}]` ); if (result.name && result.confidence == "HIGH") { try { const response = await fetch( "http://167.99.189.49:8000/detect", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ name: result.name, }), } ); console.log(await response.json()); [truncated β€” 10532 more characters] ``` ### frontend/src/main.tsx ```typescript import { ConvexReactClient, ConvexProvider } from 'convex/react'; import { StrictMode } from 'react'; import { createRoot } from 'react-dom/client'; import './index.css'; import { Suspense } from 'react'; import { BrowserRouter, useRoutes } from 'react-router-dom'; import routes from '~react-pages'; const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string); function App() { return ( Loading...}>{useRoutes(routes)} ); } createRoot(document.getElementById('root') as HTMLElement).render( , ); ``` [85 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]