# Project export: Blinket This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: We allow those with ALS to interact with the internet through an OS controllable only by the eyes. - Devpost: https://devpost.com/software/renaissance-845het - GitHub: https://github.com/idalmas/Blinkit-med - Demo: https://seattle-sepia.vercel.app/apps - Video: https://www.youtube.com/embed/NoRGMwNrWNU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Pranav (61 commits), Claude Opus 4.6 (32 commits), idalmas (23 commits), Yash Dagade (5 commits) ## Devpost submission (written by the team) ### Inspiration ALS is a devastating neurodegenerative disease in which one gradually looses motor function over time, for which there is no known cure. However, those experiencing ALS retain control of their eyes, which can be used to speak and interact with. the world. There exist solutions for this (e.g. https://us.tobiidynavox.com/products/td-i-series), however these are often $7000-$1500+. And these solutions do not allows those with ALS to interface with the internet. We wanted to use LLMs, software, and cheap hardware to build an interface to allow those with ALS to communicate in-person, online, and on the internet in a way that is radically cheaper (<$200). ### What it does Blinkit is a web app which can be entirely controlled either in (1) Blink mode using CV, or (2) EOG Mode using electrodes. Our web app allows users to communicate easily with others and interface with the internet. We took an OS approach and built an app store. We currently support: Amazon, ChatGPT, Google Maps, Web Search, Flappy Bird, Books, Talk and Zoom. To increase communication speed, beyond using the combination of blinking and EOG, our apps use long running context to provide personalized suggestions with LLMs. For the purposes of this demo, we processed the entire script of the Harry Potter movies, and used all instances of Hagrid talking as context. So our model would speak and make suggestions in the same way Hagrid does, preserving an ALS patient's true voice ### How we built it There is a frontend, backend, and hardware component. We used Elasticsearch as our vector DB for long running context, and we used BrightData to collect data for some of our apps, such as Amazon. We also allow those with ALS to communicate Hardware: We place 3 electrodes on a person's face. Left, right, and center. We can pick up with 1–3 millivolt differences, which are identified through a denoising and amplification circuit. Frontend: We built an intuitive frontend built around left/right eye movement (or blinks), double blinks, and triple blinks, allowing one to navigate through Amazon pages, participate in conversation, Play Flappy bird, read classic novels, read pages on the web, and more. Backend: We used Cerebras with GPT-oss for responsive inference. We also used serverless Elastic Cloud to host large amounts of context to inform application suggestions through kNN cosine similarity search. ### Challenges we ran into Tuning the blinking was very difficult Each person has unique electrical signals. Tuning that in onboarding was very challenging so was coming up with the circuit diagram ### What's next for Blinkit ## README (from the GitHub repository) # Blinket **Hands-free internet access for people with ALS and motor impairments.** Blinket is an assistive technology platform that lets users navigate the internet, shop, search, read, chat, join video calls, and have conversations — all controlled entirely through eye blinks and winks detected via a standard webcam. **Live demo:** [seattle-sepia.vercel.app](https://seattle-sepia.vercel.app) **Backend:** [revive-1-ef6k.onrender.com](https://revive-1-ef6k.onrender.com) --- ## Why We Built This One of our teammates spent a summer in South Africa working directly with patients living with ALS. He saw firsthand how these patients — sharp, intelligent people with full cognitive ability — were effectively locked out of the internet. The tools that did exist, like the Tobii Dynavox, cost between **$6,000 and $15,000**, putting them out of reach for most families, especially in developing countries. We built Blinket to change that. Our system runs on **any laptop with a webcam** using open-source MediaPipe face tracking. No special hardware. No expensive eye-tracking devices. Total cost: **under $200 in API credits** for the full stack. --- ## How It Works Blinket detects six distinct eye gestures using MediaPipe facial landmarks from a standard webcam: | Gesture | Action | |---------|--------| | **Wink left** | Navigate left / previous | | **Wink right** | Navigate right / next | | **Double blink** | Select / confirm | | **Triple blink** | Go back / cancel | | **Quadruple blink** | Send (Morse keyboard) | | **Long close** (2+ sec) | Exit to home | An alternative **EOG (electrooculogram) input mode** is also supported for users who prefer signal-based eye tracking. Every interaction is personalized through a **RAG pipeline**: user context is embedded and stored in Elasticsearch, then retrieved via kNN search and fed to Cerebras LLM to generate suggestions tailored to the individual — their interests, needs, and history. --- ## Apps ### Talk Real-time voice conversation assistant. Deepgram transcribes speech live, and on a double-blink, Blinket generates contextual response options using RAG. Select a response with a wink and it's spoken aloud via Fish Audio TTS. Includes a Morse keyboard for typing custom messages through blink patterns. ### Amazon Shopping Personalized product discovery. Blinket generates search queries based on user context, scrapes Amazon via BrightData, and presents results in a 3D carousel. Double-blink to email yourself a product link. ### Google Maps Location-aware place discovery. Uses geolocation + personalized queries to find relevant businesses and places. Browse results with winks, double-blink to open in Google Maps. ### ChatGPT Streaming chat interface with personalized conversation starters. Blinket suggests what you might want to ask based on your context. Wink to browse suggestions, double-blink to send. Full conversation history with scroll navigation. ### Web Search Personalized web search. Generates queries from user context, scrapes search results via BrightData, and displays them in a browsable carousel with preview modals. ### Books Curated classic literature reader with page-flip animations. Wink to turn pages, browse chapters, read full books — all hands-free. ### Flappy Bird Blink-controlled game. Blink to jump. A fun way to practice blink detection calibration. ### Zoom Meetings Join Zoom meetings directly in the browser via the Zoom Embedded SDK. Live meeting transcripts appear in a side panel via Zoom RTMS. Double-blink to generate contextual responses during the call and speak them via TTS. --- ## Tech Stack | Layer | Technology | |-------|-----------| | **Frontend** | React 19, TypeScript, Vite, React Router | | **Blink Detection** | MediaPipe Face Landmarker (webcam) | | **Speech-to-Text** | Deepgram Nova-2 (real-time WebSocket) | | **Text-to-Speech** | Fish Audio | | **LLM** | Cerebras (fast inference for response generation) | | **Embeddings** | Cohere | | **Vector Search** | Elasticsearch (kNN) | | **Web Scraping** | BrightData (Amazon, SERP, Maps) | | **Video Calls** | Zoom Meeting SDK + RTMS | | **Email** | Resend | | **Backend** | Hono + Bun | | **Frontend Hosting** | Vercel | | **Backend Hosting** | Render | --- ## Cost Comparison | Solution | Cost | |----------|------| | Tobii Dynavox | $6,000 – $15,000 | | Other AAC devices | $3,000 – $10,000 | | **Blinket** | **< $200 in API credits** | Blinket requires only a laptop with a webcam — hardware most people already own. All the intelligence runs through affordable cloud APIs (Cerebras for LLM, Deepgram for transcription, Cohere for embeddings). No proprietary hardware. No licensing fees. --- ## Running Locally ### Backend ```bash cd backend bun install bun run dev ``` Requires a `.env` file with API keys for Deepgram, Cerebras, Elasticsearch, BrightData, Fish Audio, Cohere, Resend, and optionally Zoom. ### Frontend ```bash cd frontend npm install npm run dev ``` The frontend proxies API requests to `localhost:3003` in development. Set `VITE_API_BASE` to override. ### RTMS (Zoom transcript logging) ```bash cd backend bun run rtms ``` Runs the Zoom RTMS webhook listener on port 8080 for real-time meeting transcript capture. --- ## Architecture ``` ┌─────────────────────────────────────────────┐ │ Frontend │ │ (React + MediaPipe + Webcam) │ │ │ │ Blink Detection ──► App Navigation │ │ Deepgram Audio ──► Live Transcription │ │ Zoom SDK ──► Embedded Meetings │ └──────────────┬──────────────────────────────┘ │ HTTP + WebSocket ┌──────────────▼──────────────────────────────┐ │ Backend │ │ (Hono + Bun) │ │ │ │ /getContext ──► Elasticsearch kNN + LLM │ │ /ws ──► Deepgram + EOG + RTMS │ │ /apps/* ──► BrightData, TTS, Email │ │ /zoom/* ──► Meeting SDK auth │ └─────────────────────────────────────────────┘ ``` --- ## Team Built at a hackathon by a team that believes internet access is a right, not a privilege — especially for people whose bodies have stopped cooperating but whose minds never did. ## Detected evidence (automated analysis) Indexed codebase: 52 recognized source files, 447 KB. - Express (technology) — detected in the code - HTML (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - TypeScript (language) — detected in the code - CSS (language) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (66 of 66) ``` .cursor/plans/create_context_files_1835df39.plan.md .gitignore backend/.gitignore backend/bun.lock backend/context.md backend/package.json backend/README.md backend/src/index.ts backend/src/lib/cerebras.ts backend/src/lib/elasticsearch.ts backend/src/lib/embeddings.ts backend/src/lib/supabase.ts backend/src/routes/apps.ts backend/src/routes/documents.ts backend/src/routes/generate.ts backend/src/routes/getContext.ts backend/src/routes/long.ts backend/src/routes/upload.ts backend/src/routes/zoom.ts backend/src/rtms.ts backend/supabase/migration.sql backend/tsconfig.json data/Characters.csv data/dataprocess.py data/Dialogue.csv frontend/.gitignore frontend/app/components/AppCard.tsx frontend/app/components/AppGrid.tsx frontend/context.md frontend/index.html frontend/next-env.d.ts frontend/package.json frontend/server.ts frontend/src/AmazonSearchPage.tsx frontend/src/AppsPage.tsx frontend/src/BooksPage.tsx frontend/src/ChatPage.tsx frontend/src/components/GlobalEogPanel.tsx frontend/src/components/InputModeToggle.tsx frontend/src/components/SignalVisualizer.tsx frontend/src/config.ts frontend/src/DiarizationPanel.tsx frontend/src/FlappyBirdPage.tsx frontend/src/inputMode.ts frontend/src/main.tsx frontend/src/MapsPage.tsx frontend/src/MorseKeyboard.tsx frontend/src/TalkPage.tsx frontend/src/types.ts frontend/src/useBlinkDetection.ts frontend/src/useRealtimeTranscription.ts frontend/src/vite-env.d.ts frontend/src/WebSearchPage.tsx frontend/src/ZoomPage.tsx frontend/src/ZoomTranscriptPanel.tsx frontend/tsconfig.json frontend/tsconfig.tsbuildinfo frontend/vite.config.ts hardware/bridge.py hardware/context.md hardware/main.py hardware/plotter.py new-frontend/.vite/deps/_metadata.json new-frontend/.vite/deps/package.json README.md vercel.json ``` ### Dependencies - backend/package.json: @cerebras/cerebras_cloud_sdk@^1, @deepgram/sdk@^4.11.3, @elastic/elasticsearch@^9.3.1, @supabase/supabase-js@^2, @types/bun@latest, @types/jsrsasign@^10.5.15, @zoom/rtms@^1.0.2, hono@^4, jsrsasign@^11.1.0, jszip@^3.10.1, openai@^4, serialport@^13.0.0 - frontend/package.json: @deepgram/sdk@^4.11.3, @mediapipe/tasks-vision@^0.10.32, @types/express@^5.0.6, @types/react@^19.2.14, @types/react-dom@^19.2.3, @types/ws@^8.18.1, @vitejs/plugin-react@^5.1.4, @zoom/meetingsdk@^5.1.2, dotenv@^17.3.1, express@^5.2.1, react@^19.2.4, react-dom@^19.2.4, react-icons@^5.5.0, react-router-dom@^7.13.0, react-webcam@^7.2.0, tsx@^4.21.0, typescript@~5.9.3, vite@^7.3.1, ws@^8.19.0 ### Recent commits (newest first) - Merge pull request #30 from idalmas/add-vercel-config - Rename project from Revive to Blinket - Add comprehensive README with project overview, features, and cost comparison - Clean up local changes: webcam position, remove EogTuningPanel, add .vercel to gitignore - Merge pull request #29 from idalmas/add-vercel-config - Remove speaker diarization from Talk page - Merge pull request #28 from idalmas/add-vercel-config - Add SPA rewrite rule for client-side routing - attempt to fix this - Merge pull request #27 from idalmas/add-vercel-config - Point frontend API_BASE to Render backend - Merge pull request #26 from idalmas/add-vercel-config - Make @zoom/rtms import conditional for cloud deployment - Merge pull request #25 from idalmas/add-vercel-config - Add Vercel deployment config for frontend - Merge pull request #24 from idalmas/add-zoom-rtms-transcript-logging - Add Zoom meeting page with embedded SDK, live transcript, and blink responses - Merge pull request #23 from idalmas/add-zoom-rtms-transcript-logging - Add Zoom RTMS real-time transcript logging - Merge pull request #22 from idalmas/RationalChestnut/talk-context-flow ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/context.md ```markdown # Frontend Context: Hands-Free Interface ## Overview The Blinket frontend is a React-based web application designed for hands-free interaction. It uses computer vision (MediaPipe) to translate facial gestures—specifically eye blinks and winks—into navigation and control commands across a suite of specialized applications. ## Core Technology Stack - **Framework**: React 18 with Vite - **Routing**: React Router - **Gestures**: MediaPipe Face Landmarker - **Styling**: Tailwind CSS ## Gesture Detection System (`src/hooks/useBlinkDetection.ts`) The heart of the frontend is the `useBlinkDetection` hook, which processes video frames to detect: - **Single Blink**: Basic trigger. - **Double Blink**: Open/Select action. - **Triple Blink**: Reset or special action. - **Wink Left / Wink Right**: Navigation (previous/next). - **Long Close**: State change or "back" action. The system uses Exponential Moving Averages (EMA) to smooth noisy data from the camera and relative eye-opening ratios to distinguish between intentional winks and natural blinks. ## Application Suite The frontend is organized into several "apps," each accessible from the main launcher (`AppsPage.tsx`): ### 1. Apps Launcher (`AppsPage.tsx`) A grid-based launcher where users browse apps using winks and select them with a double blink. ### 2. Talk Page (`TalkPage.tsx`) A voice-based conversation interface. - Supports real-time transcription and diarization. - Uses a state machine (Idle -> Listening -> Thinking -> Speaking) controlled by blinks. - Integrates with Fish Audio for voice cloning. ### 3. Books Page (`BooksPage.tsx`) An EPUB reader. - Page flipping is controlled by left/right winks. - Hands-free reading experience. ### 4. Search Apps - **Amazon Search**: Browse products and send links via email. - **Web Search**: Scroll through Google search results using winks. - **Maps**: Navigate locations hands-free. ### 5. Games - **Flappy Bird**: Control the bird's jump using blinks. ## Backend Integration - **API Base**: Configured in `src/config.ts` (typically `http://localhost:3001`). - **Communication**: Uses standard `fetch` for REST and WebSockets for real-time features. ## Planned Integration: EOG The frontend is being prepared to accept input from the **Hardware EOG System**. This will allow users to navigate by looking left or right (EOG mode) in addition to the existing camera-based blink detection. ``` ### backend/context.md ```markdown # Backend Context: Blinket RAG & App Services ## Overview The Blinket backend is a high-performance TypeScript service built on **Bun** and the **Hono** web framework. It serves as the central intelligence hub for the Blinket ecosystem, providing Retrieval-Augmented Generation (RAG) capabilities, specialized app integrations, and real-time communication support. ## Core Architecture - **Runtime**: [Bun](https://bun.sh/) - **Framework**: [Hono](https://hono.dev/) - **Database**: [Elasticsearch](https://www.elastic.co/) (Vector Store for kNN search) - **LLM Provider**: Cerebras (using `gpt-oss-120b`) - **Embeddings**: OpenAI (`text-embedding-3-small`, 1536 dimensions) ## Key Components ### 1. Vector Database (Elasticsearch) Located in `backend/src/lib/elasticsearch.ts`. - Manages the `person-context` index. - Stores text chunks with their corresponding vector embeddings. - Supports kNN (k-Nearest Neighbors) search for semantic retrieval. - Includes a `person` field to scope context to specific users (e.g., "ian"). ### 2. RAG Pipeline - **Ingestion**: `POST /upload` and `POST /long` endpoints handle text chunking, embedding generation via OpenAI, and indexing into Elasticsearch. - **Retrieval**: `POST /generate` performs kNN search to find relevant context before querying the Cerebras LLM. ### 3. App Services (`backend/src/routes/apps.ts`) The backend provides specialized endpoints for various frontend "apps": - **Amazon Search**: BrightData integration for scraping product data. - **Maps Search**: Google Maps API integration. - **Web Search**: Google SERP API for real-time information. - **Voice/Talk**: Fish Audio integration for voice cloning and TTS. - **Books**: EPUB content extraction and management. ### 4. Real-time Communication - **WebSocket (`/ws`)**: A minimal WebSocket implementation in `index.ts` designed for real-time speech-to-text (STT) streaming (placeholder for transcription engine). ## API Endpoints Summary - `POST /upload`: Embed and index a single text chunk. - `POST /long`: Bulk ingest and index long documents. - `POST /generate`: RAG-powered chat completion. - `POST /getContext`: Retrieve context chunks for specific apps. - `GET /documents`: Debug endpoint to list all stored documents. - `POST /apps/*`: Sub-routes for Amazon, Maps, Web Search, and Voice services. ## Development Commands - `bun run dev`: Start with hot-reloading. - `bun run start`: Production server. ``` ### backend/package.json ``` { "name": "blinket-backend", "version": "0.1.0", "private": true, "scripts": { "dev": "bun run --hot src/index.ts", "start": "bun run src/index.ts", "rtms": "bun run src/rtms.ts" }, "dependencies": { "@cerebras/cerebras_cloud_sdk": "^1", "@deepgram/sdk": "^4.11.3", "@elastic/elasticsearch": "^9.3.1", "@supabase/supabase-js": "^2", "@zoom/rtms": "^1.0.2", "hono": "^4", "jsrsasign": "^11.1.0", "jszip": "^3.10.1", "openai": "^4", "serialport": "^13.0.0" }, "devDependencies": { "@types/bun": "latest", "@types/jsrsasign": "^10.5.15" } } ``` ### frontend/package.json ``` { "name": "blink-detector", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "server": "tsx server.ts", "build": "tsc -b && vite build", "preview": "vite preview", "postinstall": "mkdir -p public/lib && cp -r node_modules/@zoom/meetingsdk/dist/lib/av/* public/lib/ 2>/dev/null || true" }, "dependencies": { "@deepgram/sdk": "^4.11.3", "@mediapipe/tasks-vision": "^0.10.32", "@zoom/meetingsdk": "^5.1.2", "dotenv": "^17.3.1", "express": "^5.2.1", "react": "^19.2.4", "react-dom": "^19.2.4", "react-icons": "^5.5.0", "react-router-dom": "^7.13.0", "react-webcam": "^7.2.0", "ws": "^8.19.0" }, "devDependencies": { "@types/express": "^5.0.6", "@types/react": "^19.2.14", "@types/react-dom": "^19.2.3", "@types/ws": "^8.18.1", "@vitejs/plugin-react": "^5.1.4", "tsx": "^4.21.0", "typescript": "~5.9.3", "vite": "^7.3.1" } } ``` ### new-frontend/.vite/deps/package.json ``` { "type": "module" } ``` ### hardware/main.py ```python import machine import time from machine import Pin, ADC # Configuration based on hardware/context.md ADC_PIN = 34 # GPIO34 LED_PIN = 2 # Standard onboard LED for most ESP32s SAMPLING_RATE = 200 # Hz SLEEP_MS = 1000 // SAMPLING_RATE # Initialize ADC # GPIO34 is an input-only pin, safe for ADC1 adc = ADC(Pin(ADC_PIN)) # Initialize LED led = Pin(LED_PIN, Pin.OUT) # Configure ADC for 0-3.3V range # ATTN_11DB allows full range up to ~3.6V (clamped at 3.3V) adc.atten(ADC.ATTN_11DB) # 12-bit resolution (0-4095) adc.width(ADC.WIDTH_12BIT) print("EOG System Initialized") print(f"Sampling GPIO{ADC_PIN} at {SAMPLING_RATE}Hz") def run_loop(): print("Starting sampling loop... Press Ctrl+C to stop.") counter = 0 try: while True: # Read raw ADC value raw_val = adc.read() # Convert to voltage (approximate) # 4095 corresponds to ~3.3V voltage = (raw_val / 4095.0) * 3.3 # Blink LED every 100 samples (~0.5 seconds at 200Hz) if counter % 100 == 0: led.value(not led.value()) # Print data in format readable by serial plotters # Format: raw_value, voltage print(f"{raw_val},{voltage:.4f}") counter += 1 # Maintain sampling rate time.sleep_ms(SLEEP_MS) except KeyboardInterrupt: print("\nSampling stopped by user.") if __name__ == "__main__": run_loop() ``` ### frontend/server.ts ```typescript import 'dotenv/config'; import express from 'express'; import http from 'http'; import { WebSocketServer, WebSocket } from 'ws'; import { createClient, LiveTranscriptionEvents } from '@deepgram/sdk'; const PORT = 3003; const app = express(); const server = http.createServer(app); app.get('/api/health', (_req, res) => { res.json({ status: 'ok' }); }); const wss = new WebSocketServer({ server, path: '/ws' }); wss.on('connection', (browserSocket: WebSocket) => { console.log('[ws] Browser connected'); const deepgram = createClient(process.env.DEEPGRAM_API_KEY!); const dgConnection = deepgram.listen.live({ model: 'nova-3', diarize: true, smart_format: true, punctuate: true, interim_results: false, encoding: 'linear16', sample_rate: 16000, }); dgConnection.on(LiveTranscriptionEvents.Open, () => { console.log('[deepgram] Connection opened'); }); dgConnection.on(LiveTranscriptionEvents.Transcript, (data: any) => { const words = data.channel?.alternatives?.[0]?.words; const transcript = data.channel?.alternatives?.[0]?.transcript; if (transcript && words?.length > 0) { browserSocket.send( JSON.stringify({ type: 'transcript', transcript, words: words.map((w: any) => ({ word: w.punctuated_word || w.word, speaker: w.speaker, start: w.start, end: w.end, confidence: w.confidence, })), is_final: data.is_final, }) ); } }); dgConnection.on(LiveTranscriptionEvents.Error, (err: any) => { console.error('[deepgram] Error:', err); browserSocket.send(JSON.stringify({ type: 'error', message: String(err) })); }); dgConnection.on(LiveTranscriptionEvents.Close, () => { console.log('[deepgram] Connection closed'); }); browserSocket.on('message', (data: Buffer) => { if (dgConnection.getReadyState() === WebSocket.OPEN) { dgConnection.send(data); } }); browserSocket.on('close', () => { console.log('[ws] Browser disconnected'); dgConnection.requestClose(); }); }); server.listen(PORT, () => { console.log(`Server running on http://localhost:${PORT}`); }); ``` ### frontend/src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import { BrowserRouter, Routes, Route, Navigate } from 'react-router-dom' import AppsPage from './AppsPage.tsx' import AmazonSearchPage from './AmazonSearchPage.tsx' import MapsPage from './MapsPage.tsx' import ChatPage from './ChatPage.tsx' import WebSearchPage from './WebSearchPage.tsx' import FlappyBirdPage from './FlappyBirdPage.tsx' import BooksPage from './BooksPage.tsx' import TalkPage from './TalkPage.tsx' import ZoomPage from './ZoomPage.tsx' import InputModeToggle from './components/InputModeToggle.tsx' import GlobalEogPanel from './components/GlobalEogPanel.tsx' createRoot(document.getElementById('root')!).render( } /> } /> } /> } /> } /> } /> } /> } /> } /> } /> , ) ``` ### backend/src/index.ts ```typescript /** * index.ts — Hono App Entry Point * * Sets up the Hono web server with CORS middleware, bootstraps the * Elasticsearch index, mounts the API routes, and exports the app for * Bun to serve. * * Routes: * POST /upload — upload text → embed → index into Elasticsearch * POST /generate — dialog in → kNN RAG + Cerebras → personalized response * POST /long — bulk ingest: chunk long text → embed all → bulk index * POST /getContext — retrieve relevant context chunks for a given app (+optional text) * GET /documents — browse everything stored in the vector DB (debug/demo) * POST /apps/amazon-search — kick off a BrightData Amazon scrape * GET /apps/amazon-status/:id — poll for scrape results * POST /apps/send-email — email a product link via Resend * * On startup: * - Calls ensureIndex() to create the Elasticsearch `person-context` index * if it doesn't already exist. * - Calls ensurePersonField() to add the `person` keyword field to the * mapping if the index already existed before the field was introduced. * * Run with: * bun run dev (hot-reload) * bun run start (production) */ import { Hono } from "hono"; import { createBunWebSocket } from "hono/bun"; import { cors } from "hono/cors"; import { logger } from "hono/logger"; import { createClient, LiveTranscriptionEvents } from "@deepgram/sdk"; import { ensureIndex, ensurePersonField } from "./lib/elasticsearch"; import upload from "./routes/upload"; import generate from "./routes/generate"; import long from "./routes/long"; import documents from "./routes/documents"; import getContext from "./routes/getContext"; import apps from "./routes/apps"; import zoom from "./routes/zoom"; const DEEPGRAM_API_KEY = process.env.DEEPGRAM_API_KEY || ""; const app = new Hono(); const { upgradeWebSocket, websocket } = createBunWebSocket(); const signalClients = new Set(); type Direction = -1 | 0 | 1; interface SignalInput { type: "signal"; raw: number; voltage: number; timestamp?: number; } interface ProcessedSignal extends SignalInput { movingAvg: number; lowerBound: number; upperBound: number; convScore: number; convThreshold: number; direction: Direction; calibrated: boolean; calibrationRemainingMs: number; } class EogProcessor { private readonly calibrationMs = 15_000; private readonly kernel = [-1, -0.5, 0, 0.5, 1]; private readonly refractoryMs = 180; private readonly directionHoldMs = 160; private readonly minConvThreshold = 12; private readonly minMargin = 35; private startedAt = 0; private lastSampleAt = 0; private isCalibrated = false; private movingAvg = 0; private noiseEma = 0; private convNoiseEma = 0; private detrendedWindow: number[] = []; private calibrationValues: number[] = []; private calibratedStd = 18; private calibratedMean = 0; private lastEventAt = 0; private holdUntil = 0; private direction: Direction = 0; private readyForNextEvent = true; reset(nowMs: number) { this.startedAt = nowMs; this.lastSampleAt = nowMs; this.isCalibrated = false; this.movingAvg = 0; this.noiseEma = 0; this.convNoiseEma = 0; this.detrendedWindow = []; this.calibrationValues = []; this.calibratedStd = 18; this.calibratedMean = 0; this.lastEventAt = 0; this.holdUntil = 0; this.direction = 0; this.readyForNextEvent = true; } process(input: SignalInput): ProcessedSignal { const nowMs = Date.now(); if (!this.startedAt) this.reset(nowMs); if (this.lastSampleAt && nowMs - this.lastSampleAt > 2_000) { // Stream gap usually means a new user/session. Recalibrate automatically. this.reset(nowMs); } this.lastSampleAt = nowMs; const raw = Number(input.raw); const voltage = Number(input.voltage); if (this.movingAvg === 0) this.movingAvg = raw; const avgAlpha = 0.02; // moving average baseline this.movingAvg = this.movingAvg + avgAlpha * (raw - this.movingAvg); const detrended = raw - this.movingAvg; this.noiseEma = this.noiseEma + 0.05 * (Math.abs(detrended) - this.noiseEma); this.detrendedWindow.push(detrended); if (this.detrendedWindow.length > this.kernel.length) this.detrendedWindow.shift(); let convScore = 0; if (this.detrendedWindow.length === this.kernel.length) { for (let i = 0; i < this.kernel.length; i++) { convScore += this.kernel[i] * this.detrendedWindow[i]; } } this.convNoiseEma = this.convNoiseEma + 0.05 * (Math.abs(convScore) - this.convNoiseEma); if (!this.isCalibrated) { this.calibrationValues.push(raw); if (nowMs - this.startedAt >= this.calibrationMs && this.calibrationValues.length > 50) { let sum = 0; for (const v of this.calibrationValues) sum += v; this.calibratedMean = sum / this.calibrationValues.length; let varSum = 0; for (const v of this.calibrationValues) varSum += (v - this.calibratedMean) ** 2; this.calibratedStd = Math.sqrt(varSum / this.calibrationValues.length) || 18; this.isCalibrated = true; } } const calibratedMargin = Math.max( this.minMargin, this.calibratedStd * 4, this.noiseEma * 7 ); const lowerBound = this.movingAvg - calibratedMargin; const upperBound = this.movingAvg + calibratedMargin; const convThreshold = Math.max( this.minConvThreshold, this.convNoiseEma * 3.6, this.calibratedStd * 0.9 ); if (Math.abs(convScore) < convThreshold * 0.35) { this.readyForNextEvent = true; } if ( this.readyForNextEvent && nowMs - this.lastEventAt > this.refractoryMs && Math.abs(convScore) > convThreshold ) { this.direction = convScore > 0 ? -1 : 1; this.lastEventAt = nowMs; this.holdUntil = nowMs + this.directionHoldMs; this.readyForNextEvent = false; } else if (nowMs > this.holdUntil) { [truncated — 8345 more characters] ``` ### frontend/next-env.d.ts ```typescript /// /// import "./.next/dev/types/routes.d.ts"; // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. ``` [43 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]