# Project export: ctrl+f This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Physical version control for the real world - ctrl+f watches your workspace, tracks what moved and when, and lets you search past events in plain English to instantly find what changed. - Devpost: https://devpost.com/software/ctrl-f-oc1yr9 - GitHub: https://github.com/alemanb/treehacks-2026 - Video: https://www.youtube.com/embed/QuoFo0_ZrEs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — alemanb (9 commits), Abhinav Srivatsa (6 commits) ## Devpost submission (written by the team) ### Inspiration Physical debugging is still painfully manual. When software breaks, we have Git history, diffs, and blame. When a real world workflow breaks, teams usually scrub hours of video and guess what happened. That gap inspired ctrl+f: a system that treats physical environments like a version controlled timeline, so you can ask “what changed, when, and why?” instead of manually hunting through footage. Our idea originated from struggles in maintaining persistent memory in long context video modalities especially when passed to a vision language model. We wanted a good way to identify key frames of references and represent the world state as efficient as possible. ### What it does ctrl+f is physical version control for real spaces. It continuously observes a workspace, tracks object level changes, and builds a semantic event history. Users can then run natural language investigations such as: “Where did the blue book go?” “When was the laptop moved?” “What changed before the failure?” The system returns ranked, timestamped evidence in an investigation interface with: Table view for structured evidence review Calendar view for time based exploration Paginated results for deeper investigation Instead of raw clips only, users get queryable, structured memory. ### How we built it We built ctrl+f as a three layer pipeline. 1. Edge Perception Layer (Jetson class stack) DeepStream + YOLO inference for real time detections Custom multi frame tracking for stable identity and movement state On device VLM checks for movement, disappearance, and reappearance reasoning This layer converts raw pixels into structured object level events. 2. Semantic Memory Backend FastAPI service deployed with Modal Text + metadata ingestion endpoints Embedding generation Vector indexing with Elasticsearch for semantic retrieval This transforms scene changes into searchable memory. 3. Investigation Frontend React + TypeScript interface Query → process → results investigation flow Timeline style date exploration Paginated evidence display We focused on building a practical investigation UX, not just a model demo. ### Challenges we ran into Detection instability in real scenes Occlusions, temporary missed detections, and class label drift created noisy event streams. We had to design tracking logic that tolerated real world imperfections. Latency vs reasoning depth We needed richer semantic reasoning without blocking edge responsiveness. This required a hybrid approach: fast detection first, deeper reasoning selectively. Search quality depends on schema quality Retrieval improved only after refining event text structure and metadata consistency. Good embeddings alone were not enough. Full stack integration pressure Synchronizing edge outputs, backend contracts, and frontend UX under hackathon constraints required tight coordination across the stack. Jetson Nano Struggles Trying to get our pipelines to work on the jetson nano for the first time as we have had no exposure to such Nvidia devices prior ### Accomplishments we're proud of Built a true end to end prototype from live perception to semantic investigation Implemented robust tracking logic that handles physical world noise Turned scene changes into a queryable memory layer Delivered a practical investigation interface, not just model outputs Framed and validated a strong product concept: “Git Blame for Reality” ### What we learned In edge AI systems, reliability engineering matters as much as raw model accuracy Hybrid pipelines combining fast detectors and selective deeper reasoning work well in practice Explainable, time grounded logs are essential for trust and usability Great demos require tight coupling between ML outputs and user facing workflows ### What's next Add stricter privacy first mode with semantic retention by default and configurable media retention Expand to multi camera and multi agent coordination Improve causal diagnostics so the system explains why something likely happened, not only what changed Add deployment hardening: monitoring, evaluation benchmarks, and failover behavior Extend to mobile sensing workflows for broader real world coverage ctrl+f is our first step toward making physical environments as debuggable as software systems. ## README (from the GitHub repository) # Ctrl-F: Intelligent Video Observation & Search **Built for TreeHacks 2026** — [Full Project Details on Devpost](https://devpost.com/software/ctrl-f-oc1yr9) Ctrl-F is an end-to-end, edge-to-cloud intelligent surveillance system that transforms raw video feeds into a searchable, semantic database. Instead of just recording footage, it "understands" the scene locally and allows users to query history using natural language. ## 👥 The Team * **Benjamin Aleman** ([alemanb](https://github.com/alemanb)) * **Abhinav Srivatsa** ([adsrivatsa](https://github.com/adsrivatsa)) * **Wei Jiang** ([mr2wei](https://github.com/mr2wei)) * **Preet Sojitra** ([preetsojitra2712](https://github.com/preetsojitra2712)) --- ## 🚀 System Architecture The system is split into a high-performance edge pipeline and a scalable cloud intelligence layer: ### 1. Edge Compute (NVIDIA Jetson Orin Nano Super) Located in the `deepstream/` directory, the edge node handles the heavy lifting of computer vision: * **Vision Pipeline:** DeepStream + YOLO for real-time object detection and multi-object tracking (MOT). * **Local Reasoning:** NanoLLM / local VLM service provides semantic enrichment, turning raw bounding boxes into natural-language observations. * **Frame Server:** A lightweight HTTP server serves captured frames directly from the edge for UI inspection. * **Ingestion:** Asynchronously pushes enriched observations to the cloud backend. ### 2. Cloud Backend (Modal) Located in the `backend/` directory, deployed as serverless functions on Modal: * **FastAPI:** Provides `/ingest` and `/search` endpoints. * **Intelligent Search:** A multi-agent RAG workflow involving query expansion and temporal reasoning. * **Embedding & Storage:** Uses Jina embeddings to store data in Elasticsearch Cloud (Vector + Metadata store). ### 3. Frontend (React + TypeScript) Located in the `frontend/` directory: * **Investigation UI:** A modern dashboard for searching through historical observations. * **Visualization:** Calendar and table views with direct deep-links to frame images hosted on the Jetson edge device. ## 🛠️ Tech Stack - **Hardware:** NVIDIA Jetson Orin Nano Super - **Edge AI:** NVIDIA DeepStream, NanoLLM, YOLO - **Cloud/Infra:** Modal (Serverless), Elasticsearch Cloud - **LLM/Embeddings:** OpenAI GPT-4o, Jina AI - **Frontend:** React, TypeScript, Vite, Tailwind CSS ## 🔄 Data Flow 1. **Capture:** Webcam feed processed by DeepStream on the Jetson. 2. **Analyze:** VLM generates natural-language descriptions of tracked objects. 3. **Sync:** Observations are POSTed to the Modal backend. 4. **Index:** Modal embeds the text and indexes it into Elasticsearch. 5. **Query:** User enters a natural language query in the React UI. 6. **Retrieve:** Backend agents perform a vector search + re-ranking and return the most relevant video frames. ## Detected evidence (automated analysis) Indexed codebase: 84 recognized source files, 528 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - Ollama (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (99 of 99) ``` .gitignore deepstream/.gitignore deepstream/config_infer_primary_yolo26.txt deepstream/deepstream_app_config.txt deepstream/frame_image_server.py deepstream/labels.txt deepstream/nano_llm_openai_server.py deepstream/object_tracker.py deepstream/tests/live_detection_test.py deepstream/tests/stream.py deepstream/vlm_nano_llm.py deepstream/vlm_ollama.py deepstream/webcam_yolo.py docs/01_vector_search_moda_implementation/design-vector-search.md docs/01_vector_search_moda_implementation/implementation-guide.md docs/02_frontend_boilerplate/design-document.md docs/02_frontend_boilerplate/implementation-plan.md docs/03_vector_search_api/design-search-api.md docs/03_vector_search_api/implementation-search-api.md docs/04_pagination_feature/design-pagination.md docs/04_pagination_feature/implementation-pagination.md frontend/.gitignore frontend/components.json frontend/eslint.config.js frontend/index.html frontend/package.json frontend/README.md frontend/src/App.tsx frontend/src/components/calendar/CalendarView.tsx frontend/src/components/calendar/DateDetailCard.tsx frontend/src/components/data-table/columns.tsx frontend/src/components/data-table/DataTableView.tsx frontend/src/components/data-table/LikelihoodBadge.tsx frontend/src/components/investigation/InputPhase.tsx frontend/src/components/investigation/InvestigationPage.tsx frontend/src/components/investigation/ProcessingPhase.tsx frontend/src/components/investigation/ResultsPhase.tsx frontend/src/components/ui/badge.tsx frontend/src/components/ui/button.tsx frontend/src/components/ui/calendar.tsx frontend/src/components/ui/card.tsx frontend/src/components/ui/popover.tsx frontend/src/components/ui/progress.tsx frontend/src/components/ui/table.tsx frontend/src/components/ui/textarea.tsx frontend/src/data/mock-results.ts frontend/src/hooks/useSearch.ts frontend/src/hooks/useUrlState.ts frontend/src/index.css frontend/src/lib/likelihood.ts frontend/src/lib/utils.ts frontend/src/main.tsx frontend/src/types/search.ts frontend/tsconfig.app.json frontend/tsconfig.json frontend/tsconfig.node.json frontend/vite.config.ts modal/.python-version modal/data/.gitignore modal/data/README.md modal/deploy.sh modal/main.py modal/pyproject.toml modal/README.md modal/scripts/__init__.py modal/scripts/generate_fake_data.py modal/scripts/README_DATA_GENERATION.md modal/src/__init__.py modal/src/agents/__init__.py modal/src/agents/condition.py modal/src/agents/matching_agent.py modal/src/agents/matching_function.py modal/src/agents/matching.py modal/src/agents/query_expansion.py modal/src/api/__init__.py modal/src/api/routes.py modal/src/config.py modal/src/models/__init__.py modal/src/models/agent_schemas.py modal/src/models/schemas.py modal/src/services/__init__.py modal/src/services/embeddings.py modal/src/services/temporal_detector.py modal/src/services/vectordb.py modal/src/workflows/__init__.py modal/src/workflows/intelligent_rag.py modal/src/workflows/README.md modal/tests/test_agents_individually.py modal/tests/test_ai_matching.py modal/tests/test_condition_agent_phase2.py modal/tests/test_intelligent_search.py modal/tests/test_intelligent_search.sh modal/tests/test_matching_integration_phase3.py modal/tests/test_phase8_integration.sh modal/tests/test_production.sh modal/tests/test_temporal_detector.py modal/tests/test_temporal_integration.sh modal/uv.lock README.md ``` ### Dependencies - frontend/package.json: @eslint/js@^9.39.1, @tailwindcss/vite@^4.1.18, @tanstack/react-table@^8.21.3, @types/node@^24.10.1, @types/react@^19.2.7, @types/react-dom@^19.2.3, @vitejs/plugin-react@^5.1.1, class-variance-authority@^0.7.1, clsx@^2.1.1, date-fns@^4.1.0, eslint@^9.39.1, eslint-plugin-react-hooks@^7.0.1, eslint-plugin-react-refresh@^0.4.24, globals@^16.5.0, lucide-react@^0.564.0, radix-ui@^1.4.3, react@^19.2.0, react-day-picker@^9.13.2, react-dom@^19.2.0, shadcn@^3.8.4, tailwind-merge@^3.4.0, tailwindcss@^4.1.18, tw-animate-css@^1.4.0, typescript@~5.9.3, typescript-eslint@^8.48.0, vite@^7.3.1 - modal/pyproject.toml: agno@>=2.5.0, dateparser@>=1.3.0, elasticsearch@>=8.17, fastapi[standard], modal, openai@>=2.21.0, pytest@>=9.0.2, requests@>=2.31 ### Recent commits (newest first) - updated readme - added readme - completed - updated timestamp from uuid - update frame links from timestamps to uuids - prompt engineering, added server to get files from the jetson from frontend, added frontend viewing images from jetson - updated some small ui changes - updated some small ui changes - Merge branch 'main' of https://github.com/alemanb/treehacks-2026 - added the multi-agent system, and cleaned up some organizational issues. added temporal reasoning - added the multi-agent system, and cleaned up some organizational issues. added temporal reasoning - added deepstream pipeline featuring yolo pipeline and vlm integration - Delete .DS_Store - added performance optimization - added documentation from pagination - reorgznied structure and added pagination for the table - added search endpoint - added design and implementation documents for frontend - implement vector search ingestion pipeline. Added FastAPI routes (ingest, batch ingest, health) served on Modal, Pydantic request/response models, and bulk Elasticsearch indexing. - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### modal/scripts/README_DATA_GENERATION.md ```markdown # Fake Data Generation Script This script generates realistic fake observation data for the RAG database to test pagination and search functionality. ## Quick Start ```bash # Generate 100 observations (Feb 1-14, 2026) python generate_fake_data.py # Generate custom number of observations python generate_fake_data.py --count 200 # Custom date range python generate_fake_data.py --start-date 2026-01-01 --end-date 2026-01-31 # Save to JSON first (for review before ingesting) python generate_fake_data.py --save-json observations.json # Load from JSON and ingest python generate_fake_data.py --load-json observations.json ``` ## Options | Option | Default | Description | |--------|---------|-------------| | `--count` | 100 | Number of observations to generate | | `--start-date` | 2026-02-01 | Start date (YYYY-MM-DD) | | `--end-date` | 2026-02-14 | End date (YYYY-MM-DD) | | `--batch-size` | 10 | Batch size for API ingestion | | `--save-json` | None | Save generated data to JSON file | | `--load-json` | None | Load data from JSON file | ## Generated Data ### Objects Backpacks, laptops, phones, wallets, keys, watches, glasses, books, notebooks, headphones, chargers, tablets, cameras, bags, briefcases, jackets, and more. ### Colors Red, blue, green, black, white, gray, silver, brown, navy, maroon, purple, pink, orange, yellow, gold, bronze, teal, cyan. ### Locations Desk, shelf, table, chair, floor, counter, bench, windowsill, cabinet, drawer, couch, bookshelf, door, hallway, kitchen counter, lobby, conference room, and more. ### Sample Observation ```json { "content": "Blue backpack placed on desk near the window.", "metadata": { "object": "backpack", "color": "blue", "timestamp": "2026-02-07T14:23:45Z", "motion_vector": [0.3, -0.1], "device_id": "jetson_01" } } ``` ## Features - **Realistic Descriptions**: Combines objects, colors, locations, and actions naturally - **Random Timestamps**: Evenly distributed across specified date range - **Motion Vectors**: 70% have motion, 30% stationary - **Device IDs**: Simulates multiple cameras/sensors - **Batch Ingestion**: Efficient API usage with configurable batch sizes - **Progress Tracking**: Shows real-time progress and success/failure counts - **JSON Export**: Save generated data for review or reuse ## Examples ### Basic Usage ```bash cd modal python generate_fake_data.py ``` Output: ``` ============================================================ Fake Data Generation for RAG Database ============================================================ Count: 100 observations Date range: 2026-02-01 to 2026-02-14 Batch size: 10 Backend URL: https://alemanb--treehacks-vector-search-web.modal.run ============================================================ Generating 100 observations... Generated 10/100 observations... Generated 20/100 observations... ... Generated 100/100 observations... ✓ Generated 100 observations Sample observation: { "content": "Black laptop placed on [truncated — 2525 more characters] ``` ### docs/01_vector_search_moda_implementation/implementation-guide.md ```markdown # Implementation Guide: Vector Search Pipeline ## Project Status ### Completed - `config.py` — Constants (ES endpoint, Jina model, dimensions) - `embeddings.py` — Jina embeddings client (`get_embeddings`) - `vectordb.py` — Elasticsearch client (`get_client`, `ensure_index`, `index_document`) - `pyproject.toml` — Dependencies defined (`elasticsearch>=8.17`, `fastapi[standard]`, `modal`, `requests>=2.31`) - `.env` — Local secrets file (gitignored) ### Missing - `main.py` — Modal app + FastAPI routes (the entire API layer) - `models.py` — Pydantic request/response schemas - Modal secrets creation (`jina-secret`, `elastic-secret`) - Bulk indexing support in `vectordb.py` - Deployment + validation --- ## Target File Structure ``` modal/ ├── main.py # Modal app definition + FastAPI ASGI entrypoint ├── models.py # Pydantic schemas (request/response) ├── embeddings.py # Jina embeddings client (DONE) ├── vectordb.py # Elasticsearch operations (PARTIAL — needs bulk) ├── config.py # Constants (DONE) ├── pyproject.toml # Dependencies (DONE) ├── .env # Local secrets (DONE, gitignored) └── README.md # Usage and deployment instructions ``` --- ## Phase 1: Pydantic Models (`models.py`) **Goal**: Extract request/response schemas into a dedicated file for clarity and reuse. **Create `modal/models.py`**: ```python from pydantic import BaseModel class Metadata(BaseModel): object: str | None = None color: str | None = None timestamp: str | None = None motion_vector: list[float] | None = None device_id: str | None = None class IngestRequest(BaseModel): content: str metadata: Metadata | None = None class BatchIngestRequest(BaseModel): documents: list[IngestRequest] class IngestResponse(BaseModel): status: str id: str class BatchIngestResponse(BaseModel): indexed: int errors: int class HealthResponse(BaseModel): status: str elasticsearch: str jina: str ``` **Why separate?** Keeps `main.py` focused on routing/orchestration. Models are importable by tests or other modules. --- ## Phase 2: Bulk Indexing (`vectordb.py`) **Goal**: Add `bulk_index_documents` to `vectordb.py` using the Elasticsearch `helpers.bulk` API. **Add to `modal/vectordb.py`**: ```python from elasticsearch.helpers import bulk def bulk_index_documents( client: Elasticsearch, documents: list[dict], ) -> tuple[int, int]: """Bulk index observations. Each dict must have content, metadata, embedding. Returns (success_count, error_count). """ actions = [ { "_index": ES_INDEX, "_source": { "content": doc["content"], "metadata": doc["metadata"], "embedding": doc["embedding"], }, } for doc in documents ] success, errors = bulk(client, actions, raise_on_error=False) return success, len(errors) if isins [truncated — 5651 more characters] ``` ### modal/pyproject.toml ``` [project] name = "treehacks-vector-search" version = "0.1.0" description = "TreeHacks 2026 - Vector search with Elasticsearch and Jina embeddings" readme = "README.md" requires-python = ">=3.12" dependencies = [ "agno>=2.5.0", "dateparser>=1.3.0", "elasticsearch>=8.17", "fastapi[standard]", "modal", "openai>=2.21.0", "pytest>=9.0.2", "requests>=2.31", ] ``` ### frontend/package.json ``` { "name": "frontend", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "@tailwindcss/vite": "^4.1.18", "@tanstack/react-table": "^8.21.3", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "date-fns": "^4.1.0", "lucide-react": "^0.564.0", "radix-ui": "^1.4.3", "react": "^19.2.0", "react-day-picker": "^9.13.2", "react-dom": "^19.2.0", "tailwind-merge": "^3.4.0", "tailwindcss": "^4.1.18" }, "devDependencies": { "@eslint/js": "^9.39.1", "@types/node": "^24.10.1", "@types/react": "^19.2.7", "@types/react-dom": "^19.2.3", "@vitejs/plugin-react": "^5.1.1", "eslint": "^9.39.1", "eslint-plugin-react-hooks": "^7.0.1", "eslint-plugin-react-refresh": "^0.4.24", "globals": "^16.5.0", "shadcn": "^3.8.4", "tw-animate-css": "^1.4.0", "typescript": "~5.9.3", "typescript-eslint": "^8.48.0", "vite": "^7.3.1" } } ``` ### modal/main.py ```python """Modal app entry point for TreeHacks Vector Search API with Multi-Agent System. This file defines the Modal app configuration and serves the FastAPI application using Modal's serverless infrastructure. Includes support for intelligent search with multi-agent workflow (query expansion, condition analysis, and matching). """ import modal # Create Modal image with all required dependencies image = ( modal.Image.debian_slim() .pip_install( "fastapi[standard]", "elasticsearch", "requests", "agno", # Multi-agent framework "openai", # For Agno agents "dateparser>=1.3.0", # PHASE 1: Temporal detection (natural language date parsing) ) .add_local_python_source("src") ) # Create Modal app app = modal.App("treehacks-vector-search", image=image) @app.function( secrets=[ modal.Secret.from_name("jina-secret"), modal.Secret.from_name("elastic-secret"), modal.Secret.from_name("openai-secret"), # OpenAI API key for multi-agent system ], # Performance optimization: More compute resources cpu=4.0, # 4 CPU cores for faster request processing memory=4096, # 4 GB RAM # Keep warm to reduce cold starts min_containers=1, # Always keep 1 container running scaledown_window=300, # Keep containers alive for 5 minutes after last request # Allow concurrent requests on same container allow_concurrent_inputs=10, # Increased timeout for multi-agent processing timeout=120, # 2 minutes (up from default 60s) ) @modal.asgi_app() def web(): """Modal ASGI app function that returns the FastAPI application.""" from src.api import create_app return create_app() ``` ### frontend/src/App.tsx ```typescript import { InvestigationPage } from "@/components/investigation/InvestigationPage" function App() { return } export default App ``` ### frontend/src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import './index.css' import App from './App.tsx' createRoot(document.getElementById('root')!).render( , ) ``` ### frontend/vite.config.ts ```typescript import path from "path" import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' import tailwindcss from "@tailwindcss/vite" // https://vite.dev/config/ export default defineConfig({ plugins: [react(), tailwindcss()], resolve: { alias: { "@": path.resolve(__dirname, "./src"), }, }, }) ``` ### frontend/eslint.config.js ```javascript import js from '@eslint/js' import globals from 'globals' import reactHooks from 'eslint-plugin-react-hooks' import reactRefresh from 'eslint-plugin-react-refresh' import tseslint from 'typescript-eslint' import { defineConfig, globalIgnores } from 'eslint/config' export default defineConfig([ globalIgnores(['dist']), { files: ['**/*.{ts,tsx}'], extends: [ js.configs.recommended, tseslint.configs.recommended, reactHooks.configs.flat.recommended, reactRefresh.configs.vite, ], languageOptions: { ecmaVersion: 2020, globals: globals.browser, }, }, ]) ``` ### frontend/index.html ```html frontend
``` [70 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]