Project Info
Inspiration
The idea came from watching how people used our search product. After finding a clip, they immediately wanted to know more about it: What's happening here? Who's speaking? What's the intent or tone of this scene? Where does this specific object appear? They weren't searching across many videos anymore, they were interrogating a single one. We realized these were two different problems. Our indexing pipeline was designed for breadth: processing large libraries and making them searchable. Once you've found a specific video, however, you want depth. We'd been treating those as the same feature, but they're not. The image search capability came from another common request. If you're looking for every shot containing a product, logo, or person, describing it with words isn't always the best interface. Sometimes you already know exactly what you're looking for, you should be able to simply show it.
What it does
Apparat makes video archives searchable by what's actually inside them. Media companies own decades of footage that cost real money to produce, but most of it can only be found if someone tagged it correctly years ago. Our core product ingests a video library, analyzes each clip across visual, audio, and dialogue signals, and lets users find any moment by simply describing it. For this hackathon, we built a complementary feature: Video Analysis. Instead of searching across an entire library, it analyzes a single video in depth.
How we built it
The feature runs on our existing technology stack. Backend: TypeScript services running on Google Cloud Run Agent layer: Python services deployed as Cloud Run Jobs Frontend: React Models: Gemini Flash and Flash Lite Vector database: NeonDB with pgvector Storage: Google Cloud Storage The biggest architectural decision was keeping Video Analysis separate from our indexing pipeline instead of adding it to the existing workflow. Indexing is optimized for processing many videos and building a persistent knowledge graph. That overhead isn't necessary when analyzing a single video. We built a lighter processing path that skips the persistent knowledge graph and works directly on one file. The result is faster, simpler, and easier to maintain. For image search, users upload a reference image, which our multimodal agent uses to locate every matching appearance within the video. Building that capability was relatively straightforward because the underlying foundation already existed. Fine-tuning it was much harder. Throughout the project we used Codex, primarily for scaffolding services, wiring backend endpoints, and implementing frontend components. It significantly accelerated implementation, but it didn't make the key product or engineering decisions. We still designed the architecture, decided to separate the analysis pipeline from indexing, and determined how the matching system should work. Codext made development faster once we knew exactly what we wanted to build.
Challenges we ran into
Matching Thresholds Finding the right balance for image matching proved difficult. If the thresholds were too loose, the system returned visually similar frames that weren't actually matches. Too strict, and it missed valid matches because the object appeared from a different angle or was partially occluded. There wasn't a perfect answer, only repeated testing against real footage until the trade-offs became acceptable. Trustworthy Answers Natural-language analysis is only valuable if users can trust the results. A confident but incorrect answer is worse than no answer at all. We spent more time than expected grounding responses with precise timestamps so users could quickly verify the output themselves. Scope Midway through the hackathon, we cut two features that we realized we couldn't finish at the quality we wanted. It was the right decision, even if it was frustrating.
Accomplishments we're proud of
We shipped a fully working product. Video analysis, natural-language questions, summaries, speaker identification, emotional and intent analysis, and image-based search all work end to end. Nothing shown in the demo is mocked or staged. Midway through the hackathon, we deliberately cut two planned features so we could focus on delivering a smaller set of capabilities at production quality. Image-Based Search Image-based search was the feature we were least certain would work when we started. Matching a reference image reliably across an entire video (despite changes in angle, lighting, scale, and partial occlusion) required more tuning than any other part of the project. The result is a feature that can locate a specific visual element anywhere in a video, making it genuinely useful rather than just a technical demonstration. Performance We kept the experience fast. Video ingestion completes in under one minute per hour of footage. Single-video analysis returns results in seconds. It would have been easy to build a system that worked but felt too slow for real users. Separating single-video analysis from the archive indexing pipeline proved to be the key architectural decision that made this possible. Trustworthy Results Every answer is grounded in specific timestamps. Instead of simply returning an answer, Apparat shows users exactly where the supporting evidence appears in the video, allowing them to verify the result immediately. Making the system verifiable required considerably more work, but it's one of the decisions we're most confident in. Development Process We built the entire feature in under one week. Without Codex, this would likely have taken several weeks of focused engineering, much of it spent scaffolding services, wiring APIs, and implementing frontend infrastructure. Codex dramatically accelerated implementation, but it didn't replace the important engineering decisions. We still designed the architecture, chose to separate the analysis pipeline from the indexing pipeline, determined the image-matching thresholds, and decided that every answer should be grounded in timestamps. Those decisions shaped the product. The coding agent helped us build it much faster.
What we learned
The biggest lesson wasn't technical, it was about defining the problem correctly. We'd been thinking of "understanding video" as a single challenge. It's actually at least two: Finding content across a large archive Understanding a single video in depth Separating those workflows made both products better. The second lesson was about working with Codex. They dramatically reduced implementation time, but they also made upfront thinking even more important. When we had a clear plan, development moved quickly. When our requirements were vague, the agent produced plausible but incorrect solutions, and we lost time correcting them.
What's next
Over time, single-video analysis and archive-wide search should become one continuous workflow. Users should be able to search an archive, open a result, analyze it in depth, and use those insights to continue searching, all within the same experience. Today those are separate features. Connecting them seamlessly is our next major milestone. We're especially excited about image-based search. Once it scales from a single video to an entire archive, users will be able to ask: "Find every frame containing this." That's a fundamentally different capability from traditional metadata or tag-based search, and it opens up entirely new ways of working with large video libraries.
This project did not link a GitHub repository.
Analysis
No indexed repository for this project, so there are no commit stats to show.
Technology
- FirebaseUnchecked
- PythonUnchecked
- ReactUnchecked
- TypeScriptUnchecked
No repository was indexed for this project, so these Devpost claims have not been checked against code.
AI coding agents
No repository was indexed, so agent usage could not be checked.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
No repository was indexed, so there is no codebase to measure.
This project did not link a GitHub repository, so there is nothing to diagram.
This project did not link a GitHub repository, so its feature claims have not been checked against code.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.