# Project export: Magifab This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: Every Story, for everyone - Devpost: https://devpost.com/software/magifab - GitHub: https://github.com/dr1810/Magifab - Demo: https://magifab.vercel.app/ - Video: https://www.youtube.com/embed/zC6QzNIvNLs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — dr1810 (83 commits) ## Devpost submission (written by the team) ### Inspiration Walt Disney believed that technology should always serve the story, and that entertainment should be accessible to everyone. That philosophy became one of the foundations of MagiFab. I envision a world where everyone can understand and enjoy stories, regardless of how complex they are. MagiFab's accessibility experience was inspired by applications like MapHabit and BeeVisual, which demonstrate how thoughtful design can make information easier to understand. I wanted anyone—whether they were watching Dune, reading A Memory Called Empire, or experiencing any complex story—to be able to follow and visualize what was happening. One of my biggest inspirations was Mahit Dzmare's relationship with her imago, Yskandr Aghavn: a neurological companion that helps her understand, remember, and reason throughout the story. I imagined MagiFab becoming that kind of companion for every user. I noticed a gap between AI assistants and accessibility tools. Most AI systems expect users to know what questions to ask and often respond with long blocks of text. Many accessibility tools, on the other hand, help people access content but not truly understand the narrative. MagiFab bridges that gap with simple prompt bubbles, visual explanations, and personalized guidance that appear naturally as the story unfolds. Imagine watching a movie with friends when someone struggles to understand what's happening. Instead of pausing the film or relying on someone else to explain every scene, MagiFab quietly provides contextual guidance in real time, helping them catch up and stay engaged without interrupting the experience. To understand what the application does we can divide it into 2 parts: UI/Profile Personalised user onboarding based on what the user struggles with while watching movies, series or reading books. Allows the user to create a personalised AI companion to assist them throughout the experience. A questionnaire which understands how the companion could help them better. If the user wants to understand who a character is and clicks on a prompt, a small water bubble glows, immersing into the colour scheme of the movie or book. If the user wants to understand what is happening in a scene and why, the companion provides contextual explanations supported by simple visual aids. The user simply moves toward the side of the screen and the companion panel appears with intelligent prompt bubbles. Minimalistic visual aids designed specifically for people with disabilities using simple shapes, relationships, timelines and contextual explanations instead of overwhelming diagrams. Model Functionality/Specification Predicts what difficulties the user might face based on their accessibility preferences and answers during onboarding. Analyses movies and books into semantic intervals rather than waiting until the user becomes confused. Builds contextual story memory as the story progresses. Creates intelligent prompt bubbles based on predicted moments of confusion instead of waiting for the user to ask questions. Maintains story state including: Characters Relationships Emotions Timeline Memory Cause & Effect Objects Conversations Story Summary Characters Relationships Emotions Timeline Memory Cause & Effect Objects Conversations Story Summary Generates contextual visual drawers that evolve naturally with the story. Technologies OpenAI Codex with the $100 credits provided through Devpost GPT-5.6 (Terra Medium) TypeScript Python FastAPI HTML CSS JavaScript Gemini 2.5 Flash Google Search Codex with GPT-5.6 I explored how OpenAI Codex and GPT-5.6 could be used to build an application almost entirely through AI-assisted development. Used Codex as an AI software engineering partner throughout the project. Developed the application using GPT-5.6 Terra Medium with structured development workflows instead of one large prompt. Broke development into phases including system architecture, backend pipelines, frontend experience, accessibility features, API integration, debugging, and iterative UI improvements. Used Codex to rapidly generate components, refine architecture, debug issues, and iterate on both the frontend and backend. Backend I designed the complete system architecture before implementing the application. Rather than preprocessing content during playback, MagiFab creates durable accessibility artifacts before a user begins watching a movie or reading a book. During playback or reading, the application simply retrieves the appropriate stored artifact. Movie pipeline The movie pipeline performs: Upload processing FFmpeg 90-second chunk generation Gemini video understanding Google Search evidence retrieval for uncertain entities OpenAI accessibility reasoning using the companion profile Prompt bubble generation Visual Drawer generation Scene artifact generation Persistent scene storage Timestamp-based retrieval during playback Gemini analyzes continuous video chunks rather than isolated frames. When uncertain about identifiable entities, Google Search provides supporting evidence. OpenAI then combines these observations with the user's accessibility profile, companion personality, explanation preferences, and learning needs to generate contextual scene artifacts. Each stored artifact contains: Prompt bubbles Scene explanations Character information Memory cues Visual aids Timeline information Relationship data Companion responses During playback, MagiFab retrieves the closest scene artifact using the movie timestamp. No additional preprocessing or Gemini calls occur while the user watches the movie. Book pipeline Books follow a completely separate pipeline and are never processed as videos. The book workflow performs: PDF, EPUB, or text extraction Front-matter filtering Narrative start detection Chapter and section segmentation Accessibility reasoning Relationship construction Chapter artifact generation Reading-time retrieval Each chapter generates durable accessibility artifacts including: Chapter summaries Simplified explanations Character cards Relationship maps Important events Difficult concepts Memory aids Companion questions During reading, the application retrieves stored chapter artifacts instead of repeatedly reasoning over the original document. Novel Architecture One of MagiFab's primary architectural contributions is separating preprocessing from the user experience. Traditional multimodal assistants repeatedly send content to an LLM whenever a user asks a question or advances through a story. This increases latency, API usage, and the likelihood of inconsistent explanations. Instead, MagiFab performs a one-time preprocessing pipeline that constructs durable accessibility artifacts before the experience begins. For movies, the reasoning engine combines: Gemini video understanding Google Search evidence Companion accessibility profile User explanation preferences Accessibility requirements to generate contextual scene artifacts that can be retrieved instantly during playback. For books, the reasoning engine constructs chapter-level accessibility artifacts that provide summaries, simplified explanations, relationship maps, timelines, memory aids, and companion interactions without reprocessing the original text. This retrieval-first architecture means playback and reading only access stored artifacts through dedicated APIs rather than repeatedly invoking Gemini or OpenAI. The result is: Faster responses Lower API usage Consistent explanations Reduced repeated reasoning Personalized accessibility guidance A scalable retrieval-first architecture Frontend GPT-5.6 helped produce polished, accessible user interfaces with a strong visual hierarchy. The interface was intentionally designed to reduce cognitive overload while allowing assistance to appear only when needed. For movies, users receive contextual prompt bubbles synchronized to the current playback timestamp together with a Visual Drawer containing: Characters Relationships Timeline Objects Memory Emotion Cause & Effect For books, the interface provides a dedicated reading experience with: Chapter navigation Simplified explanations Companion chat Character cards Relationship maps Timelines Memory aids Visual maps By separating preprocessing from retrieval, the frontend remains lightweight and responsive while delivering personalized accessibility assistance exactly when it is needed. Debugging I identified bugs while testing the application. Codex analysed stack traces, logs and backend behaviour to diagnose and resolve issues. Multiple architecture refactors were performed as the reasoning pipeline evolved. ### Challenges we ran into Video storage and selecting the right cloud service to stream large movie files. Designing the UI, especially the prompt panel where there were conflicting interaction states while reaching for the scrollbar. Debugging the UI and making it thoughtful, accessible and intuitive. Making sure API calls were optimised for only $5 worth of credits while still demonstrating complete functionality. Creating prompt bubbles that appear naturally without interrupting immersion. Designing an interval pipeline that analyses the story ahead of the user. Iterating the reasoning system and pipeline multiple times. It should follow the analogy I gave — it shouldn't just create the map, it should continuously learn from that map as the story progresses. Handling hallucinations (for example calling Big Buck Bunny a bear). Multiple architecture refactors while improving reasoning quality. Improving visual recognition for difficult scenes such as Sprite Fright ("The Secret Life of Pets", "young boy", "British flag shirt", "train", etc.). Balancing accessibility with preserving the original storytelling experience. Building a companion that understands context rather than simply recognising objects. ### Accomplishments we're proud of Fully functional UI with playback controls. Personalised accessibility onboarding. AI companion creation workflow. Designed and implemented a novel interval-based story reasoning architecture. Built persistent story memory that evolves throughout the movie. Context-aware prompt generation. Dynamic visual drawers driven by story state. Optimised API usage with limited billing. Successfully built a working end-to-end prototype almost entirely using Codex and GPT-5.6. ### What we learned How to optimise UI/UX for people with different accessibility needs. How to design minimalistic interfaces that reduce cognitive overload. How to architect an AI reasoning pipeline instead of relying on a single prompt. How persistent story memory dramatically improves contextual explanations. How prompt engineering and system architecture directly influence accessibility. How rapidly complex AI systems can be prototyped using Codex and GPT-5.6. ### What's next Improve the movie reasoning pipeline. Extend the same AI companion experience to books and educational content. Improve multimodal understanding across movies, books and visual media. Better predict moments where users may struggle before confusion occurs. Richer interactive visual explanations with lightweight diagrams and animations. More personalised companions that adapt to users over time. Support additional accessibility needs and learning styles. Reduce preprocessing latency while improving explanation quality. ## README (from the GitHub repository) # MagiFab MagiFab is an accessibility companion for movies and books. It creates durable, personalized story artifacts before a person starts watching or reading. Playback and reading only retrieve those artifacts; they never call Gemini or OpenAI to preprocess content. ## Architecture ### Movie pipeline ```text Upload movie → FFmpeg 90-second chunks → Gemini video understanding → Google Search evidence for uncertain entities only → OpenAI accessibility reasoning (companion profile + accessibility needs) → Stored scene artifacts → Playback-time retrieval ``` Gemini receives each continuous video chunk, never a frame-analysis pipeline. Google Search is evidence for uncertain, identifiable entities only. The OpenAI reasoning step receives Gemini observations, that evidence, and the user’s companion profile (personality, accessibility needs, difficulties, and explanation style). It generates prompt bubbles, scene explanations, character information, memory cues, Visual Drawer data, and visual aids. During playback the frontend calls only `GET /api/v1/movies/{movie_id}/scene?timestamp=…` to retrieve the closest stored artifact. Prompt clicks reuse it. Direct questions use the stored-context companion-chat endpoint; neither route triggers preprocessing or Gemini. ### Book pipeline Books are a separate text pipeline and are never treated as movies. ```text PDF / EPUB / text upload → text extraction per page → front-matter filtering (cover, copyright, publisher, TOC) → narrative start detection → chapter/section/page-range segmentation → accessibility reasoning and relationship construction → stored chapter artifacts → reading-time retrieval ``` The repository includes `books/Frank Herbert - Dune 1 - Dune.pdf`. The backend registers it as the Dune example, and the frontend’s Dune tile starts its book-specific process. Artifacts include chapter summaries, simplified explanations, character cards, directional relationship maps, important events, difficult concepts, memory aids, and chapter-level companion questions. The book API stores chapter metadata (chapter number/title and page range) and serves a dedicated reading UI layout: ```text Left panel: cover/progress/chapter list Center panel: chapter summary + simplified explanation Right panel: companion chat and quick questions Bottom tabs: Characters, Relationships, Timeline, Memory Aid, Visual Map ``` ## User experience For movies, an unprocessed upload shows **“Creating your MagiFab companion experience…”** and reports chunking, Gemini analysis, accessibility reasoning, and artifact progress. Once complete, the player uses timestamped prompt bubbles and a Visual Drawer with Characters, Timeline, Objects, Memory, Emotion, and Cause tabs. For Dune and uploaded books, the loading screen says **“Creating your MagiFab reading companion…”** and reports text extraction, chapter understanding, relationship building, and accessibility explanations. Readers can move through chapters, view the simplified artifact, open relationship information, and ask their companion a stored-context question. ## API ### Movies | Endpoint | Purpose | | --- | --- | | `POST /api/v1/movies/upload` | Upload a video and return `movie_id`. | | `POST /api/v1/movies/{movie_id}/preprocess` | Queue the one-time movie pipeline with `companion_profile`. | | `GET /api/v1/movies/{movie_id}/processing-status` | Returns `queued`, `chunking`, `analyzing`, `reasoning`, `complete`, or `failed`, plus progress and percentage. | | `GET /api/v1/movies/{movie_id}/scene?timestamp=` | Returns the closest stored scene artifact. | | `POST /api/v1/movies/{movie_id}/companion/chat` | Answers from stored scene context, companion profile, and question. | | `GET /api/v1/movies/{movie_id}/video` | Development video streaming endpoint. | ### Books | Endpoint | Purpose | | --- | --- | | `POST /api/v1/books/upload` | Upload a PDF, EPUB, or text document. | | `POST /api/v1/books/{book_id}/preprocess` | Queue separate book extraction and accessibility processing. | | `GET /api/v1/books/{book_id}/processing-status` | Read extraction/reasoning progress. | | `GET /api/v1/books/{book_id}/chapters` | Retrieve chapter metadata list and page ranges. | | `GET /api/v1/books/{book_id}/chapter?chapter=` | Retrieve a stored chapter artifact. | | `POST /api/v1/books/{book_id}/companion/chat` | Answer from the stored chapter context. | `GET /api/v1/books/examples/dune` resolves the locally supplied Dune example for the frontend tile. ## Local setup Requirements: Node.js 20+, Python 3.11+, FFmpeg/FFprobe, and Python dependencies from `backend/requirements.txt`. ```bash cd backend python -m venv .venv .venv/bin/pip install -r requirements.txt cp .env.example .env .venv/bin/uvicorn app:app --reload --port 8000 ``` In another terminal: ```bash npm install npm run dev ``` Frontend API calls are always absolute and always use `VITE_MAGIFAB_BACKEND_URL`. Create a frontend env file (for local dev or Vercel): ```dotenv VITE_MAGIFAB_BACKEND_URL=https://magifab.onrender.com ``` No frontend route or Vite proxy should be used for backend API traffic. ### Environment variables Set these in `backend/.env`, never in the frontend: ```dotenv OPENAI_API_KEY=... OPENAI_MODEL=gpt-5.6 GEMINI_API_KEY=... GEMINI_MODEL=gemini-2.5-flash MOVIE_PIPELINE_DIR=cache/movie-pipeline CORS_ORIGINS=http://localhost:5173 ``` The browser has no provider credentials. ### Frontend runtime variables Set this in your frontend environment (`.env.local`, Vercel Project Settings, etc.): ```dotenv VITE_MAGIFAB_BACKEND_URL=https://magifab.onrender.com ``` ## Deployment architecture ```text Frontend (Vercel) ↓ Render FastAPI backend ↓ Gemini ↓ Google Search ↓ OpenAI ↓ Scene cache ``` The development implementation uses SQLite and local blobs beneath `MOVIE_PIPELINE_DIR`. In production replace those storage adapters with object storage and a durable database, run preprocessing through a job worker, and keep the API deployment stateless. The player and reader should remain retrieval-only clients. Provider keys stay in the backend worker/API environment; never expose them through Vite or the browser. ## Detected evidence (automated analysis) Indexed codebase: 263 recognized source files, 1018 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 345) ``` .env.example .gitattributes .gitignore .vscode/extensions.json .vscode/settings.json backend/__init__.py backend/.dockerignore backend/.env.example backend/adapters/__init__.py backend/adapters/florence_adapter.py backend/adapters/gemini_answer_generator.py backend/adapters/gemini_book_semantic_extractor.py backend/adapters/gemini_embeddings.py backend/adapters/gemini_video_provider.py backend/adapters/google_search_provider.py backend/adapters/grounding_dino_adapter.py backend/adapters/openai_answer_generator.py backend/adapters/openai_book_reasoner.py backend/adapters/openai_personalizer.py backend/adapters/openai_scene_reasoner.py backend/adapters/perception_evidence.py backend/adapters/retinaface_arcface_adapter.py backend/adapters/yolo_adapter.py backend/app.py backend/assets/books/.gitkeep backend/assets/books/README.md backend/cache/.gitkeep backend/cache/movie-knowledge/v21/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d.json backend/cache/movie-knowledge/v21/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000000.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000001.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000002.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000003.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000004.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000005.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000006.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000007.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000008.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000009.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000010.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000011.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000012.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000013.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000014.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000015.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000016.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000017.json backend/cache/movie-knowledge/v21/interval-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/00000018.json backend/cache/movie-knowledge/v21/interval-states/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef/00000000.json backend/cache/movie-knowledge/v21/interval-states/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef/00000001.json backend/cache/movie-knowledge/v21/movie-memories/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d.json backend/cache/movie-knowledge/v21/movie-memories/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/10ec6d4366643cbbbda1bc0d65406706e21a4a7e586dc90c0c62c2b474f5a75f.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/4649f8f12b11cd37ca81fb7a15643b973503fb3a132bbac5fb92935d2c63cb1a.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/515ebe5e3d1cfc8d890eab009654361176db41982dcf58b25c4b970d4bf056dc.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/5a7c66f6baa1c69e5ad0464b89df89fefa31927cebcb5fc75586a058f12fd029.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/6977aa56f5214e165c1528088473a7a1dbb7f7869c671ec0a6f3cb3370483409.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/857959aa58ef97b29747deaca303a79720847c24414e3e68cc96746995d388c5.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/9f0676edc2a56df25f4de5cc19cd3ef54e6df80c26d3cf25604d95f6f170ba78.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/a44c3ba40637103ce7f2f439dbc6205e3f583cbb2f9721100f0cb856e50ca17c.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/aab97179e3fc2e8fc625e6597126b289db17bb0898226d8876ba0bcc6aac08bf.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/b3549698818fc52f05bef6d19669dda74a7e5d68e8321c758a0c7aa2fa5ecc41.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/d27dbaacea050a117f79d4ee04063d16a031a6a7046741a3675731383f15b079.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/da293082cba7aa9425143e5a7ae355200170d8237de797aca51e42b4f32dbafe.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/dcdc393f0d470b9891f50ee9c735ddd05a7bb0f02470d284c8f1e17917dfdac0.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/dd2460200f46fcd08c862f95211834f870df3c4e29eb01b079e45dc579b02fa4.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/e879db9213517176e5facda531ae4769b7300541aa02717cac0a5258b8b07540.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/eb1fd8e0736e44a5c87546b606a2aabc12afad1e88afa8fa23f5960790cb81c8.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/f690a354733272a54ecebb310d10fd451f2b4bbf2166b6684379016285862439.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/f9c5418d858695b552af1cab45f3e189b04ce83832561ff5279f764401e9083a.json backend/cache/movie-knowledge/v21/prepared-contexts/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d/fc56214d3b81cf78408cf053271238626035625d02aab5b393d792a38152d838.json backend/cache/movie-knowledge/v21/prepared-contexts/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef/9df7f0299acbf960756e1c991f9cd2ee660c20ecb6dab9f6c8f9f82a22b09249.json backend/cache/movie-knowledge/v21/prepared-contexts/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef/e4f8ac7d412139ab26a444a417fd47188b1f7c48749bceb6544be84760153b54.json backend/cache/movie-knowledge/v21/story-states/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d.json backend/cache/movie-knowledge/v21/story-states/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef.json backend/cache/movie-knowledge/v21/timeline-memory/2024f13514a8bb144397a6b4f7b791c00dd27fb37748ca2ba509bf04664d253d.json backend/cache/movie-knowledge/v21/timeline-memory/b4c0bd1c1d972538092dcbd49f6ae0eba761452618c83424752f8e011bed37ef.json backend/config.py backend/data/movie_knowledge/bigbuckbunny.json backend/data/movie_knowledge/insideout.json backend/data/movie_knowledge/spritefright.json backend/Dockerfile backend/INTERVAL_STATE_ARCHITECTURE.md backend/knowledge_engine/__init__.py backend/knowledge_engine/embeddings.py backend/knowledge_engine/engine.py backend/knowledge_engine/models.py backend/knowledge_engine/README.md backend/knowledge_engine/store.py backend/models/__init__.py backend/models/accessibility_reasoner.py backend/models/answer_generator.py backend/models/book_semantic_extractor.py backend/models/face_embedding_extractor.py backend/models/knowledge_store.py backend/models/language_personalizer.py backend/models/movie_pipeline.py backend/models/object_detector.py backend/models/perception_evidence_adapter.py backend/models/semantic_matcher.py backend/models/text_guided_object_localizer.py backend/models/vision_language_model.py backend/README.md backend/render.yaml backend/requirements.txt backend/routers/__init__.py backend/routers/accessibility_reasoning.py backend/routers/books.py backend/routers/companion_pipeline.py backend/routers/debug.py backend/routers/detect.py backend/routers/face_verification.py backend/routers/fusion.py backend/routers/grounding.py backend/routers/health.py backend/routers/knowledge_expansion.py backend/routers/knowledge.py backend/routers/match.py backend/routers/movies.py backend/routers/personalization.py [225 more files omitted for size] ``` ### Dependencies - backend/requirements.txt: fastapi@==0.115.12, google-genai@==1.29.0, openai@==1.58.1, pydantic-settings@==2.7.1, pypdf@==5.7.0, python-dotenv@==1.0.1, python-multipart@==0.0.20, uvicorn[standard]@==0.34.0 - package.json: @supabase/supabase-js@^2.110.7, @types/node@^22.10.2, @types/react@^19.0.2, @types/react-dom@^19.0.2, autoprefixer@^10.4.20, framer-motion@^12.4.1, lucide-react@^0.468.0, pdfjs-dist@^5.4.54, postcss@^8.4.49, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^3.4.17, tus-js-client@^4.3.1, typescript@^5.7.2, vite@^6.0.5 ### Recent commits (newest first) - patch - book companion fix - backend fix - Configured the backed to have the book - Backend fix for book - (fix): Backend fix - (fix): Fixing the UI of the Book Companion - (fix):Fixing the pipeline - (fix): Fixed the underlying issues - (fix): removed old dependencies - frontend update - feat: integrate Gemini, OpenAI, and search-powered accessibility pipeline - Fresh retrieval runs for every question; no prior answer/prompt reuse path. - Added a collapsible development-only panel showing all six stages plus the original question. - Added semantic intent routing before Gemini answer generation. - Replaced hash/whole-interval lookup with persisted typed chunks: scenes, events, dialogue, characters, relationships, objects, timelines, and book paragraphs. - Adds a planning pass before retrieval: intent, evidence needs, search queries, timeline scope, and conversation-memory use. - Replaced generic single-interval answer branch with LLM-grounded retrieval - (refactor): Refactoring the AI architecture - Centralized mutually-exclusive overlays ## Key source files (fetched from GitHub, selected and truncated for size) ### CodexUsage.md ```markdown # **How Codex and GPT 5.6 were used to build the project** - Create boiler template for the project ``` ### backend/INTERVAL_STATE_ARCHITECTURE.md ```markdown # IntervalState ownership `IntervalState` is MagiFab's only browser-facing playback object. Movie load preprocesses every fixed 30-second interval in chronological order. Each `POST /api/v1/companion/intervals/prepare` call samples one interval and persists its frozen snapshot. Playback and seeking only select that stored snapshot by timestamp; they never invoke semantic reasoning or prompt generation. `POST /api/v1/companion/respond` loads the already-persisted snapshot and may return a transient prompt answer beside its immutable prompt set. A missing snapshot is a preprocessing failure (409), not a fallback to live generation. ## Field ownership | Field | Owner | Consumers | | --- | --- | --- | | `metadata` | interval assembly in `CompanionPipelineService` | all UI surfaces; identifies the exact movie interval and knowledge revision | | `metadata.interval_id`, `start_time`, `end_time`, `interval_number` | fixed 30-second `IntervalStateRepository` boundary | cache invalidation and time-aware rendering | | `metadata.catalog_scene_id`, `movie_id`, `knowledge_revision` | optional catalog enrichment and interval assembly | diagnostics; never playback identity | | `prompts.prompt_bubbles` | prompt ranking over the resolved story interval | prompt panel and bubbles | | `prompts.prompt_answers` | prompt-response assembly | companion bubble/widget | | `prompts.suggested_questions` | prompt ranking | future question affordances | | `visualDrawer` | `StoryStatePresenter` projection copied into the snapshot | Visual Drawer only | | `storyState` | `StoryStatePresenter` projection copied into the snapshot | story summary and timeline context | | `characters` | presenter-derived active characters | character cards | | `relationships` | resolved story relationships | relationship cards and drawer | | `memory` | resolved story memory reminders | memory UI | | `conversationContext` | current scene explanation and simplified dialogue | companion response text | | `accessibilityHints` | profile-adapted vocabulary and emotion guidance | accessibility affordances | | `semanticMemoryBefore` | frozen cumulative memory before the interval | preprocessing validation and traceability | | `semanticMemoryAfter` | frozen cumulative memory after the interval | all interval reconstruction and validation | | `cacheMetadata` | interval assembly | cache provenance and semantic revision diagnostics | `PreprocessingStoryBuilder` and `TimelineMemoryService` are chronological construction aids used only while generating IntervalStates. They are not API contracts and are never consulted during playback. `IntervalStateRepository` is the runtime store: it validates interval identity/range and retrieves an immutable snapshot for any timestamp without forward-only assumptions. There are no public drawer, live-story, character-card, timeline, or prompt response envelopes. The two companion endpoints each serialize a single `IntervalState` object at their response root. A prompt ans [truncated — 2666 more characters] ``` ### package.json ``` { "name": "magifab", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "preview": "vite preview", "upload:movies": "node scripts/upload-movies.mjs", "validate:timeline": "node scripts/validate-story-beats.mjs" }, "devDependencies": { "@types/node": "^22.10.2", "@types/react": "^19.0.2", "@types/react-dom": "^19.0.2", "autoprefixer": "^10.4.20", "postcss": "^8.4.49", "tailwindcss": "^3.4.17", "typescript": "^5.7.2", "vite": "^6.0.5" }, "dependencies": { "@supabase/supabase-js": "^2.110.7", "framer-motion": "^12.4.1", "lucide-react": "^0.468.0", "pdfjs-dist": "^5.4.54", "react": "^19.0.0", "react-dom": "^19.0.0", "tus-js-client": "^4.3.1" } } ``` ### backend/requirements.txt ``` # The retired local YOLO / Florence / Grounding DINO stack is intentionally absent. fastapi==0.115.12 uvicorn[standard]==0.34.0 pydantic-settings==2.7.1 python-dotenv==1.0.1 openai==1.58.1 google-genai==1.29.0 python-multipart==0.0.20 pypdf==5.7.0 ``` ### backend/Dockerfile ``` FROM python:3.12-slim WORKDIR /app ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 COPY requirements.txt ./ RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg \ && rm -rf /var/lib/apt/lists/* \ && pip install --no-cache-dir -r requirements.txt # Ensure bundled example books are present in the runtime image. COPY assets/books ./assets/books COPY . ./ EXPOSE 8000 CMD ["sh", "-c", "uvicorn app:app --host 0.0.0.0 --port ${PORT:-8000}"] ``` ### src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import App from './App' import { AccessibilityProvider } from './accessibility-context' import './index.css' import './movie-experience.css' createRoot(document.getElementById('app')!).render( , ) ``` ### backend/app.py ```python """MagiFab's server-only movie preprocessing API.""" from __future__ import annotations from contextlib import asynccontextmanager import logging from functools import lru_cache from fastapi import FastAPI, HTTPException, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse from config import Settings, get_settings from services.movie_pipeline_retry import RetryExecutor from services.movie_pipeline_service import MoviePipelineService from services.movie_pipeline_storage import LocalMovieBlobStorage, SqliteMoviePipelineRepository from services.book_pipeline_service import BookPipelineService from services.example_books import BOOKS_DIR, discover_example_books from services.gemini_client import GeminiClient, GeminiClientConfigurationError, validate_gemini_sdk_import from services.video_chunk_service import FfmpegVideoChunker def configure_logging(settings: Settings) -> None: logging.basicConfig(level=settings.log_level.upper(), format="%(asctime)s %(levelname)s %(name)s %(message)s") @lru_cache def get_movie_blob_storage() -> LocalMovieBlobStorage: return LocalMovieBlobStorage(get_settings().movie_pipeline_dir) @lru_cache def get_movie_pipeline_repository() -> SqliteMoviePipelineRepository: return SqliteMoviePipelineRepository(get_settings().movie_pipeline_dir) @lru_cache def get_movie_pipeline_service() -> MoviePipelineService: """The only AI pipeline: Gemini video → Google Search → OpenAI scene reasoning.""" from adapters.gemini_video_provider import GeminiVideoProvider from adapters.google_search_provider import GoogleSearchGroundingProvider from adapters.openai_scene_reasoner import OpenAISceneReasoner settings = get_settings() gemini_client = GeminiClient.from_settings(settings) return MoviePipelineService( repository=get_movie_pipeline_repository(), blobs=get_movie_blob_storage(), chunker=FfmpegVideoChunker(get_movie_blob_storage()), visual_provider=GeminiVideoProvider(settings, gemini_client=gemini_client), search_provider=GoogleSearchGroundingProvider(settings, gemini_client=gemini_client), reasoning_provider=OpenAISceneReasoner(settings), chunk_duration_seconds=settings.movie_chunk_duration_seconds, retry_executor=RetryExecutor(settings.movie_pipeline_retry_attempts, settings.movie_pipeline_retry_base_seconds), model_versions={"gemini": settings.gemini_model, "openai": settings.openai_model, "scene_schema": "magifab-scene-v1"}, gemini_client=gemini_client, ) @lru_cache def get_book_pipeline_service() -> BookPipelineService: from adapters.openai_book_reasoner import OpenAIBookReasoner settings = get_settings() service = BookPipelineService(settings.movie_pipeline_dir / "book-pipeline", OpenAIBookReasoner(settings)) logger = logging.getLogger(__name__) for key, path in discover_example_books().items(): registered = service.register_example(path, title=_display_title_from_stem(path.stem)) if registered: logger.info("Registered bundled example '%s' from %s", key, path) return service def create_app(settings: Settings | None = None) -> FastAPI: active_settings = settings or get_settings() configure_logging(active_settings) @asynccontextmanager async def lifespan(_: FastAPI): _validate_startup_dependencies(active_settings) yield application = FastAPI( title=active_settings.app_name, version=active_settings.api_version, description="MagiFab offline movie preprocessing and stored-scene retrieval API.", lifespan=lifespan, ) origins = [origin.strip() for origin in active_settings.cors_origins.split(",") if origin.strip()] application.add_middleware( CORSMiddleware, allow_origins=origins, allow_credentials=False, allow_methods=["GET", "POST", "OPTIONS"], allow_headers=["Content-Type", "Authorization"], ) @application.exception_handler(Exception) async def unhandled_exception(_: Request, error: Exception) -> JSONResponse: if isinstance(error, HTTPException): return JSONResponse(status_code=error.status_code, content={"detail": error.detail}) logging.getLogger(__name__).exception("Unhandled request failure", exc_info=error) return JSONResponse(status_code=500, content={"detail": "Internal server error"}) from routers.health import router as health_router from routers.movies import router as movies_router from routers.books import router as books_router application.include_router(health_router) application.include_router(movies_router) application.include_router(books_router) return application def _validate_startup_dependencies(settings: Settings) -> None: logger = logging.getLogger(__name__) if not settings.gemini_api_key or not settings.gemini_api_key.get_secret_value().strip(): raise RuntimeError("Backend startup validation failed: GEMINI_API_KEY is required but missing.") try: validate_gemini_sdk_import() except GeminiClientConfigurationError as error: raise RuntimeError(f"Backend startup validation failed: {error}") from error try: from openai import OpenAI except Exception as error: raise RuntimeError( "Backend startup validation failed: OpenAI SDK import failed. Install dependencies with `pip install -r backend/requirements.txt`." ) from error if OpenAI is None: raise RuntimeError("Backend startup validation failed: OpenAI SDK import returned no client type.") _log_example_books_catalog(logger) logger.info("Startup dependency validation succeeded for Gemini and OpenAI SDKs.") def _log_example_books_catalog(logger: logging.Logger) -> None: logger.info("Books directory:\n%s", BOOKS_DIR.resolve()) discovered = discover_example_books() if [truncated — 622 more characters] ``` ### src/movie-data/index.ts ```typescript import { bigBuckBunnyData } from './bigBuckBunny' import { spriteFrightData } from './spriteFright' import type { MovieData, MovieId, SceneData } from '../types/movie' export const demoMovies: MovieData[] = [bigBuckBunnyData, spriteFrightData] export const movieById: Partial> = { bigBuckBunny: bigBuckBunnyData, spriteFright: spriteFrightData, } export function getSceneAtTimestamp(movie: MovieData, timestamp: number): SceneData { const sorted = [...movie.scenes].sort((a, b) => a.timestamp - b.timestamp) const match = sorted.filter((scene) => scene.timestamp <= timestamp).at(-1) return match ?? sorted[0] } ``` ### src/App.tsx ```typescript import { motion } from 'framer-motion' import { useState } from 'react' import { Accessibility, Check, ChevronLeft, Eye, Film, RotateCcw, Save, Sparkles, Volume2 } from 'lucide-react' import { presets, useAccessibility, type Settings } from './accessibility-context' import { Onboarding } from './onboarding' import { MovieSelector } from './components/MovieSelector' import { MovieViewer } from './movie-viewer' import { BookViewer } from './book-viewer' import type { MovieId } from './types/movie' import { useAccessibilityProfile } from './hooks/useAccessibilityProfile' import { HomeProfileButton } from './components/HomeProfileButton' import { LandingPage } from './landing-page' import { ProfileEditor } from './components/ProfileEditor' import { resetPlaybackTimestamp } from './services/playbackSessionService' import { CompanionDebugPanel } from './components/CompanionDebugPanel' type Key = keyof Settings const fontSizes: Settings['fontSize'][] = ['Small', 'Medium', 'Large', 'Extra Large'] const fontFamilies: Settings['fontFamily'][] = ['Default', 'OpenDyslexic', 'Atkinson Hyperlegible', 'Lexend', 'Inter'] const withDebugPanel = (view: React.ReactNode) => <>{view} function Segmented({ label, setting, options }: { label: string; setting: K; options: Settings[K][] }) { const { settings, update } = useAccessibility() return
{options.map((option) => )}
 } function Toggle({ label, setting, description }: { label: string; setting: K; description?: string }) { const { settings, update } = useAccessibility(); const active = Boolean(settings[setting]) return } function Slider({ label, setting, min, max, step = 1, suffix = '' }: { label: string; setting: K; min: number; max: number; step?: number; suffix?: string }) { const { settings, update } = useAccessibility(); const value = settings[setting] as number return update(setting, Number(event.target.value) as Settings[K])} className="accent-indigo-600" /> } function Control({ label, value, children }: { label: string; value?: string; children: React.ReactNode }) { return
{label}{value && {value}}
{children}
} function SettingCard({ title, icon, children }: { title: string; icon: React.ReactNode; children: React.ReactNode }) { return
{icon}

{title}

{children}
} function Preview() { const { settings } = useAccessibility() const dark = settings.theme === 'Dark' || (settings.theme === 'System' && window.matchMedia('(prefers-color-scheme: dark)').matches) const previewStyle = { fontFamily: settings.fontFamily === 'Lexend' ? 'Lexend, sans-serif' : settings.fontFamily === 'Inter' ? 'Inter, sans-serif' : settings.fontFamily === 'Atkinson Hyperlegible' ? 'Arial, sans-serif' : settings.fontFamily === 'OpenDyslexic' ? 'Verdana, sans-serif' : 'system-ui, sans-serif', fontSize: settings.fontSize === 'Small' ? 12 : settings.fontSize === 'Large' ? 16 : settings.fontSize === 'Extra Large' ? 18 : 14, letterSpacing: `${settings.letterSpacing}px`, lineHeight: settings.lineHeight, maxWidth: `${settings.readingWidth}%`, } const radius = `${settings.roundedCorners}px`, subtitleTop = settings.subtitlePosition === 'Top' const subtitleBackground = settings.subtitleBackground === 'Solid' ? 'rgba(0,0,0,.93)' : settings.subtitleBackground === 'Semi Transparent' ? 'rgba(0,0,0,.62)' : 'transparent' const promptClass = settings.promptPosition === 'Bottom' ? 'bottom-3 left-1/2 -translate-x-1/2' : settings.promptPosition === 'Left' ? 'bottom-5 left-4' : 'bottom-5 right-4' const highlight = settings.highlightStyle === 'Glow' ? 'ring-4 ring-amber-300/80 shadow-[0_0_24px_8px_rgba(251,191,36,.55)]' : settings.highlightStyle === 'Outline' ? 'ring-4 ring-cyan-300' : settings.highlightStyle === 'Pulse' ? 'ring-4 ring-indigo-300 animate-pulse' : settings.highlightStyle === 'Arrow' ? 'ring-4 ring-rose-300 before:content-["↑"]' : settings.highlightStyle === 'Underline' ? 'border-b-4 border-amber-300' : '' return
; companion: { personality: string; conversationStyle: string; detailLevel: string }; }; type PersonalizeResponse = { explanation: string; emotion: string }; type ExplainSceneResponse = { explanation: string; emotion: string; character: string | null; characterFound: boolean; confidence: number; anchor: Anchor; visualAidType: "magnifier" | "highlight"; }; type OpenAIResponsePayload = { id?: unknown; status?: unknown; output_text?: unknown; output?: unknown; error?: unknown }; const MAX_FRAME_DATA_URL_LENGTH = 3_000_000; const corsHeaders = { "Access-Control-Allow-Headers": "authorization, x-client-info, apikey, content-type", "Access-Control-Allow-Methods": "POST, OPTIONS", "Access-Control-Allow-Origin": "*", }; const responseSchema = { type: "object", additionalProperties: false, required: ["explanation", "emotion", "character", "characterFound", "confidence", "anchor", "visualAidType"], properties: { explanation: { type: "string", description: "A concise, accessible explanation in plain language." }, emotion: { type: "string", description: "The character's clearest current emotion." }, character: { type: ["string", "null"], description: "The identified character's name. Use null when no person is visible or identity cannot be determined." }, characterFound: { type: "boolean", description: "True only when a visible person can be identified from the movie frame with confidence." }, confidence: { type: "number", minimum: 0, maximum: 1 }, anchor: { type: "object", additionalProperties: false, required: ["x", "y", "width", "height"], properties: { x: { type: "number", minimum: 0, maximum: 100, description: "Character center x as a percentage of the image width." }, y: { type: "number", minimum: 0, maximum: 100, description: "Character center y as a percentage of the image height." }, width: { type: "number", minimum: 1, maximum: 100, description: "Character bounding-box width as a percentage of image width." }, height: { type: "number", minimum: 1, maximum: 100, description: "Character bounding-box height as a percentage of image height." }, }, }, visualAidType: { type: "string", enum: ["magnifier", "highlight"] }, }, }; const personalizeResponseSchema = { type: "object", additionalProperties: false, required: ["explanation", "emotion"], properties: { explanation: { type: "string", description: "A concise, accessible explanation based only on supplied facts." }, emotion: { type: "string", description: "The supported emotion, or neutral when none is supplied." }, }, }; function json(body: unknown, status = 200) { return Response.json(body, { status, headers: corsHeaders }); } function readText(value: unknown, limit = 2_000): string | null { if (typeof value !== "string") return null; const text = value.trim(); return text && text.length <= limit ? text : null; } function readAnchor(value: unknown): Anchor | null { if (!value || typeof value !== "object") return null; const anchor = value as Record; const values = [anchor.x, anchor.y, anchor.width, anchor.height]; if (!values.every((item) => typeof item === "number" && Number.isFinite(item))) return null; const [x, y, width, height] = values as number[]; if (x < 0 || x > 100 || y < 0 || y > 100 || width <= 0 || width > 100 || height <= 0 || height > 100) return null; return { x, y, width, height }; } function validateRequest(value: unknown): ExplainSceneRequest | null { if (!value || typeof value !== "object") return null; const request = value as Record; const companion = request.companion; const frame = request.frame; if (!companion || typeof companion !== "object" || !frame || typeof frame !== "object") return null; const companionValue = companion as Record; const frameValue = frame as Record; const question = readText(request.question); const scene = readText(request.scene); const personality = readText(companionValue.personality); const conversationStyle = readText(companionValue.conversationStyle); const detailLevel = readText(companionValue.detailLevel); const dataUrl = readText(frameValue.dataUrl, MAX_FRAME_DATA_URL_LENGTH); const timestamp = request.timestamp; const width = frameValue.width; const height = frameValue.height; if (!question || !scene || !personality || !conversationStyle || !detailLevel || !dataUrl?.startsWith("data:image/") || typeof timestamp !== "number" || !Number.isFinite(timestamp) || typeof width !== "number" || typeof height !== "number" || width <= 0 || height <= 0) return null; return { question, scene, timestamp, frame: { dataUrl, width, height }, companion: { personality, conversationStyle, detailLevel }, }; } function validatePersonalizeRequest(value: unknown): PersonalizeRequest | null { if (!value || typeof value !== "object") return null; const request = value as Record; if (request.mode !== "personalize" || !request.companion || typeof request.companion !== "object" || !request.semanticContext || typeof request.semanticContext !== "object") return null; const companion = request.companion as Record; const question = readText(request.question); const personality = readText(companion.personality); const conversationStyle = readText(companion.conversationStyle); [truncated — 9014 more characters] ``` [252 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]