# Project export: Vidscribe This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: An interactive study notebook and video lecture annotation tool - Devpost: https://devpost.com/software/vid-scribe - GitHub: https://github.com/adrian-perez121/VidScribe - Demo: https://vidscribe-vdk5.onrender.com/ - Video: https://www.youtube.com/embed/fTBgzcah3Ho?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 5 GitHub contributor(s) — Adrian Perez (22 commits), Halren (20 commits), David (19 commits), Claude Sonnet 4.6 (15 commits), angelinedo (1 commits) ## Devpost submission (written by the team) ### Inspiration If students can write notes in the margins of books and PDFs, why not do the same for video lectures? Students increasingly use AI transcript tools to convert lectures, class recordings, and educational videos into notes. But current systems have several problems: AI transcripts frequently misinterpret words because of accents, technical vocabulary, poor audio quality, fast speech, or speaker pronunciation. Students use little to no effort in learning when using AI transcribing and note-taking tools, leading them to have insufficient knowledge. Important visual content—diagrams, equations, slides, charts, demonstrations—often gets lost entirely. Students constantly pause and rewind lectures to manually write notes. Notes become disconnected from the original explanation, making it difficult to revisit concepts later. ### What it does With Vidscribe, students can create notes attached directly to any timestamp while watching a lecture or educational video. Students can also annotate directly on the video using a select-and-annotate tool, allowing them to highlight important concepts, diagrams, equations, images, or specific areas on the screen. To improve accessibility, Vidscribe also includes a speech-to-text microphone feature powered by Deepgram, allowing students to verbally record notes instead of typing. This is especially beneficial for students with disabilities, including dyslexic students, by making note-taking faster and more accessible. If students need additional clarification on a topic, an integrated AI agent is available to answer questions in real time. Using the select tool, students can highlight equations, diagrams, images, or sections of the video and ask for further explanation or context. Once the student finishes watching the video, Vidscribe generates a transcript and combines it with the student’s annotations, timestamped notes, spoken notes, and AI-provided explanations. This information is then automatically compiled into organized study notes, creating a personalized study guide directly connected to the original lecture. Students can also generate flashcards, review questions, and other study materials from their compiled notes to reinforce learning and improve retention. ### How we built it To build Vidscribe, we combined several AI and infrastructure tools into one seamless, intelligent learning experience. Deepgram — we used Deepgram's Speech-to-Text API to accurately transcribe lectures into timestamped text that merges with student notes and annotations. It also powers the microphone feature, letting students speak their notes instead of typing — improving accessibility for students with disabilities, including dyslexic learners. Deepgram — we used Deepgram's Speech-to-Text API to accurately transcribe lectures into timestamped text that merges with student notes and annotations. It also powers the microphone feature, letting students speak their notes instead of typing — improving accessibility for students with disabilities, including dyslexic learners. Anthropic Claude — we used Claude Code to accelerate development across the front-end and back-end, enabling rapid prototyping and feature implementation. At runtime, the Claude API powers the product's intelligence: explaining screenshots captured from videos, answering questions in the study chatbot, and generating study guides and flashcards — all grounded strictly in the student's own notes and transcripts so it never invents facts. Anthropic Claude — we used Claude Code to accelerate development across the front-end and back-end, enabling rapid prototyping and feature implementation. At runtime, the Claude API powers the product's intelligence: explaining screenshots captured from videos, answering questions in the study chatbot, and generating study guides and flashcards — all grounded strictly in the student's own notes and transcripts so it never invents facts. Redis — Redis 8 is the backbone of our study layer. We use it as a vector database (RediSearch) to embed and semantically search a student's notes, transcripts, research, and visual explanations, powering a retrieval-augmented chatbot that cites the exact video and timestamp an answer came from. Redis also stores per-session conversation memory, caches generated study guides and answers for instant reuse, and schedules spaced-repetition flashcards with sorted sets so "what's due now" is an O(log n) lookup. Redis — Redis 8 is the backbone of our study layer. We use it as a vector database (RediSearch) to embed and semantically search a student's notes, transcripts, research, and visual explanations, powering a retrieval-augmented chatbot that cites the exact video and timestamp an answer came from. Redis also stores per-session conversation memory, caches generated study guides and answers for instant reuse, and schedules spaced-repetition flashcards with sorted sets so "what's due now" is an O(log n) lookup. OpenAI — we use OpenAI's text-embedding-3-small model to turn notes and transcript chunks into vector embeddings, which Redis indexes for fast semantic retrieval. OpenAI — we use OpenAI's text-embedding-3-small model to turn notes and transcript chunks into vector embeddings, which Redis indexes for fast semantic retrieval. Browserbase — we integrated Browserbase to search the web for resources related to lecture topics, identifying key concepts, visiting relevant pages, and generating concise summaries. This encourages students to explore further without leaving the lecture page, reducing opportunities for distraction. Browserbase — we integrated Browserbase to search the web for resources related to lecture topics, identifying key concepts, visiting relevant pages, and generating concise summaries. This encourages students to explore further without leaving the lecture page, reducing opportunities for distraction. Google Gemini — we used the Gemini API to condense larger chunks of text into concise 5–6 word topic labels, which are then passed to Browserbase for more targeted searches. Google Gemini — we used the Gemini API to condense larger chunks of text into concise 5–6 word topic labels, which are then passed to Browserbase for more targeted searches. MongoDB — we used MongoDB (with GridFS) to store uploaded videos, timestamped notes, annotations, transcripts, and flashcards, enabling efficient organization and retrieval across learning sessions. MongoDB — we used MongoDB (with GridFS) to store uploaded videos, timestamped notes, annotations, transcripts, and flashcards, enabling efficient organization and retrieval across learning sessions. Hono — a lightweight Hono server on Node.js runs the backend API, with a React + Vite + TypeScript front-end. Students can also export their notes and study guides to Word (.docx). Hono — a lightweight Hono server on Node.js runs the backend API, with a React + Vite + TypeScript front-end. Students can also export their notes and study guides to Word (.docx). ### Challenges we ran into During development, we encountered several technical and workflow challenges. One of our biggest issues was unreliable Wi-Fi connectivity, which prevented us from consistently connecting to our database server and forced us to temporarily rely on mobile hotspots to continue development. We also spent time deciding between using a SQL or NoSQL database architecture. Since Vidscribe needs to manage larger files and flexible user-generated content such as annotations, notes, and media-related data, we ultimately chose MongoDB because its document-based structure better fit our needs. Another challenge involved Redis Cloud, which unexpectedly stopped working during development. Initially, we struggled to identify the problem because an important update regarding the issue had been posted earlier in Slack and was easy to miss. We also encountered difficulties with AI development workflows, particularly when running out of conversation context during coding sessions, which caused us to lose parts of our progress and planning history. This required us to spend additional time reconstructing our ideas and reestablishing development context before moving forward. ### Accomplishments we're proud of We are proud that we successfully achieved the core goals we initially scoped for Vidscribe and were able to turn our idea into a functional product within a limited timeframe. Throughout development, our team maintained a smooth workflow and experienced very few merge conflicts, which allowed us to spend more time building features rather than resolving technical issues. We were also able to successfully integrate a strong variety of sponsor tools into the project, leveraging their capabilities to enhance functionality and create a more intelligent and seamless user experience. Most importantly, our collaboration remained efficient throughout the project, with strong communication and teamwork allowing us to rapidly iterate on ideas and overcome challenges together. ### What we learned Throughout the development of Vidscribe, we learned how to integrate and coordinate multiple APIs within a single application to create a seamless user experience. Since our project relied on several tools and services working together, we gained experience connecting AI models, speech-to-text systems, memory services, databases, and web automation tools into one workflow. We also learned how important it is to design systems where each tool has a specific role and can effectively communicate with others. Beyond the technical side, we gained experience in debugging integrations, managing development workflows, and collaborating as a team while building a project under time constraints. ### What's next Moving forward, we plan to expand Vidscribe beyond educational lectures into film studies and media analysis, allowing students to use timestamped annotations to analyze scenes, themes, dialogue, and cinematography. We also plan to develop Vid Scribe into a Chrome extension that integrates with platforms like Canvas and other video-hosting sites, making it easier to use within existing learning environments. ## README (from the GitHub repository) # Vidscribe A video annotation tool that lets you take notes on video lectures and automatically research concepts from the transcript using live web sources. Notes are timestamped to the exact moment in the video, giving you a study notebook that remembers where every idea came from. On top of that, a Redis-powered study layer turns your notes and transcripts into a RAG chatbot, study guides, and spaced-repetition flashcards — all exportable to Word. Demo video [here](https://www.youtube.com/watch?v=fTBgzcah3Ho). ## Features **Text notes** — write a note at any playback position; the timestamp is saved automatically. **Voice notes** — record a quick observation via microphone; Deepgram transcribes it and the text is saved as a note. **Visual notes** — draw a crop box over any frame; Claude explains what is in that region in plain language. **AI research** — highlight a chunk of transcript; the server distills keywords, searches the web via Browserbase, and returns a teacher-style summary plus the source links. The summary is explicitly tied back to what is on screen. **Lecture transcripts** — Deepgram transcribes the full video into timestamped segments, stored once per video and reused by the study tools below. **Study chatbot** — ask questions across your videos. A Redis vector search (RAG) retrieves the most relevant notes, research summaries, visual explanations, and transcript chunks; Claude answers grounded in them and returns the source video(s) — with a timestamp to jump to when the answer comes from the transcript. Conversation memory and first-turn answers are kept in Redis; if nothing relevant is found it says so instead of inventing an answer. **Study guide** — generate a structured guide (overview + themed sections) from a single video or your whole library, drawn from notes, visual explanations, research, and transcript. Cached in Redis (invalidated when notes change) with a one-click Regenerate. **Flashcards with spaced repetition** — auto-generate question/answer cards from the same material and review them with the SM-2 algorithm. Due-date scheduling is backed by a Redis sorted set, so "what's due now" is an O(log n) lookup. **DOCX export** — download all your notes (with their summaries and timestamps) or the generated study guide as Word documents. **Video library** — upload MP4 or WebM files to a MongoDB GridFS store. The dashboard shows thumbnails (captured in-browser before upload), durations, and per-video note counts. Videos can be deleted and their notes are removed from both the database and localStorage. **Demo mode** — a bundled lecture video works with no credentials at all (notes go to localStorage only). ## Tech stack | Layer | Technology | |---|---| | Frontend | React 19 + Vite + TypeScript + Tailwind CSS | | Backend | Hono on Node.js (run with tsx, no compile step) | | Video storage | MongoDB Atlas + GridFS (streaming, range-request aware) | | Notes | localStorage (live store) + MongoDB (mirrored for persistence) | | Speech-to-text | Deepgram Nova | | Visual explain / chatbot / study guide / flashcards | Anthropic Claude (`claude-sonnet-4-6`) | | Web research | Browserbase + Stagehand + Gemini | | Vector search, caching, session memory, due-queue | Redis 8 (RediSearch / sorted sets) | | Embeddings | OpenAI `text-embedding-3-small` (1536-dim) | | Document export | `docx` | ## Requirements - Node.js 20+ (22 recommended) - MongoDB Atlas cluster (or any MongoDB 6+ instance) - Redis 8 with the search module — local `redis-server` 8.x or Redis Stack (powers the chatbot, caching, and flashcard due-queue) - API keys: Anthropic, OpenAI, Deepgram, Browserbase, Gemini (see below) ## Setup ```sh npm install cp packages/server/.env.sample packages/server/.env ``` Fill in `packages/server/.env`: ``` # MongoDB MONGODB_URI="mongodb+srv://..." MONGODB_USERNAME="..." MONGODB_PASSWORD="..." MONGODB_DB="vidmark" # AI services ANTHROPIC_API_KEY="sk-ant-..." OPENAI_API_KEY="sk-proj-..." # embeddings for the chatbot/study tools DEEPGRAM_API_KEY="..." GEMINI_API_KEY="..." BROWSERBASE_API_KEY="..." # Redis (vector index, caches, sessions, due-queue) REDIS_URL="redis://localhost:6379" ``` If you want to run without external API keys during a demo, set the mock flags: ``` MOCK_DEEPGRAM="true" # voice notes return a canned transcript MOCK_BROWSERBASE="true" # research returns a placeholder summary ``` ## Running ```sh # Development (hot reload on both frontend :5173 and server :3000) npm run dev # Production (build then serve everything from :3000) npm run build npm run start ``` In development the Vite dev server proxies `/api` to the Hono server, so there is no CORS to configure. In production the Hono server serves the built frontend and the API from the same origin. ## Redis & the search index The study chatbot searches a Redis vector index (`idx:study`) of your notes, research, visual explanations, and transcript chunks. Caches, chat session memory, and the flashcard due-queue also live in Redis. Run Redis 8 locally (Ubuntu example — needs the bundled `search` module): ```sh redis-cli ping # PONG redis-cli MODULE LIST # should include "search" ``` The chatbot reads from the index, which is built by a batch script. After adding notes / generating transcripts, (re)build it: ```sh cd packages/server npx tsx scripts/ingest.ts # embeds notes + transcripts into idx:study npx tsx scripts/healthcheck.ts # checks Redis + OpenAI + Claude connectivity ``` Re-running `ingest.ts` is safe — it drops and rebuilds the index from MongoDB and clears the dependent caches. The study guide, flashcards, and DOCX exports read notes from MongoDB directly, so they reflect changes immediately without a re-ingest; only the chatbot depends on the index. ## Project layout ``` packages/ web/ React frontend (src/pages, src/components, src/lib) server/ Hono API (src/routes, lib/) shared/ TypeScript types imported by both (no build step) ``` ## API overview | Method | Path | Description | |---|---|---| | GET | /api/health | Server status | | POST | /api/explain | Explain a cropped video frame (multipart: image + prompt) | | POST | /api/deepgram/voice-note | Transcribe audio (multipart: audio blob) | | POST | /api/research | Research a transcript chunk; returns keywords + summary + links | | GET | /api/videos | List all uploaded videos (metadata + thumbnails) | | GET | /api/videos/:id | Video metadata + its notes | | GET | /api/videos/:id/stream | Range-aware video stream (for `