# Project export: Noteworthy: Visualize, Capture, Remember. This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: AI-powered lecture companion: Transforms speech into real-time visualizations, incorporates notes, and enhances learning with summaries, resources, and chatbot integration—effortless and efficient. - Devpost: https://devpost.com/software/noteworthy-visualize-capture-remember - GitHub: https://github.com/arushisharmaa/stanford-tree-hacks- - Video: https://www.youtube.com/embed/m_1f7-KvGmQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Zoom: Best Use of Zoom APIs ($250 Git Card + Herschel Duffle Bags [1st] & $100 Git Card + Hoodies [2nd] & $50 Git Card + Bottles [3rd])) - Team: 3 GitHub contributor(s) — arushisharmaa (21 commits), Japleen Kaur (14 commits), mansapatel111 (11 commits) ## Devpost submission (written by the team) ### Inspiration and What it Does 🌟 We've all been in lectures where the professor quickly sketches a vague diagram, expecting instant understanding, then erases the board while moving onto a completely new topic. Most students struggle to keep up, especially in technical subjects where visualizations are crucial. While we have had many text-to-speech and text-to-image integrations in educational resources, we rarely see speech-to-image transformations. Our team wanted to redefine this learning experience by building an AI-powered lecture companion that dynamically generates real-time visuals based on lectures. This tool also provides students with a comprehensive learning platform where their notes are efficiently stored and organized, as well as allowing students to access summaries, Q&As, chatbots, and other useful resources. What We Learned 💡 Throughout the development process, we discovered several key insights about multimodal communication, efficient information transmission, and integration of AI models: Reading APIs and understanding the unique offerings of each technology to develop innovative applications. Extensive verbal information could be represented with concise illustrations. Vertical integration of AI-based technologies requires a clear formal definition of expected inputs and outputs. Visual Learning Matters as students absorb information faster when concepts are accompanied by meaningful visuals. Personalized Learning Improves Retention as students benefit from a tool adaptive to their pace and note-taking style. Collaboration Enhances Education as the ability to share AI-generated notes and visuals fosters deeper discussions and understanding. How We Built It 🛠️ Our project utilizes cutting-edge AI technologies to create an intelligent, real-time assistive educational application. The development process unfolded through these innovative stages: 🔹Speech-to-Text Processing We engineered a speech recognition system by: Leveraging Zoom's powerful servers to capture and process audio in real-time. Utilizing data generated from Zoom’s WebSocket Technology for text extraction. Crafting a custom JSON-to-text converter that dynamically updates transcripts as lectures progress. 🔹AI-Generated Visuals We created lecture visualizations by: Analyzing speech transcripts to categorize word functions and highlight key concepts. Integrating Large Language Models (LLMs) to transform extracted data into structured visual prompts. Engineering prompts to receive strictly structured outputs redirected into further scripts. Standardizing the style of visual representations of concepts. Interfacing with OpenAI's API to generate P5.js code, enabling real-time sketch creation. Utilized client-server communication to switch between JS and Python. Implementing a dynamic rendering system that evolves visuals as the lecture unfolds. 🔹Intelligent Note-Taking We implemented a versatile digital workspace by: Designing an intuitive interface for seamless image insertion and manipulation. Creating a powerful multi-source merger capable of integrating diverse media types (chatbots, PDFs, etc.). 🔹AI Summaries & Resource Suggestions We enhanced learning opportunity through: Integrating Perplexity AI API to generate comprehensive summaries and power an intelligent Q&A system. Developing a sophisticated resource recommendation engine using Perplexity AI API for curated video, book, and article suggestions. Incorporating Gemini AI API to provide instant, context-aware answers to student queries. 🔹Collaboration, Sharing, and Organized Access We created a collaborative learning environment by: Designing an intelligent storage system that categorizes notes by course, lecture, and date. Seamlessly integrating Zoom's Calendar API to auto-populate the app with students’ calendar data. Developing a versatile note management system allowing anytime access, editing, printing, and sharing. Challenges We Faced 🚧 Building an advanced AI-powered tool in a limited timeframe posed several challenges: Evaluating the capabilities and limitations of each cutting-edge AI model and leveraging sample sets to make a final decision on an image-generation tool. Ensuring Visual Accuracy – Mapping abstract concepts to the right AI-generated diagrams was complex. Real-Time Syncing – Keeping students and professors on the same page without lags. Balancing Automation & Control – Allowing AI to generate content while keeping pace with professors’ lectures. Project Impact 🌙 Education should be accessible, engaging, and tailored to every student’s needs. Our tool: Supports Different Learning Styles – Visual, textual, and interactive elements enhance comprehension. Bridges the Gap for Struggling Students – Helps those who have difficulty keeping up with fast-paced lectures. Promotes Inclusive Learning – Works across devices, even for students without premium note-taking tools. Encourages Deeper Engagement – AI-generated questions and insights stimulate critical thinking, leveraging the best technologies available for student growth. Why It Stands Out 🎉 Our AI-powered lecture companion aligns perfectly with Zoom’s Education Grand Prize criteria: Creativity – A groundbreaking way to transform passive lectures into interactive learning experiences. One of the very few real-time speech-to-image education tools on the market! Technical Complexity – Integrates real-time AI speech processing, APIs, visual generation, chatbot integrations and multi-source syncing. Social Impact – Empowers students worldwide by making education more accessible, personalized, and engaging. It helps all students learn comfortably at their own pace. Future Directions 🔮🔜 Rendering Video and Images on the Same Platform Deploy the app as an extension in platforms like Zoom Apps and Microsoft Apps to seamlessly integrate video and image rendering during lessons and presentations. Utilizing Additional Tools for Diverse Illustration Styles Leverage tools like Mermaid AI to create structured flowcharts and diagrams, while integrating advanced models to visualize more complex topics, enhancing the learning experience. Implementing the React 360 Framework Explore how this platform can fit into a VR space by implementing the React 360 framework, providing a more immersive and interactive learning environment. The Future of Learning Starts Now 🚀 This is just the beginning. Our AI-powered lecture companion redefines how we learn, connect, and grow—ensuring that every student, no matter their learning style or background, has the tools they need to succeed in the classroom and beyond. Try Us Out! 📱 📹 GitHub: https://github.com/arushisharmaa/stanford-tree-hacks- 📹 Youtube Video: https://youtu.be/m_1f7-KvGmQ ## README (from the GitHub repository) # stanford-tree-hacks-NoteWorthy # This application does real time transcription and creates illustrations/diagrams as the speaker is talking over a video. It lets the user create notes using these images and use additional tools perplexity, openAI and/or gemini apis. ## Detected evidence (automated analysis) Indexed codebase: 42 recognized source files, 6861 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - Google Gemini (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - React (technology) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (66 of 66) ``` .DS_Store .github/workflows/main.yml .gitignore backend/data/transcript.txt backend/package.json backend/server.js frontend/.gitignore frontend/package.json frontend/public/answers.txt frontend/public/index.html frontend/public/manifest.json frontend/public/questions.txt frontend/public/resources.txt frontend/public/robots.txt frontend/public/summary.txt frontend/README.md frontend/src/AnswerDisplay.jsx frontend/src/App.css frontend/src/App.js frontend/src/App.test.js frontend/src/CalendarPage.css frontend/src/CalendarPage.js frontend/src/CoursePage.css frontend/src/CoursePage.jsx frontend/src/FullQuizComponent.jsx frontend/src/index.css frontend/src/index.js frontend/src/NotesPage.css frontend/src/NotesPages.jsx frontend/src/PrintComponent.css frontend/src/PrintComponent.jsx frontend/src/QuizDisplay.css frontend/src/QuizDisplay.jsx frontend/src/reportWebVitals.js frontend/src/Resources.css frontend/src/Resources.jsx frontend/src/setupTests.js frontend/src/StudentView.css frontend/src/StudentView.jsx frontend/src/SummaryDisplay.css frontend/src/SummaryDisplay.jsx frontend/src/WelcomePage.css frontend/src/WelcomePage.jsx package.json perplexity/generate_quiz.js perplexity/package.json perplexity/resources.js perplexity/summary.js README.md text-to-img/index.html text-to-img/nb-bio.ipynb text-to-img/nb-bio.nbconvert.ipynb text-to-img/nb-script.ipynb text-to-img/nb-script.nbconvert.ipynb text-to-img/p5/addons/p5.sound.js text-to-img/p5/addons/p5.sound.min.js text-to-img/p5/empty-example/.gitignore text-to-img/p5/p5.js text-to-img/p5/p5.min.js text-to-img/p5/README.txt text-to-img/prompt-bio.txt text-to-img/prompt-ds.txt text-to-img/run-nb-script.sh text-to-img/style.css text-to-img/transcript-bio.txt text-to-img/transcript-ds.txt ``` ### Dependencies - backend/package.json: axios@^1.7.9, body-parser@^1.20.3, cors@^2.8.5, dotenv@^16.4.7, express@^4.21.2, socket.io@^4.8.1 - frontend/package.json: @google/generative-ai@^0.21.0, @testing-library/dom@^10.4.0, @testing-library/jest-dom@^6.6.3, @testing-library/react@^16.2.0, @testing-library/user-event@^13.5.0, date-fns@^4.1.0, react@^19.0.0, react-calendar@^5.1.0, react-dom@^19.0.0, react-router-dom@^7.1.5, react-scripts@5.0.1, react-slick@^0.30.3, react-to-pdf@^1.0.1, react-to-print@^3.0.5, slick-carousel@^1.8.1, socket.io-client@^4.8.1, web-vitals@^2.1.4 - package.json: dotenv@^16.4.7, node-fetch@^3.3.2, react-to-print@^3.0.5, sqlite3@^5.1.7 - perplexity/package.json: axios@^1.7.9, dotenv@^16.4.7, node-fetch@^2.7.0 ### Recent commits (newest first) - fix README.md - Update README.md - organize into ds and bio examples. - fix prompt - add bio example - Merge branch 'main' of https://github.com/arushisharmaa/stanford-tree-hacks- - Add sample transcript - fix - fix header - fix bug - weird conflits - updates to perplexity - Merge branch 'main' of https://github.com/arushisharmaa/stanford-tree-hacks- - changes - Clean extraneous and byproduct files - final commits - perplexity?? - perplexity - Merge branch 'main' of https://github.com/arushisharmaa/stanford-tree-hacks- - gemini api integrated with chatbossssss ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "dependencies": { "dotenv": "^16.4.7", "node-fetch": "^3.3.2", "react-to-print": "^3.0.5", "sqlite3": "^5.1.7" } } ``` ### perplexity/package.json ``` { "name": "perplexity-summary", "version": "1.0.0", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "description": "", "dependencies": { "axios": "^1.7.9", "dotenv": "^16.4.7", "node-fetch": "^2.7.0" } } ``` ### backend/package.json ``` { "name": "backend", "version": "1.0.0", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "description": "", "dependencies": { "axios": "^1.7.9", "body-parser": "^1.20.3", "cors": "^2.8.5", "dotenv": "^16.4.7", "express": "^4.21.2", "socket.io": "^4.8.1" } } ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "dependencies": { "@google/generative-ai": "^0.21.0", "@testing-library/dom": "^10.4.0", "@testing-library/jest-dom": "^6.6.3", "@testing-library/react": "^16.2.0", "@testing-library/user-event": "^13.5.0", "date-fns": "^4.1.0", "react": "^19.0.0", "react-calendar": "^5.1.0", "react-dom": "^19.0.0", "react-router-dom": "^7.1.5", "react-scripts": "5.0.1", "react-slick": "^0.30.3", "react-to-pdf": "^1.0.1", "react-to-print": "^3.0.5", "slick-carousel": "^1.8.1", "socket.io-client": "^4.8.1", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### backend/server.js ```javascript // express server // import required modules const express = require('express'); const bodyParser = require('body-parser'); const http = require('http'); const socketIo = require('socket.io'); const cors = require('cors'); // create express app + http server const app = express(); const server = http.createServer(app); const io = socketIo(server, { cors: { origin: "http://localhost:3000", methods: ["GET", "POST"] } }); // set up middleware to parse json bodies from incoming requests app.use(cors()); app.use(bodyParser.json()); // route to handle incoming webhook events from zoom app.post('/webhook', (req, res) => { const transcriptionData = req.body; // process transcription data console.log(transcriptionData); // send transcription data to react frontend using websocket io.emit('transcriptionData', transcriptionData); res.sendStatus(200); }); // start the server const PORT = process.env.PORT || 9002; server.listen(PORT, () => { console.log(`Server is running on port ${PORT}`); }); ``` ### frontend/src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### frontend/src/App.js ```javascript import React, { useState, useEffect } from 'react'; import { BrowserRouter as Router, Route, Routes, Link } from 'react-router-dom'; import WelcomePage from './WelcomePage'; import StudentView from './StudentView'; import io from 'socket.io-client'; import './App.css'; import logo from './assets/logo.png'; import CalendarPage from './CalendarPage'; import CoursePage from './CoursePage'; import NotesPage from './NotesPages'; import PrintComponent from './PrintComponent'; import QuizDisplay from './QuizDisplay'; const socket = io('http://localhost:9002'); function App() { const [transcription, setTranscription] = useState(''); useEffect(() => { socket.on('transcriptionData', (data) => { setTranscription(data.text); }); return () => { socket.off('transcriptionData'); }; }, []); return (
} /> } /> } /> } /> } /> } />
); } export default App; ``` ### text-to-img/style.css ```css html, body { margin: 0; padding: 0; } canvas { display: block; } ``` ### text-to-img/run-nb-script.sh ```shell #!/bin/bash echo "Starting local server..." python3 -m http.server 8000 & SERVER_PID=$! sleep 2 echo "Running Playwright script..." jupyter nbconvert --to notebook --execute nb-script.ipynb # nb-bio.ipynb echo "Stopping local server..." kill $SERVER_PID echo "Done!" ``` ### text-to-img/index.html ```html
``` [29 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]