# Project export: Engage: Real-time AI Agents for Smarter Classrooms This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Engage is a real-time multimodal agentic platform that proactively observes and interacts with teaching environments, maximizing learning outcomes with personalized data-driven per-student insights. - Devpost: https://devpost.com/software/engage-real-time-ai-agents-for-smarter-classrooms - GitHub: https://github.com/luca-borletti/engage-ai - Video: https://www.youtube.com/embed/1y4bA36OYp8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — SiddhBamb (10 commits), Luca Borletti (6 commits) ## Devpost submission (written by the team) ### Inspiration As teaching assistants for large courses, we understand the difficulty of addressing individual student needs in real time. With resource constraints and high student-to-faculty ratios, traditional classroom analytics tend to rely on surveys or assessments, failing to capture live student comprehension and causing a disconnect between teachers and their students. Engage changes that. Unlike previous solutions, Engage leverages real-world physical interactions—body language, speech, and classroom behaviors—to create a proactive AI teaching assistant that provides instant insights into student engagement and understanding. Engage also enables teachers with powerful presentation features capable of evolving to meet student needs during lectures, using agentic AI to dynamically insert information and provide additional resources. ### What it does Engage is powered by a real-time multimodal AI, analyzing video, audio, and text to assess student attentiveness, participation, and comprehension, both per-class and long term. It also offers live content generation features to assist teachers by instantly creating learning aids, allowing for evolving presentations which answer students’ questions with increased clarity. From a live classroom feed, the system: Captures events—recording structured teacher actions (ideas, feedback) and student responses (questions, distractions, confusing points), generating logs that are mapped to specific lecture sections for detailed per-topic analysis. Offers live Q&A—following along with classroom interactions and capable of performing retrieval-augmented generation (RAG) queries to retrieve course-specific content. The topic is drawn directly from uploaded course materials, ensuring consistent nomenclature/terminology – a huge improvement over internet searches. Assesses engagement—detects attentiveness levels, off-topic discussions, and areas of confusion, allowing teachers to adjust their approach mid-lesson and gain insights which can be used to guide their teaching style long-term. By offering a data-driven view of each student’s engagement with the class, Engage empowers teachers to refine lectures, assignments, and support systems to better serve students’ needs. ### How we built it Engage is built on Google Gemini 2.0 Flash Experimental, optimized for low-latency streaming inference. Engage dynamically calls custom tools in response to observed classroom events (e.g., detecting confusion, tracking student participation). Engage’s RAG-based content generation agent uses InterSystems IRIS for its backend, serving as a vector search-enabled SQL database which stores course materials for easy semantic access. SBERT is used to calculate vector embeddings for course content. The application itself is built using TypeScript, React.JS, and Node.JS, with Python used for some APIs. ### Challenges we ran into One of the biggest challenges was ensuring that each tool for the agent served a distinct yet complementary role within the system. Engagement tracking, event logging, and Q&A assistance needed to operate independently yet cohesively, with each agent serving a specific, specialized purpose, yet being able to interact with other agents effectively. We tested extensively to tune our prompts and injected heuristic logic where possible to ensure consistent output in a variety of classroom situations. We expect that as language models continue to become more accurate and cheaper, Engage will be able to derive further utility from the models it is built on. Another challenge laid in integrating the multiple technologies comprising our stack. Between multimodal models, vector databases, and agent tool calling, we found it difficult to properly link all components even with extensive planning between teammates, especially with time pressure of a 36-hour hackathon. Most of our team was also inexperienced in web development, which made for a challenging but an overall enjoyable experience as we learned new technologies and frameworks while building our project. ### Accomplishments we're proud of We’re proud of building our intuitive user experience within the time constraints. We’ve never previously worked with multimodal language models, especially in an educational context, and it was exciting for us to explore this technology and apply it to an area that we believe benefits from additional data at the physical level. ### What we learned We learned a lot about orchestrating and planning agent tools effectively, as well as how to work with multimodal data. We also picked up skills in web development as two of our team members lacked prior experience, and combining the various elements of the project involved significant debugging and integration skills which we are sure will be useful for future projects. ### What's next for Engage We feel confident that real-time AI-powered classrooms will soon become the norm. We hope to integrate Engage with existing LMS systems, seamlessly linking content that students see in the classroom with online sources. We’re also interested in incorporating wearable data into our system, adding metrics like heart rate, fatigue, and readiness to better understand student well-being in the context of education. By aligning real-time data streams, proactive assistance, and personalized analytics, we’re excited to contribute to building the classroom of the future. ## README (from the GitHub repository) # Engage – Real-time AI Agents for Smarter Classrooms Engage is a multimodal agentic platform for realtime educational experiences. It aims to maximize learning outcomes with personalized student insights and empower educators with in-class assistance. (200 chars. max) Screenshot 2025-02-16 at 8 29 14 AM Screenshot 2025-02-16 at 8 29 47 AM Screenshot 2025-02-16 at 8 23 58 AM Screenshot 2025-02-16 at 8 30 01 AM ## Inspiration As teaching assistants for large courses, we understand the difficulty of addressing individual student needs in real time. With resource constraints and high student-to-faculty ratios, traditional classroom analytics tend to rely on surveys or assessments, failing to capture live student comprehension and causing a disconnect between teachers and their students. Engage changes that. Unlike previous solutions, Engage leverages real-world physical interactions—body language, speech, and classroom behaviors—to create a proactive AI teaching assistant that provides instant insights into student engagement and understanding. Engage also enables teachers with powerful presentation features capable of evolving to meet student needs during lectures, using agentic AI to dynamically insert information and provide additional resources. ## What it does Engage is powered by a real-time multimodal AI, analyzing video, audio, and text to assess student attentiveness, participation, and comprehension, both per-class and long term. It also offers live content generation features to assist teachers by instantly creating learning aids, allowing for evolving presentations which answer students’ questions with increased clarity. From a live classroom feed, the system: - Captures events—recording structured teacher actions (ideas, feedback) and student responses (questions, distractions, confusing points), generating logs that are mapped to specific lecture sections for detailed per-topic analysis. - Offers live Q&A—following along with classroom interactions and capable of performing retrieval-augmented generation (RAG) queries to retrieve course-specific content. The topic is drawn directly from uploaded course materials, ensuring consistent nomenclature/terminology – a huge improvement over internet searches. - Assesses engagement—detects attentiveness levels, off-topic discussions, and areas of confusion, allowing teachers to adjust their approach mid-lesson and gain insights which can be used to guide their teaching style long-term. By offering a data-driven view of each student’s engagement with the class, Engage empowers teachers to refine lectures, assignments, and support systems to better serve students’ needs. ## How we built it Engage is built on Google Gemini 2.0 Flash Experimental, optimized for low-latency streaming inference. Engage dynamically calls custom tools in response to observed classroom events (e.g., detecting confusion, tracking student participation). Engage’s RAG-based content generation agent uses InterSystems IRIS for its backend, serving as a vector search-enabled SQL database which stores course materials for easy semantic access. SBERT is used to calculate vector embeddings for course content. The application itself is built using TypeScript, React.JS, and Node.JS, with Python used for some APIs. ## Challenges we ran into One of the biggest challenges was ensuring that each tool for the agent served a distinct yet complementary role within the system. Engagement tracking, event logging, and Q&A assistance needed to operate independently yet cohesively, with each agent serving a specific, specialized purpose, yet being able to interact with other agents effectively. We tested extensively to tune our prompts and injected heuristic logic where possible to ensure consistent output in a variety of classroom situations. We expect that as language models continue to become more accurate and cheaper, Engage will be able to derive further utility from the models it is built on. Another challenge laid in integrating the multiple technologies comprising our stack. Between multimodal models, vector databases, and agent tool calling, we found it difficult to properly link all components even with extensive planning between teammates, especially with time pressure of a 36-hour hackathon. Most of our team was also inexperienced in web development, which made for a challenging but an overall enjoyable experience as we learned new technologies and frameworks while building our project. ## Accomplishments that we're proud of We’re proud of building our intuitive user experience within the time constraints. We’ve never previously worked with multimodal language models, especially in an educational context, and it was exciting for us to explore this technology and apply it to an area that we believe benefits from additional data at the physical level. ## What we learned We learned a lot about orchestrating and planning agent tools effectively, as well as how to work with multimodal data. We also picked up skills in web development as two of our team members lacked prior experience, and combining the various elements of the project involved significant debugging and integration skills which we are sure will be useful for future projects. ## What's next for Engage We feel confident that real-time AI-powered classrooms will soon become the norm. We hope to integrate Engage with existing LMS systems, seamlessly linking content that students see in the classroom with online sources. We’re also interested in incorporating wearable data into our system, adding metrics like heart rate, fatigue, and readiness to better understand student well-being in the context of education. By aligning real-time data streams, proactive assistance, and personalized analytics, we’re excited to contribute to building the classroom of the future. ## Available Scripts In the project directory, you can run: ### `npm start` Runs the app in the development mode.\ Open [http://localhost:3000](http://localhost:3000) to view it in the browser. The page will reload if you make edits.\ You will also see any lint errors in the console. ### `npm run build` Builds the app for production to the `build` folder.\ It correctly bundles React in production mode and optimizes the build for the best performance. ## Detected evidence (automated analysis) Indexed codebase: 37 recognized source files, 149 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - Google Gemini (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - LangChain (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (56 of 56) ``` .env .env.template .gcloudignore .gitignore app.yaml CONTRIBUTING.md LICENSE package.json public/index.html public/robots.txt README.md src/App.css src/App.css.map src/App.scss src/App.test.tsx src/App.tsx src/components/audio-pulse/audio-pulse.css src/components/audio-pulse/audio-pulse.css.map src/components/audio-pulse/audio-pulse.scss src/components/audio-pulse/AudioPulse.tsx src/components/control-tray/control-tray.css src/components/control-tray/control-tray.css.map src/components/control-tray/control-tray.scss src/components/control-tray/ControlTray.tsx src/components/logger/logger.css src/components/logger/logger.css.map src/components/logger/logger.scss src/components/logger/Logger.tsx src/components/logger/mock-logs.ts src/components/side-panel/side-panel.css src/components/side-panel/side-panel.css.map src/components/side-panel/side-panel.scss src/components/side-panel/SidePanel.tsx src/contexts/LiveAPIContext.tsx src/hooks/use-live-api.ts src/hooks/use-media-stream-mux.ts src/hooks/use-screen-capture.ts src/hooks/use-webcam.ts src/index.css src/index.tsx src/lib/api/pdf-search-api.py src/lib/api/supabase_api.ts src/lib/audio-recorder.ts src/lib/audio-streamer.ts src/lib/audioworklet-registry.ts src/lib/multimodal-live-client.ts src/lib/store-logger.ts src/lib/utils.ts src/lib/worklets/audio-processing.ts src/lib/worklets/vol-meter.ts src/multimodal-live-types.ts src/react-app-env.d.ts src/reportWebVitals.ts src/requirements.txt src/setupTests.ts tsconfig.json ``` ### Dependencies - package.json: @emotion/react@^11.14.0, @emotion/styled@^11.14.0, @google/generative-ai@^0.21.0, @mui/icons-material@^6.4.4, @mui/material@^6.4.4, @supabase/supabase-js@^2.48.1, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @types/jest@^27.5.2, @types/lodash@^4.17.13, @types/node@^16.18.126, @types/react@^18.3.12, @types/react-dom@^18.3.1, @types/react-syntax-highlighter@^15.5.13, axios@^1.7.9, classnames@^2.5.1, dotenv-flow@^4.1.0, eventemitter3@^5.0.1, googleapis@^144.0.0, lodash@^4.17.21, node-fetch@^3.3.2, react@^18.3.1, react-dom@^18.3.1, react-draggable@^4.4.6, react-icons@^5.3.0, react-markdown@^9.0.3, react-pdf@^9.2.1, react-scripts@5.0.1, react-select@^5.8.3, react-syntax-highlighter@^15.6.1, sass@^1.80.6, supabase@^2.12.1, ts-node@^10.9.2, typescript@^5.6.3, vega@^5.30.0, vega-embed@^6.29.0, vega-lite@^5.22.0, web-vitals@^2.1.4, zustand@^5.0.1 - src/requirements.txt: acres@==0.2.0, aiofiles@==24.1.0, aiohappyeyeballs@==2.4.6, aiohttp@==3.11.12, aiosignal@==1.3.2, annotated-types@==0.7.0, anyio@==4.8.0, asttokens@==3.0.0, async-timeout@==4.0.3, attrs@==25.1.0, beautifulsoup4@==4.13.3, blinker@==1.9.0, cachetools@==5.5.1, certifi@==2025.1.31, cffi@==1.17.1, charset-normalizer@==3.4.1, ci-info@==0.3.0, click@==8.1.8, comm@==0.2.2, configobj@==5.0.9, configparser@==7.1.0, coverage@==7.6.12, cryptography@==44.0.1, dataclasses-json@==0.6.7, decorator@==5.1.1, docstring_parser@==0.16, etelemetry@==0.3.1, exceptiongroup@==1.2.2, executing@==2.2.0, fastapi@==0.115.8, filelock@==3.17.0, fitz@==0.0.1.dev2, Flask@==3.1.0, frontend@==0.0.3, frozenlist@==1.5.0, fsspec@==2025.2.0, gemini_ai@==0.0.5, google@==3.0.0, google-ai-generativelanguage@==0.6.15, google-api-core@==2.24.1, google-api-python-client@==2.161.0, google-auth@==2.38.0, google-auth-httplib2@==0.2.0, google-cloud-aiplatform@==1.80.0, google-cloud-bigquery@==3.29.0, google-cloud-core@==2.4.1, google-cloud-resource-manager@==1.14.0, google-cloud-storage@==2.19.0, google-crc32c@==1.6.0, google-genai@==1.2.0, google-generativeai@==0.8.4, google-resumable-media@==2.7.2, googleapis-common-protos@==1.67.0, grpc-google-iam-v1@==0.14.0, grpcio@==1.70.0, grpcio-status@==1.70.0, h11@==0.14.0, httpcore@==1.0.7, httplib2@==0.22.0, httpx@==0.27.2, httpx-sse@==0.4.0, huggingface-hub@==0.28.1, idna@==3.10, importlib_resources@==6.5.2, infinity_client@==0.0.75, iniconfig@==2.0.0, intersystems-iris@@ https://github.com/intersystems-community/intersystems-irispython/releases/download/3.8.0/intersystems_iris-3.8.0-py3-none-any.whl, ipython@==8.32.0, ipywidgets@==8.1.5, iris@@ https://github.com/grongierisc/iris-embedded-python-wrapper/releases/download/v0.0.5/iris-0.0.5-py3-none-any.whl, isodate@==0.6.1, itsdangerous@==2.2.0, jedi@==0.19.2, Jinja2@==3.1.5, joblib@==1.4.2, jsonpatch@==1.33, jsonpointer@==3.0.0, jupyterlab_widgets@==3.0.13, langchain@==0.3.18, langchain-community@==0.3.17, langchain-core@==0.3.35, langchain-iris@==0.2.1, langchain-text-splitters@==0.3.6, langsmith@==0.3.8, looseversion@==1.3.0, lxml@==5.3.1, MarkupSafe@==3.0.2, marshmallow@==3.26.1, matplotlib-inline@==0.1.7, mpmath@==1.3.0, multidict@==6.1.0, mypy-extensions@==1.0.0, networkx@==3.4.2, nibabel@==5.3.2, nipype@==1.9.2, numpy@==1.26.4, orjson@==3.10.15, packaging@==24.2, pandas@==2.2.3, parso@==0.8.4, pathlib@==1.0.1, pdfminer.six@==20231228, pdfplumber@==0.11.5, pexpect@==4.9.0, pillow@==11.1.0, pluggy@==1.5.0, prompt_toolkit@==3.0.50, propcache@==0.2.1, proto-plus@==1.26.0, protobuf@==5.29.3, prov@==2.0.1, ptyprocess@==0.7.0, pure_eval@==0.2.3, puremagic@==1.28, pyasn1@==0.6.1, pyasn1_modules@==0.4.1, pycparser@==2.22, pydantic@==2.10.6, pydantic_core@==2.27.2, pydantic-settings@==2.7.1, pydot@==3.0.4, Pygments@==2.19.1, PyMuPDF@==1.25.3, pyparsing@==3.2.1, pypdfium2@==4.30.1, pytest@==8.3.4, pytest-cov@==6.0.0, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, pytz@==2025.1, pyxnat@==1.6.3, PyYAML@==6.0.2, rdflib@==6.3.2, regex@==2024.11.6, requests@==2.32.3, requests-toolbelt@==1.0.0, rsa@==4.9, safetensors@==0.5.2, scikit-learn@==1.6.1, scipy@==1.15.1, sentence-transformers@==3.4.1, shapely@==2.0.7, simplejson@==3.20.1, six@==1.17.0, sniffio@==1.3.1, soupsieve@==2.6, SQLAlchemy@==2.0.38, sqlalchemy-iris@==0.15.5, stack-data@==0.6.3, starlette@==0.45.3, sympy@==1.13.1, tenacity@==9.0.0, threadpoolctl@==3.5.0, tokenizers@==0.21.0, tomli@==2.2.1, torch@==2.6.0, tqdm@==4.67.1, traitlets@==5.14.3, traits@==7.0.2, transformers@==4.48.3, typing_extensions@==4.12.2, typing-inspect@==0.9.0, tzdata@==2025.1, uritemplate@==4.1.1, urllib3@==2.3.0, uvicorn@==0.34.0, wcwidth@==0.2.13, websockets@==14.2, Werkzeug@==3.1.3, widgetsnbextension@==4.0.13, yarl@==1.18.3, zstandard@==0.23.0 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - fixed search - final edits - new updates to API - fix slide annotation, support intelsys, etc - pdf api updates - add final touches to control try - add newest version - fix - fix weird pdf viewer bug - push prelim pdf / slides / new interface - rebased on new changes - add book / slides for demo; engage component - supabase api made - Merge pull request #1 from google-gemini/dependabot/npm_and_yarn/multi-6bc014718a - Merge pull request #2 from google-gemini/dependabot/npm_and_yarn/nanoid-3.3.8 - fix typo in success: true example tool response - fixes #43 typo in isServerContentMessage ## Key source files (fetched from GitHub, selected and truncated for size) ### CONTRIBUTING.md ```markdown # How to contribute We'd love to accept your patches and contributions to this project. ## Before you begin ### Sign our Contributor License Agreement Contributions to this project must be accompanied by a [Contributor License Agreement](https://cla.developers.google.com/about) (CLA). You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project. If you or your current employer have already signed the Google CLA (even if it was for a different project), you probably don't need to do it again. Visit to see your current agreements or to sign a new one. ### Review our community guidelines This project follows [Google's Open Source Community Guidelines](https://opensource.google/conduct/). ## Contribution process ### Code reviews All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more information on using pull requests. ``` ### package.json ``` { "name": "multimodal-live-api-web-console", "version": "0.1.0", "homepage": ".", "dependencies": { "@emotion/react": "^11.14.0", "@emotion/styled": "^11.14.0", "@mui/icons-material": "^6.4.4", "@mui/material": "^6.4.4", "@supabase/supabase-js": "^2.48.1", "axios": "^1.7.9", "classnames": "^2.5.1", "dotenv-flow": "^4.1.0", "eventemitter3": "^5.0.1", "googleapis": "^144.0.0", "lodash": "^4.17.21", "node-fetch": "^3.3.2", "react": "^18.3.1", "react-dom": "^18.3.1", "react-draggable": "^4.4.6", "react-icons": "^5.3.0", "react-markdown": "^9.0.3", "react-pdf": "^9.2.1", "react-scripts": "5.0.1", "react-select": "^5.8.3", "react-syntax-highlighter": "^15.6.1", "sass": "^1.80.6", "supabase": "^2.12.1", "vega": "^5.30.0", "vega-embed": "^6.29.0", "vega-lite": "^5.22.0", "web-vitals": "^2.1.4", "zustand": "^5.0.1" }, "scripts": { "start-https": "HTTPS=true react-scripts start", "start": "react-scripts start & npm run start-pdf_api", "start-pdf_api": "uvicorn src.lib.api.pdf-search-api:app --host 0.0.0.0 --port 8080", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] }, "devDependencies": { "@google/generative-ai": "^0.21.0", "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "@types/jest": "^27.5.2", "@types/lodash": "^4.17.13", "@types/node": "^16.18.126", "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "@types/react-syntax-highlighter": "^15.5.13", "ts-node": "^10.9.2", "typescript": "^5.6.3" }, "overrides": { "typescript": "^5.6.3" } } ``` ### src/requirements.txt ``` acres==0.2.0 aiofiles==24.1.0 aiohappyeyeballs==2.4.6 aiohttp==3.11.12 aiosignal==1.3.2 annotated-types==0.7.0 anyio==4.8.0 asttokens==3.0.0 async-timeout==4.0.3 attrs==25.1.0 beautifulsoup4==4.13.3 blinker==1.9.0 cachetools==5.5.1 certifi==2025.1.31 cffi==1.17.1 charset-normalizer==3.4.1 ci-info==0.3.0 click==8.1.8 comm==0.2.2 configobj==5.0.9 configparser==7.1.0 coverage==7.6.12 cryptography==44.0.1 dataclasses-json==0.6.7 decorator==5.1.1 docstring_parser==0.16 etelemetry==0.3.1 exceptiongroup==1.2.2 executing==2.2.0 fastapi==0.115.8 filelock==3.17.0 fitz==0.0.1.dev2 Flask==3.1.0 frontend==0.0.3 frozenlist==1.5.0 fsspec==2025.2.0 gemini_ai==0.0.5 google==3.0.0 google-ai-generativelanguage==0.6.15 google-api-core==2.24.1 google-api-python-client==2.161.0 google-auth==2.38.0 google-auth-httplib2==0.2.0 google-cloud-aiplatform==1.80.0 google-cloud-bigquery==3.29.0 google-cloud-core==2.4.1 google-cloud-resource-manager==1.14.0 google-cloud-storage==2.19.0 google-crc32c==1.6.0 google-genai==1.2.0 google-generativeai==0.8.4 google-resumable-media==2.7.2 googleapis-common-protos==1.67.0 grpc-google-iam-v1==0.14.0 grpcio==1.70.0 grpcio-status==1.70.0 h11==0.14.0 httpcore==1.0.7 httplib2==0.22.0 httpx==0.27.2 httpx-sse==0.4.0 huggingface-hub==0.28.1 idna==3.10 importlib_resources==6.5.2 infinity_client==0.0.75 iniconfig==2.0.0 intersystems-iris @ https://github.com/intersystems-community/intersystems-irispython/releases/download/3.8.0/intersystems_iris-3.8.0-py3-none-any.whl ipython==8.32.0 ipywidgets==8.1.5 iris @ https://github.com/grongierisc/iris-embedded-python-wrapper/releases/download/v0.0.5/iris-0.0.5-py3-none-any.whl isodate==0.6.1 itsdangerous==2.2.0 jedi==0.19.2 Jinja2==3.1.5 joblib==1.4.2 jsonpatch==1.33 jsonpointer==3.0.0 jupyterlab_widgets==3.0.13 langchain==0.3.18 langchain-community==0.3.17 langchain-core==0.3.35 langchain-iris==0.2.1 langchain-text-splitters==0.3.6 langsmith==0.3.8 looseversion==1.3.0 lxml==5.3.1 MarkupSafe==3.0.2 marshmallow==3.26.1 matplotlib-inline==0.1.7 mpmath==1.3.0 multidict==6.1.0 mypy-extensions==1.0.0 networkx==3.4.2 nibabel==5.3.2 nipype==1.9.2 numpy==1.26.4 orjson==3.10.15 packaging==24.2 pandas==2.2.3 parso==0.8.4 pathlib==1.0.1 pdfminer.six==20231228 pdfplumber==0.11.5 pexpect==4.9.0 pillow==11.1.0 pluggy==1.5.0 prompt_toolkit==3.0.50 propcache==0.2.1 proto-plus==1.26.0 protobuf==5.29.3 prov==2.0.1 ptyprocess==0.7.0 pure_eval==0.2.3 puremagic==1.28 pyasn1==0.6.1 pyasn1_modules==0.4.1 pycparser==2.22 pydantic==2.10.6 pydantic-settings==2.7.1 pydantic_core==2.27.2 pydot==3.0.4 Pygments==2.19.1 PyMuPDF==1.25.3 pyparsing==3.2.1 pypdfium2==4.30.1 pytest==8.3.4 pytest-cov==6.0.0 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 pytz==2025.1 pyxnat==1.6.3 PyYAML==6.0.2 rdflib==6.3.2 regex==2024.11.6 requests==2.32.3 requests-toolbelt==1.0.0 rsa==4.9 safetensors==0.5.2 scikit-learn==1.6.1 scipy==1.15.1 sentence-transformers==3.4.1 shapely==2.0.7 simplejson==3.20.1 six==1.17.0 sniffio==1.3.1 soupsieve==2.6 SQLAlchemy==2.0.38 sqlalchemy-iris==0.15.5 stack-data==0.6.3 starlette==0.45.3 sympy==1.13.1 tenacity==9.0.0 threadpoolctl==3.5.0 tokenizers==0.21.0 tomli==2.2.1 torch==2.6.0 tqdm==4.67.1 traitlets==5.14.3 traits==7.0.2 transformers==4.48.3 typing-inspect==0.9.0 typing_extensions==4.12.2 tzdata==2025.1 uritemplate==4.1.1 urllib3==2.3.0 uvicorn==0.34.0 wcwidth==0.2.13 websockets==14.2 Werkzeug==3.1.3 widgetsnbextension==4.0.13 yarl==1.18.3 zstandard==0.23.0 ``` ### src/index.tsx ```typescript /** * Copyright 2024 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot( document.getElementById('root') as HTMLElement ); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### src/App.tsx ```typescript /** * Copyright 2024 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ import "react-pdf/dist/esm/Page/TextLayer.css"; import 'react-pdf/dist/Page/AnnotationLayer.css'; import React, { useEffect, useRef, useState } from "react"; import { LiveAPIProvider, useLiveAPIContext } from "./contexts/LiveAPIContext"; import cn from "classnames"; import Draggable from "react-draggable"; import { ExpandMore, ExpandLess } from "@mui/icons-material"; import Markdown from 'react-markdown' import axios from "axios" import { RequestInit } from 'node-fetch'; // Material UI components import Container from "@mui/material/Container"; import AppBar from "@mui/material/AppBar"; import Toolbar from "@mui/material/Toolbar"; import Typography from "@mui/material/Typography"; import TextField from "@mui/material/TextField"; import Button from "@mui/material/Button"; import Box from "@mui/material/Box"; import { List, ListItem, ListItemText } from "@mui/material"; import ToggleButton from "@mui/material/ToggleButton"; import ToggleButtonGroup from "@mui/material/ToggleButtonGroup"; import Paper from "@mui/material/Paper"; import Grid from "@mui/material/Grid"; import Table from "@mui/material/Table"; import TableContainer from "@mui/material/TableContainer"; import TableHead from "@mui/material/TableHead"; import TableRow from "@mui/material/TableRow"; import TableCell from "@mui/material/TableCell"; import TableBody from "@mui/material/TableBody"; import Slider from "@mui/material/Slider"; import { VideocamOff } from "@mui/icons-material"; import { createTheme, ThemeProvider } from "@mui/material/styles"; // For Gemini API function declarations import { type FunctionDeclaration, SchemaType } from "@google/generative-ai"; import { ToolCall } from "./multimodal-live-types"; import ControlTray from "./components/control-tray/ControlTray"; import SidePanel from "./components/side-panel/SidePanel"; import { IconButton, Modal } from "@mui/material"; // If you use react-pdf, ensure that you have installed it and its peer dependencies. // For simplicity, we assume here that the Document and Page components work as expected. import { Document, Page } from "react-pdf"; import { pdfjs } from 'react-pdf'; pdfjs.GlobalWorkerOptions.workerSrc = new URL( 'pdfjs-dist/build/pdf.worker.min.mjs', import.meta.url, ).toString(); // ----------------------------------------------------------------------------- // Color Scheme / MUI Theme Setup using your colors: // "231942", "5e548e", "9f86c0", "be95c4", "e0b1cb" const theme = createTheme({ palette: { primary: { main: "#231942", }, secondary: { main: "#5e548e", }, // Using one of the lighter colors for the background background: { default: "#e0b1cb", }, info: { main: "#9f86c0", }, warning: { main: "#be95c4", }, }, }); function SearchContentForIdeaModal({ open, content, onClose }: { open: boolean, content: string, onClose: () => void }) { return ( Search Content for Idea {content} ); } // ----------------------------------------------------------------------------- // Data and Function Declarations (unchanged) // ----------------------------------------------------------------------------- const API_KEY = process.env.REACT_APP_GEMINI_API_KEY as string; if (typeof API_KEY !== "string") { throw new Error("set REACT_APP_GEMINI_API_KEY in .env"); } const host = "generativelanguage.googleapis.com"; const uri = `wss://${host}/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent`; const INTERSYST_URL = process.env.INTERSYST_URL as string; const lecture_sections = [ { title: "Intro to Calculus", description: "This section covers the basics of calculus, including limits, derivatives, and integrals.", }, { title: "Derivatives", description: "This section covers the basics of derivatives, including how to find the derivative of a function at a point.", }, { title: "Applications of Derivatives", description: "This section covers the applications of derivatives, including optimization and related rates.", }, { title: "Integration and the Fundamental Theorem of Calculus", description: "This section covers the basics of integrals, including how to find the integral of a function at a point.", }, { title: "Applications of Integration", description: "This section covers the applications of integration, including area under a curve and volume of a solid.", }, ]; // Represents a person tracked by the system. interface Person { name: string; physical_description: string; role: "student" | "teacher"; } // Represents a teaching event. interface StudentEvent { name: string; description: string; type: "correction" | "misunderstanding" | "clarification" | "comment" | "distraction"; section: string; } interface TeacherEvent { description: string; type: "idea" | "feedback"; section: string; } interface Timestamped { timestamp: number; } interface TPerson extends Person, Timestamped {} interface TStudentEvent extends StudentEve [truncated — 32464 more characters] ``` ### app.yaml ```yaml # Copyright 2024 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. runtime: nodejs20 env: standard handlers: # serve static files - url: /(.*\..+)$ static_files: build/\1 upload: build/(.*\..+)$ # Catch all handler to index.html - url: /.* static_files: build/index.html secure: always redirect_http_response_code: 301 upload: buid/index.html ``` ### src/index.css ```css body { margin: 0; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } code { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; } .App { width: 100%; height: 100%; } ``` ### src/react-app-env.d.ts ```typescript /** * Copyright 2024 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ /// ``` ### src/setupTests.ts ```typescript /** * Copyright 2024 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ // jest-dom adds custom jest matchers for asserting on DOM nodes. // allows you to do things like: // expect(element).toHaveTextContent(/react/i) // learn more: https://github.com/testing-library/jest-dom import '@testing-library/jest-dom'; ``` ### src/App.test.tsx ```typescript /** * Copyright 2024 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ import React from 'react'; import { render, screen } from '@testing-library/react'; import App from './App'; test('renders learn react link', () => { render(); const linkElement = screen.getByText(/learn react/i); expect(linkElement).toBeInTheDocument(); }); ``` [28 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]