# Project export: Eduverse This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: AI-Powered Education Assistant - Devpost: https://devpost.com/software/eduverse-nzuaqj - GitHub: https://github.com/pollyliu1/EDUVerse - Video: https://www.youtube.com/embed/iKDCFYKw5uk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Victor Wei (33 commits), hargunmujral (20 commits), Polly Liu (19 commits), Leo Huang (15 commits) ## Devpost submission (written by the team) ### Inspiration Traditional learning lacks interactivity, making engagement difficult. Eduverse leverages AI to create an inclusive, accessible learning experience through speech transcription, natural language processing, and voice synthesis, pushing the boundaries of AI in education. ### What it does Eduverse enhances learning with AI-powered assistance for note-taking and interactive studying: AI Speech Transcription: Converts speech to text using Groq Whisper for seamless note capture. AI Tutor: Uses a visual LLM Agent to analyze and explain content on the screen. Speech Synthesis: Eleven Labs transforms text responses into natural speech. You can clone any voice! We cloned Grant Sanderson's voice: 3Blue1Brown Interactive Canvas: Displays notes, images, and educational materials with AI-driven explanations. Gesture-Based Controls: Scroll through content with hand gestures and clap to interact with the AI assistant. ### How we built it Developed for Meta Quest 2 integrated with ThreeJS WebVR. Groq Whisper for real-time speech-to-text conversion. OpenAI Visual LM for intelligent content interpretation. Eleven Labs for AI-generated speech responses. Interactive Canvas for real-time educational engagement. ### Challenges we ran into Our team had no prior experience developing in VR and had to start from scratch. We ended up pivoting from Unity to using ThreeJS after realizing that Unity had a lot of developmental overhead. Real-time AI responses were hard to achieve due to long inference times with LLMs. We minimized the latency by using the Groq API and by parallelizing steps in the AI agent. Hand gesture detection using the WebXR Device API, which was quite challenging due to its sparse and complex documentation. We overcame this by referencing community examples and by fine-tuning gesture recognition to ensure smooth and responsive interactions. ### Accomplishments we're proud of Successfully integrated AI for speech recognition, image comprehension, and voice synthesis. Created an interactive learning assistant that enhances engagement. Developed gesture-based controls for a more immersive learning experience. Designed an engaging VR-based educational assistant despite limited experience. ### What we learned AI voice technologies unlock new possibilities for interactive education. AI-driven note-taking and comprehension improve student engagement. Real-time speech and text processing enhance learning accessibility. Gesture-based interactions make VR learning more intuitive and effective. ### What's next Advanced AI Capabilities: Enhancing AI’s ability to handle complex educational queries. Expanded Learning Tools: Support more document formats such as spreadsheets and handwritten notes for real-time AI assistance. Personalized AI Tutors: We'd like to adapt AI model to cater towards different learning preferences. We could support different learning styles, whether visual, auditory, or kinesthetic. Study Tool Integration: Connecting with existing note-taking applications would improve the experience and allow for automatic tagging and organization of study materials for easier retrieval. Improved Gestures: Expanding the set of supported gestures, allowing users to navigate, select, and interact with AI more intuitively. Examples include having a laser pointer gesture as well as an ability to highlight notes directly within the VR world. ## README (from the GitHub repository) # EDUVerse Submission for TreeHacks 2025 Devpost: https://devpost.com/software/eduverse-nzuaqj [![Youtube Video](https://img.youtube.com/vi/iKDCFYKw5uk/0.jpg)](https://www.youtube.com/watch?v=iKDCFYKw5uk) ## Installation #### Frontend ```bash yarn ``` #### Backend ```bash pip install -r pip-freeze.txt ``` ## Run #### Frontend ``` yarn run dev ``` #### Backend: ``` ./run_server.sh ``` ## Detected evidence (automated analysis) Indexed codebase: 29 recognized source files, 65 KB. - C# (language) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (115 of 115) ``` .env.example .gitignore eduverse/.gitignore eduverse/eslint.config.js eduverse/index.html eduverse/package.json eduverse/public/gltf/background.hdr eduverse/public/gltf/cafe-background.exr eduverse/public/gltf/microphone.glb eduverse/public/gltf/microphone.gltf eduverse/public/gltf/rubber_duck.glb eduverse/README.md eduverse/src/AILoader.ts eduverse/src/App.css eduverse/src/App.tsx eduverse/src/AudioManager.ts eduverse/src/constants.ts eduverse/src/dragger.ts eduverse/src/Game.ts eduverse/src/index.css eduverse/src/main.tsx eduverse/src/ModelLoader.ts eduverse/src/pdf.ts eduverse/src/Services.ts eduverse/src/SlideLoader.ts eduverse/src/ThreeScene.ts eduverse/src/vite-env.d.ts eduverse/tsconfig.app.json eduverse/tsconfig.json eduverse/tsconfig.node.json eduverse/vite.config.ts python/.gitignore python/clear_outputs.py python/convert_to_images.py python/examples/agent_example.sh python/outputs/README.md python/pip-freeze.txt python/README.md python/requirements.txt python/run_server.sh python/server.py README.md unity/.gitignore unity/.vscode/extensions.json unity/.vscode/launch.json unity/.vscode/settings.json unity/Assets/InputSystem_Actions.inputactions unity/Assets/InputSystem_Actions.inputactions.meta unity/Assets/Oculus.meta unity/Assets/Oculus/OculusProjectConfig.asset unity/Assets/Oculus/OculusProjectConfig.asset.meta unity/Assets/Readme.asset unity/Assets/Readme.asset.meta unity/Assets/Resources.meta unity/Assets/Resources/ImmersiveDebuggerSettings.asset unity/Assets/Resources/ImmersiveDebuggerSettings.asset.meta unity/Assets/Resources/OculusRuntimeSettings.asset unity/Assets/Resources/OculusRuntimeSettings.asset.meta unity/Assets/Resources/OVRPlatformToolSettings.asset unity/Assets/Resources/OVRPlatformToolSettings.asset.meta unity/Assets/Scenes.meta unity/Assets/Scenes/SampleScene.unity unity/Assets/Scenes/SampleScene.unity.meta unity/Assets/Settings.meta unity/Assets/Settings/DefaultVolumeProfile.asset unity/Assets/Settings/DefaultVolumeProfile.asset.meta unity/Assets/Settings/Mobile_Renderer.asset unity/Assets/Settings/Mobile_Renderer.asset.meta unity/Assets/Settings/Mobile_RPAsset.asset unity/Assets/Settings/Mobile_RPAsset.asset.meta unity/Assets/Settings/PC_Renderer.asset unity/Assets/Settings/PC_Renderer.asset.meta unity/Assets/Settings/PC_RPAsset.asset unity/Assets/Settings/PC_RPAsset.asset.meta unity/Assets/Settings/SampleSceneProfile.asset unity/Assets/Settings/SampleSceneProfile.asset.meta unity/Assets/Settings/UniversalRenderPipelineGlobalSettings.asset unity/Assets/Settings/UniversalRenderPipelineGlobalSettings.asset.meta unity/Assets/TutorialInfo.meta unity/Assets/TutorialInfo/Icons.meta unity/Assets/TutorialInfo/Icons/URP.png.meta unity/Assets/TutorialInfo/Layout.wlt unity/Assets/TutorialInfo/Layout.wlt.meta unity/Assets/TutorialInfo/Scripts.meta unity/Assets/TutorialInfo/Scripts/Editor.meta unity/Assets/TutorialInfo/Scripts/Editor/ReadmeEditor.cs unity/Assets/TutorialInfo/Scripts/Editor/ReadmeEditor.cs.meta unity/Assets/TutorialInfo/Scripts/Readme.cs unity/Assets/TutorialInfo/Scripts/Readme.cs.meta unity/Packages/manifest.json unity/Packages/packages-lock.json unity/ProjectSettings/AudioManager.asset unity/ProjectSettings/ClusterInputManager.asset unity/ProjectSettings/DynamicsManager.asset unity/ProjectSettings/EditorBuildSettings.asset unity/ProjectSettings/EditorSettings.asset unity/ProjectSettings/GraphicsSettings.asset unity/ProjectSettings/InputManager.asset unity/ProjectSettings/MemorySettings.asset unity/ProjectSettings/MultiplayerManager.asset unity/ProjectSettings/NavMeshAreas.asset unity/ProjectSettings/PackageManagerSettings.asset unity/ProjectSettings/Physics2DSettings.asset unity/ProjectSettings/PresetManager.asset unity/ProjectSettings/ProjectSettings.asset unity/ProjectSettings/ProjectVersion.txt unity/ProjectSettings/QualitySettings.asset unity/ProjectSettings/ShaderGraphSettings.asset unity/ProjectSettings/TagManager.asset unity/ProjectSettings/TimeManager.asset unity/ProjectSettings/UnityConnectSettings.asset unity/ProjectSettings/URPProjectSettings.asset unity/ProjectSettings/VersionControlSettings.asset unity/ProjectSettings/VFXManager.asset unity/ProjectSettings/XRSettings.asset ``` ### Dependencies - eduverse/package.json: @dimforge/rapier3d@^0.14.0, @eslint/js@^9.19.0, @types/react@^19.0.8, @types/react-dom@^19.0.3, @types/three@^0.173.0, @vitejs/plugin-react@^4.3.4, axios@^1.7.9, eslint@^9.19.0, eslint-plugin-react-hooks@^5.0.0, eslint-plugin-react-refresh@^0.4.18, globals@^15.14.0, howler@^2.2.4, pdfjs-dist@^4.10.38, react@^19.0.0, react-dom@^19.0.0, three@^0.173.0, three-stdlib@^2.35.13, typescript@~5.7.2, typescript-eslint@^8.22.0, vite@^6.1.0 - python/requirements.txt: annotated-types@==0.7.0, anyio@==4.8.0, Aspose.Slides@==25.2.0, cachetools@==5.5.1, certifi@==2025.1.31, charset-normalizer@==3.4.1, click@==8.1.8, distro@==1.9.0, elevenlabs@==1.51.0, fastapi@==0.115.8, google-api-core@==2.24.1, google-api-python-client@==2.161.0, google-auth@==2.38.0, google-auth-httplib2@==0.2.0, googleapis-common-protos@==1.67.0, groq@==0.18.0, h11@==0.14.0, httpcore@==1.0.7, httplib2@==0.22.0, httpx@==0.28.1, idna@==3.10, jiter@==0.8.2, lxml@==5.3.1, openai@==1.63.0, pdf2image@==1.17.0, pillow@==11.1.0, proto-plus@==1.26.0, protobuf@==5.29.3, pyasn1@==0.6.1, pyasn1_modules@==0.4.1, pydantic@==2.10.6, pydantic_core@==2.27.2, pyparsing@==3.2.1, python-dotenv@==1.0.1, python-multipart@==0.0.20, python-pptx@==1.0.2, requests@==2.32.3, rsa@==4.9, sniffio@==1.3.1, starlette@==0.45.3, tqdm@==4.67.1, typing_extensions@==4.12.2, uritemplate@==4.1.1, urllib3@==2.3.0, uvicorn@==0.34.0, websockets@==14.2, XlsxWriter@==3.2.2 ### Recent commits (newest first) - Update README.md - add clipping plane - fix: rebase changes - moved gltf imports into rightful folder - gave duck rotations - improved python readme - made duck slightly smaller - shifted reset and exit buttons to the right - add to vite - Merge branch 'main' of https://github.com/pollyliu1/EDUVerse - replaced blue blob with rubber duck - add notes - add notes - cs notes - cancel howl if new - support dynamic pages - add constants url - fuck this shit - added room and cafe background - agent working on local ## Key source files (fetched from GitHub, selected and truncated for size) ### python/requirements.txt ``` annotated-types==0.7.0 anyio==4.8.0 Aspose.Slides==25.2.0 cachetools==5.5.1 certifi==2025.1.31 charset-normalizer==3.4.1 click==8.1.8 distro==1.9.0 elevenlabs==1.51.0 fastapi==0.115.8 google-api-core==2.24.1 google-api-python-client==2.161.0 google-auth==2.38.0 google-auth-httplib2==0.2.0 googleapis-common-protos==1.67.0 groq==0.18.0 h11==0.14.0 httpcore==1.0.7 httplib2==0.22.0 httpx==0.28.1 idna==3.10 jiter==0.8.2 lxml==5.3.1 openai==1.63.0 pdf2image==1.17.0 pillow==11.1.0 proto-plus==1.26.0 protobuf==5.29.3 pyasn1==0.6.1 pyasn1_modules==0.4.1 pydantic==2.10.6 pydantic_core==2.27.2 pyparsing==3.2.1 python-dotenv==1.0.1 python-multipart==0.0.20 python-pptx==1.0.2 requests==2.32.3 rsa==4.9 sniffio==1.3.1 starlette==0.45.3 tqdm==4.67.1 typing_extensions==4.12.2 uritemplate==4.1.1 urllib3==2.3.0 uvicorn==0.34.0 websockets==14.2 XlsxWriter==3.2.2 ``` ### eduverse/package.json ``` { "name": "eduverse", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "@dimforge/rapier3d": "^0.14.0", "axios": "^1.7.9", "howler": "^2.2.4", "pdfjs-dist": "^4.10.38", "react": "^19.0.0", "react-dom": "^19.0.0", "three": "^0.173.0", "three-stdlib": "^2.35.13" }, "devDependencies": { "@eslint/js": "^9.19.0", "@types/react": "^19.0.8", "@types/react-dom": "^19.0.3", "@types/three": "^0.173.0", "@vitejs/plugin-react": "^4.3.4", "eslint": "^9.19.0", "eslint-plugin-react-hooks": "^5.0.0", "eslint-plugin-react-refresh": "^0.4.18", "globals": "^15.14.0", "typescript": "~5.7.2", "typescript-eslint": "^8.22.0", "vite": "^6.1.0" } } ``` ### python/server.py ```python from fastapi import FastAPI, HTTPException, UploadFile, File, Form from fastapi.responses import StreamingResponse from pydantic import BaseModel import openai import os from dotenv import load_dotenv from groq import Groq import base64 import io from elevenlabs.client import ElevenLabs from elevenlabs import play, stream from fastapi.middleware.cors import CORSMiddleware import sys load_dotenv('../.env') app = FastAPI() origins = [ "http://localhost:5173", "http://localhost:8080", "https://grey-need-permission-disable.trycloudflare.com", "https://asked-literally-lt-mattress.trycloudflare.com", ] app.add_middleware( CORSMiddleware, allow_origins=origins, allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) openai_client = openai.OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), ) groq_client = openai.OpenAI( api_key=os.environ.get("GROQ_API_KEY"), base_url="https://api.groq.com/openai/v1", ) eleven_labs_client = ElevenLabs( api_key=os.environ.get("ELEVENLABS_API_KEY"), ) class ChatRequest(BaseModel): llm: str = "openai" prompt: str max_tokens: int = None temperature: float = None top_p: float = None class GenerateSpeechRequest(BaseModel): input: str stream: bool = True voice_id: str = "qDazFCguyJ6M5CH0mFuN" # 3blue1brown voice id class ImageRequest(BaseModel): prompt: str = "Describe the content of this image." max_tokens: int | None = None temperature: float | None = None top_p: float | None = None # -------------------------------------- Chat -------------------------------------- def send_message(prompt: str, llm: str = "openai", max_tokens: int = None, temperature: float = None, top_p: float = None): try: request_payload = { "model": "gpt-4o", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt}, ] } if max_tokens is not None: request_payload["max_tokens"] = max_tokens if temperature is not None: request_payload["temperature"] = temperature if top_p is not None: request_payload["top_p"] = top_p if llm == "openai": response= openai_client.chat.completions.create(**request_payload) return response.choices[0].message.content elif llm == "groq": request_payload["model"] = "llama-3.3-70b-versatile" response = groq_client.chat.completions.create(**request_payload) return response.choices[0].message.content else: raise HTTPException(status_code=400, detail="Invalid LLM") except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.post("/chat") async def chat(request: ChatRequest): response = send_message(request.prompt, request.llm, request.max_tokens, request.temperature, request.top_p) return {"response": response} # -------------------------------------- Speech -------------------------------------- def transcribe_audio(file: UploadFile, provider: str = "groq"): try: audio_bytes = file.file.read() # await? audio_file = io.BytesIO(audio_bytes) audio_file.name = file.filename print(audio_file) if provider == "groq": transcript = groq_client.audio.transcriptions.create( model="whisper-large-v3-turbo", file=(audio_file.name, audio_file), response_format="json", language="en" ) return {"transcript": transcript.text} else: transcript = openai_client.audio.transcriptions.create( model="whisper-1", file=audio_file, ) return {"transcript": transcript["text"].strip()} except Exception as e: # print stack trace sys.exc_info() print(sys.exc_info()) raise HTTPException(status_code=500, detail=str(e)) @app.post("/transcribe") async def transcribe(file: UploadFile = File(...), provider: str = "groq"): return transcribe_audio(file, provider) def generate_speech_audio(input: str, voice_id: str = "qDazFCguyJ6M5CH0mFuN", stream: bool = True, do_play: bool = False): try: if not stream: # currently only supports streaming response = eleven_labs_client.text_to_speech.convert( text=input, voice_id=voice_id, model_id="eleven_turbo_v2_5", output_format="mp3_44100_128", ) audio_data = io.BytesIO() for chunk in response: audio_data.write(chunk) audio_data.seek(0) audio_data.name = "output.mp3" return StreamingResponse(audio_data, media_type="audio/mpeg", headers={"Content-Disposition": "attachment; filename=output.mp3"}) else: audio_stream = eleven_labs_client.text_to_speech.convert_as_stream( text=input, voice_id=voice_id, model_id="eleven_turbo_v2_5", output_format="mp3_44100_128", ) if do_play: play(audio_stream) return StreamingResponse(audio_stream, media_type="audio/mpeg") except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.post("/generate_speech") async def generate_speech(request: GenerateSpeechRequest): audio = generate_speech_audio(request.input, request.voice_id, request.stream) return audio # -------------------------------------- Image -------------------------------------- def send_image_message(image_data: bytes, prompt: str = "Describe the content of this image.", max_tokens: int = None, temperature: float = None, top_p: float = None): image_base64 = base64.b64encode(image_data).decode("utf-8") request_payload = [truncated — 3296 more characters] ``` ### eduverse/src/App.tsx ```typescript import React from "react"; import "./App.css"; const App: React.FC = () => { return
; }; export default App; ``` ### eduverse/src/main.tsx ```typescript import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import App from "./App"; import Game from "./Game"; createRoot(document.getElementById("root")!).render( ); export const game = new Game(); game.init(); // eslint-disable-next-line @typescript-eslint/no-explicit-any (globalThis as any).game = game; ``` ### eduverse/index.html ```html Eduverse
``` ### python/clear_outputs.py ```python import os import glob def clear_png_files(output_folder): png_files = glob.glob(os.path.join(output_folder, '*.png')) for file in png_files: os.remove(file) print(f"Removed {file}") if __name__ == "__main__": output_folder = 'outputs' # Change this to your specific output folder if needed clear_png_files(output_folder) print("All .png files have been removed from the outputs folder.") ``` ### eduverse/vite.config.ts ```typescript import react from "@vitejs/plugin-react"; import { defineConfig } from "vite"; // https://vite.dev/config/ export default defineConfig({ plugins: [react()], server: { allowedHosts: [ "localhost", "127.0.0.1", "grey-need-permission-disable.trycloudflare.com", "dx-invention-curves-sam.trycloudflare.com", "asked-literally-lt-mattress.trycloudflare.com", "el-producing-songs-aaron.trycloudflare.com", "virginia-aviation-slovakia-tv.trycloudflare.com", "believed-vb-hand-challenged.trycloudflare.com", ], }, }); ``` ### eduverse/eslint.config.js ```javascript import js from '@eslint/js' import globals from 'globals' import reactHooks from 'eslint-plugin-react-hooks' import reactRefresh from 'eslint-plugin-react-refresh' import tseslint from 'typescript-eslint' export default tseslint.config( { ignores: ['dist'] }, { extends: [js.configs.recommended, ...tseslint.configs.recommended], files: ['**/*.{ts,tsx}'], languageOptions: { ecmaVersion: 2020, globals: globals.browser, }, plugins: { 'react-hooks': reactHooks, 'react-refresh': reactRefresh, }, rules: { ...reactHooks.configs.recommended.rules, 'react-refresh/only-export-components': [ 'warn', { allowConstantExport: true }, ], }, }, ) ``` ### python/run_server.sh ```shell #!/bin/bash # Check if the current directory is /python if [ "$(basename "$PWD")" != "python" ]; then echo "Error: Please navigate to the /python directory and try again." exit 1 fi # Check if the virtual environment folder exists if [ ! -d "venv" ]; then echo "Virtual environment not found." echo "Please create one by running:" echo "python3 -m venv venv" echo "Then activate it with:" echo "source venv/bin/activate" exit 1 fi # check if they've run source venv/bin/activate if [ -z "$VIRTUAL_ENV" ]; then echo "Error: Please activate the virtual environment by running:" echo "source venv/bin/activate" exit 1 fi # Install dependencies pip install -r pip-freeze.txt # Run the FastAPI server uvicorn server:app --reload ``` [16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]