HackStack

Project Info

Eduverse

Devpost

Inspiration

Traditional learning lacks interactivity, making engagement difficult. Eduverse leverages AI to create an inclusive, accessible learning experience through speech transcription, natural language processing, and voice synthesis, pushing the boundaries of AI in education.

What it does

Eduverse enhances learning with AI-powered assistance for note-taking and interactive studying: AI Speech Transcription: Converts speech to text using Groq Whisper for seamless note capture. AI Tutor: Uses a visual LLM Agent to analyze and explain content on the screen. Speech Synthesis: Eleven Labs transforms text responses into natural speech. You can clone any voice! We cloned Grant Sanderson's voice: 3Blue1Brown Interactive Canvas: Displays notes, images, and educational materials with AI-driven explanations. Gesture-Based Controls: Scroll through content with hand gestures and clap to interact with the AI assistant.

How we built it

Developed for Meta Quest 2 integrated with ThreeJS WebVR. Groq Whisper for real-time speech-to-text conversion. OpenAI Visual LM for intelligent content interpretation. Eleven Labs for AI-generated speech responses. Interactive Canvas for real-time educational engagement.

Challenges we ran into

Our team had no prior experience developing in VR and had to start from scratch. We ended up pivoting from Unity to using ThreeJS after realizing that Unity had a lot of developmental overhead. Real-time AI responses were hard to achieve due to long inference times with LLMs. We minimized the latency by using the Groq API and by parallelizing steps in the AI agent. Hand gesture detection using the WebXR Device API, which was quite challenging due to its sparse and complex documentation. We overcame this by referencing community examples and by fine-tuning gesture recognition to ensure smooth and responsive interactions.

Accomplishments we're proud of

Successfully integrated AI for speech recognition, image comprehension, and voice synthesis. Created an interactive learning assistant that enhances engagement. Developed gesture-based controls for a more immersive learning experience. Designed an engaging VR-based educational assistant despite limited experience.

What we learned

AI voice technologies unlock new possibilities for interactive education. AI-driven note-taking and comprehension improve student engagement. Real-time speech and text processing enhance learning accessibility. Gesture-based interactions make VR learning more intuitive and effective.

What's next

Advanced AI Capabilities: Enhancing AI’s ability to handle complex educational queries. Expanded Learning Tools: Support more document formats such as spreadsheets and handwritten notes for real-time AI assistance. Personalized AI Tutors: We'd like to adapt AI model to cater towards different learning preferences. We could support different learning styles, whether visual, auditory, or kinesthetic. Study Tool Integration: Connecting with existing note-taking applications would improve the experience and allow for automatic tagging and organization of study materials for easier retrieval. Improved Gestures: Expanding the set of supported gestures, allowing users to navigate, select, and interact with AI more intuitively. Examples include having a laser pointer gesture as well as an ability to highlight notes directly within the VR world.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • C#In code
  • CSSIn code
  • FastAPIIn code
  • HTMLIn code
  • JavaScriptIn code
  • OpenAIIn code
  • PythonIn code
  • ReactIn code
  • TypeScriptIn code
  • Node.jsClaimed

9 of 10 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

65 KB

Source files

29

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars