# Project export: TeachXR

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: World's First AI Study Assistant in Extended Reality
- Devpost: https://devpost.com/software/teachxr
- GitHub: https://github.com/VietRocHack/TeachXR-frontend/
- Demo: https://github.com/VietRocHack/TeachXR-backend-dataflow
- Video: https://www.youtube.com/embed/qVoCHS9BaIc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Cal Hacks: Hackers' Choice; Vapi: Show us your Voice AI)
- Team: 3 GitHub contributor(s) — Hoang Le (14 commits), VuDuc1610 (6 commits), Vuong Ho (2 commits)

## Devpost submission (written by the team)

### Inspiration

Across the globe, a critical shortage of qualified teachers poses a significant challenge to education. The average student-to-teacher ratio in primary schools worldwide stands at an alarming 23:1! In some regions of Africa, this ratio skyrockets to an astonishing 40:1. Research 1 and Research 2 As populations continue to explode, the demand for quality education has never been higher, yet the supply of capable teachers is dwindling. This results in students receiving neither the attention nor the personalized support they desperately need from their educators. Moreover, a staggering 20% of students experience social anxiety when seeking help from their teachers. This anxiety can severely hinder their educational performance and overall learning experience. Research 3 While many educational platforms leverage generative AI to offer personalized support, we envision something even more revolutionary. Introducing TeachXR—a fully voiced, interactive, and hyper-personalized AI teacher that allows students to engage just like they would with a real educator, all within the immersive realm of extended reality. Imagine a world where every student has access to a dedicated tutor who can cater to their unique learning styles and needs. With TeachXR, we can transform education, making personalized learning accessible to all. Join us on this journey to revolutionize education and bridge the gap in teacher shortages!

### What it does

Introducing TeachVR: Your Interactive XR Study Assistant TeachVR is not just a simple voice-activated Q&A AI; it’s a fully interactive extended reality study assistant designed to enhance your learning experience. Here’s what it can do: Intuitive Interaction: Use natural hand gestures to circle the part of a textbook page that confuses you. Focused Questions: Ask specific questions about the selected text for summaries, explanations, or elaborations. Human-like Engagement: Interact with TeachVR just like you would with a real person, enjoying milliseconds response times and a human voice powered by Vapi.ai. Multimodal Learning: Visualize the concepts you’re asking about, aiding in deeper understanding. Personalized and Private: All interactions are tailored to your unique learning style and remain completely confidential. How to Ask Questions: Circle the Text: Point your finger and circle the paragraph you want to inquire about. OK Gesture: Use the OK gesture to crop the image and submit your question. TeachVR's Capabilities: Summarization: Gain a clear understanding of the paragraph's meaning. TeachVR captures both book pages to provide context. Examples: Receive relevant examples related to the paragraph. Visualization: When applicable, TeachVR can present a visual representation of the concepts discussed. Unlimited Queries: Feel free to ask anything! If it’s something your teacher can answer, TeachVR can too! Interactive and Dynamic: TeachVR operates just like a human. You can even interrupt the AI if you feel it’s not addressing your needs effectively!

### How we built it

TeachXR: A Technological Innovation in Education TeachXR is the culmination of advanced technologies, built on a microservice architecture. Each component focuses on delivering essential functionalities: 1. Gesture Detection and Image Cropping We have developed and fine-tuned a hand gesture detection system that reliably identifies gestures for cropping based on MediaPipe gesture detection. Additionally, we created a custom bounding box cropping algorithm to ensure that the desired paragraphs are accurately cropped by users for further Q&A. 2. OCR (Word Detection) Utilizing Google AI OCR service, we efficiently detect words within the cropped paragraphs, ensuring speed, accuracy, and stability. Given our priority on latency—especially when simulating interactions like pointing at a book—this approach aligns perfectly with our objectives. 3. Real-time Data Orchestration Our goal is to replicate the natural interaction between a student and a teacher as closely as possible. As mentioned, latency is critical. To facilitate the transfer of image and text data, as well as real-time streaming from the OCR service to the voiced assistant, we built a robust data flow system using the SingleStore database. Its powerful real-time data processing and lightning-fast queries enable us to achieve sub-1-second cropping and assistant understanding for prompt question-and-answer interactions. 4. Voiced Assistant To ensure a natural interaction between students and TeachXR, we leverage Vapi, a natural voice interaction orchestration service that enhances our feature development. By using DeepGram for transcription, Google Gemini 1.5 flash model as the AI “brain,” and Cartesia for a natural voice, we provide a unique and interactive experience with your virtual teacher—all within TeachXR.

### Challenges we ran into

Challenges in Developing TeachXR Building the architecture to keep the user-cropped image in sync with the chat on the frontend posed a significant challenge. Due to the limitations of the Meta Quest 3, we had to run local gesture detection directly on the headset and stream the detected image to another microservice hosted in the cloud. This required us to carefully adjust the size and details of the images while deploying a hybrid model of microservices. Ultimately, we successfully navigated these challenges. Another difficulty was tuning our voiced assistant. The venue we were working in was quite loud, making background noise inevitable. We had to fine-tune several settings to ensure our assistant provided a smooth and natural interaction experience.

### Accomplishments we're proud of

Achievements We are proud to present a complete and functional MVP! The cropped image and all related processes occur in under 1 second, significantly enhancing the natural interaction between the student and TeachVR.

### What we learned

Developing a Great AI Application We successfully transformed a solid idea into reality by utilizing the right tools and technologies. There are many excellent pre-built solutions available, such as Vapi, which has been invaluable in helping us implement a voice interface. It provides a user-friendly and intuitive experience, complete with numerous settings and plug-and-play options for transcription, models, and voice solutions.

### What's next

We’re excited to think of the future of TeachXR holds even greater innovations! we’ll be considering*adaptive learning algorithms* that tailor content in real-time based on each student’s progress and engagement. Additionally, we will work on integrating multi-language support to ensure that students from diverse backgrounds can benefit from personalized education. With these enhancements, TeachXR will not only bridge the teacher shortage gap but also empower every student to thrive, no matter where they are in the world!

## README (from the GitHub repository)

# TeachXR-frontend


## Detected evidence (automated analysis)

Indexed codebase: 22 recognized source files, 25 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- MongoDB (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (24 of 24)

```
README.md
teachxr/.gitignore
teachxr/eslint.config.js
teachxr/index.html
teachxr/package.json
teachxr/postcss.config.js
teachxr/README.md
teachxr/src/App.jsx
teachxr/src/classes/ChatMessage.js
teachxr/src/components/CenterButton.jsx
teachxr/src/components/ChatBox.jsx
teachxr/src/components/ChatField.jsx
teachxr/src/components/ConfirmationBox.jsx
teachxr/src/components/Mute.jsx
teachxr/src/components/NavigationBar.jsx
teachxr/src/components/NavItem.jsx
teachxr/src/HomePage.jsx
teachxr/src/index.css
teachxr/src/Learn.jsx
teachxr/src/main.jsx
teachxr/src/SocketTestScreen.jsx
teachxr/src/utils/index.js
teachxr/tailwind.config.js
teachxr/vite.config.js
```

### Dependencies

- teachxr/package.json: @emotion/react@^11.13.3, @emotion/styled@^11.13.0, @eslint/js@^9.11.1, @mui/material@^6.1.4, @mui/styled-engine-sc@^6.1.4, @types/react@^18.3.10, @types/react-dom@^18.3.0, @vapi-ai/web@^2.1.4, @vitejs/plugin-react@^4.3.2, autoprefixer@^10.4.20, dotenv@^16.4.5, eslint@^9.11.1, eslint-plugin-react@^7.37.0, eslint-plugin-react-hooks@^5.1.0-rc.0, eslint-plugin-react-refresh@^0.4.12, express@^4.21.1, globals@^15.9.0, mongoose@^8.7.2, postcss@^8.4.47, react@^18.3.1, react-bootstrap@^2.10.5, react-dom@^18.3.1, react-error-overlay@^6.0.9, react-icons@^5.3.0, react-on-screen@^2.1.1, react-router-dom@^6.27.0, socket.io-client@^4.8.0, styled-components@^6.1.13, tailwindcss@^3.4.14, vite@^5.4.8

### Recent commits (newest first)

- Merge branch 'main' of https://github.com/VietRocHack/TeachXR-frontend
- Reverse chat
- add all avatars
- avatar
- test avaters
- Merge branch 'main' of https://github.com/VietRocHack/TeachXR-frontend
- update UI
- update socket test
- add ai avater
- socket test screen
- Connect data to here
- Mute something semoethii
- remove some things
- css changes for chatbox
- many changes
- fix some css in learn screen
- feat: update layout Learn screen
- Duc: Merge with Hoang 2
- Duc: Update chatbot in Learn.jsx
- Merge branch 'main' of https://github.com/VietRocHack/TeachXR-frontend

## Key source files (fetched from GitHub, selected and truncated for size)

### teachxr/package.json

```
{
  "name": "teachxr",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "lint": "eslint .",
    "preview": "vite preview"
  },
  "dependencies": {
    "@emotion/react": "^11.13.3",
    "@emotion/styled": "^11.13.0",
    "@mui/material": "^6.1.4",
    "@vapi-ai/web": "^2.1.4",
    "@mui/styled-engine-sc": "^6.1.4",
    "dotenv": "^16.4.5",
    "express": "^4.21.1",
    "mongoose": "^8.7.2",
    "react": "^18.3.1",
    "react-bootstrap": "^2.10.5",
    "react-dom": "^18.3.1",
    "react-icons": "^5.3.0",
    "react-on-screen": "^2.1.1",
    "react-router-dom": "^6.27.0",
    "socket.io-client": "^4.8.0",
    "styled-components": "^6.1.13"
  },
  "devDependencies": {
    "@eslint/js": "^9.11.1",
    "@types/react": "^18.3.10",
    "@types/react-dom": "^18.3.0",
    "@vitejs/plugin-react": "^4.3.2",
    "autoprefixer": "^10.4.20",
    "eslint": "^9.11.1",
    "eslint-plugin-react": "^7.37.0",
    "eslint-plugin-react-hooks": "^5.1.0-rc.0",
    "eslint-plugin-react-refresh": "^0.4.12",
    "globals": "^15.9.0",
    "postcss": "^8.4.47",
    "react-error-overlay": "^6.0.9",
    "tailwindcss": "^3.4.14",
    "vite": "^5.4.8"
  }
}

```

### teachxr/src/main.jsx

```javascript
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import App from './App.jsx'
import './index.css'

createRoot(document.getElementById('root')).render(
  <StrictMode>
    <App />
  </StrictMode>,
)

```

### teachxr/src/App.jsx

```javascript
import HomePage from './HomePage'
import Learn from './Learn'
import SocketTestScreen from './SocketTestScreen'
import { BrowserRouter as Router, Routes, Route } from 'react-router-dom';

function App() {
  
  return (
    <Router>
      <Routes>
        <Route path="/" element={<HomePage />} />
        <Route path="/learn" element={<Learn />} />
        <Route path="/socket" element={<SocketTestScreen />} />
      </Routes>
    </Router>
  )
}

export default App

```

### teachxr/src/utils/index.js

```javascript
export const isPublicKeyMissingError = ({ vapiError }) => {
    return !!vapiError && vapiError.error.statusCode === 403 && vapiError.error.error === "Forbidden";
  };
  

export const VAPI_KEY = "34bf4314-0f47-40d2-afa2-316ed5cc4826";

export const PRIVATE_VAPI_KEY = "3b9f9908-c81d-487f-9343-7bf868fc2e3e"
```

### teachxr/postcss.config.js

```javascript
export default {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
}

```

### teachxr/vite.config.js

```javascript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react()],
})

```

### teachxr/tailwind.config.js

```javascript
/** @type {import('tailwindcss').Config} */
export default {
  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {
      colors: {
        darkPurple: '#180D3E',
        deepIndigo: '#4023A4',
      },
    },
  },
  plugins: [],
}


```

### teachxr/index.html

```html
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite + React</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

```

### teachxr/eslint.config.js

```javascript
import js from '@eslint/js'
import globals from 'globals'
import react from 'eslint-plugin-react'
import reactHooks from 'eslint-plugin-react-hooks'
import reactRefresh from 'eslint-plugin-react-refresh'

export default [
  { ignores: ['dist'] },
  {
    files: ['**/*.{js,jsx}'],
    languageOptions: {
      ecmaVersion: 2020,
      globals: globals.browser,
      parserOptions: {
        ecmaVersion: 'latest',
        ecmaFeatures: { jsx: true },
        sourceType: 'module',
      },
    },
    settings: { react: { version: '18.3' } },
    plugins: {
      react,
      'react-hooks': reactHooks,
      'react-refresh': reactRefresh,
    },
    rules: {
      ...js.configs.recommended.rules,
      ...react.configs.recommended.rules,
      ...react.configs['jsx-runtime'].rules,
      ...reactHooks.configs.recommended.rules,
      'react/jsx-no-target-blank': 'off',
      'react-refresh/only-export-components': [
        'warn',
        { allowConstantExport: true },
      ],
    },
  },
]

```

### teachxr/src/index.css

```css
@tailwind base;
@tailwind components;
@tailwind utilities;

:root {
  font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
  line-height: 1.5;
  font-weight: 400;

  color-scheme: light dark;
  color: rgba(255, 255, 255, 0.87);
  background-color: #242424;

  font-synthesis: none;
  text-rendering: optimizeLegibility;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

a {
  font-weight: 500;
  color: #646cff;
  text-decoration: inherit;
}
a:hover {
  color: #535bf2;
}

body {
  margin: 0;
  background-color: #242424; /* For dark mode */
}

h1 {
  font-size: 3.2em;
  line-height: 1.1;
  color: rgba(255, 255, 255, 0.87); /* Use root color */
}

button {
  border-radius: 8px;
  border: 1px solid transparent;
  padding: 0.6em 1.2em;
  font-size: 1em;
  font-weight: 500;
  font-family: inherit;
  background-color: #1a1a1a;
  cursor: pointer;
  transition: border-color 0.25s;
}
button:hover {
  border-color: #646cff;
}
button:focus,
button:focus-visible {
  outline: 4px auto -webkit-focus-ring-color;
}

@media (prefers-color-scheme: light) {
  :root {
    color: #213547;
    background-color: #ffffff;
  }
  a:hover {
    color: #747bff;
  }
  button {
    background-color: #f9f9f9;
  }
}

```

[11 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]