# Project export: mentora

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: A Voice AI Tutor made simple, fun, and accessible for elementary and middle school students!
- Devpost: https://devpost.com/software/edupal-ai
- GitHub: https://github.com/subham2006/calhacks
- Video: https://www.youtube.com/embed/liTClp33-I4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 5 GitHub contributor(s) — subham2006 (34 commits), Edan Gortzak (11 commits), Andrew Falcon (7 commits), Dennis Koh (6 commits), Rohan Gopalam (1 commits)

## Devpost submission (written by the team)

### Inspiration

AI is a super powerful tool for those who know how to prompt it and utilize it for guidance and education rather than just for a final answer. As AI becomes increasingly more accessible to everyone, it is clear that teaching the younger generation to use it properly is incredibly important, so that it does not have a negative impact on their learning and development. This thought process inspired us to create an app that allows a younger child to receive AI assistance in a way that is both fun and engaging, while preventing them from skipping steps in their learning process.

### What it does

mentora is an interactive Voice AI Tutor geared towards elementary and middle school aged students which takes on the form of their favorite fictional characters from movies and TV shows. Users are provided with the ability to write their work onto a whiteboard within the web application while chatting with an emotionally intelligent AI who sounds exactly like the character of their choice. The tutor is specifically engineered to guide the user towards a solution to their problem without revealing or explaining too many steps at a time. mentora gives children a platform to learn how to use AI the right way, highlighting it as a powerful and useful tool for learning rather than a means for taking short cuts.

### How we built it

We built mentora to be a full-stack web application utilizing React for the frontend and Node.js for the backend, with the majority of our code being written in javascript and typescript. Our project required integrating several APIs into a seamless workflow to create an intuitive, voice-driven educational tool. We started by implementing Deepgram, which allowed us to capture and transcribe students' voice inputs in real time. Beyond transcription, Deepgram’s sentiment analysis feature helped detect emotions like frustration or confusion in the child’s tone, enabling our AI to adjust its responses accordingly and provide empathetic assistance. Next, we integrated Cartesia to clone character voices, making interactions more engaging by allowing children to talk to their favorite characters. This feature gave our AI a personalized feel, as it responded using the selected character’s voice, making the learning experience more enjoyable and relatable for younger users. For visual interaction, we used Tldraw to develop a dynamic whiteboard interface. This allowed children to upload images or draw directly on the screen, which the AI could interpret to provide relevant feedback. The whiteboard input was synchronized with the audio input, creating a multi-modal learning environment where both voice and visuals were processed together. Finally, we used the OpenAI API to tie everything together. The API parsed contextual information from previous conversations and the whiteboard to generate thoughtful, step-by-step guidance. This integration ensured the AI could provide appropriate hints without giving away full solutions, fostering meaningful learning while maintaining real-time responsiveness.

### Challenges we ran into

A summary of our biggest challenges: Combining data from our whiteboard feature with our microphone feature to make a single openAI API call. Learning how to use and integrate Deepgram and Cartesia APIs to emotionally analyze and describe our audio inputs, and voice clone for AI responses Finding a high quality photo of Aang from Avatar the Last Airbender

### Accomplishments we're proud of

We are really proud of the fact that we successfully brought to life the project we set out to build and brainstormed for, while expanding on our ideas in ways that we wouldn’t have even imagined before this weekend. We are also proud of the fact that we created an application that could benefit the next generation by shedding a positive light on the use of AI for students who are just becoming familiar with it.

### What we learned

Building mentora taught us how to integrate multiple APIs into a seamless workflow. We gained hands-on experience with Deepgram, using it to transcribe voice inputs and perform sentiment analysis. We also integrated Cartesia for voice cloning, allowing the AI to respond in the voice of the character selected by the user. Using Tldraw, we created a functional whiteboard interface where students could upload images or write directly, providing visual input alongside audio input for a smoother learning experience. Finally, we used an OpenAI API call to integrate the entire functionality. The most valuable part of the process was learning how to design a workflow where multiple technologies interacted harmoniously—from capturing voice input and analyzing emotions to generating thoughtful responses through avatars. We also learned how important it was to plan the integration ahead of time. We had many ideas, and we had to try out all of them to see what would work and what would not. While this was initially challenging due to all the moving pieces, creating a structure for what we wanted the final project to look like allowed us to keep the final goal in mind. On the other hand, it was important that we were willing to change focus when better ideas were created and when old ideas had flaws. Ultimately, this project gave us deeper insights into full-stack development and reinforced the balance of structure vs. adaptability when creating a new product.

### What's next

There are many next steps we could take and directions we could go with mentora. Ideas we have discussed are deploying the website, creating a custom character creation menu that allows the users to input new characters and voices, improve latency up to real-time speed for back and forth conversation, and broaden the range of subjects that the tutor is well prepared to assist with.

## README (from the GitHub repository)

# Getting Started

## Install & Start Frontend

##### `git clone git@github.com:subham2006/calhacks.git`
##### `cd calhacks`
##### `cd frontend`
##### `npm install`
##### `npm start`

## Install & Start Backend

##### `cd backend`
##### `npm install`
##### `node server.js`

## env file

Create a .env file in the frontend and backend directory and add the following:

REACT_APP_LOCAL_IP = 'YOUR_IP_ADDRESS'

You can get your IP address by running `ipconfig getifaddr en0` in the terminal.

Alteratively, when you run `npm start`, the terminal will print your IP address.

## Detected evidence (automated analysis)

Indexed codebase: 24 recognized source files, 44 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Node.js (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (32 of 32)

```
.gitignore
backend/.gitignore
backend/package.json
backend/server.js
frontend/.gitignore
frontend/package.json
frontend/postcss.config.js
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/README.md
frontend/src/App.css
frontend/src/App.js
frontend/src/App.test.js
frontend/src/components/About.js
frontend/src/components/AIContext.js
frontend/src/components/AITool.ts
frontend/src/components/deprecated-whiteboard.js
frontend/src/components/ExtractImage.tsx
frontend/src/components/Home.js
frontend/src/components/Interact.js
frontend/src/components/Navbar.js
frontend/src/components/QRCodeInteraction.js
frontend/src/components/SpeechBubble.css
frontend/src/components/Whiteboard.tsx
frontend/src/index.css
frontend/src/index.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
frontend/tailwind.config.js
package.json
README.md
```

### Dependencies

- backend/package.json: cors@^2.8.5, dotenv@^16.4.5, express@^4.21.1, openai@^4.68.1
- frontend/package.json: @cartesia/cartesia-js@^1.1.0, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @tldraw/sync@^3.3.2, @tldraw/tldraw@^3.3.2, autoprefixer@^10.4.20, axios@^1.7.7, hume@^0.9.1, katex@^0.16.11, openai@^4.68.1, postcss@^8.4.47, process@^0.11.10, qrcode.react@^4.0.1, react@^18.3.1, react-dom@^18.3.1, react-katex@^3.0.1, react-latex-next@^3.0.0, react-router-dom@^6.27.0, react-scripts@5.0.1, react-spinners@^0.14.1, tailwindcss@^3.4.14, tldraw@^3.3.2, web-vitals@^2.1.4
- package.json: ws@^8.18.0

### Recent commits (newest first)

- more ui changes
- final working sol
- changed favicon
- merged and added loader when converting tts
- final integration fixed
- final integration
- merge prep
- merged with capture image on end of voice input
- added Latex for text
- new ai work
- fixed console to text box output
- ui fixes
- ui fixed
- added speech bubble and better image quality
- UI updated to character panel
- first commit
- edited to work better
- fantastic merge
- good merge
- removed env from commit lol

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "dependencies": {
    "ws": "^8.18.0"
  }
}

```

### backend/package.json

```
{
  "name": "backend",
  "version": "1.0.0",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "cors": "^2.8.5",
    "dotenv": "^16.4.5",
    "express": "^4.21.1",
    "openai": "^4.68.1"
  }
}

```

### frontend/package.json

```
{
  "name": "frontend",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@cartesia/cartesia-js": "^1.1.0",
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "@tldraw/sync": "^3.3.2",
    "@tldraw/tldraw": "^3.3.2",
    "axios": "^1.7.7",
    "hume": "^0.9.1",
    "katex": "^0.16.11",
    "openai": "^4.68.1",
    "process": "^0.11.10",
    "qrcode.react": "^4.0.1",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "react-katex": "^3.0.1",
    "react-latex-next": "^3.0.0",
    "react-router-dom": "^6.27.0",
    "react-scripts": "5.0.1",
    "react-spinners": "^0.14.1",
    "tldraw": "^3.3.2",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "autoprefixer": "^10.4.20",
    "postcss": "^8.4.47",
    "tailwindcss": "^3.4.14"
  }
}

```

### backend/server.js

```javascript
const express = require('express');
const cors = require('cors');
const { OpenAI } = require('openai');
const dotenv = require('dotenv');
dotenv.config();

const app = express();
const port = 3001;

// Middleware
app.use(cors());
app.use(express.json());

const openai = new OpenAI();

// Routes
app.post('/analyze-whiteboard', async (req, res) => {
    try {
        const { base64Image, transcript } = req.body;

        if (!base64Image) {
            return res.status(400).json({ error: 'No image provided' });
        }

        const response = await openai.chat.completions.create({
            model: "gpt-4o-mini",
            messages: [
            {
                role: "user",
                content: [
                { type: "text", text: `Pretend that I am a young child learning math and I am having a conversation with you. Here are some of the questions that I am asking you: ${transcript}. These questions refer to the math problems on the attached image. Please teach / tutor me how to solve these problems. Try to refrain from giving me the exact answer and instead drop hints and lay out potential steps / approaches. Feel free to use analogies if you see fit. The answer to the problem should not be revealed within a single prompt. Please keep the response to less than 3 sentences.` },
                {
                    type: "image_url",
                    image_url: {
                    url: base64Image,
                    },
                },
                ],
            },
            ],
        });
          
        res.json({"chatgpt_response" : response.choices[0].message.content});

    } catch (error) {
        console.log(error);
    }
});

app.listen(port, () => {
    console.log(`Server is running on port ${port}`);
});
```

### frontend/src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
import 'katex/dist/katex.min.css';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

reportWebVitals();
```

### frontend/src/App.js

```javascript
// App.js
import React, { useState } from 'react';
import { BrowserRouter as Router, Route, Routes } from 'react-router-dom';
import { AIProvider } from './components/AIContext';
import Navbar from './components/Navbar';
import Home from './components/Home';
import About from './components/About';
import Interact from './components/Interact';
import QRCodeInteraction from './components/QRCodeInteraction';
import Whiteboard from './components/Whiteboard.tsx';
import './App.css';
import 'katex/dist/katex.min.css';

function App() {
  const [connected, setConnected] = useState(false);

  const handleConnect = () => setConnected(true);

  return (
    <AIProvider>
      <Router>
        <div className="App">
          <Navbar />
          <Routes>
            <Route path="/" element={<Home />} />
            <Route path="/about" element={<About />} />
            <Route path="/whiteboard" element={<Whiteboard />} />
            <Route
              path="/connect"
              element={
                !connected ? (
                  <QRCodeInteraction onConnect={handleConnect} />
                ) : (
                  <Interact />
                )
              }
            />
          </Routes>
        </div>
      </Router>
    </AIProvider>
  );
}

export default App;
```

### frontend/postcss.config.js

```javascript
module.exports = {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
}

```

### frontend/tailwind.config.js

```javascript
module.exports = {
  content: [
    "./src/**/*.{js,jsx,ts,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}


```

### frontend/src/App.css

```css
.App {
  text-align: center;
}

nav ul {
  list-style-type: none;
  padding: 0;
}

nav ul li {
  display: inline;
  margin-right: 10px;
}

nav ul li a {
  text-decoration: none;
  color: #333;
}
```

### frontend/src/setupTests.js

```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';

```

[15 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]