Project Info

WhiteboardAI

Inspiration

We all know the power of standing next to a good teacher at a white board as they smoother all your confusions through step by step guided walk through

What it does

We do anything the same magic that teacher + white board can do it, using AI!

How we built it

We reversed engineering current white board software's, creating our own unique coordinate system and ability for multi-modal ability + dual system updates. From here we used Google Cloud Vision API to analyze the current state of the white board (OCR) and used an efficient coordinate algorithm overlay to convert the photo not only to real world user data but also with accurate spatial relations of all the strokes on the white board. Doing this we were able to have real time data of users strokes and inputs, feeding all of the individual data into Gemini flash and Vapi. By doing so we were able to give Gemini's as well as Vapi context to provide feed back for the user in Json / Speech format. We used in depth prompt engineering to turn Gemini's and Vapi's outputs and walk through the users problem like an exceptional teacher providing direct hints and guidance directly through the whiteboard and speech. All of this happened in a matter of a few seconds, the low latency thanks to the OCR Coordinate System + Flash set up we choose.

Challenges we ran into

We took a major risk with this project due to its many technical layers. The first was accurately converting the our whiteboard into a machine readable data with proper spatial relations, so that we can provide feedback, comments, and highlights in the correct locations. We decided to do OCR due to its speed over an LLM in order to quickly covert to text and then be able to use our algorithm to find coordinates (something an LLM is incredibly unreliable for). This coordinate algorithm itself took many hours to streamline but was the back bone between proper communication through the white board between teacher ai and user. Once we had the correct coordinates we had to work to perfectly prompt our models to provide useful guidance as well as having it printing on the white board in the correct locations so the user could seamlessly understand how our AI was aiding them, just like a teacher! Finally we added the ability for Vapi to also help you through understanding the context and answering any questions through voice.

Accomplishments we're proud of

We almost perfected this difficult coordinate grid being able to identify and directly highlight specific characters we want to address that the user wrote. For example if the user accidentally added a - sign, we could isolate in on the white board, highlight it, and provide feed back.

What we learned

We learned the power AI has to providing in depth education to all, but the meticulous steps that must be completed in order to provide a thorough learning experience. We worked with many workflows and systematically improved to the most efficient version for updates. Providing helpful and fast guidance.

What's next

for Whiteboard teacher Implementing with tools such as Kahn academy, One notes, Good notes, and other white board related tools or even providing this as its own service could be very beneficial for individuals to have their own teachers.

Whiteboard App

A simple and modern whiteboard application built with React, TypeScript, and tldraw.

Features

🎨 Interactive drawing canvas
🖌️ Multiple drawing tools (pen, brush, eraser, etc.)
📝 Text and shape tools
🎯 Selection and manipulation tools
📱 Responsive design
🌈 Color picker and style options
🔍 Zoom and pan functionality
🤖 AI-Powered OCR Analysis - Extract text from your whiteboard and get intelligent feedback
🧮 Mathematical Analysis - AI suggestions for math equations and calculations
📊 Smart Positioning - AI suggestions positioned contextually on your whiteboard

Getting Started

Prerequisites

Node.js (version 16 or higher)
npm or yarn

Installation

Clone or download this project
Install dependencies:
```
npm install
cd backend
npm install
```

API Keys Setup

To enable AI features, you'll need to set up API keys:

Google Cloud Vision API: Place your service account JSON file in the backend/ directory
Gemini AI API:
- Get your API key from Google AI Studio
- In the backend/ directory, create a .env file:
```
GEMINI_API_KEY=your-gemini-api-key-here
```

Running the Application

Start the backend server:
```
cd backend
node server.js
```
In a new terminal, start the frontend development server:
```
npm run dev
```
Open your browser and navigate to http://localhost:5173
Start drawing on your whiteboard!

Using AI Features

Draw some mathematical content on your whiteboard (equations, numbers, text)
Click the "Process Whiteboard (OCR)" button
The AI will:
- Extract all text from your whiteboard
- Analyze mathematical expressions
- Generate intelligent suggestions and feedback
- Position these suggestions contextually on your whiteboard
- Download a detailed JSON analysis file

Building for Production

To create a production build:

npm run build

To preview the production build:

npm run preview

Usage

Draw: Use the pen tool to draw freehand
Shapes: Add rectangles, circles, and other shapes
Text: Add text annotations
Select: Click and drag to select and move objects
Zoom: Use the zoom controls or mouse wheel to zoom in/out
Pan: Hold space and drag to pan around the canvas
Undo/Redo: Use Ctrl+Z and Ctrl+Y (or Cmd+Z and Cmd+Y on Mac)

Technologies Used

React 18 - UI framework
TypeScript - Type safety
Vite - Build tool and dev server
tldraw - Drawing library
CSS3 - Styling with modern features

Project Structure

whiteboard-app/
├── src/
│   ├── App.tsx          # Main application component
│   ├── main.tsx         # React entry point
│   └── index.css        # Global styles
├── index.html           # HTML template
├── package.json         # Dependencies and scripts
├── tsconfig.json        # TypeScript configuration
├── vite.config.ts       # Vite configuration
└── README.md           # This file

Customization

You can customize the whiteboard by modifying the Tldraw component props in src/App.tsx:

showMenu: Show/hide the main menu
showPages: Show/hide page management
showStyles: Show/hide style options
showUI: Show/hide the entire UI
showZoom: Show/hide zoom controls
showTools: Show/hide the toolbar

License

This project is open source and available under the MIT License.

Analysis

Compare with all teams

View

Metric

M-A-A-R
5

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
ExpressIn code
Google GeminiIn code
HTMLIn code
JavaScriptIn code
ReactIn code
TypeScriptIn code
PythonClaimed

7 of 8 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

62 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.