# Project export: WhiteboardAI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: Enabling real world teachers for anyone, anytime
- Devpost: https://devpost.com/software/whiteboard-zlcny2
- GitHub: https://github.com/JaydenDhaliwal/Whiteboard-Frontend
- Video: https://www.youtube.com/embed/OlQ1ERSPytg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — M-A-A-R (5 commits)
## Devpost submission (written by the team)
### Inspiration
We all know the power of standing next to a good teacher at a white board as they smoother all your confusions through step by step guided walk through
### What it does
We do anything the same magic that teacher + white board can do it, using AI!
### How we built it
We reversed engineering current white board software's, creating our own unique coordinate system and ability for multi-modal ability + dual system updates. From here we used Google Cloud Vision API to analyze the current state of the white board (OCR) and used an efficient coordinate algorithm overlay to convert the photo not only to real world user data but also with accurate spatial relations of all the strokes on the white board. Doing this we were able to have real time data of users strokes and inputs, feeding all of the individual data into Gemini flash and Vapi. By doing so we were able to give Gemini's as well as Vapi context to provide feed back for the user in Json / Speech format. We used in depth prompt engineering to turn Gemini's and Vapi's outputs and walk through the users problem like an exceptional teacher providing direct hints and guidance directly through the whiteboard and speech. All of this happened in a matter of a few seconds, the low latency thanks to the OCR Coordinate System + Flash set up we choose.
### Challenges we ran into
We took a major risk with this project due to its many technical layers. The first was accurately converting the our whiteboard into a machine readable data with proper spatial relations, so that we can provide feedback, comments, and highlights in the correct locations. We decided to do OCR due to its speed over an LLM in order to quickly covert to text and then be able to use our algorithm to find coordinates (something an LLM is incredibly unreliable for). This coordinate algorithm itself took many hours to streamline but was the back bone between proper communication through the white board between teacher ai and user. Once we had the correct coordinates we had to work to perfectly prompt our models to provide useful guidance as well as having it printing on the white board in the correct locations so the user could seamlessly understand how our AI was aiding them, just like a teacher! Finally we added the ability for Vapi to also help you through understanding the context and answering any questions through voice.
### Accomplishments we're proud of
We almost perfected this difficult coordinate grid being able to identify and directly highlight specific characters we want to address that the user wrote. For example if the user accidentally added a - sign, we could isolate in on the white board, highlight it, and provide feed back.
### What we learned
We learned the power AI has to providing in depth education to all, but the meticulous steps that must be completed in order to provide a thorough learning experience. We worked with many workflows and systematically improved to the most efficient version for updates. Providing helpful and fast guidance.
### What's next
for Whiteboard teacher Implementing with tools such as Kahn academy, One notes, Good notes, and other white board related tools or even providing this as its own service could be very beneficial for individuals to have their own teachers.
## README (from the GitHub repository)
# Whiteboard App
A simple and modern whiteboard application built with React, TypeScript, and tldraw.
## Features
- 🎨 Interactive drawing canvas
- 🖌️ Multiple drawing tools (pen, brush, eraser, etc.)
- 📝 Text and shape tools
- 🎯 Selection and manipulation tools
- 📱 Responsive design
- 🌈 Color picker and style options
- 🔍 Zoom and pan functionality
- 🤖 **AI-Powered OCR Analysis** - Extract text from your whiteboard and get intelligent feedback
- 🧮 **Mathematical Analysis** - AI suggestions for math equations and calculations
- 📊 **Smart Positioning** - AI suggestions positioned contextually on your whiteboard
## Getting Started
### Prerequisites
- Node.js (version 16 or higher)
- npm or yarn
### Installation
1. Clone or download this project
2. Install dependencies:
```bash
npm install
cd backend
npm install
```
### API Keys Setup
To enable AI features, you'll need to set up API keys:
1. **Google Cloud Vision API**: Place your service account JSON file in the `backend/` directory
2. **Gemini AI API**:
- Get your API key from [Google AI Studio](https://makersuite.google.com/app/apikey)
- In the `backend/` directory, create a `.env` file:
```
GEMINI_API_KEY=your-gemini-api-key-here
```
### Running the Application
1. Start the backend server:
```bash
cd backend
node server.js
```
2. In a new terminal, start the frontend development server:
```bash
npm run dev
```
3. Open your browser and navigate to `http://localhost:5173`
4. Start drawing on your whiteboard!
### Using AI Features
1. Draw some mathematical content on your whiteboard (equations, numbers, text)
2. Click the "Process Whiteboard (OCR)" button
3. The AI will:
- Extract all text from your whiteboard
- Analyze mathematical expressions
- Generate intelligent suggestions and feedback
- Position these suggestions contextually on your whiteboard
- Download a detailed JSON analysis file
### Building for Production
To create a production build:
```bash
npm run build
```
To preview the production build:
```bash
npm run preview
```
## Usage
- **Draw**: Use the pen tool to draw freehand
- **Shapes**: Add rectangles, circles, and other shapes
- **Text**: Add text annotations
- **Select**: Click and drag to select and move objects
- **Zoom**: Use the zoom controls or mouse wheel to zoom in/out
- **Pan**: Hold space and drag to pan around the canvas
- **Undo/Redo**: Use Ctrl+Z and Ctrl+Y (or Cmd+Z and Cmd+Y on Mac)
## Technologies Used
- **React 18** - UI framework
- **TypeScript** - Type safety
- **Vite** - Build tool and dev server
- **tldraw** - Drawing library
- **CSS3** - Styling with modern features
## Project Structure
```
whiteboard-app/
├── src/
│ ├── App.tsx # Main application component
│ ├── main.tsx # React entry point
│ └── index.css # Global styles
├── index.html # HTML template
├── package.json # Dependencies and scripts
├── tsconfig.json # TypeScript configuration
├── vite.config.ts # Vite configuration
└── README.md # This file
```
## Customization
You can customize the whiteboard by modifying the `Tldraw` component props in `src/App.tsx`:
- `showMenu`: Show/hide the main menu
- `showPages`: Show/hide page management
- `showStyles`: Show/hide style options
- `showUI`: Show/hide the entire UI
- `showZoom`: Show/hide zoom controls
- `showTools`: Show/hide the toolbar
## License
This project is open source and available under the MIT License.
## Detected evidence (automated analysis)
Indexed codebase: 9 recognized source files, 62 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- Google Gemini (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code
- Python (language) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (13 of 13)
```
.gitignore
backend/server.js
COORDINATE_MAPPING.md
index.html
package.json
README.md
src/App.tsx
src/index.css
src/main.tsx
test-json-export.html
tsconfig.json
tsconfig.node.json
vite.config.ts
```
### Dependencies
- package.json: @google-cloud/vision@^5.2.0, @google/generative-ai@^0.24.1, @tldraw/tldraw@^2.0.0-alpha.17, @types/react@^18.2.43, @types/react-dom@^18.2.17, @vitejs/plugin-react@^4.2.1, cors@^2.8.5, express@^5.1.0, react@^18.2.0, react-dom@^18.2.0, typescript@^5.2.2, vite@^5.0.8
### Recent commits (newest first)
- update
- krishnaupdates
- 30%baby
- basicfunc
- ocrandspatialrealtion
- Added snapshot
- initial
## Key source files (fetched from GitHub, selected and truncated for size)
### COORDINATE_MAPPING.md
```markdown
# Coordinate Mapping System for AI Integration
## Overview
This system captures whiteboard screenshots with precise coordinate mapping, allowing AI to analyze the content and draw suggestions back onto the exact same locations on the whiteboard.
## How It Works
### 1. Screenshot Capture with Coordinate Mapping
When you click "Process Whiteboard (OCR)":
- Captures the current viewport bounds from tldraw
- Takes a screenshot of the visible area
- Records the mapping between image pixels and tldraw coordinates
- Sends both image and coordinate mapping to the backend
### 2. OCR Processing with Coordinate Conversion
The backend:
- Processes the image with Google Cloud Vision AI
- Gets text locations in image pixel coordinates
- Converts these to exact tldraw coordinates using the mapping
- Returns JSON with both coordinate systems
### 3. AI Integration Ready
The exported JSON contains:
- `wordAnnotations[].tldraw_coords`: Exact tldraw coordinates for each detected text
- `coordinateMapping`: Complete mapping information
- `spatialAnalysis`: Grouped equations with tldraw coordinates
## JSON Structure
```json
{
"fullText": "Combined detected text",
"wordAnnotations": [
{
"text": "x",
"bbox": { /* image pixel coordinates */ },
"center_x": 150, // image pixels
"center_y": 200, // image pixels
"tldraw_coords": {
"center_x": 45.5, // exact tldraw coordinate
"center_y": 67.3, // exact tldraw coordinate
"bbox": { /* tldraw coordinate bounding box */ }
}
}
],
"coordinateMapping": {
"imageWidth": 800,
"imageHeight": 600,
"tldrawBounds": {
"x": -100,
"y": -50,
"width": 400,
"height": 300
},
"exportScale": 2,
"camera": { /* camera info */ }
}
}
```
## For AI Systems
### Input to AI
Send the complete JSON from OCR processing. The AI can:
- Analyze text content and spatial relationships
- Understand mathematical equations and their positions
- Generate suggestions with specific coordinate targets
### AI Response Format
AI should return suggestions in this format:
```json
{
"suggestions": [
{
"type": "text",
"text": "Solution: x = 5",
"tldraw_coords": {
"center_x": 100.5,
"center_y": 200.3
}
},
{
"type": "arrow",
"start": { "x": 50, "y": 100 },
"end": { "x": 150, "y": 120 }
}
]
}
```
### Drawing AI Suggestions
Use the `drawAISuggestions(editor, suggestions)` function to draw AI responses directly onto the whiteboard at exact coordinates.
## Testing
1. Draw some math/text on the whiteboard
2. Click "Process Whiteboard (OCR)" - downloads JSON with coordinates
3. Click "Test AI Draw" - demonstrates drawing at specific coordinates
4. Feed JSON to your AI system
5. Use the returned suggestions with `drawAISuggestions()`
## Coordinate System Notes
- Tldraw uses a continuous coordinate system (not pixels)
- Coordinates can be negative (viewport can be anywhere
[truncated — 219 more characters]
```
### package.json
```
{
"name": "whiteboard-app",
"version": "1.0.0",
"description": "A simple whiteboard application using tldraw",
"main": "index.js",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"@google-cloud/vision": "^5.2.0",
"@google/generative-ai": "^0.24.1",
"@tldraw/tldraw": "^2.0.0-alpha.17",
"cors": "^2.8.5",
"express": "^5.1.0",
"react": "^18.2.0",
"react-dom": "^18.2.0"
},
"devDependencies": {
"@types/react": "^18.2.43",
"@types/react-dom": "^18.2.17",
"@vitejs/plugin-react": "^4.2.1",
"typescript": "^5.2.2",
"vite": "^5.0.8"
}
}
```
### src/main.tsx
```typescript
import React from 'react'
import ReactDOM from 'react-dom/client'
import App from './App.tsx'
import './index.css'
ReactDOM.createRoot(document.getElementById('root')!).render(
,
)
```
### src/App.tsx
```typescript
import React, { useState } from 'react';
import { Tldraw, Editor, TLGeoShape, TLTextShape, TLUiOverrides, createShapeId } from '@tldraw/tldraw';
import '@tldraw/tldraw/tldraw.css';
import './index.css'
const DEFAULT_CAMERA_STEPS = [0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8];
// UI overrides to lock camera zoom and pan - defined outside component to avoid re-creation
const uiOverrides: TLUiOverrides = {
actions(editor, actions) {
actions.lockCameraZoom = {
id: 'lock-camera-zoom',
kbd: 'shift+k',
onSelect() {
const isCameraZoomLockedAlready = editor.getCameraOptions().zoomSteps.length === 1
editor.setCameraOptions({
zoomSteps: isCameraZoomLockedAlready ? DEFAULT_CAMERA_STEPS : [editor.getZoomLevel()],
})
},
}
return actions
},
};
// New component to encapsulate the OCR button logic
interface OCRButtonProps {
editor: Editor | null;
}
// Helper function to draw AI suggestions back onto the whiteboard
function drawAISuggestions(editor: Editor, suggestions: any[]) {
console.log("Drawing AI suggestions:", suggestions);
console.log("Current viewport bounds:", editor.getViewportPageBounds());
console.log("Current camera:", editor.getCamera());
// CLEAR ALL EXISTING AI SUGGESTIONS FIRST
console.log("Clearing old AI suggestions...");
const allShapes = editor.getCurrentPageShapes();
const suggestionsToDelete = allShapes.filter(shape => {
// Primary method: Check for metadata tag
if (shape.meta?.isAISuggestion) {
return true;
}
// Fallback method: Pattern matching for older suggestions without metadata
if (shape.type === 'text') {
const textShape = shape as any;
const text = textShape.props.text?.toLowerCase() || '';
// Broader detection - any text that looks like AI feedback or highlight labels
return text.includes('solve') || text.includes('subtract') || text.includes('divide') ||
text.includes('good') || text.includes('show') || text.includes('work') ||
text.includes('next') || text.includes('step') || text.includes('keep') ||
text.includes('great') || text.includes('add') || text.includes('multiply') ||
text.includes('hint') || text.includes('try') || text.includes('correct') ||
text.includes('check') || text.includes('verify') || text.includes('complete') ||
// Check for highlight labels (text with quotes)
(text.startsWith('"') && text.endsWith('"')) ||
// Check if text has colored styling (AI suggestions have specific colors)
(textShape.props.color && ['red', 'orange', 'blue', 'green'].includes(textShape.props.color)) ||
// Check for mono font (used for highlight labels)
(textShape.props.font === 'mono');
}
if (shape.type === 'geo') {
const geoShape = shape as any;
// Remove ALL dashed rectangles (suggestion backgrounds) and highlight boxes
return geoShape.props.dash === 'dashed' ||
(geoShape.props.fill === 'none' && geoShape.props.w < 500) || // Small rectangles are likely suggestion boxes
(geoShape.props.fill === 'semi') || // Old highlight rectangles
(shape.meta?.suggestionType === 'highlight') ||
(shape.meta?.suggestionType === 'highlight_border') ||
(shape.meta?.suggestionType === 'highlight_box') ||
(shape.meta?.suggestionType === 'underline'); // New highlight shapes
}
if (shape.type === 'arrow') {
// Remove pointer arrows
return shape.meta?.suggestionType === 'pointer_arrow';
}
return false;
});
// Delete old suggestions
if (suggestionsToDelete.length > 0) {
console.log(`Deleting ${suggestionsToDelete.length} old suggestions`);
console.log("Suggestion texts being deleted:",
suggestionsToDelete
.filter(s => s.type === 'text')
.map(s => (s as any).props.text)
);
editor.deleteShapes(suggestionsToDelete.map(shape => shape.id));
}
suggestions.forEach((suggestion, index) => {
// Only draw text suggestions, skip arrows completely
if (suggestion.type === 'text' && suggestion.tldraw_coords) {
console.log(`Drawing suggestion ${index} at:`, suggestion.tldraw_coords);
console.log(`Backend thinks text is at: x=${suggestion.tldraw_coords.x}, y=${suggestion.tldraw_coords.y}`);
// Position suggestions directly under user's text vertically
const forcedCoords = {
x: suggestion.tldraw_coords.x, // Use same X position as detected text
y: suggestion.tldraw_coords.y + 150 + (index * 60) // Much more spacing: 150px below text, 60px between suggestions
};
console.log(`Positioning suggestion ${index} under text at:`, forcedCoords);
// Choose color based on priority and category
let color = 'red'; // default
let size = 'm'; // default
switch (suggestion.priority) {
case 'high':
color = 'red';
size = 'l';
break;
case 'medium':
color = 'orange';
size = 'm';
break;
case 'low':
color = 'blue';
size = 's';
break;
}
// Different styling for different categories
switch (suggestion.category) {
case 'correction':
color = 'red';
break;
case 'clarification':
color = 'orange';
break;
case 'next_step':
color = 'blue';
break;
case 'encouragement':
color = 'green';
break;
}
// Draw a clear box around the referenced text (if provided)
if (suggestion.highlightRegion && suggestion.highlightRegion.bbox) {
console.log(`Drawing highlight box for suggestion ${index}:`, suggestion.highlightRegion);
const bbox = suggestion.highlight
[truncated — 19259 more characters]
```
### backend/server.js
```javascript
const express = require('express');
const { ImageAnnotatorClient } = require('@google-cloud/vision');
const { GoogleGenerativeAI } = require('@google/generative-ai');
const path = require('path');
const cors = require('cors'); // Required for cross-origin requests from your frontend
const app = express();
const port = 3001; // Choose a port that isn't already in use by your frontend (Vite usually uses 5173)
// Explicitly set the path to your service account key file
// Make sure this path is correct relative to where your server.js is located
process.env.GOOGLE_APPLICATION_CREDENTIALS = path.join(__dirname, 'berkeleyaihackathon-f5e4780fd20e.json');
const client = new ImageAnnotatorClient();
// Initialize Gemini AI
// Replace 'your-gemini-api-key-here' with your actual Gemini API key from Google AI Studio
const genAI = new GoogleGenerativeAI('AIzaSyDgX4ncN87Ef5UlJj81pVRijdAh1ENwF_E');
app.use(cors()); // Enable CORS for requests from your frontend
app.use(express.json({ limit: '10mb' })); // Increase limit for larger image data
app.post('/api/ocr-vision', async (req, res) => {
try {
console.log('OCR request received');
const imageData = req.body.imageData;
const coordinateMapping = req.body.coordinateMapping;
if (!imageData) {
console.log('No image data provided');
return res.status(400).json({ error: 'No image data provided.' });
}
console.log('Image data received, length:', imageData.length);
console.log('Coordinate mapping received:', coordinateMapping);
console.log('Sending to Google Cloud Vision AI...');
// Google Cloud Vision API request
const [result] = await client.textDetection({
image: {
content: imageData,
},
});
console.log('Google Cloud Vision AI response received');
const detections = result.textAnnotations;
if (detections && detections.length > 0) {
console.log('Text detections found:', detections.length);
const fullText = detections[0].description;
const wordAnnotations = detections.slice(1).map(annotation => ({
text: annotation.description,
bbox: {
x_min: Math.min(...annotation.boundingPoly.vertices.map(v => v.x || 0)),
y_min: Math.min(...annotation.boundingPoly.vertices.map(v => v.y || 0)),
x_max: Math.max(...annotation.boundingPoly.vertices.map(v => v.x || 0)),
y_max: Math.max(...annotation.boundingPoly.vertices.map(v => v.y || 0)),
},
center_x: 0, // Will be calculated below
center_y: 0, // Will be calculated below
}));
// Calculate center points and convert to tldraw coordinates
wordAnnotations.forEach(annotation => {
annotation.center_x = (annotation.bbox.x_min + annotation.bbox.x_max) / 2;
annotation.center_y = (annotation.bbox.y_min + annotation.bbox.y_max) / 2;
// Convert from image coordinates to tldraw coordinates if mapping is provided
if (coordinateMapping) {
annotation.tldraw_coords = convertImageToTldrawCoords(annotation, coordinateMapping);
}
});
// Spatial Analysis for Mathematical Expressions
const spatialAnalysis = performSpatialAnalysis(wordAnnotations);
// Create vertical text order for easier AI model reading
const verticalTextOrder = createVerticalTextOrder(wordAnnotations);
console.log('Full text detected:', fullText);
console.log('Spatial analysis completed');
// Generate AI suggestions using Gemini
console.log('Generating AI suggestions with Gemini...');
const aiSuggestions = await generateAISuggestions(fullText, spatialAnalysis, wordAnnotations, coordinateMapping, verticalTextOrder);
console.log('AI suggestions generated:', aiSuggestions.length);
const responseData = {
// Easy-to-read section for AI model
verticalTextOrder: verticalTextOrder,
// Detailed technical data
fullText: fullText,
wordAnnotations: wordAnnotations,
spatialAnalysis: spatialAnalysis,
aiSuggestions: aiSuggestions,
coordinateMapping: coordinateMapping,
tldrawCoordinates: coordinateMapping ? 'included' : 'not_provided',
};
// Log response data for debugging
console.log('Response data structure:', {
fullTextLength: fullText?.length || 0,
wordAnnotationsCount: wordAnnotations?.length || 0,
spatialAnalysisCount: spatialAnalysis?.equations?.length || 0,
coordinateMappingPresent: !!coordinateMapping,
});
res.json(responseData);
} else {
console.log('No text detected in image');
res.json({ fullText: '', wordAnnotations: [], spatialAnalysis: null });
}
} catch (error) {
console.error('Google Cloud Vision AI Error Details:');
console.error('Error message:', error.message);
console.error('Error code:', error.code);
console.error('Error details:', error.details);
console.error('Full error:', error);
res.status(500).json({ error: 'Failed to process image with Vision AI.' });
}
});
// Spatial Analysis Function for Mathematical Expressions
function performSpatialAnalysis(wordAnnotations) {
console.log('Starting spatial analysis...');
// Group elements by approximate horizontal lines (equations)
const equations = groupIntoEquations(wordAnnotations);
// Analyze each equation for mathematical structure
const analyzedEquations = equations.map(equation => analyzeEquation(equation));
console.log(`Spatial analysis found ${equations.length} equations:`);
analyzedEquations.forEach((eq, index) => {
console.log(` Equation ${index}: "${eq.reconstructedText}" (type: ${eq.type})`);
console.log(` Elements: ${eq.originalElements.map(el => `"${el.text}"`).join(', ')}`);
console.log(` Y-range: ${Math.round(eq.boundingBox.y_min)} to ${Math.round(eq.boundingBox.y_max)}`);
});
return
[truncated — 20491 more characters]
```
### vite.config.ts
```typescript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
// https://vitejs.dev/config/
export default defineConfig({
plugins: [react()],
server: {
port: 3000,
open: true
}
})
```
### index.html
```html
Whiteboard App
```
### test-json-export.html
```html
JSON Export Test