# Project export: The Magic Table This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Any sufficiently advanced technology looks like magic. With VLMs, LLMs, and a custom magnetic gantry, we built a table that aims to change how physically disadvantaged people interact with the world. - Devpost: https://devpost.com/software/the-magic-table - GitHub: https://github.com/Jaybear411/treehacks2026 - Video: https://www.youtube.com/embed/zQGj1xWwJHQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Jay Khemchandani (1 commits) ## Devpost submission (written by the team) No Devpost description available. ## README (from the GitHub repository) # Magic Table - Voice-Controlled Object Retrieval A voice-controlled robotic system for accessibility. Uses computer vision to identify objects on a tabletop and an XY magnet gantry to drag them to a pickup zone. **Core use case:** Accessibility for blind/mobility-impaired users — "voice-controlled physical retrieval" in a clean, magical demo. ## How It Works 1. **Camera + Vision Model** watches the tabletop, identifies objects (keys, phone, glasses, etc.), and tracks their (x, y) position 2. **XY Gantry** with powerful magnets moves under the table surface 3. **Metal pucks** attached to objects couple with magnets underneath 4. **Voice commands** like "bring me the keys" route the magnet to pull that object to the pickup zone ## Hardware Requirements ### Electronics Stack | Component | Purpose | |-----------|---------| | Arduino Uno | Controller (runs GRBL firmware) | | CNC Shield V3 | Motor driver interface (plugs onto Uno) | | DRV8825 Drivers (x3) | Stepper motor drivers (X, Y, A slots) | | 24V Power Supply | Motor power (NOT to Arduino!) | | NEMA 17 Steppers (x3) | Two for Y-axis (ganged), one for X-axis | | Webcam | Overhead view of table surface | ### Mechanical Stack | Component | Specification | |-----------|---------------| | 2020 Aluminum Extrusion | 400mm length (x3) | | Linear Rails/V-wheels | For smooth gantry movement | | GT2 Belts + Pulleys | Motion transmission | | Strong Neodymium Magnets | Under-table magnet carriage | | Metal Pucks | Attached to objects for magnetic coupling | ### Wiring Diagram ``` ┌─────────────────┐ │ 24V Power │ │ Supply │ └────────┬────────┘ │ ┌────────────────────────┴────────────────────────┐ │ CNC Shield V3 │ │ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ │ │ X │ │ Y │ │ Z │ │ A │ ← Drivers │ │ │DRV │ │DRV │ │empty│ │DRV │ │ │ └──┬──┘ └──┬──┘ └─────┘ └──┬──┘ │ │ │ │ │ │ │ Motor Motor Motor │ │ #3 #1 #2 │ │ (X) (Y) (Y clone) │ │ │ │ Set A→Y jumper to clone A axis to Y! │ └─────────────────────────────────┬──────────────┘ │ (sits on top) ┌─────────────────┴──────────────┐ │ Arduino Uno │ │ (GRBL) │ └─────────────────┬──────────────┘ │ USB ┌─────────────────┴──────────────┐ │ Computer │ │ (runs this Python code) │ └────────────────────────────────┘ ``` ## Software Setup ### 1. Flash GRBL to Arduino 1. Download [GRBL](https://github.com/grbl/grbl) 2. Flash to Arduino Uno via Arduino IDE 3. Configure GRBL settings (see below) ### 2. Install Python Dependencies ```bash # Create virtual environment python3 -m venv .venv source .venv/bin/activate # Linux/Mac # .venv\Scripts\activate # Windows # Install dependencies ``` **Note for macOS:** PyAudio may require portaudio: ```bash brew install portaudio pip install pyaudio ``` ### 3. Configure Serial Port Edit `config.py` and set the correct serial port: ```python # macOS port: str = "/dev/tty.usbmodem1101" # Linux port: str = "/dev/ttyUSB0" # Windows port: str = "COM3" ``` Find your port: ```bash # macOS/Linux ls /dev/tty.* # Or use Python python -c "from grbl_controller import GRBLController; print(GRBLController.list_ports())" ``` ### 4. GRBL Configuration Connect via serial terminal (115200 baud) and configure: ``` $100=160 # X steps/mm (adjust for your setup) $101=160 # Y steps/mm $110=5000 # X max rate mm/min $111=5000 # Y max rate mm/min $120=500 # X acceleration mm/sec^2 $121=500 # Y acceleration $130=400 # X max travel mm $131=400 # Y max travel mm ``` ## Usage ### Run Full System ```bash python main.py ``` ### Test Individual Components ```bash # Test vision only (no motor/voice) python main.py --test-vision # Test voice recognition python main.py --test-voice # Test motor control python main.py --test-motor # Run without voice python main.py --no-voice # Run without motor (vision demo) python main.py --no-motor ``` ### Calibrate Camera-to-Gantry Mapping ```bash python main.py --calibrate ``` This walks you through clicking four corners to map camera pixels to physical coordinates. ### Motor Sweep Calibration (recommended for motion accuracy) ```bash # Default 4x3 sweep grid (12 points) python main.py --calibrate-motor # Denser grid for higher accuracy python main.py --calibrate-motor --cal-grid-x 5 --cal-grid-y 4 ``` The gantry automatically moves across the board. At each stop, click the magnet in the camera view. This records many pixel->physical pairs and saves an improved homography in `calibration_data.json`. ## Voice Commands | Command | Action | |---------|--------| | "bring me the keys" | Fetch keys to pickup zone | | "get my phone" | Fetch phone to pickup zone | | "where is the wallet" | Highlight wallet location | | "stop" | Emergency stop | | "go home" | Return magnet to home position | You can also just say the object name: "keys", "phone", "glasses" ## Detectable Objects Default objects (configurable in `config.py`): - keys - airpods - phone - wallet - pill bottle - glasses - remote - pen - cup ## File Structure ``` treehacks2026/ ├── main.py # Main orchestration ├── config.py # Configuration settings ├── grbl_controller.py # Arduino/GRBL serial control ├── object_tracker.py # Vision-based object detection ├── voice_control.py # Speech recognition ├── calibration.py # Camera-to-gantry calibration ├── requirements.txt # Python dependencies └── README.md # This file ``` ## Troubleshooting ### Camera not found - Check camera index in `config.py` (try 0, 1, 2) - Ensure no other app is using the camera ### Motor not responding - Check USB connection - Verify serial port in `config.py` - Ensure GRBL is flashed correctly - Check 24V power to CNC shield ### Voice not recognized - Ensure microphone is working - Calibrate ambient noise (automatic on startup) - Speak clearly and at normal volume - Check internet connection (uses Google Speech API) ### Object detection poor - Improve lighting (even, diffused) - Adjust `score_threshold` in config (lower = more sensitive) - Try different objects/backgrounds - Camera should be overhead with clear view ## Development ### Architecture ``` ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Voice │ │ Vision │ │ Motor │ │ Controller │ │ Tracker │ │ Controller │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ ┌───────────────┴───────────────┐ │ └────► MagicTable (main.py) ◄────┘ │ - Orchestrates components │ │ - Coordinate transforms │ │ - Command processing │ └──────────────────────────────┘ ``` ### Adding New Objects Edit `config.py`: ```python @dataclass class DetectionConfig: prompts: list = None def __post_init__(self): if self.prompts is None: self.prompts = [ "keys", "phone", "your_new_object", # Add here ] ``` ## License MIT License - TreeHacks 2026 ## Credits - Zero-shot object detection: [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO) - Motor control: [GRBL](https://github.com/grbl/grbl) - [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 22 recognized source files, 269 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (29 of 29) ``` .gitignore axis_lengths.json calibration_data.json calibration.py config.py goto.py grbl_controller.py home.py magic-table-web/.gitignore magic-table-web/app/chat/page.tsx magic-table-web/app/globals.css magic-table-web/app/layout.tsx magic-table-web/app/page.tsx magic-table-web/next-env.d.ts magic-table-web/next.config.js magic-table-web/package.json magic-table-web/postcss.config.js magic-table-web/tailwind.config.ts magic-table-web/tsconfig.json main.py measure_axis.py motortest.py nlp_voice.py object_tracker.py README.md requirements.txt switch_diag.py voice_control.py web_server.py ``` ### Dependencies - magic-table-web/package.json: @types/node@^20, @types/react@^18, @types/react-dom@^18, autoprefixer@^10, next@^14.2.0, postcss@^8, react@^18.3.0, react-dom@^18.3.0, tailwindcss@^3.4.0, typescript@^5 - requirements.txt: anthropic@>=0.39.0, elevenlabs@>=1.0.0, flask@>=3.0.0, flask-cors@>=4.0.0, numpy@>=1.24.0, openai@>=1.0.0, opencv-python@>=4.8.0, pillow@>=10.0.0, pyaudio@>=0.2.14, pynput@>=1.7.7, pyserial@>=3.5, python-dotenv@>=1.0.0, SpeechRecognition@>=3.10.0, torch@>=2.2, torchvision, transformers@>=4.36.0 ### Recent commits (newest first) - final edits - buttons - Add venv/ to .gitignore to prevent large files from being tracked - many changes to software and calibration - Initial commit: Magic Table project ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` # Magic Table - Voice-Controlled Object Retrieval System # Python 3.10+ recommended # Core ML/Vision torch>=2.2 torchvision transformers>=4.36.0 # Computer Vision opencv-python>=4.8.0 pillow>=10.0.0 numpy>=1.24.0 # Speech Recognition SpeechRecognition>=3.10.0 pyaudio>=0.2.14 # Required for microphone input pynput>=1.7.7 # Key hold detection for push-to-talk # OpenAI (vision tasks — scan_table_objects) openai>=1.0.0 python-dotenv>=1.0.0 # Anthropic Claude (text NLP — object extraction, conversation) anthropic>=0.39.0 # ElevenLabs TTS (spoken responses in conversation mode) elevenlabs>=1.0.0 # Web API server (remote command interface) flask>=3.0.0 flask-cors>=4.0.0 # Serial Communication (Arduino/GRBL) pyserial>=3.5 # Optional: Text-to-Speech feedback # pyttsx3>=2.90 # Uncomment for TTS # Development/Testing # pytest>=7.0.0 # black>=23.0.0 ``` ### magic-table-web/package.json ``` { "name": "magic-table-web", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start" }, "dependencies": { "next": "^14.2.0", "react": "^18.3.0", "react-dom": "^18.3.0" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10", "postcss": "^8", "tailwindcss": "^3.4.0", "typescript": "^5" } } ``` ### main.py ```python """ Magic Table - Voice-Controlled Object Retrieval System A voice-controlled robotic system that uses computer vision to identify objects on a tabletop and moves them to a pickup zone using an XY gantry with magnets. Core Use Case: Accessibility for blind/mobility-impaired users. Hardware Stack: - Arduino Uno + CNC Shield V3 running GRBL - DRV8825 stepper drivers (X, Y, Z slots — Z mirrors Y in software for dual-Y gantry) - 24V power supply - 3x NEMA 17 stepper motors (1x X-axis, 2x Y-axis) - 400mm x 400mm gantry (2020 aluminum extrusion) - Camera (overhead view of table) - Magnets under table surface + metal pucks on objects Usage: python main.py # Run full system python main.py --no-voice # Run without voice control python main.py --no-motor # Run without motor control (vision only) python main.py --calibrate # Run calibration wizard """ import argparse import os import subprocess import sys import time from typing import Optional, Tuple import cv2 try: from pynput import keyboard as pynput_keyboard except Exception: pynput_keyboard = None # Audio file paths (relative to this script) _SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) AUDIO_YES_RIGHT_AWAY = os.path.join(_SCRIPT_DIR, "yesrightaway.mp3") AUDIO_SHUTTING_DOWN = os.path.join(_SCRIPT_DIR, "shuttingdown.mp3") AUDIO_HAPPY_TO_HELP = os.path.join(_SCRIPT_DIR, "happytohelp.mp3") def play_audio(filepath: str, block: bool = False): """ Play an mp3 file using macOS afplay. Args: filepath: Path to the audio file. block: If True, wait for playback to finish before returning. """ if not os.path.exists(filepath): print(f"Audio file not found: {filepath}") return try: proc = subprocess.Popen( ["afplay", filepath], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) if block: proc.wait() except Exception as e: print(f"Audio playback error: {e}") from config import GANTRY, GRBL, CAMERA, DETECTION, print_config from grbl_controller import GRBLController from object_tracker import ObjectTracker, TrackedObject from voice_control import VoiceController, VoiceCommand, CommandType from nlp_voice import NLPVoiceController # Calibration is no longer used — pixel→mm conversion is simple flip+scale. # from calibration import CoordinateTransformer, CalibrationWizard, AutoCalibrator class MagicTable: """ Main controller that orchestrates vision, voice, and motor control. This is the central coordinator that: 1. Tracks objects via computer vision 2. Listens for voice commands 3. Translates pixel coordinates to physical coordinates 4. Controls the gantry to retrieve objects """ def __init__(self, enable_voice: bool = True, enable_motor: bool = True, target_label: str = None): """ Initialize the Magic Table system. Args: enable_voice: Enable voice recognition (NLP push-to-talk) enable_motor: Enable motor control target_label: Fixed label for 'V' key detection (optional) """ self.enable_voice = enable_voice self.enable_motor = enable_motor self.target_label = target_label # Components self.tracker: Optional[ObjectTracker] = None self.nlp_voice: Optional[NLPVoiceController] = None self.voice: Optional[VoiceController] = None self.motor: Optional[GRBLController] = None self.transformer = None # No longer used (flip+scale instead) # State self.running = False self._busy = False # Currently executing a command self._last_command: Optional[VoiceCommand] = None # Last detected object coordinates (pixel) self._last_result: Optional[Tuple[str, float, float]] = None # Status message for display self._status_message = "Initializing..." # Hold-to-talk keyboard state (global listener) self._keyboard_listener = None self._keys_down: set[str] = set() self._space_hold_latched = False self._c_hold_latched = False def _start_key_listener(self): """Start global key listener for hold-to-talk.""" if pynput_keyboard is None or self._keyboard_listener is not None: return def on_press(key): if key == pynput_keyboard.Key.space: self._keys_down.add("space") return try: ch = key.char.lower() if key.char else None if ch == "c": self._keys_down.add("c") except Exception: pass def on_release(key): if key == pynput_keyboard.Key.space: self._keys_down.discard("space") return try: ch = key.char.lower() if key.char else None if ch == "c": self._keys_down.discard("c") except Exception: pass self._keyboard_listener = pynput_keyboard.Listener( on_press=on_press, on_release=on_release, ) self._keyboard_listener.daemon = True self._keyboard_listener.start() def _stop_key_listener(self): """Stop global key listener.""" if self._keyboard_listener: self._keyboard_listener.stop() self._keyboard_listener = None self._keys_down.clear() def _is_key_held(self, key_name: str) -> bool: return key_name in self._keys_down def initialize(self) -> bool: """ Initialize all components. Returns True if successful. """ print("\n" + "="*60) print("MAGIC TABLE - Voice-Controlled [truncated — 33383 more characters] ``` ### magic-table-web/app/layout.tsx ```typescript import type { Metadata, Viewport } from "next"; import "./globals.css"; export const metadata: Metadata = { title: "Magic Table", description: "Voice-controlled object retrieval — web interface", }; export const viewport: Viewport = { width: "device-width", initialScale: 1, maximumScale: 5, userScalable: true, }; export default function RootLayout({ children, }: { children: React.ReactNode; }) { return ( {children} ); } ``` ### magic-table-web/app/page.tsx ```typescript "use client"; import { useState, FormEvent } from "react"; import { useRouter } from "next/navigation"; export default function LoginPage() { const [username, setUsername] = useState(""); const [serverUrl, setServerUrl] = useState( process.env.NEXT_PUBLIC_API_URL || "http://localhost:5050" ); const [error, setError] = useState(""); const router = useRouter(); function handleSubmit(e: FormEvent) { e.preventDefault(); if (username.trim().toLowerCase() !== "treehacks") { setError("Invalid username."); return; } // Store auth + server URL in sessionStorage sessionStorage.setItem("authed", "true"); sessionStorage.setItem("serverUrl", serverUrl.replace(/\/+$/, "")); router.push("/chat"); } return (

Magic Table

{ setUsername(e.target.value); setError(""); }} className="border border-gray-300 rounded px-3 py-2 text-base focus:outline-none focus:ring-2 focus:ring-black min-h-[44px]" autoFocus /> setServerUrl(e.target.value)} className="border border-gray-300 rounded px-3 py-2 text-base focus:outline-none focus:ring-2 focus:ring-black text-gray-500 min-h-[44px]" /> {error &&

{error}

}
); } ``` ### magic-table-web/app/chat/page.tsx ```typescript "use client"; import { useState, useEffect, useRef, FormEvent } from "react"; import { useRouter } from "next/navigation"; interface Message { role: "user" | "assistant"; text: string; } export default function ChatPage() { const [messages, setMessages] = useState([]); const [input, setInput] = useState(""); const [loading, setLoading] = useState(false); const [serverUrl, setServerUrl] = useState(""); const bottomRef = useRef(null); const inputRef = useRef(null); const router = useRouter(); // Auth gate useEffect(() => { const authed = sessionStorage.getItem("authed"); if (authed !== "true") { router.replace("/"); return; } setServerUrl( sessionStorage.getItem("serverUrl") || "http://localhost:5050" ); }, [router]); // Auto-scroll to bottom useEffect(() => { bottomRef.current?.scrollIntoView({ behavior: "smooth" }); }, [messages]); async function handleSend(e: FormEvent) { e.preventDefault(); const text = input.trim(); if (!text || loading) return; // Add user message const userMsg: Message = { role: "user", text }; setMessages((prev) => [...prev, userMsg]); setInput(""); setLoading(true); try { const res = await fetch(`${serverUrl}/api/command`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text }), }); const data = await res.json(); const reply = data.reply || data.error || "No response from server."; setMessages((prev) => [...prev, { role: "assistant", text: reply }]); } catch (err) { setMessages((prev) => [ ...prev, { role: "assistant", text: `Could not reach server at ${serverUrl}. Is it running?`, }, ]); } finally { setLoading(false); inputRef.current?.focus(); } } async function handleAction(endpoint: string, label: string) { if (loading) return; setMessages((prev) => [...prev, { role: "user", text: label }]); setLoading(true); try { const res = await fetch(`${serverUrl}${endpoint}`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({}), }); const data = await res.json(); const reply = data.reply || data.error || "No response from server."; setMessages((prev) => [...prev, { role: "assistant", text: reply }]); } catch (err) { setMessages((prev) => [ ...prev, { role: "assistant", text: `Could not reach server at ${serverUrl}. Is it running?`, }, ]); } finally { setLoading(false); inputRef.current?.focus(); } } function handleLogout() { sessionStorage.clear(); router.replace("/"); } return (
{/* Header */}

Magic Table

{/* Messages */}
{messages.length === 0 && (

Type a command like "fetch the red bottle" or just chat with Jarvis.

)} {messages.map((msg, i) => (
{msg.text}
))} {loading && (
Thinking...
)}
{/* Quick-action buttons */}
{/* Input */}
setInput(e.target.value)} disabled={loading} className="flex-1 border border-gray-300 rounded px-3 py-2 text-base focus:outline-none focus:ring-2 focus:ring-black disabled:opacity-50 min-h-[44px]" autoFocus />