# Project export: Maestro This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Turn an everyday object into infinite instruments. - Devpost: https://devpost.com/software/maestro-n0uqyz - GitHub: https://github.com/markmusic27/treehacks - Video: https://www.youtube.com/embed/7tlVRvY_3n8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([Suno] Best Musical Hack) - Team: 3 GitHub contributor(s) — Mark Music (15 commits), Vansh Gadhia (8 commits), Cursor (2 commits) ## Devpost submission (written by the team) ### Inspiration We’re an international team, and music is one of the few languages everyone on our team shares. No matter where we’re from, we’ve all used everyday objects, desks, bottles, and brooms to make rhythm or pretend instruments. We wanted to build something that lets anyone express themselves musically and develop their musical skills without needing expensive gear or formal training. ### What it does Maestro turns a broom into a playable guitar and music studio. Using a camera, hand tracking, and sound mapping, we detect a broom and map your hand movements to musical notes so you can instantly play it like a real instrument. An iPhone companion detects pressed “strings,” letting you form chords and control pitch while strumming in the air. If you have a broom, you have an instrument. Maestro also acts as a music coach and generator. After you play, the system analyzes your posture, timing, and sound, then gives short, actionable feedback through an AI tutor. In generation mode, what you play on the broom becomes the seed for a fully produced track: your performance is analyzed and expanded into new music. The goal is to make music creation accessible, expressive, and global, from a single everyday object to unlimited instruments and songs. ### How we built it We built Maestro as a real-time multimodal system combining computer vision, hand tracking, audio analysis, and multi-agent AI. Instrument mapping + tracking: MediaPipe hand tracking provides 21 landmarks per hand each frame Wrist + strum fingertip are smoothed to form a virtual guitar “neck” Magenta tape on the broom pole is detected with OpenCV Fretting hand position projected along the pole gives a 0–1 value that drives pitch/octave Strums are detected when the strumming fingertip crosses the neck line with enough velocity An iPhone app detects pressed “strings” and sends chord data to the system Real-time system + web interface: A Python server streams webcam frames over WebSocket Browser UI sends commands (like “get coaching”) and recorded audio The system captures sound-to-feature data and maps gestures to notes live Dual-agent AI coaching: On “stop playing,” the latest fra:me and audio are sent to a GX10 server Qwen2.5-VL analyzes posture and technique from video NVIDIA Music Flamingo analyzes rhythm, timing, and musical style Both agents run in parallel and return structured feedback Feedback is merged into a single script and delivered via on-screen text and TTS Hardware + generation pipeline: Models run on an ASUS GX10 with an NVIDIA Blackwell GPU Separate services handle vision+audio and audio-only flows In generation mode, Music Flamingo analyzes your performance The output is passed to a Suno API pipeline to generate a full track based on what you played ### Challenges we ran into Getting low-latency tracking so strums feel instant and musical Making hand tracking stable across lighting, camera angles, and different brooms Mapping gestures to notes in a way that feels intuitive Syncing iPhone string detection with webcam tracking in real time Running vision + audio models together without slowing feedback Designing short, useful coaching instead of overwhelming users ### Accomplishments we're proud of Turning a literal broom into a playable guitar that feels responsive Creating a system anyone can understand in seconds Making something expressive across cultures and skill levels Getting real-time interaction working end-to-end ### What we learned Latency matters more than model size for musical interaction Multimodal AI (vision + audio) creates much richer feedback Real-time systems require tight coordination between frontend, backend, and models The best demos are immediately understandable and fun ### What's next Support more objects beyond a broom Add multiplayer and collaborative jam mode Personal practice tracking and adaptive coaching Expand generation mode into full song creation tools ## README (from the GitHub repository)

Maestro

✶ Real-Time Air Instrument & AI Music Coach ✶
Devpost »  ·  Run locally: uv run treehacks-server + cd website && npm run dev

Maestro is a real-time **air instrument** system that combines **computer vision** (hand tracking + strum detection on a laptop) with a **mobile fretboard app** (multi-touch on iPhone). The two talk over WebSocket; on each strum, MIDI notes are generated and played through the laptop via FluidSynth. An optional **Next.js web app** streams the camera feed to the browser and connects to an **AI coaching backend** (Qwen2.5-VL + Music Flamingo on an ASUS GX10) for form and performance feedback. ![Project Demo](demo/original.png) ## Motivation Learning an instrument is hard—and getting *actionable* feedback on posture, hand position, and rhythm usually requires a teacher in the room. We wanted to turn a laptop webcam and a phone into a playable string instrument *and* hook it into an AI coach that can see and hear you. Maestro takes the “air guitar” idea seriously: you form a **neck line** with your two wrists in front of the camera, **strum** with one hand across that line, and use the **phone** as a multi-touch fretboard to choose strings and frets. The system fuses hand distance, strum velocity, and touch data into MIDI and plays it in real time. When the browser bridge and GX10 are in the loop, a single frame + audio clip can be sent to a dual-agent coach (visual form + audio performance) for structured, culturally aware feedback—so practice becomes more than just playing into the void. ## Technical Overview - **Python backend (`src/`)** — MediaPipe Hand Landmarker (LIVE_STREAM), OpenCV overlay, strum detection (perpendicular crossing velocity), note engine (string + fret + pole position → MIDI), FluidSynth playback. WebSocket server on port 8765 for the phone fretboard. Run with `uv run treehacks` (OpenCV window) or `uv run treehacks-server` (headless, for browser). - **Browser bridge** — `treehacks-server` runs a second WebSocket on port 8766: binary JPEG frames out, JSON commands in. Lets the Next.js app show the camera feed and control start/stop without a local OpenCV window. - **Next.js 15 web app (`website/`)** — TypeScript/Tailwind app for play mode, tutor flows, analysis, and integration with the GX10 dual-agent coach (video frame + audio → visual + audio feedback). - **GX10 dual-agent coach** — Optional Python service on the ASUS GX10: Qwen2.5-VL for visual form (posture, hands, technique) and Music Flamingo for audio (tempo, rhythm, style). See [GX10_INTEGRATION.md](GX10_INTEGRATION.md). - **iOS fretboard (`fretboard/`)** — Native Swift/SwiftUI app: multi-touch strings, WebSocket client to the Python backend, CoreMotion optional. Connect to the laptop’s IP and play. ## File structure ```text treehacks/ src/ # Python backend (air instrument + MIDI) main.py # OpenCV window entry (uv run treehacks) server.py # Browser bridge + GX10 (uv run treehacks-server) config.py # Paths, thresholds, ports, note tuning hand_tracking.py # Hand ID, strum detection, landmark math drawing.py # OpenCV overlay (neck line, panels, notes) note_engine.py # String/fret/pole/velocity → MIDI audio_engine.py # FluidSynth wrapper websocket_server.py # Phone fretboard WS (port 8765) pole_detection.py # Magenta-tape pole (hand position → octave) models.py # FretboardState, PhoneState, PoleState, etc. website/ # Next.js 15 app app/ api/ # audio-coach, analysis, realtime, suno, etc. play/ # Play / live camera tutor/ # Session, personality, summary analysis/, remixes/, ... components/ lib/ fretboard/ # iOS app (Swift/SwiftUI) fretboard/ ContentView.swift # WebSocketManager, WelcomeScreen, FretboardScreen RawTouchView # Multi-touch → normalized (x,y) + string demo/ original.png # Demo screenshot hand_landmarker.task # MediaPipe model (project root) soundfont.sf2 # FluidSynth SoundFont (project root) pyproject.toml # Python deps (uv) CONTEXT.md # Architecture, protocol, module roles GX10_INTEGRATION.md # GX10 endpoints, env, usage README.md ``` ## Running the stack **Python (from project root):** - OpenCV + camera + phone WS: `uv run treehacks` or `uv run python -m src.main` - Headless browser bridge (for web app): `uv run treehacks-server` or `uv run python -m src.server` **System:** `brew install fluidsynth`. Place `hand_landmarker.task` and a `soundfont.sf2` at project root (see CONTEXT.md). **Web app:** `cd website && npm install && npm run dev` **iOS:** Open `fretboard/` in Xcode, build, enter the laptop’s IP on the welcome screen. ## API (summary) - **WebSocket 8765** — Phone fretboard → Python. JSON `{ "touches": [ { "id", "x", "y", "string" } ] }`. - **WebSocket 8766** — Browser ↔ `treehacks-server`. Binary: JPEG frames. JSON: `{"action":"start"|"stop"}`, status, MIDI events. - **GX10** — `POST /dual-coach` with `image_base64`, `audio_base64`, `culture`, `instrument`. Returns `visual_coach` (Qwen2.5-VL) and `audio_coach` (Music Flamingo). See [GX10_INTEGRATION.md](GX10_INTEGRATION.md). ## Conventions - Python: `src/` package, relative imports, config and assets at project root. - Entry points: `uv run treehacks` (OpenCV), `uv run treehacks-server` (browser + GX10). - Next.js: App Router, TypeScript, Tailwind. ## License Apache-2.0. See `LICENSE` for details. ## Detected evidence (automated analysis) Indexed codebase: 263 recognized source files, 1705 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 304) ``` .gitignore app/__init__.py app/__main__.py app/cv_session.py app/fretboard.xcodeproj/project.pbxproj app/fretboard.xcodeproj/project.xcworkspace/contents.xcworkspacedata app/fretboard.xcodeproj/project.xcworkspace/xcuserdata/mmusic.xcuserdatad/UserInterfaceState.xcuserstate app/fretboard.xcodeproj/xcuserdata/mmusic.xcuserdatad/xcschemes/xcschememanagement.plist app/fretboard/AppDelegate.swift app/fretboard/Assets.xcassets/AccentColor.colorset/Contents.json app/fretboard/Assets.xcassets/AppIcon.appiconset/Contents.json app/fretboard/Assets.xcassets/Contents.json app/fretboard/ContentView.swift app/fretboard/fretboardApp.swift app/fretboard/FretboardView.swift app/fretboard/FretboardViewController.swift app/fretboard/Info.plist app/fretboard/MotionManager.swift app/fretboard/SceneDelegate.swift app/fretboard/WebSocketManager.swift app/instrument_setup.py app/main.py app/requirements.txt app/session.py hand_landmarker.task JAM_BAND/jam_band.py JAM_BAND/requirements.txt MIDI_TO_SOUNDFONT/FluidSynth_Player/fluidsynth_player.py MIDI_TO_SOUNDFONT/FluidSynth_Player/play_midi.py MIDI_TO_SOUNDFONT/FluidSynth_Player/test_midi.py MIDI_TO_SOUNDFONT/full_band_pipeline.py MIDI_TO_SOUNDFONT/MIDI_generation/drum_transcriber.py MIDI_TO_SOUNDFONT/MIDI_generation/generate_MIDI.py MIDI_TO_SOUNDFONT/MIDI_generation/manage_instruments.py MIDI_TO_SOUNDFONT/MIDI_generation/perplexity_search.py MIDI_TO_SOUNDFONT/MIDI_generation/stem_separator.py MIDI_TO_SOUNDFONT/MIDI_generation/youtube_to_midi.py MIDI_TO_SOUNDFONT/midi_to_mp3.py MIDI_TO_SOUNDFONT/pipeline.py MIDI_TO_SOUNDFONT/realtime_player.py MIDI_TO_SOUNDFONT/requirements.txt MIDI_TO_SOUNDFONT/REST MIDI_TO_SOUNDFONT/riff_bank.py MIDI_TO_SOUNDFONT/test_instrument_notes.py pyproject.toml README.md SONG_GENERATION/generate_song.py SONG_GENERATION/render_midi_to_wav.py SONG_GENERATION/requirements.txt src/__init__.py src/audio_engine.py src/config.py src/drawing.py src/hand_tracking.py src/main.py src/models.py src/note_engine.py src/pole_detection.py src/server.py src/websocket_server.py uv.lock vision/.python-version vision/audio_engine.py vision/config.py vision/CONTEXT.md vision/drawing.py vision/fretboard/fretboard.xcodeproj/project.pbxproj vision/fretboard/fretboard.xcodeproj/project.xcworkspace/contents.xcworkspacedata vision/fretboard/fretboard.xcodeproj/project.xcworkspace/xcuserdata/mmusic.xcuserdatad/UserInterfaceState.xcuserstate vision/fretboard/fretboard.xcodeproj/xcuserdata/mmusic.xcuserdatad/xcschemes/xcschememanagement.plist vision/fretboard/fretboard/AppDelegate.swift vision/fretboard/fretboard/Assets.xcassets/AccentColor.colorset/Contents.json vision/fretboard/fretboard/Assets.xcassets/AppIcon.appiconset/Contents.json vision/fretboard/fretboard/Assets.xcassets/Contents.json vision/fretboard/fretboard/ContentView.swift vision/fretboard/fretboard/fretboardApp.swift vision/fretboard/fretboard/FretboardView.swift vision/fretboard/fretboard/FretboardViewController.swift vision/fretboard/fretboard/Info.plist vision/fretboard/fretboard/MotionManager.swift vision/fretboard/fretboard/SceneDelegate.swift vision/fretboard/fretboard/WebSocketManager.swift vision/hand_landmarker.task vision/hand_tracking.py vision/main.py vision/models.py vision/note_engine.py vision/pole_detection.py vision/pyproject.toml vision/README.md vision/server.py vision/uv.lock vision/website/.gitignore vision/website/app/analysis/page.tsx vision/website/app/api/analyze/route.ts vision/website/app/api/audio-coach/route.ts vision/website/app/api/claude/artist/route.ts vision/website/app/api/claude/instrument/route.ts vision/website/app/api/feedback/route.ts vision/website/app/api/instructor/chat/route.ts vision/website/app/api/remix/route.ts vision/website/app/api/sketchfab/route.ts vision/website/app/api/songs/search/route.ts vision/website/app/api/soundfont/search/route.ts vision/website/app/api/suno/clips/route.ts vision/website/app/api/suno/generate/route.ts vision/website/app/api/youtube/route.ts vision/website/app/globals.css vision/website/app/home/page.tsx vision/website/app/layout.tsx vision/website/app/learn/page.tsx vision/website/app/page.tsx vision/website/app/play/page.tsx vision/website/app/remixes/page.tsx vision/website/app/results/page.tsx vision/website/app/select/page.tsx vision/website/components.json vision/website/components/AnimatedCharacter.tsx vision/website/components/CulturalInstructor.tsx vision/website/components/maestro/confetti-burst.tsx [184 more files omitted for size] ``` ### Dependencies - app/requirements.txt: basic-pitch, demucs, mediapipe, numpy@<2, onnxruntime, opencv-python, pretty_midi, pyfluidsynth, resampy@<0.4.3, scipy@<1.13, setuptools@<70, torchcodec, websockets, yt-dlp - JAM_BAND/requirements.txt: pygame - MIDI_TO_SOUNDFONT/requirements.txt: basic-pitch, demucs, numpy@<2, onnxruntime, pyfluidsynth, resampy@<0.4.3, scipy@<1.13, setuptools@<70, torchcodec, yt-dlp - pyproject.toml: httpx, mediapipe, numpy, opencv-python, pyfluidsynth, websockets - SONG_GENERATION/requirements.txt: pretty_midi - vision/pyproject.toml: httpx, mediapipe, numpy, opencv-python, pyfluidsynth, websockets - vision/website/package.json: @hookform/resolvers@^3.10.0, @radix-ui/react-accordion@1.2.2, @radix-ui/react-alert-dialog@1.1.4, @radix-ui/react-aspect-ratio@1.1.1, @radix-ui/react-avatar@1.1.2, @radix-ui/react-checkbox@1.1.3, @radix-ui/react-collapsible@1.1.2, @radix-ui/react-context-menu@2.2.4, @radix-ui/react-dialog@1.1.4, @radix-ui/react-dropdown-menu@2.1.4, @radix-ui/react-hover-card@1.1.4, @radix-ui/react-label@2.1.1, @radix-ui/react-menubar@1.1.4, @radix-ui/react-navigation-menu@1.2.3, @radix-ui/react-popover@1.1.4, @radix-ui/react-progress@1.1.1, @radix-ui/react-radio-group@1.2.2, @radix-ui/react-scroll-area@1.2.2, @radix-ui/react-select@2.1.4, @radix-ui/react-separator@1.1.1, @radix-ui/react-slider@1.2.2, @radix-ui/react-slot@1.1.1, @radix-ui/react-switch@1.1.2, @radix-ui/react-tabs@1.1.2, @radix-ui/react-toast@1.2.4, @radix-ui/react-toggle@1.1.1, @radix-ui/react-toggle-group@1.1.1, @radix-ui/react-tooltip@1.1.6, @tailwindcss/postcss@^4.1.9, @types/canvas-confetti@^1.6.4, @types/node@^22, @types/react@^19, @types/react-dom@^19, @vercel/analytics@1.3.1, autoprefixer@^10.4.20, canvas-confetti@^1.9.3, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@1.0.4, date-fns@4.1.0, embla-carousel-react@8.5.1, framer-motion@^11.18.2, input-otp@1.4.1, lucide-react@^0.454.0, next@16.0.10, next-themes@^0.4.6, postcss@^8.5, react@19.2.0, react-day-picker@9.8.0, react-dom@19.2.0, react-hook-form@^7.60.0, react-resizable-panels@^2.1.7, recharts@2.15.4, sonner@^1.7.4, tailwind-merge@^3.3.1, tailwindcss@^4.1.9, tailwindcss-animate@^1.0.7, tw-animate-css@1.3.3, typescript@^5, vaul@^1.1.2, zod@3.25.76 - website/package.json: @hookform/resolvers@^3.10.0, @openai/agents@^0.4.10, @radix-ui/react-accordion@1.2.2, @radix-ui/react-alert-dialog@1.1.4, @radix-ui/react-aspect-ratio@1.1.1, @radix-ui/react-avatar@1.1.2, @radix-ui/react-checkbox@1.1.3, @radix-ui/react-collapsible@1.1.2, @radix-ui/react-context-menu@2.2.4, @radix-ui/react-dialog@1.1.4, @radix-ui/react-dropdown-menu@2.1.4, @radix-ui/react-hover-card@1.1.4, @radix-ui/react-label@2.1.1, @radix-ui/react-menubar@1.1.4, @radix-ui/react-navigation-menu@1.2.3, @radix-ui/react-popover@1.1.4, @radix-ui/react-progress@1.1.1, @radix-ui/react-radio-group@1.2.2, @radix-ui/react-scroll-area@1.2.2, @radix-ui/react-select@2.1.4, @radix-ui/react-separator@1.1.1, @radix-ui/react-slider@1.2.2, @radix-ui/react-slot@1.1.1, @radix-ui/react-switch@1.1.2, @radix-ui/react-tabs@1.1.2, @radix-ui/react-toast@1.2.4, @radix-ui/react-toggle@1.1.1, @radix-ui/react-toggle-group@1.1.1, @radix-ui/react-tooltip@1.1.6, @tailwindcss/postcss@^4.1.9, @types/canvas-confetti@^1.6.4, @types/node@^22, @types/react@^19, @types/react-dom@^19, @vercel/analytics@1.3.1, autoprefixer@^10.4.20, canvas-confetti@^1.9.3, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@1.0.4, date-fns@4.1.0, embla-carousel-react@8.5.1, framer-motion@^11.18.2, input-otp@1.4.1, lucide-react@^0.454.0, next@16.0.10, next-themes@^0.4.6, postcss@^8.5, react@19.2.0, react-day-picker@9.8.0, react-dom@19.2.0, react-hook-form@^7.60.0, react-resizable-panels@^2.1.7, recharts@2.15.4, sonner@^1.7.4, tailwind-merge@^3.3.1, tailwindcss@^4.1.9, tailwindcss-animate@^1.0.7, tw-animate-css@1.3.3, typescript@^5, vaul@^1.1.2, zod@3.25.76 ### Recent commits (newest first) - Adding to recovered - README file - feat: finish music generation - Add GX10 AI coaching integration, CV overlay toggle, and UI updates - Merge branch 'main' of https://github.com/markmusic27/treehacks - Changes to UI flow - done - App pipeline - chore: remove gaston directory - feat: update site - Added app pipeline - MIDI to MP3 - feat: DONE - feat: fix pole logic - feat: fix pole logic - feat: LETS FUCKING GOOOO - Deleted non useful files - feat: connect with website - feat: improve UI - feat: add website ## Key source files (fetched from GitHub, selected and truncated for size) ### website/RUN.md ```markdown # Run the Maestro app Open PowerShell and run these commands **one by one**: ```powershell cd "C:\Users\gba09\OneDrive\Bureau\duolingo-style-webpage" npm install npm run dev ``` Then open in your browser: **http://localhost:3000** --- ### Instructor chat (Pick & Play) — OpenAI / ChatGPT When you **pause** during a session, you can talk to your instructor like a real personal coach. This uses the **OpenAI API** (ChatGPT): 1. Get an API key: [OpenAI API keys](https://platform.openai.com/api-keys) → Create new secret key. 2. In the **project folder**, create or edit **`.env.local`** and add: ``` OPENAI_API_KEY=sk-proj-xxxxx... ``` 3. Save, then **restart** the dev server (Ctrl+C, then `npm run dev`). ### Optional: Learn & Grow (artist/instrument info) — Anthropic For Claude-powered artist and instrument info in Learn & Grow, add to `.env.local`: ``` ANTHROPIC_API_KEY=sk-ant-... ``` **Note:** `.env.local` must be in the project root. Restart the dev server after any change. --- **Why:** `npm install` and `npm run dev` must be run from the folder that contains `package.json` (the project folder). Your prompt was at `C:\Users\gba09\`, so npm looked for `package.json` there and didn’t find it. ``` ### vision/CONTEXT.md ```markdown # Vision -- Air Instrument System ## What This Is A real-time air instrument system that combines **computer vision** (hand tracking + strum detection on a laptop) with a **mobile fretboard app** (multi-touch string controller on an iPhone). The two communicate over **WebSocket** on the local network. On each strum, MIDI notes are generated and played through the laptop speakers via FluidSynth. ## Architecture ``` ┌──────────────────────────────┐ WebSocket (port 8765) ┌────────────────────────┐ │ Python Backend │ <────────────────────────────── │ iOS Fretboard App │ │ (vision/) │ JSON touch + motion data │ (vision/fretboard/) │ │ │ │ │ │ - MediaPipe hand detection │ │ - Multi-touch strings │ │ - Strum detection │ │ - Haptic feedback │ │ - Note engine (MIDI) │ │ - WebSocket client │ │ - Audio engine (FluidSynth) │ │ - CoreMotion sensors │ │ - OpenCV visualization │ │ │ │ - WebSocket server │ │ │ └──────────────────────────────┘ └────────────────────────┘ ``` ## Python Backend **Dependencies:** MediaPipe, OpenCV, NumPy, WebSockets, pyfluidsynth (see `pyproject.toml`) **System dependency:** `brew install fluidsynth` **Run:** `uv run main.py` (prints the local IP to enter on the phone) **What it does:** - Opens the webcam and detects up to 2 hands using MediaPipe's Hand Landmarker (task-based API, `LIVE_STREAM` mode) - Draws a **neck line** between the two wrists (smoothed with exponential moving average) - Detects **strums** when the strum hand's index fingertip crosses the neck line perpendicularly with sufficient velocity - On each strum, combines phone touch data + hand distance + strum velocity to compute MIDI notes, which are played through the laptop speakers via FluidSynth - Runs a **WebSocket server** on port 8765 that receives touch/motion data from the phone - Renders an overlay showing: hand landmarks, neck line (flashes on strum), strum count, last played notes, phone connection status, active touches, FPS ### Note Generation **MIDI note = base_note[string] + fret_semitones(y) + octave(hand_distance) * 12** - **String** (from phone touch): determines the base note (C, D, E, F, G, A for 6 strings) - **Fret position** (touch y-coordinate): adds 0-4 semitones - **Hand distance** (wrist-to-wrist): determines octave (farther apart = lower octave, range 2-6) - **Strum velocity** (crossing speed): determines MIDI velocity (40-127) and note duration (0.2-1.5s) ### Module Structure | Module | Responsibility | |---|---| | `main.py` | Entry point — wires camer [truncated — 3886 more characters] ``` ### pyproject.toml ``` [project] name = "treehacks" version = "0.2.0" description = "Air instrument system — hand tracking + phone fretboard + MIDI audio" readme = "README.md" requires-python = ">=3.12" dependencies = [ "mediapipe", "opencv-python", "numpy", "websockets", "pyfluidsynth", "httpx", ] [project.scripts] treehacks = "src.main:main" treehacks-server = "src.server:main" [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src"] ``` ### JAM_BAND/requirements.txt ``` pygame ``` ### SONG_GENERATION/requirements.txt ``` # Song Generation dependencies # # System dependency: # brew install fluid-synth (for MIDI → WAV rendering) # # Install: # pip install -r requirements.txt pretty_midi ``` ### vision/pyproject.toml ``` [project] name = "vision" version = "0.2.0" description = "Air instrument system — hand tracking + phone fretboard + MIDI audio" readme = "README.md" requires-python = ">=3.12" dependencies = [ "mediapipe", "opencv-python", "numpy", "websockets", "pyfluidsynth", "httpx", ] ``` ### MIDI_TO_SOUNDFONT/requirements.txt ``` # MIDI_TO_SOUNDFONT dependencies # # System dependencies (install first): # macOS: brew install fluid-synth ffmpeg # Linux: sudo apt-get install fluidsynth libfluidsynth-dev ffmpeg # # Install Python dependencies: # pip install -r requirements.txt pyfluidsynth basic-pitch yt-dlp onnxruntime scipy<1.13 demucs torchcodec setuptools<70 resampy<0.4.3 numpy<2 ``` ### app/requirements.txt ``` # Air Guitar App — all dependencies across the pipeline # # Install: pip install -r requirements.txt # Also requires: brew install fluid-synth ffmpeg # ── Vision / CV ────────────────────────────────────────────────────── mediapipe opencv-python numpy websockets # ── Audio synthesis ────────────────────────────────────────────────── pyfluidsynth # ── MIDI processing ────────────────────────────────────────────────── pretty_midi basic-pitch onnxruntime # ── Audio processing ───────────────────────────────────────────────── scipy<1.13 resampy<0.4.3 demucs torchcodec setuptools<70 # ── YouTube ────────────────────────────────────────────────────────── yt-dlp # ── Utilities ──────────────────────────────────────────────────────── numpy<2 ``` ### website/package.json ``` { "name": "my-v0-project", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint ." }, "dependencies": { "@hookform/resolvers": "^3.10.0", "@openai/agents": "^0.4.10", "@radix-ui/react-accordion": "1.2.2", "@radix-ui/react-alert-dialog": "1.1.4", "@radix-ui/react-aspect-ratio": "1.1.1", "@radix-ui/react-avatar": "1.1.2", "@radix-ui/react-checkbox": "1.1.3", "@radix-ui/react-collapsible": "1.1.2", "@radix-ui/react-context-menu": "2.2.4", "@radix-ui/react-dialog": "1.1.4", "@radix-ui/react-dropdown-menu": "2.1.4", "@radix-ui/react-hover-card": "1.1.4", "@radix-ui/react-label": "2.1.1", "@radix-ui/react-menubar": "1.1.4", "@radix-ui/react-navigation-menu": "1.2.3", "@radix-ui/react-popover": "1.1.4", "@radix-ui/react-progress": "1.1.1", "@radix-ui/react-radio-group": "1.2.2", "@radix-ui/react-scroll-area": "1.2.2", "@radix-ui/react-select": "2.1.4", "@radix-ui/react-separator": "1.1.1", "@radix-ui/react-slider": "1.2.2", "@radix-ui/react-slot": "1.1.1", "@radix-ui/react-switch": "1.1.2", "@radix-ui/react-tabs": "1.1.2", "@radix-ui/react-toast": "1.2.4", "@radix-ui/react-toggle": "1.1.1", "@radix-ui/react-toggle-group": "1.1.1", "@radix-ui/react-tooltip": "1.1.6", "@vercel/analytics": "1.3.1", "autoprefixer": "^10.4.20", "canvas-confetti": "^1.9.3", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "1.0.4", "date-fns": "4.1.0", "embla-carousel-react": "8.5.1", "framer-motion": "^11.18.2", "input-otp": "1.4.1", "lucide-react": "^0.454.0", "next": "16.0.10", "next-themes": "^0.4.6", "react": "19.2.0", "react-day-picker": "9.8.0", "react-dom": "19.2.0", "react-hook-form": "^7.60.0", "react-resizable-panels": "^2.1.7", "recharts": "2.15.4", "sonner": "^1.7.4", "tailwind-merge": "^3.3.1", "tailwindcss-animate": "^1.0.7", "vaul": "^1.1.2", "zod": "3.25.76" }, "devDependencies": { "@tailwindcss/postcss": "^4.1.9", "@types/canvas-confetti": "^1.6.4", "@types/node": "^22", "@types/react": "^19", "@types/react-dom": "^19", "postcss": "^8.5", "tailwindcss": "^4.1.9", "tw-animate-css": "1.3.3", "typescript": "^5" } } ``` ### vision/website/package.json ``` { "name": "my-v0-project", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint ." }, "dependencies": { "@hookform/resolvers": "^3.10.0", "@radix-ui/react-accordion": "1.2.2", "@radix-ui/react-alert-dialog": "1.1.4", "@radix-ui/react-aspect-ratio": "1.1.1", "@radix-ui/react-avatar": "1.1.2", "@radix-ui/react-checkbox": "1.1.3", "@radix-ui/react-collapsible": "1.1.2", "@radix-ui/react-context-menu": "2.2.4", "@radix-ui/react-dialog": "1.1.4", "@radix-ui/react-dropdown-menu": "2.1.4", "@radix-ui/react-hover-card": "1.1.4", "@radix-ui/react-label": "2.1.1", "@radix-ui/react-menubar": "1.1.4", "@radix-ui/react-navigation-menu": "1.2.3", "@radix-ui/react-popover": "1.1.4", "@radix-ui/react-progress": "1.1.1", "@radix-ui/react-radio-group": "1.2.2", "@radix-ui/react-scroll-area": "1.2.2", "@radix-ui/react-select": "2.1.4", "@radix-ui/react-separator": "1.1.1", "@radix-ui/react-slider": "1.2.2", "@radix-ui/react-slot": "1.1.1", "@radix-ui/react-switch": "1.1.2", "@radix-ui/react-tabs": "1.1.2", "@radix-ui/react-toast": "1.2.4", "@radix-ui/react-toggle": "1.1.1", "@radix-ui/react-toggle-group": "1.1.1", "@radix-ui/react-tooltip": "1.1.6", "@vercel/analytics": "1.3.1", "autoprefixer": "^10.4.20", "canvas-confetti": "^1.9.3", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "1.0.4", "date-fns": "4.1.0", "embla-carousel-react": "8.5.1", "framer-motion": "^11.18.2", "input-otp": "1.4.1", "lucide-react": "^0.454.0", "next": "16.0.10", "next-themes": "^0.4.6", "react": "19.2.0", "react-day-picker": "9.8.0", "react-dom": "19.2.0", "react-hook-form": "^7.60.0", "react-resizable-panels": "^2.1.7", "recharts": "2.15.4", "sonner": "^1.7.4", "tailwind-merge": "^3.3.1", "tailwindcss-animate": "^1.0.7", "vaul": "^1.1.2", "zod": "3.25.76" }, "devDependencies": { "@tailwindcss/postcss": "^4.1.9", "@types/canvas-confetti": "^1.6.4", "@types/node": "^22", "@types/react": "^19", "@types/react-dom": "^19", "postcss": "^8.5", "tailwindcss": "^4.1.9", "tw-animate-css": "1.3.3", "typescript": "^5" } } ``` [254 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]