# Project export: ChromaChord This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Real-time music chord suggestions based on the elegant theory of 24D chroma vectors, enhanced with track generation features. - Devpost: https://devpost.com/software/chromachord - GitHub: https://github.com/J4Joshua/JASS-APP - Video: https://www.youtube.com/embed/niFOu62-IwU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Treehacks Grand Prize (3rd); [Suno] Best Musical Hack) - Team: 4 GitHub contributor(s) — Lucy Wu (17 commits), joeeliang (14 commits), Roger You (11 commits), J4Joshua (9 commits) ## Devpost submission (written by the team) ### Inspiration The inspiration behind this idea is to have the most extensive overview of music theory, especially in Jazz. Using a physics based approach, we could correctly identify all ~4000 chords in an octave and rank them by dissonance. With the power of LLMs in today's world, it is easier than ever to offload the burden of improvisation to LLMs. However, it was important to us in the beginning of this process to stray away from the LLM models that have little interpretability. This way of constructing the chords and ranking their dissonance from the ground up builds concrete understanding of which chord works well with what. Now, introducing ChromaChord, a solution at the intersection of acoustics, graph theory, and a strong desire to have better tools for learning how to play jazz! ### What it does ChromaChord is a real-time AI jazz assistant that listens to your improvisation and helps you navigate harmony. As you play, it analyzes incoming notes using chroma vector representations, maps harmonic relationships onto a graph of possible chord transitions, and suggests musically coherent next chords. Of course, humans have limitations, and there are only so many paths we can physically traverse. So after the extensive sifting with the chroma vectors, we can use AI to generate harmony, melodies, matching the progression that was played. ### How we built it Part 1: Chroma Vectors MIDI input is captured live from guitar/keyboard. Notes are converted into chroma vectors to represent pitch class content. A graph-based harmonic model encodes possible chord transitions and overtone relationships. Deterministic tools handle chord detection, key inference, and progression suggestions. We did our literature review for this project to convert the theory to code. Here are some papers we read: A multi-level tonal interval space for modelling pitch relatedness and musical consonance Detecting Harmonic Change In Musical Audio A Computational Model of Tonal Tension Profile of Chord Progressions in the Tonal Interval Space Autochord An Efficient Algorithm for the Calculation of a Constant Q Transform Part 2: AI Features We wanted to think creatively about how to incorporate the best AI tools to enhance our product. The Perplexity Sonar API is queried to retrieve popular songs that also follow a similar chord progression. This helps with inspiration and enhancing how you think about future directions. We separated fast local harmonic analysis from slower AI reasoning to maintain responsiveness during live play. ### Challenges we ran into Real-time Translation Problem: Signal processing for instruments is noisy and still an open problem. Solution: Using direct MIDI input from our keyboard Chord ambiguity Problem: Translating chroma vectors into user input. Multiple harmonic interpretations can exist for the same pitch set. Problem: Avoiding random suggestions and diluting our product. Solution: Incorporating the Perplexity Sonar API and Claude Agent SDK in an intentional way. Synchronization ### Accomplishments we're proud of Real-time harmonic graph visualization driven by live chroma data. Generating backing tracks dynamically from user-generated chord progressions. Integrating signal processing and graph theory into one coherent system. Building a system that feels like collaboration, not automation. All of the musical friends we made along the way! ### What we learned Creative AI tools are most compelling when they enhance, not replace, human expression. Harmony is surprisingly graph-like — functional relationships translate naturally into network structures. AI works best as a decision layer on top of deterministic musical analysis. Real-time systems require architectural separation between fast analysis and slower reasoning. ### What's next Personalized skill modeling and long-term learning for music. Support for even more musical instruments, like guitars. Expanded harmonic graph modeling using learned embeddings. Voice-based interaction (“make it darker,” “more tension”) ## README (from the GitHub repository) # JASS-APP Real-time MIDI chord detection and suggestion engine. Play chords on a MIDI piano and get intelligent next-chord suggestions streamed to a web UI via WebSocket. ## Prerequisites - Python 3.10+ - Node.js 18+ - A MIDI controller (optional — the backend can run without one) ## Setup ### Backend ```bash cd backend pip install -r requirements.txt ``` ### Frontend ```bash cd frontend npm install ``` ## Running ### 1. Start the backend ```bash python backend/main.py ``` The WebSocket server starts on `ws://localhost:8000/ws`. To run without a MIDI device: ```bash DISABLE_MIDI=1 python backend/main.py ``` ### 2. Start the frontend ```bash cd frontend npm run dev ``` Opens at `http://localhost:3000`. ## Configuration ### Environment Variables Create a `.env` file in the `backend` directory with the following variables: ```bash # Perplexity API Key (required for song recommendations) PERPLEXITY_API_KEY=your_api_key_here # Spotify API Credentials (optional - for album art and artist info) SPOTIFY_CLIENT_ID=your_client_id_here SPOTIFY_CLIENT_SECRET=your_client_secret_here # Optional: Disable MIDI input DISABLE_MIDI=1 ``` #### Getting Perplexity API Key 1. Visit https://www.perplexity.ai/ 2. Sign up or log in 3. Navigate to your account settings and generate an API key #### Getting Spotify Credentials 1. Visit https://developer.spotify.com/dashboard 2. Log in or create a developer account 3. Create a new application 4. Copy the **Client ID** and **Client Secret** 5. Add them to your `.env` file **Note**: Spotify credentials are optional. Without them, song recommendations will still work but won't display album art or artist information. ## Project Structure ``` backend/ main.py # FastAPI WebSocket server + MIDI capture jass/ # Chord suggestion & tonal tension engine pianomidi/ # MIDI input & chord detection utilities requirements.txt frontend/ # Next.js + React + Tailwind CSS src/app/ ``` ## Detected evidence (automated analysis) Indexed codebase: 39 recognized source files, 260 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (49 of 49) ``` .gitignore backend/.gitignore backend/CLAUDE_SDK_GUIDE.md backend/constants.py backend/jass/__init__.py backend/jass/chord_suggestion.py backend/jass/chroma_index.py backend/jass/tis_index_basic_jazz.npz backend/jass/tis_index.npz backend/jass/tis_index.py backend/jass/tis_metrics.py backend/jass/tonal_tension/__init__.py backend/jass/tonal_tension/dissonance.py backend/jass/tonal_tension/features.py backend/jass/tonal_tension/model.py backend/jass/tonal_tension/theory.py backend/jass/tonal_tension/weights.py backend/main.py backend/modal_app.py backend/modal_integration.py backend/MODAL_README.md backend/pianomidi/consumer.py backend/pianomidi/pianomidi/__init__.py backend/pianomidi/pianomidi/detector.py backend/requirements.txt backend/test.py frontend/.gitignore frontend/eslint.config.mjs frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/src/app/coach/page.tsx frontend/src/app/globals.css frontend/src/app/history/page.tsx frontend/src/app/layout.tsx frontend/src/app/page.tsx frontend/src/components/Background/Background.tsx frontend/src/components/ChordGraph/ChordEdge.tsx frontend/src/components/ChordGraph/ChordGraph.tsx frontend/src/components/ChordGraph/ChordNode.tsx frontend/src/components/ChordGraph/GlowDefs.tsx frontend/src/components/ChordKeyboard.tsx frontend/src/types/chord.ts frontend/src/utils/chordColors.ts frontend/src/utils/chordNotes.ts frontend/src/utils/keyboardToMidi.ts frontend/tsconfig.json README.md ``` ### Dependencies - backend/requirements.txt: aiohappyeyeballs@==2.6.1, aiohttp@==3.13.3, aiosignal@==1.4.0, annotated-doc@==0.0.4, annotated-types@==0.7.0, anyio@==4.12.1, attrs@==25.4.0, cbor2@==5.8.0, certifi@==2026.1.4, claude-code-sdk@>=0.1.0, click@==8.3.1, distro@==1.9.0, frozenlist@==1.8.0, grpclib@==0.4.9, h11@==0.16.0, h2@==4.3.0, hpack@==4.1.0, httpcore@==1.0.9, httpx@==0.28.1, hyperframe@==6.1.0, idna@==3.11, markdown-it-py@==4.0.0, mdurl@==0.1.2, midiutil@>=1.2.1, modal@==1.3.3, multidict@==6.7.1, perplexityai@==0.29.1, propcache@==0.4.1, protobuf@==6.33.5, pydantic@==2.12.5, pydantic_core@==2.41.5, pygame, Pygments@==2.19.2, rich@==14.3.2, shellingham@==1.5.4, sniffio@==1.3.1, spotipy@==2.24.0 - frontend/package.json: @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, babel-plugin-react-compiler@1.0.0, eslint@^9, eslint-config-next@16.1.6, framer-motion@^12.34.0, next@16.1.6, react@19.2.3, react-dom@19.2.3, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - my changes - last - graph - Merge pull request #18 from J4Joshua/feature/weightFix - merging together - hopefully one of our last commits - . - Merge pull request #16 from J4Joshua/suno-button - passing in chords to button to be used for track generation - Merge pull request #15 from J4Joshua/preloaded-chords - Merge branch 'main' into preloaded-chords - . - Merge pull request #14 from J4Joshua/preloaded-chords - . - . - Merge pull request #13 from J4Joshua/song-rec - ignore env files - adding env file - adding env example - Merge branch 'main' into song-rec ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/MODAL_README.md ```markdown # Modal Multi-Turn Chord Coaching This directory contains the Modal deployment for multi-turn chord coaching with state management. ## Quick Start ```bash # 1. Install Modal pip install modal # 2. Authenticate modal token new # 3. Create secrets modal secret create perplexity-api PERPLEXITY_API_KEY=your_key_here modal secret create spotify-api SPOTIFY_CLIENT_ID=xxx SPOTIFY_CLIENT_SECRET=yyy # 4. Test locally modal serve modal_app.py # Server runs at http://localhost:8000 # 5. Deploy to production modal deploy modal_app.py # Outputs live URLs for your endpoints ``` ## Architecture ### State Management Options **Current: In-Memory Dict** (Good for demos/hackathons) - Persists ~10 minutes after last use - No setup required - Resets on container restart **Upgrade: Modal Dict** (Production-ready) ```python from modal import Dict session_dict = Dict.lookup("jass-sessions", create_if_missing=True) ``` **Upgrade: Modal Volume** (Large-scale) ```python from modal import Volume vol = Volume.lookup("jass-data", create_if_missing=True) ``` ## Multi-Turn Flow ``` Turn 1: User plays Dm7 ↓ Agent: "Try guide tones (3rd and 7th)" Suggestions: [G7, A7, Fmaj7] Turn 2: User plays G7 (follows ii-V!) ↓ Agent: "Nice ii-V! Try tritone sub Db7 next time" Difficulty: 1 → 2 Suggestions: [Cmaj7, Db7, Em7] Turn 3: User plays Db7 (tritone sub!) ↓ Agent: "Excellent! Keep E-B guide tones, resolve to Cmaj9" Difficulty: 2 → 3 Pattern Detected: "tritone substitution" ``` ## API Endpoints ### POST `/process_turn` Process a chord and get coaching feedback. **Request:** ```json { "session_id": "uuid-string", "chord_name": "Dm7", "chroma": [0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0], "key": "C" } ``` **Response:** ```json { "suggestions": [ {"name": "G7", "notes": ["G", "B", "D", "F"], "tension": 0.65} ], "exercise": "Try guide tones...", "feedback": "Great! You played a ii-V progression.", "pattern_detected": "ii-V progression", "difficulty": 2, "turn_number": 3 } ``` ### GET `/get_session?session_id=xxx` Get session history and state. ### POST `/reset_session` Reset a session. ### POST `/recommend_songs_async` Async song recommendation (handles high load). ## Integration with FastAPI Add to your `backend/main.py`: ```python from modal_integration import ModalCoachClient # Initialize modal_coach = ModalCoachClient(os.environ["MODAL_ENDPOINT_URL"]) @app.post("/coach-turn") async def coach_turn(data: dict): return await modal_coach.process_turn( session_id=data["session_id"], chord_name=data["chord_name"], chroma=data["chroma"] ) ``` Add to `.env`: ```bash MODAL_ENDPOINT_URL=https://yourname--jass-chord-coach-process-turn.modal.run ``` ## Frontend Integration ```typescript // Start coaching session const sessionId = crypto.randomUUID(); // When chord detected fetch("http://localhost:8000/coach-turn", { method: "POST", body: JSON.stringify({ session_id: sessionId, chord_n [truncated — 1085 more characters] ``` ### backend/CLAUDE_SDK_GUIDE.md ```markdown # Claude SDK Agents for JASS-APP Use Claude SDK agents to automate code review, testing, and documentation tasks. ## Setup ```bash # Install Claude SDK pip install anthropic # Set API key export ANTHROPIC_API_KEY=your_api_key_here ``` ## Example Agents ### 1. Code Review Agent Review chord detection logic, API security, and TIS index performance. ```python from anthropic import Anthropic import os client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"]) def review_jass_codebase(): """Review JASS-APP codebase for improvements.""" # Read code files with open("jass/chord_suggestion.py") as f: chord_logic = f.read() with open("main.py") as f: api_code = f.read() # Create review prompt message = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=4000, messages=[{ "role": "user", "content": f"""Review this chord detection system for: 1. Performance optimizations (TIS index queries) 2. Security issues (API endpoints, WebSocket) 3. Music theory accuracy (chord progressions) 4. Code quality (naming, structure) CHORD SUGGESTION CODE: {chord_logic} API CODE: {api_code} Provide specific, actionable recommendations.""" }] ) return message.content[0].text # Run review if __name__ == "__main__": review = review_jass_codebase() print(review) # Save to file with open("code_review.md", "w") as f: f.write(review) ``` ### 2. Test Generation Agent Generate Jest/Pytest tests for critical paths. ```python def generate_tests_for_modal(): """Generate tests for Modal deployment.""" with open("modal_app.py") as f: modal_code = f.read() message = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=3000, messages=[{ "role": "user", "content": f"""Generate pytest tests for this Modal app: {modal_code} Focus on: 1. SessionState management (history, difficulty) 2. Pattern detection (ii-V, tritone subs) 3. Difficulty escalation logic 4. Edge cases (empty sessions, invalid chords) Return complete pytest code with fixtures.""" }] ) test_code = message.content[0].text # Save test file with open("test_modal_app.py", "w") as f: f.write(test_code) print("✓ Generated test_modal_app.py") return test_code ``` ### 3. Documentation Agent Auto-generate API docs and setup guides. ```python def generate_api_docs(): """Generate OpenAPI-style documentation.""" with open("main.py") as f: api_code = f.read() message = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=3000, messages=[{ "role": "user", "content": f"""Generate OpenAPI 3.0 documentation for these FastAPI endpoints: {api_code} Include: 1. Request/response schemas with examples 2. WebSocket protoco [truncated — 5185 more characters] ``` ### backend/requirements.txt ``` aiohappyeyeballs==2.6.1 aiohttp==3.13.3 aiosignal==1.4.0 annotated-doc==0.0.4 annotated-types==0.7.0 anyio==4.12.1 attrs==25.4.0 cbor2==5.8.0 certifi==2026.1.4 click==8.3.1 distro==1.9.0 frozenlist==1.8.0 grpclib==0.4.9 h11==0.16.0 h2==4.3.0 hpack==4.1.0 httpcore==1.0.9 httpx==0.28.1 hyperframe==6.1.0 idna==3.11 markdown-it-py==4.0.0 mdurl==0.1.2 modal==1.3.3 multidict==6.7.1 perplexityai==0.29.1 propcache==0.4.1 protobuf==6.33.5 pydantic==2.12.5 pydantic_core==2.41.5 Pygments==2.19.2 rich==14.3.2 shellingham==1.5.4 sniffio==1.3.1 spotipy==2.24.0 claude-code-sdk>=0.1.0 midiutil>=1.2.1 pygame ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --webpack", "build": "next build --webpack", "start": "next start", "lint": "eslint" }, "dependencies": { "framer-motion": "^12.34.0", "next": "16.1.6", "react": "19.2.3", "react-dom": "19.2.3" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "babel-plugin-react-compiler": "1.0.0", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### backend/main.py ```python #!/usr/bin/env python3 """Minimal MIDI → WebSocket streaming server. MIDI piano → chord detection → JSON over WebSocket to frontend. Run: python backend/pianomidi/ws_server.py """ import asyncio import base64 import io import json import os import re import sys import tempfile import threading import time import uuid from dataclasses import dataclass, asdict from pathlib import Path from fastapi import FastAPI, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel import uvicorn from perplexity import Perplexity from dotenv import load_dotenv try: from claude_code_sdk import query, ClaudeCodeOptions CLAUDE_SDK_AVAILABLE = True except ImportError: CLAUDE_SDK_AVAILABLE = False try: from midiutil import MIDIFile MIDIUTIL_AVAILABLE = True except ImportError: MIDIUTIL_AVAILABLE = False try: import pygame PYGAME_AVAILABLE = True except ImportError: PYGAME_AVAILABLE = False load_dotenv() # Ensure backend/ is on sys.path so jass package resolves _backend_dir = str(Path(__file__).resolve().parent.parent) if _backend_dir not in sys.path: sys.path.insert(0, _backend_dir) from jass.tis_index import TISIndex from jass.chord_suggestion import suggest_chords from jass.tonal_tension.weights import PAPER_WEIGHTS_TABLE1, PAPER_WEIGHTS_TABLE2 from jass.tonal_tension.theory import roman_to_chord from constants import CURATED_SERIES app = FastAPI() # --- CORS Configuration --- app.add_middleware( CORSMiddleware, allow_origins=["*"], # Allow all origins for development allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # --- Graph Data Structures --- @dataclass class Node: """Represents a chord node in the graph.""" uuid: str name: str depth: int chroma: list[int] @dataclass class Relation: """Represents a directed edge from one chord to another.""" uuid: str source_node: str target_node: str NOTE_NAMES = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"] # --- Global state --- clients: set[WebSocket] = set() loop: asyncio.AbstractEventLoop | None = None tis_idx: TISIndex | None = None queue: asyncio.Queue[frozenset[int]] = asyncio.Queue() spotify_client = None # Graph tracking state last_chroma: tuple[int, ...] | None = None graph_depth: int = 0 nodes: list[Node] = [] relations: list[Relation] = [] allowed_suggestions: set[tuple[int, ...]] = set() # Curated series state series_cursor: int = 0 resolved_series: list[dict] | None = None current_difficulty: str = "easy" DIFFICULTY_WEIGHTS = { "easy": dict(PAPER_WEIGHTS_TABLE1), "hard": dict(PAPER_WEIGHTS_TABLE2), } # --- Stage 1: MIDI capture (rtmidi thread) --- class MidiCapture: """Captures MIDI note-on/off, debounces, and pushes snapshots to an asyncio queue.""" def __init__(self, q: asyncio.Queue, event_loop: asyncio.AbstractEventLoop): self.held: set[int] = set() self.q = q self.loop = event_loop self._timer: threading.Timer | None = None self._lock = threading.Lock() def callback(self, event, _data=None): message, _ = event status = message[0] & 0xF0 note = message[1] velocity = message[2] if len(message) > 2 else 0 if status == 0x90 and velocity > 0: self.held.add(note) elif status == 0x80 or (status == 0x90 and velocity == 0): self.held.discard(note) else: return self._debounce() def _debounce(self, delay: float = 0.03): with self._lock: if self._timer: self._timer.cancel() self._timer = threading.Timer(delay, self._emit) self._timer.daemon = True self._timer.start() def _emit(self): snapshot = frozenset(self.held) self.loop.call_soon_threadsafe(self.q.put_nowait, snapshot) # --- Stage 2: chord detection + suggestion worker (async) --- async def chord_worker(): """Reads note snapshots from the queue, detects chords, broadcasts.""" global last_chroma, graph_depth, nodes, relations, allowed_suggestions, series_cursor from pychord.analyzer import find_chords_from_notes while True: try: snapshot = await queue.get() # drain to latest — skip stale intermediate states while not queue.empty(): snapshot = queue.get_nowait() # build chroma + note names pitch_classes = {n % 12 for n in snapshot} chroma = [1 if i in pitch_classes else 0 for i in range(12)] chroma_key = tuple(chroma) names = [NOTE_NAMES[pc] for pc in sorted(pitch_classes)] chord_name: str | None = None if len(names) >= 2: chords = find_chords_from_notes(names) if chords: chord_name = str(chords[0]) if chord_name is None: chord_name = "-".join(names) if names else "?" print(f"[chord_worker] detected: {chord_name!r} notes: {names} chroma={list(chroma_key)}", file=sys.stderr) # Gate: if we have suggestions, only accept chords whose chroma matches if allowed_suggestions and chroma_key not in allowed_suggestions: print(f"[chord_worker] REJECTED {chord_name!r} played={list(chroma_key)} allowed={[list(c) for c in allowed_suggestions]}", file=sys.stderr) continue print(f"[chord_worker] ACCEPTED {chord_name!r} chroma={list(chroma_key)}", file=sys.stderr) # Advance curated series cursor if the played chord matches if resolved_series and list(chroma_key) == resolved_series[series_cursor]["chroma"]: print(f"[chord_worker] series match: {resolved_series[series_cursor]['name']}, advancing cursor", file=sys.stderr) series_cursor = (series_cursor + 1) % len(resolved_series) # Get suggestions for accepted chord weights = DIFFICULTY_WEIGHTS[current_difficult [truncated — 26743 more characters] ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono, Patrick_Hand } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); const patrickHand = Patrick_Hand({ variable: "--font-patrick-hand", subsets: ["latin"], weight: ["400"], }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### frontend/src/app/page.tsx ```typescript "use client"; import { useCallback, useEffect, useRef, useState } from "react"; import Link from "next/link"; import { Background } from "@/components/Background/Background"; import { ChordGraph } from "@/components/ChordGraph/ChordGraph"; import type { ChordGraphState, HistorySeed } from "@/types/chord"; import { KEY_TO_MIDI, midiNotesToNames } from "@/utils/keyboardToMidi"; const HISTORY_SEED_KEY = "historySeed"; type ChordMsg = { type: "chord"; chord: { name: string | null; notes: string[]; chroma: number[] }; suggestions: { name: string; notes: string[]; chroma: number[]; tension: number }[]; }; export default function Home() { const [status, setStatus] = useState<"disconnected" | "connecting" | "connected">("disconnected"); const [historyBannerChord, setHistoryBannerChord] = useState(null); const [chord, setChord] = useState(null); const [lastChord, setLastChord] = useState(null); const [suggestions, setSuggestions] = useState([]); const [lastSuggestions, setLastSuggestions] = useState([]); const [log, setLog] = useState([]); const [difficulty, setDifficulty] = useState<"easy" | "hard">("easy"); const wsRef = useRef(null); const logRef = useRef(null); const playAreaRef = useRef(null); const [keyboardHeldNotes, setKeyboardHeldNotes] = useState>(new Set()); /** Live notes from physical piano (backend MIDI) - for real-time visual feedback */ const [livePianoNotes, setLivePianoNotes] = useState(null); const liveNotesThrottleRef = useRef<{ timer: ReturnType | null; pending: string[] | null }>({ timer: null, pending: null, }); // Live chord graph state — starts empty until user plays a chord or clicks Demo const [chordGraphState, setChordGraphState] = useState(null); const [notesMap, setNotesMap] = useState>({}); const prevChordGraphStateRef = useRef(null); useEffect(() => { prevChordGraphStateRef.current = chordGraphState; }); // Hydrate from history seed when navigating from History page ("play from here") useEffect(() => { try { const raw = sessionStorage.getItem(HISTORY_SEED_KEY); if (!raw) return; const seed: HistorySeed = JSON.parse(raw); sessionStorage.removeItem(HISTORY_SEED_KEY); setChordGraphState({ current: seed.current, previous: seed.previous ?? [], next: seed.next, }); setNotesMap(seed.notesMap); setHistoryBannerChord(seed.current.chordId); demoModeRef.current = false; // Allow WebSocket to take over when user plays const t = setTimeout(() => setHistoryBannerChord(null), 3000); return () => clearTimeout(t); } catch { // Ignore parse errors or missing key } }, []); // Transform WebSocket data to ChordGraphState format function transformToChordGraphState(chordMsg: ChordMsg): ChordGraphState | null { // If no chord name, return null (no visualization) if (!chordMsg.chord.name) { return null; } // Create current chord node const current = { id: `current-${Date.now()}`, chordId: chordMsg.chord.name, }; // Transform suggestions to next nodes // Convert tension to probability: lower tension = higher probability const maxTension = Math.max(...chordMsg.suggestions.map(s => s.tension), 0.01); const next = chordMsg.suggestions.slice(0, 3).map((suggestion, i) => ({ id: `next-${i}-${Date.now()}`, chordId: suggestion.name, // Invert tension: lower tension = better = higher probability probability: 1 - (suggestion.tension / maxTension), })); return { current, previous: [], // Not using previous for now next, }; } // Create notes map from WebSocket data function createNotesMap(chordMsg: ChordMsg): Record { const notesMap: Record = {}; // Add current chord notes if (chordMsg.chord.name) { notesMap[chordMsg.chord.name] = chordMsg.chord.notes; } // Add suggestion notes chordMsg.suggestions.forEach(suggestion => { notesMap[suggestion.name] = suggestion.notes; }); return notesMap; } function sendDifficulty(diff: "easy" | "hard") { setDifficulty(diff); if (wsRef.current?.readyState === WebSocket.OPEN) { wsRef.current.send(JSON.stringify({ type: "set_difficulty", difficulty: diff })); } } function addLog(msg: string) { setLog((prev) => [...prev.slice(-99), `${new Date().toLocaleTimeString()} ${msg}`]); } function connect() { if (wsRef.current) { wsRef.current.close(); } setStatus("connecting"); addLog("Connecting to ws://localhost:8000/ws ..."); const ws = new WebSocket("ws://localhost:8000/ws"); wsRef.current = ws; ws.onopen = () => { setStatus("connected"); demoModeRef.current = false; // Switch to live mode when connected addLog("Connected!"); console.log("✅ WebSocket CONNECTED"); ws.send(JSON.stringify({ type: "set_difficulty", difficulty })); }; ws.onmessage = (e) => { console.log("📨 Raw WebSocket message received:", e.data); try { const data = JSON.parse(e.data) as ChordMsg | { type: "live_notes"; notes: string[] }; console.log("📦 Parsed data:", data); if (data.type === "live_notes") { const notes = data.notes as string[]; const throttle = liveNotesThrottleRef.current; throttle.pending = notes; if (throttle.timer === null) { setLivePianoNotes(notes); throttle.timer = setTimeout(() => { if (throttle.pending !== null) { setLivePianoNotes(throttle.pending); th [truncated — 17816 more characters] ``` ### frontend/src/app/coach/page.tsx ```typescript "use client"; import { useCallback, useEffect, useRef, useState } from "react"; import Link from "next/link"; import { Background } from "@/components/Background/Background"; type Message = { id: string; type: "user" | "coach"; content: string; timestamp: number; chord?: string; suggestions?: Array<{ name: string; tension: number }>; pattern?: string; difficulty?: number; }; export default function Coach() { const [sessionId, setSessionId] = useState(""); const [messages, setMessages] = useState([]); const [isLoading, setIsLoading] = useState(false); const [inputChord, setInputChord] = useState(""); const [currentDifficulty, setCurrentDifficulty] = useState(1); const [error, setError] = useState(null); const messagesEndRef = useRef(null); // Initialize session ID useEffect(() => { const newSessionId = crypto.randomUUID(); setSessionId(newSessionId); // Add welcome message setMessages([ { id: "welcome", type: "coach", content: "👋 Welcome to the AI Coaching Session! I'll help you master chord progressions and voice leading.", timestamp: Date.now(), }, { id: "start", type: "coach", content: "Try playing a chord to get started. Type the chord name (e.g., 'Dm7') or press keys on your keyboard.", timestamp: Date.now() + 100, }, ]); }, []); // Scroll to latest message useEffect(() => { messagesEndRef.current?.scrollIntoView({ behavior: "smooth" }); }, [messages]); const sendChord = useCallback(async (chordName: string) => { if (!chordName.trim() || !sessionId) return; setError(null); setIsLoading(true); // Add user message const userMsg: Message = { id: `user-${Date.now()}`, type: "user", content: chordName, timestamp: Date.now(), chord: chordName, }; setMessages((prev) => [...prev, userMsg]); setInputChord(""); try { // Get modal endpoint from env or use default const modalUrl = process.env.NEXT_PUBLIC_MODAL_ENDPOINT || "http://localhost:8000"; const response = await fetch(`${modalUrl}/process_turn`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ session_id: sessionId, chord_name: chordName, chroma: new Array(12).fill(0), // Placeholder: could parse actual chord key: "C", }), }); if (!response.ok) { throw new Error(`Coach error: ${response.statusText}`); } const data = await response.json(); // Add coach response const coachMsg: Message = { id: `coach-${Date.now()}`, type: "coach", content: data.exercise || data.feedback || "Great chord choice!", timestamp: Date.now(), pattern: data.pattern_detected, suggestions: data.suggestions?.slice(0, 3).map((s: any) => ({ name: s.name, tension: s.tension, })), difficulty: data.difficulty, }; setMessages((prev) => [...prev, coachMsg]); setCurrentDifficulty(data.difficulty || 1); } catch (err) { setError( err instanceof Error ? err.message : "Connection error. Make sure Modal is running: `modal serve backend/modal_app.py`" ); } finally { setIsLoading(false); } }, [sessionId]); const handleSubmit = (e: React.FormEvent) => { e.preventDefault(); if (inputChord.trim()) { sendChord(inputChord); } }; const handleKeyDown = (e: React.KeyboardEvent) => { if (e.key === "Enter" && !e.shiftKey) { e.preventDefault(); handleSubmit(e as any); } }; const resetSession = async () => { try { const modalUrl = process.env.NEXT_PUBLIC_MODAL_ENDPOINT || "http://localhost:8000"; await fetch(`${modalUrl}/reset_session`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ session_id: sessionId }), }); const newSessionId = crypto.randomUUID(); setSessionId(newSessionId); setMessages([]); setCurrentDifficulty(1); setError(null); } catch (err) { setError("Failed to reset session"); } }; return (
{/* Header */}

♪ Chord Coaching

← Play History →
{/* Main Content */}
{/* Messages Container */} [truncated — 7785 more characters] ``` ### frontend/src/app/history/page.tsx ```typescript "use client"; import { useEffect, useRef, useState } from "react"; import Link from "next/link"; import { useRouter } from "next/navigation"; import { Background } from "@/components/Background/Background"; import type { HistorySeed } from "@/types/chord"; const NOTE_NAMES = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]; const HISTORY_SEED_KEY = "historySeed"; function chromaToNotes(chroma: number[]): string[] { if (!chroma || chroma.length !== 12) return []; return NOTE_NAMES.filter((_, i) => chroma[i] === 1); } function chromaKey(chroma: number[]): string { if (!chroma || chroma.length !== 12) return ""; return chroma.join(","); } type Node = { uuid: string; name: string; depth: number; chroma: number[]; }; type Relation = { uuid: string; source_node: string; target_node: string; }; type Session = { session_number: number; filename: string; timestamp: string; total_depth: number; node_count: number; }; type SessionData = { timestamp: string; nodes: Node[]; relations: Relation[]; total_depth: number; }; type ChordEvent = { timestamp: number; chord: { name: string | null; notes: string[]; chroma: number[] }; suggestions: { name: string; notes: string[]; chroma: number[]; tension: number }[]; }; type ChordMsg = { type: "chord"; chord: { name: string | null; notes: string[]; chroma: number[] }; suggestions: { name: string; notes: string[]; chroma: number[]; tension: number }[]; }; type Song = { title: string; artists: string; album: string; image_url: string | null; spotify_url: string | null; original_query: string; }; export default function History() { const router = useRouter(); const [status, setStatus] = useState<"disconnected" | "connecting" | "connected">("disconnected"); const [chordHistory, setChordHistory] = useState([]); const [isRecording, setIsRecording] = useState(false); const [sessions, setSessions] = useState([]); const [selectedSession, setSelectedSession] = useState(null); const [recommendedSongs, setRecommendedSongs] = useState([]); const [isLoadingSongs, setIsLoadingSongs] = useState(false); const [songsError, setSongsError] = useState(null); const [chordProgression, setChordProgression] = useState(null); const [isGeneratingTrack, setIsGeneratingTrack] = useState(false); const [generateTrackError, setGenerateTrackError] = useState(null); const [generateTrackSuccess, setGenerateTrackSuccess] = useState(null); const [isCreatingSong, setIsCreatingSong] = useState(false); const [songMidiBase64, setSongMidiBase64] = useState(null); const [createSongError, setCreateSongError] = useState(null); const wsRef = useRef(null); useEffect(() => { fetchSessions(); }, []); async function fetchSessions() { try { const response = await fetch("http://localhost:8000/sessions"); const data = await response.json(); setSessions(data.sessions || []); } catch (err) { console.error("Failed to fetch sessions:", err); } } async function loadSession(filename: string) { try { const response = await fetch(`http://localhost:8000/sessions/${filename}`); const data: SessionData = await response.json(); setSelectedSession(data); setRecommendedSongs([]); setSongsError(null); setChordProgression(null); setSongMidiBase64(null); setCreateSongError(null); } catch (err) { console.error("Failed to load session:", err); } } async function fetchRecommendedSongs() { if (!selectedSession) return; // Find the filename from the selected session timestamp const sessionFile = sessions.find(s => s.timestamp === selectedSession.timestamp); if (!sessionFile) { setSongsError("Could not find session file"); return; } setIsLoadingSongs(true); setSongsError(null); setRecommendedSongs([]); try { const response = await fetch(`http://localhost:8000/recommend-songs/${sessionFile.filename}`); const data = await response.json(); if (data.error) { setSongsError(data.error); } else { setRecommendedSongs(data.songs || []); setChordProgression(data.chord_progression); } } catch (err) { setSongsError(err instanceof Error ? err.message : "Failed to fetch recommendations"); console.error("Failed to fetch recommendations:", err); } finally { setIsLoadingSongs(false); } } function getPlayedChordNames(session: SessionData): string[] { const sourceNodeIds = new Set(session.relations.map((rel) => rel.source_node)); const playedNodes = session.nodes.filter((node) => sourceNodeIds.has(node.uuid)); playedNodes.sort((a, b) => a.depth - b.depth); return playedNodes.map((node) => node.name).filter((name) => name); } async function generateTrack() { if (!selectedSession) return; const chordNames = getPlayedChordNames(selectedSession); if (chordNames.length === 0) { setGenerateTrackError("No chords found for this session."); return; } const endpoint = process.env.NEXT_PUBLIC_TRACK_API_URL || "http://localhost:8000/generate-track"; setIsGeneratingTrack(true); setGenerateTrackError(null); setGenerateTrackSuccess(null); try { const response = await fetch(endpoint, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ session_timestamp: selectedSession.timestamp, chords: chordNames, }), }); if (!response.ok) { const errText = await response.text(); throw new Error(errText || "Failed to generate track"); } const data = await response.json().catch(() => null); setGenerateTrackSuccess(data [truncated — 26220 more characters] ``` ### frontend/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ reactCompiler: true, }; export default nextConfig; ``` [27 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]