# Project export: Tactus This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: ROCKSMITH for the auditorily impaired - Devpost: https://devpost.com/software/tactus-z9a8w6 - GitHub: https://github.com/wilyan09007/Tactus - Video: https://www.youtube.com/embed/_cQWa3X3Wtk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Aritro Bhattacharjee (39 commits), Claude Opus 4.8 (31 commits), Aditya Singh (19 commits), William (1 commits) ## Devpost submission (written by the team) ### Inspiration 🎸 Our teammate Aditya loves DJing and playing guitar, and wants to help bring the language of music to the world 🎶. That led us to a bigger question: what if learning an instrument did not depend on hearing at all? Instead of treating accessibility as an add-on, we wanted to build a system where touch, sight, and AI work together as the primary teaching interface. We were also excited by the technical challenge. Guitar playing is hard to interpret from a camera and a mic alone: fingers occlude the fretboard, buzz does not directly tell you its cause, and useful feedback has to be immediate and specific. That made Tactus the kind of hackathon project we love: meaningful, ambitious, hardware-heavy, and legitimately difficult. ### What it does 🎯 Tactus is a Deaf-accessible guitar coaching system. It watches your fretting hand with a webcam, listens to your guitar with a mic, fuses those signals to infer what note you played, where you played it, which finger you used, and how cleanly you fretted it, then turns that into one precise correction you can feel on your body and see on screen. It has two core modes: LEARN 🎓: feel the target song on a 12-channel haptic wearable, play it on a real guitar, and get corrective feedback when you make a mistake. PLAY 🎵: get real-time tactile guidance for pitch, timing, and correctness while playing freely. The feedback loop is the key idea. When a player makes a mistake, Tactus can replay the correct haptic target, compare the player's attempt against the reference, highlight the relevant finger, and present a simple physical correction instead of vague advice. The goal is to recreate the "hear it, fix it" loop as a "feel it, see it, fix it" loop ✨. Redis is an important part of that experience. We use Redis as the local-first memory layer for practice: storing analyzed song references, rendered haptic patterns, target fingering, calibration, and user progress so practice sessions can run offline 📦. That matters because a coaching device should still work in a basement, rehearsal room, or hackathon venue with unreliable internet. Redis also gives us a path toward retrieving similar past mistakes so the system can say, in effect, "this is like the buzz you made last time," which opens the door to adaptive coaching rather than one-off corrections. ### How we built it 🛠️ We split the system into two halves running on a single laptop 💻. The browser side handles camera capture, the visual interface, and hand/fretboard understanding using MediaPipe Hands and geometry-based fretboard mapping 👁️. The Python side handles mic input, pitch tracking, feature extraction, fusion logic, and 12-channel haptic rendering 🎛️. Those two halves communicate over a localhost WebSocket 🔌. That let us develop the visual system and the audio/haptics engine in parallel while keeping a clean contract between them. On the hardware side, we built a 12-channel wearable haptics rig using USB multichannel audio adapters, class-D amp boards, and de-housed speaker drivers mounted on the body 🦺. Six channels map to the strings across the back, and six channels map to fret zones across the torso. Instead of generating an abstract score, we render music and corrections directly into vibration patterns. On the perception side, we combined: audio analysis for pitch, onset/offset timing, and note quality cues 🎧 vision features from the fretting hand and fretboard geometry 👋 fusion logic that reconciles what the camera sees with what the mic hears 🧠 an LLM-based coaching layer for phrase-level natural-language feedback 💬 We also designed Redis into the architecture as more than a cache. Redis acts as Tactus's memory. It stores the offline song library, user calibration, progress, and historical error context. In our design, Redis also supports nearest-neighbor retrieval on feature vectors so we can surface similar past mistakes and make coaching more personalized over time. That gave us a strong sponsor fit, but more importantly, it fit the product honestly: a wearable practice coach should remember your songs, your body, and your progress. ### Challenges we ran into 🚧 The biggest challenge was that almost every interesting part of the project sits in an awkward real-world boundary. Hardware was one 🔩. Building a wearable 12-channel haptics system quickly meant dealing with amplifier wiring, body coupling, actuator placement, power constraints, and the fact that something can be audible, visible, and physically underwhelming at the same time. Making vibrations feel distinct on the body is a much harder problem than simply making something vibrate. Perception was another 👀. Guitar technique is hard to read from raw signals. A bad note, weak pressure, poor placement, and bad timing can overlap, and the camera's view of the fretboard is often partially occluded by the player's own hand. We had to be disciplined about putting AI only where the problem is genuinely unsolved and keeping the rest of the system deterministic and measurable. Latency and architecture were also real constraints ⏱️. Immediate haptic feedback needs to feel responsive, while richer fused corrections arrive a bit later. That forced us to separate fast feedback from deeper coaching instead of pretending one model could do everything instantly. Finally, Redis introduced an honesty challenge in the best way: if we were going to talk about memory, it needed to be a real product function. That pushed us to think carefully about what must persist locally for a practice session to work offline and what kinds of retrieval would actually help a player improve, rather than adding a database just to satisfy a sponsor category. ### Accomplishments we're proud of 🏆 We are proud that Tactus is not just an accessibility pitch deck. It is a coherent system with a real hardware rig, a grounded software architecture, and a clear theory for how tactile coaching can work 💡. We are proud of the product framing: instead of claiming to "make Deaf people hear," we built around a more respectful and rigorous idea, creating a new sensory pathway for learning music through touch and vision. We are proud of the technical split between solved and unsolved problems. The system uses ML and multimodal fusion where it earns its keep, then hands off to deterministic haptic rendering so the output remains inspectable and honest. We are also proud of the Redis story. We did not treat Redis as a generic cache. We treated it as the memory of the coach: the offline library, the calibration store, the progress tracker, and eventually the retrieval layer for similar mistakes and adaptive difficulty. That made the architecture stronger and the demo more believable. ### What we learned 📚 We learned that accessibility projects get much better when they are designed around a user's actual learning loop instead of around a flashy output. The most important insight was not "how do we visualize music," but "how do we let someone notice and correct a mistake without relying on hearing?" We learned that haptics is unforgiving. Frequency choice, actuator coupling, body placement, timing, and intensity all matter, and small physical design changes have huge perceptual effects. We learned that multimodal AI becomes much more credible when each modality has a clear job. Vision is good at telling us where the hand is and which finger is involved. Audio is good at telling us what actually sounded and how cleanly it sounded. Fusing them gives us a better correction than either alone. We also learned that Redis is especially compelling in embodied systems. Memory is not just a backend feature; it changes the user experience. When a system remembers your calibration, your songs, and your recurring mistakes, it starts to feel like a real coach instead of a stateless demo 🧠. ### What's next 🚀 We want to expand the offline library, support more songs cleanly, improve the wearable form factor, and test with more real players to further develop our teaching and feedback methodology 🎸. Longer term, we see Tactus becoming a broader platform for accessible instrument learning: still rigorous, still tactile-first, and still grounded in the idea that the language of music should be available to more people. ## README (from the GitHub repository) # 🎸 Tactus **A Deaf-accessible guitar-coaching wearable.** From a webcam + a mic, Tactus recovers *what you played, where, with which finger, and how cleanly you fretted it* — by fusing vision and audio through a learned physical model — then gives you **one precise correction you feel on your body and see on your hand.** > Built for the **UC Berkeley AI Hackathon 2026 (June 20–22)**. Team of 4. Working name **Tactus** (*touch* + the conductor's *pulse*). > > 📌 **Canonical source of truth: [`truth.md`](truth.md).** If any doc disagrees with it, `truth.md` wins. --- ## The 30-second version - **Problem.** Learning guitar is a hear-and-correct loop: play → hear the buzz / wrong note → fix. Deaf and hard-of-hearing players are locked out of that loop (~466M people have disabling hearing loss, WHO). - **Tactus — two modes on one engine:** - **LEARN** — "Rocksmith for Deaf players." Feel the target song on your body, play it on a real guitar, and on any mistake the system rewinds, replays the correct haptics vs. yours, and shows the physical fix — Claude vision highlights the wrong finger. - **PLAY** — pick up the guitar and get real-time "flat / sharp / in tune / on the beat" by feel. - **Where the AI is.** The hard, unsolved part is reading an *occluded* fretting hand and a *cause-blind* buzz from a webcam + mic. The ML lives exactly there; everything downstream is a deterministic, measurable signal→vibration transform (no AI-invented "score"). That split is the rigor. - **The one correction.** Three axes: **wrong note · wrong duration · pressure** (too-light vs good; "too hard" surfaces as a sharp/choked pitch fault). Each is a measured signal, cross-checked across vision + audio. ## The rig (as-built — full detail in [`truth.md`](truth.md)) ``` LAPTOP (M4 Pro) ──USB──► 2× Vantec NBA-200U (USB→8 ch; V3 spare) └► 6× SK473 PAM8403 stereo amp boards (gutted) └► 12× KHD 3 Ω/5 W drivers (de-housed) on the body = 12 channels: 6 back (strings) + 6 torso (fret-zones) ``` - **Software split (locked):** the **browser** owns camera + MediaPipe + ArUco + AR + viz; a **Python** process owns mic + F0 + the fusion model + 12-ch haptic output; they sync over one localhost WebSocket (vision features flow browser→Python, timestamped). - **Power:** wall (10-port hub) for the judged run, or cordless (Anker 737). Only ~2–3 channels fire at once (sequential strum sweep), so real peak is ~2–3 W. ## Docs map | Doc | What's in it | |---|---| | **[`truth.md`](truth.md)** | ⭐ The canonical source of truth — direction + all hardware (parts, dimensions, cost, use) | | [`docs/00-start-here.md`](docs/00-start-here.md) | On-ramp + build-tonight TL;DR | | [`docs/01-bill-of-materials.md`](docs/01-bill-of-materials.md) | The as-purchased BOM (SKUs, qty, spares) | | [`docs/02-system-architecture.md`](docs/02-system-architecture.md) · [`03-power.md`](docs/03-power.md) | System chain + the chord-safety power math | | [`docs/04-soldering-guide.md`](docs/04-soldering-guide.md) · [`05-wiring-map.md`](docs/05-wiring-map.md) · [`06-safety.md`](docs/06-safety.md) · [`09-assembly-checklist.md`](docs/09-assembly-checklist.md) | Build: solder order, channel-by-channel wiring, safety, assembly gates | | [`docs/07-haptic-encoding.md`](docs/07-haptic-encoding.md) · [`18-tuning-and-calibration.md`](docs/18-tuning-and-calibration.md) · [`21-chord-and-sustain-rendering.md`](docs/21-chord-and-sustain-rendering.md) | How notes / chords / sustain become vibration + on-body tuning | | [`docs/08-software-architecture.md`](docs/08-software-architecture.md) · [`13_LEARN_WEB_AND_VISUALIZATION.md`](docs/13_LEARN_WEB_AND_VISUALIZATION.md) | The engine + the browser↔engine WebSocket contract | | [`docs/17-ai-rigor.md`](docs/17-ai-rigor.md) · [`20-aiml-training-design.md`](docs/20-aiml-training-design.md) · [`20-eng-review.md`](docs/20-eng-review.md) · [`23-data-and-cluster-semantics.md`](docs/23-data-and-cluster-semantics.md) | The AI core: fusion, the buzz inverse, the separability study, data + training | | [`docs/22-interface-ar-and-correction.md`](docs/22-interface-ar-and-correction.md) | The interface: live AR play + the 2D correction view | | [`docs/10-design-decisions.md`](docs/10-design-decisions.md) · [`11-ai-and-pitch.md`](docs/11-ai-and-pitch.md) · [`12-perception-references.md`](docs/12-perception-references.md) · [`13-open-questions.md`](docs/13-open-questions.md) · [`15-build-refinements.md`](docs/15-build-refinements.md) | Decisions, perception science, open items, the as-built teardown log | | [`docs/19-sponsors-refined.md`](docs/19-sponsors-refined.md) | Sponsor / prize alignment | | `docs/REF_*.md` (ml · psychophysics · sponsors) | Cited research briefs (background) | | [`cad/`](cad/) | Printable enclosure + actuator coupling pucks (FlashForge 5M) | | [`config/channel_map.json`](config/channel_map.json) | Machine-readable 12-channel routing (wiring source of truth) | | [`CHANGELOG.md`](CHANGELOG.md) | Chronological history (incl. the pre-pivot era) | --- *Pivoted from an earlier multi-pillar "music for the Deaf" concept. The abandoned Pi / ESP32 / 16-channel / 3-pillar docs were removed from this repo (recoverable from git history). The current product is **LEARN + PLAY guitar coaching** — see [`truth.md`](truth.md).* ## Detected evidence (automated analysis) Indexed codebase: 115 recognized source files, 11757 KB. - Anthropic (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - Redis (technology) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 159) ``` .gitignore cad/_puck_render.py cad/_verify_enclosure.py cad/actuator_button.stl cad/actuator_cup.stl cad/actuator_puck.scad cad/aruco/generate_marker.py cad/aruco/print_marker.html cad/contact_button.stl cad/node_barrel.stl cad/node_base.stl cad/README.md cad/tactus_box_base.stl cad/tactus_box_lid.stl cad/tactus_box.scad cad/tactus_chest_plate.scad cad/tactus_enclosure_base.3mf cad/tactus_enclosure_base.stl cad/TACTUS_enclosure_FF5M.gcode cad/tactus_enclosure_lid.stl cad/tactus_enclosure_plate.3mf cad/tactus_enclosure_plate.stl cad/tactus_enclosure.py cad/tactus_node_mount.scad cad/tactus_power_cradle.scad cad/tactus_power_cradle.stl cad/tactus_socket.scad CHANGELOG.md config/channel_map.json data/analysis/exp/beat_baseline_table.csv data/analysis/exp/e3_report.json data/analysis/exp/e3_transfer.html data/analysis/exp/e7_model_report.txt data/analysis/exp/e7_perfinger_predictions.csv data/analysis/exp/e7b_register_per_frame.csv data/analysis/exp/e7b_register_report.txt data/analysis/exp/e7b_register.json data/analysis/exp/e8_pose_chord.json data/analysis/exp/index.html data/analysis/exp/redis_retrieval_report.md data/analysis/exp/results.json data/analysis/exp/separability_3way.json data/analysis/exp/vision_perfinger.csv data/analysis/exp/viz_clean_buzz_muted_3d.html docs/00-start-here.md docs/01-bill-of-materials.md docs/02-system-architecture.md docs/03-power.md docs/04-soldering-guide.md docs/05-wiring-map.md docs/06-safety.md docs/07-haptic-encoding.md docs/08-software-architecture.md docs/09-assembly-checklist.md docs/10-design-decisions.md docs/11-ai-and-pitch.md docs/12-perception-references.md docs/13_LEARN_WEB_AND_VISUALIZATION.md docs/13-open-questions.md docs/15-build-refinements.md docs/17-ai-rigor.md docs/18-tuning-and-calibration.md docs/19-sponsors-refined.md docs/20-aiml-training-design.md docs/20-eng-review.md docs/21-chord-and-sustain-rendering.md docs/22-interface-ar-and-correction.md docs/23-data-and-cluster-semantics.md docs/24-data-collection-protocol.md docs/25-data-and-feature-format.md docs/26-aiden-handoff.md docs/27-chord-feedback-and-experiment-plan.md docs/28-analysis-results-and-methodology.md docs/REF_ml_brief.md docs/REF_psychophysics_brief.md docs/REF_sponsors_brief.md graphify-out/2026-06-20/.graphify_labels.json graphify-out/2026-06-20/cost.json graphify-out/2026-06-20/GRAPH_REPORT.md graphify-out/2026-06-20/graph.json graphify-out/2026-06-20/manifest.json graphify-out/cost.json graphify-out/GRAPH_REPORT.md graphify-out/graph.html graphify-out/graph.json graphify-out/manifest.json README.md software/ai/analysis/audit.py software/ai/analysis/collapse.py software/ai/analysis/exp/adversarial_groupcheck.py software/ai/analysis/exp/e1_viz.py software/ai/analysis/exp/e2_e4_e5.py software/ai/analysis/exp/e3_transfer.py software/ai/analysis/exp/e7_position_model.py software/ai/analysis/exp/e7_vision_extract.py software/ai/analysis/exp/e7b_register.py software/ai/analysis/exp/e8_pose_chord.py software/ai/analysis/exp/make_rotating_gif.py software/ai/analysis/exp/redis_retrieval.py software/ai/analysis/exp/separability_3way.py software/ai/analysis/extract_vision.py software/ai/analysis/features_audio.py software/ai/analysis/features_residual.py software/ai/analysis/features_vision.py software/ai/analysis/make_demo_data.py software/ai/analysis/run_pipeline.py software/ai/analysis/schema.py software/ai/analysis/segment.py software/ai/analysis/test_audit.py software/ai/analysis/test_collapse.py software/ai/analysis/test_features_audio.py software/ai/analysis/test_features_vision.py software/ai/analysis/test_pipeline.py software/ai/analysis/test_segment.py software/ai/capture/capture.html software/ai/capture/README.md software/ai/capture/record_conductor.py software/ai/capture/serve.py software/ai/vision/.gitignore software/ai/vision/annotate_video.py [39 more files omitted for size] ``` ### Dependencies - software/requirements.txt: anthropic, aubio, basic-pitch, crepe, librosa, matplotlib, mediapipe, numpy, opencv-contrib-python, pandas, redis, scikit-learn, scipy, sounddevice, soundfile ### Recent commits (newest first) - docs(vision): fix stale WebSocket port 8770->8772 in fretboard README + run comment - fix(web): emit events-wrapped tactus-tab/v1 so the haptic tab_player accepts LEARN tabs - feat: LEARN web UI + vision demo annotator - vision: anchor the neck grid to the fretboard's top edge (stable 2D rotated rect) - web+vision: auto AR via a learned YOLO-World + SAM2 neck detector backend - Merge remote-tracking branch 'origin/main' into truth-cad - web: fix AR mapping — MediaPipe hand auto-anchor + manual drag fallback - Merge origin/main into analysis branch - analysis: clean/buzz/muted + mono->poly transfer + Redis memory (methods, results, viz) - graphify: rebuild knowledge graph (1060 nodes, 93 communities) - web: Sherbet theme + markerless camera auto-lock for the LEARN UI - web: make glow.html the canonical LEARN UI; archive prototypes - capture: shuffle = 9 chords (drop G7, not learned) + 2 beats/chord (slower for changes) - Merge remote-tracking branch 'origin/worktree-analysis-pipeline' - graphify: refresh knowledge graph for consolidated main - capture: single-chord stream batch = 40 (was 80) to match collection plan - capture: chord-stream advances 1 chord/beat (BPM sets strum rate) - capture: 10-chord set (add F, G7) + 80/chord vs 100/shuffle stream lengths - Merge remote-tracking branch 'origin/haptic/resonance-check' - capture: chord-stream chord picker (single-chord vision-variety blocks) ## Key source files (fetched from GitHub, selected and truncated for size) ### CHANGELOG.md ```markdown # Changelog — Tactus > **Current state lives in [`truth.md`](truth.md), not here.** This log is the chronological record of the project, **including the abandoned pre-pivot era** (3-pillar Experience/Express; Raspberry Pi + ESP32 + MAX98357A; 16/14-channel). Older entries are kept as history — do not read them as the current build (as-built = laptop → 2× Vantec → 6× SK473/PAM8403 → 12× KHD drivers, LEARN + PLAY). ## 2026-06-20 — Framing pivot: whole-guitar capture (1080p) + offline ROI crop - **Operator constraint on the demo machine:** the tight nut→fret-7 framing forced a camera angle so low the screen was unusable. **Resolved:** capture the **WHOLE guitar** (full-screen video — also what the live AR demo + train/serve match want) at **1080p**, and **crop to the fretboard ROI offline** before MediaPipe so wide framing keeps full `d` resolution. Seeing the whole neck *strengthens* markerless registration (more fret wires + inlays at 3/5/7/9/12 + both edges). Frets **1–6** stay the only rendered/coached scope (6 motors) — the camera just sees more. - Tooling: `capture.html` + `calibrate.html` request **1080p** and the framing guide now says **whole-guitar**; `docs/25 §6d` (capture-wide-process-cropped) + `docs/26` build order add the **ROI-crop** step; `truth.md §3.8` updated. ## 2026-06-20 — docs/26 handoff + repo-wide markerless reconciliation - **`docs/26-aiden-handoff.md`** — data handoff for the offline pipeline (Aiden): exact `data/raw` + `data/calib` layout, the **authoritative full manifest schema with a real example row** (the live tool writes more fields than `docs/24 §5`'s core table), the 6 processing gotchas (pointing to `docs/25`), the build order, and the iron rules. Cross-linked from `docs/25`. - **Repo-wide registration reconciliation → MARKERLESS is primary, ArUco is OPTIONAL.** Aligned the canonical + active-path docs to the pivot (the guitar's 12-TET fret-law is its own ruler; `software/ai/vision/fretboard.py`): `truth.md` (§2 diagram, §3.8, §6 Stage 1, §9 doc map), `docs/24` (§1 rig, align-mode, §5 schema pointer, §7/§8), `docs/23` (§3 rig), `docs/17` (§1b), `docs/20` (D3 + capture rig), `docs/22` (live-AR registration → markerless-primary, ArUco/virtual-neck fallback), `cad/README` (marker = optional validator, nut/neck not headstock), `software/ai/capture/README`. `truth.md` remains the canonical tie-breaker for any background doc still describing "MediaPipe + ArUco" as a capability. ## 2026-06-20 — Markerless fretboard registration + guitar digital twin (no taped marker) - **`software/ai/vision/fretboard.py`** — markerless registration from the 12-TET fret-spacing law (`x_n = 1−2^(−n/12)`): the guitar is its own ruler, so no fiducial is needed. Fits the fretboard homography from the guitar's own geometry and validates by reprojecting the fret grid onto the real wires. Synthetic front-cam self-test passes (~1.3px median reproj, recovers signed `d`). This becomes the PRIMARY registration path; the ArUco marker is demoted [truncated — 19509 more characters] ``` ### docs/13-open-questions.md ```markdown # Open questions + to-do Live list of what's unconfirmed or unbought. Update as you close items. ## Confirm - [ ] **KHD driver diameter** — the de-housed SK473 KHD 3 Ω/5 W drivers are the actuators, but the diameter is **un-measured**; CAD uses placeholders (`spk_dia=40`). Measure the de-housed driver → set `spk_dia`/`drv_dia` in the `.scad` files → re-render the STLs. (truth.md §3.3, top open dim.) - [ ] **ALSA enumeration** — the CM6206 Vantecs reorder across reboots. Bind by `/dev/snd/by-id`, verify in-card channel order (`speaker-test`), fill `config/channel_map.json` empirically. (truth.md §3.2.) - [ ] **Webcam** — laptop cam enough, or buy an external USB webcam? Print an **ArUco marker** for the headstock for robust fretboard tracking. ## Buy / make - [ ] **2× USB-C cables** for the Mode-B bus feeds (or USB-C→USB-A-female adapters to skip the CC resistors). - [ ] **5.1 kΩ resistors** if feeding buses from bare-cut USB-C cables (see [04-soldering-guide.md](04-soldering-guide.md)). - [x] **3D-printed enclosure** — **done: see `cad/`** (`tactus_box` = 2–3 Vantec + 6 amps (+ Pi if used), vented, strain-relief comb; `tactus_power_cradle` = Anker 737 / hub; `actuator_puck` = de-housed KHD-driver coupler — `spk_dia` is a 40 mm placeholder, re-render once the driver is measured). Pre-rendered STLs included, sized to the FlashForge 5M 220³ bed. Build notes in `docs/15-build-refinements.md`. - [ ] **Mounting** — compression shirt/vest, VHB foam tape, rigid backers (corrugated plastic/acrylic), velcro, zip ties. ## Decide - [ ] **Judged run on Mode A (wall) or Mode B (cordless)?** Recommend Mode A for reliability, Mode B as the "and it goes cordless" flex. - [ ] **Real-time vs pre-processed transcription** for the demo. Live guitar → live F0 (pYIN / YIN); polished run → pre-process the song (CREPE offline). Known target song de-risks both. ## Repo / logistics - [ ] **Push to the team remote.** Currently local-only. `git remote add origin && git push -u origin `. Send the URL. - [ ] Decide branch/PR workflow for teammates. ## Known limitations (state honestly in the demo) - Polyphonic transcription accuracy degrades on dense/fast playing (vision cross-checks). - "Pressure" is a **2-class ordinal (too-light / good)** recovered by inverting the buzz surface, not measured force. "Too hard" is a separate pitch-cents fault. - Placement is **vs the fret-wire (coarse)**, not exact centimeters. - Mode B at a hard-cranked chord is tight on the 3-amp USB-C bus — run at felt level, or use Mode A. ``` ### software/requirements.txt ``` # audio basic-pitch librosa aubio crepe sounddevice numpy scipy # vision mediapipe opencv-contrib-python # includes ArUco # coach + memory anthropic redis # optional: per-user mistake history # analysis (offline pipeline: segment -> features -> PCA/LDA -> audit) scikit-learn # PCA, LDA, confusion, silhouette pandas # event/feature tables soundfile # WAV read for feature windows matplotlib # audit charts (Agg backend) ``` ### web/index.html ```html Tactus · LEARN
TACTUS · LEARN
opening Tactus…   enter →
``` ### cad/_verify_enclosure.py ```python #!/usr/bin/env python3 """Verify + visualize the TACTUS unified enclosure STLs. Outputs a PNG sheet.""" import os, numpy as np, trimesh import matplotlib; matplotlib.use("Agg") import matplotlib.pyplot as plt from matplotlib.patches import Rectangle from mpl_toolkits.mplot3d.art3d import Poly3DCollection HERE = os.path.dirname(os.path.abspath(__file__)) base = trimesh.load(os.path.join(HERE, "tactus_enclosure_base.stl")) lid = trimesh.load(os.path.join(HERE, "tactus_enclosure_lid.stl")) for n, m in (("base", base), ("lid", lid)): b = m.bounds print(f"{n}: dims={np.round(b[1]-b[0],1)} watertight={m.is_watertight} " f"bodies={m.body_count} euler={m.euler_number} vol={m.volume/1000:.1f}cm3") def iso(ax, m, title): T = m.triangles ax.add_collection3d(Poly3DCollection(T, facecolor="#7fb3d5", edgecolor="#234", linewidths=0.05, alpha=1.0)) b = m.bounds; ctr = (b[0]+b[1])/2; r = (b[1]-b[0]).max()/2 for setlim, c in ((ax.set_xlim, ctr[0]),(ax.set_ylim, ctr[1]),(ax.set_zlim, ctr[2])): setlim(c-r, c+r) ax.view_init(elev=28, azim=-58); ax.set_title(title, fontsize=9) ax.set_box_aspect((1,1,1)); ax.set_axis_off() def slc(ax, m, normal, level, ha, va, title, lim): sec = m.section(plane_origin=np.array(normal)*0+level*np.array(normal), plane_normal=normal) if sec is not None: for p in sec.discrete: ax.plot(p[:,ha], p[:,va], "-", color="#1a5276", lw=0.9) ax.set_title(title, fontsize=9); ax.set_aspect("equal") ax.set_xlim(*lim[0]); ax.set_ylim(*lim[1]); ax.grid(alpha=.25, lw=.4) fig = plt.figure(figsize=(15, 9)) ax1 = fig.add_subplot(2,3,1, projection="3d"); iso(ax1, base, "BASE (iso)") ax2 = fig.add_subplot(2,3,2, projection="3d"); iso(ax2, lid, "LID (iso)") # ---- component fit map (top-down) ------------------------------------------ ax3 = fig.add_subplot(2,3,3) WALL=2.6; IX=192; EY0=2.6; EY1=114.6; PY0=118.6; PY1=188.6; OX=197.2; OY=191.2 ax3.add_patch(Rectangle((0,0), OX, OY, fill=False, ec="k", lw=1.5)) ax3.add_patch(Rectangle((WALL,EY0), IX, EY1-EY0, fc="#eaf2f8", ec="#2980b9", lw=1)) # elec bay ax3.add_patch(Rectangle((WALL,PY0), IX, PY1-PY0, fc="#fef9e7", ec="#b9770e", lw=1)) # power bay comp = [ # x,y,w,h,label,color (4.6, 8.6, 58,100, "Vantec\n7.1 #1", "#82e0aa"), (66.6, 8.6, 58,100, "Vantec\n7.1 #2", "#82e0aa"), (128, 8.6, 64,100, "amp bank\n(7x)", "#f1948a"), (10.6,128.6,156,55, "Anker 737 (Mode B)", "#85c1e9"), ] for x,y,w,h,l,c in comp: ax3.add_patch(Rectangle((x,y),w,h, fc=c, ec="#333", lw=.8, alpha=.75)) ax3.text(x+w/2, y+h/2, l, ha="center", va="center", fontsize=6.5) # hub alternative footprint (dashed) in the power bay ax3.add_patch(Rectangle((8,121),180,65, fill=False, ec="#b9770e", lw=1, ls="--")) ax3.text(98,118.5,"or 10-port hub (Mode A, <=180x70)", ha="center", fontsize=6, color="#7e5109") ax3.set_title("Component fit map (top) — generous leeway", fontsize=9) ax3.set_aspect("equal"); ax3.set_xlim(-6, OX+6); ax3.set_ylim(-12, OY+6); ax3.grid(alpha=.25,lw=.4) slc(fig.add_subplot(2,3,4), base, [0,0,1], 1.3, 0,1, "Floor slice z=1.3 (vents/slots)", ((-6,203),(-12,197))) slc(fig.add_subplot(2,3,5), base, [0,0,1], 30, 0,1, "Mid slice z=30 (cavities/partition/bosses)", ((-6,203),(-12,197))) slc(fig.add_subplot(2,3,6), base, [1,0,0], OX/2, 1,2, "Depth section x=mid (bay heights/open top)", ((-12,197),(-6,64))) fig.suptitle("TACTUS unified enclosure — verification sheet", fontsize=12, y=.99) fig.tight_layout() out = os.path.join(HERE, "_verify_enclosure.png") fig.savefig(out, dpi=110); print("wrote", out) ``` ### cad/_puck_render.py ```python #!/usr/bin/env python3 # ============================================================================= # Headless renderer for actuator_puck.scad — faithful 1:1 port of the OpenSCAD # geometry, used because the installed OpenSCAD is an x86 2021.01 build that # won't run on this Apple-Silicon Mac (no Rosetta). Same manifold3d kernel as # tactus_enclosure.py. Mirrors the .scad EXACTLY (validated: the 40 mm .scad # output 63.4×45.4×10.4 matches this port's formula at spk_dia=40). # # Source of truth for params remains actuator_puck.scad. Keep them in sync. # RUN: /tmp/tactuscad/bin/python cad/_puck_render.py # OUT: cad/actuator_cup.stl , cad/actuator_button.stl # ============================================================================= import os, numpy as np, trimesh from trimesh import creation, transformations as tf ENG, S = "manifold", 96 # ---- params (MUST match actuator_puck.scad) --------------------------------- spk_dia, spk_depth, spk_fit_gap = 52.0, 27.0, 0.6 # SK473 driver: 52 mm OD wall, back = 2.4, 2.4 btn_dia, btn_dome_h, btn_base_h = 14.0, 4.5, 1.6 notch_w, notch_h = 6.0, 5.0 cup_id = spk_dia + spk_fit_gap # 52.6 cup_od = cup_id + 2 * wall # 57.4 cup_h = back + spk_depth + 1.0 # 30.4 def U(p): return trimesh.boolean.union(p, engine=ENG) def D(a, b): return trimesh.boolean.difference([a, b], engine=ENG) def cylz(d, h, x=0, y=0, z=0): # base at z (OpenSCAD convention) m = creation.cylinder(radius=d/2, height=h, sections=S) m.apply_translation((x, y, z + h/2)); return m def cyly(d, h, x=0, y=0, z=0): # axis along Y, centered at (x,y,z) m = creation.cylinder(radius=d/2, height=h, sections=S) m.apply_transform(tf.rotation_matrix(np.pi/2, (1, 0, 0))) m.apply_translation((x, y, z)); return m def box(sx, sy, sz, x, y, z): # min corner at (x,y,z) m = creation.box(extents=(sx, sy, sz)) m.apply_translation((x+sx/2, y+sy/2, z+sz/2)); return m def cup(): solid = U([ cylz(cup_od, cup_h), # body cylz(10, 3, cup_od/2 + 4, 0), # ear +X (overlaps body 1 mm) cylz(10, 3, -(cup_od/2 + 4), 0), # ear -X ]) cuts = U([ cylz(cup_id, cup_h, 0, 0, back), # speaker bore (closed back) box(notch_w, wall + 1.5, notch_h, -notch_w/2, cup_od/2 - wall - 0.5, back + 1.5), # wire notch (+Y) cyly(2.6, wall + 2, 0, cup_od/2 - 1.0, back + 3.5), # zip-tie strain-relief hole cylz(3.4, 5, cup_od/2 + 4, 0, -0.5), # ear hole +X cylz(3.4, 5, -(cup_od/2 + 4), 0, -0.5), # ear hole -X cylz(4, back + 1, 0, 0, -0.5), # back vent (cone breathes) ]) # NOTE: the .scad's rim_lip subtraction is a geometric no-op (it lies inside # the bore), so it is omitted here — the resulting solid is identical. return D(solid, cuts) def button(): base = cylz(btn_dia, btn_base_h) dome = creation.icosphere(subdivisions=3, radius=btn_dia/2) dome.apply_scale([1, 1, btn_dome_h / (btn_dia/2)]) dome.apply_translation((0, 0, btn_base_h)) return U([base, dome]) def main(): here = os.path.dirname(os.path.abspath(__file__)) for name, m, exp in (("actuator_cup", cup(), (75.4, 57.4, 30.4)), ("actuator_button", button(), (14.0, 14.0, 9.0))): m.process(validate=True) d = (m.bounds[1] - m.bounds[0]) p = os.path.join(here, name + ".stl") m.export(p) ok = all(abs(a - b) < 0.6 for a, b in zip(d, exp)) print(f"{name}: dims={d.round(2)} (expect {exp}) watertight={m.is_watertight} " f"bodies={m.body_count} vol={m.volume/1000:.2f}cm3 {'OK' if ok else 'CHECK'}") if __name__ == "__main__": main() ``` ### web/alphatab-loader.js ```javascript // alphatab-loader.js — sheet music (GuitarPro / MusicXML) -> Tactus chart. // // Pipeline: a .gp/.gp5/.gpx/.gp7 or .musicxml/.mxl file -> alphaTab parse (headless, // no DOM/synth) -> { bpm, notes:[{t, string, fret, dur, chordId}], end }. // Notes that sound together share the same `t` and a `chordId` -> the chord-field view // lights their cells on the same frame for free. // // Requires the alphaTab UMD bundle on the page (exposes window.alphaTab): // // (npm: `npm i @coderline/alphatab`) // // Verified against context7 /websites/alphatab_net (alphaTab agent, 2026-06-21): // alphaTab.importer.ScoreLoader.loadScoreFromBytes(Uint8Array, Settings?) -> Score // Beat.absolutePlaybackStart (ticks), Beat.playbackDuration (ticks), // Note.string (1=high-e..6=low-E), Note.fret, MasterBar.tempoAutomations, // MidiUtils.QuarterTime (= 960 ticks/quarter). const AT = () => window.alphaTab; // Parse bytes (or a URL) into an alphaTab Score, fully headless (no container, no synth). export async function loadScore(urlOrBytes) { let data; if (typeof urlOrBytes === 'string') { const res = await fetch(urlOrBytes); data = new Uint8Array(await res.arrayBuffer()); } else if (urlOrBytes instanceof ArrayBuffer) { data = new Uint8Array(urlOrBytes); } else { data = urlOrBytes; // already a Uint8Array } const settings = new (AT().Settings)(); // ScoreLoader sniffs GuitarPro-binary vs MusicXML from the bytes automatically. return AT().importer.ScoreLoader.loadScoreFromBytes(data, settings); } // Load a score and flatten it to the Tactus chart. // flipString: set true if a given file numbers strings 6=high-e..1=low-E (rare) -> uses 7-string. export async function loadChart(urlOrBytes, { trackIndex = 0, flipString = false } = {}) { const score = await loadScore(urlOrBytes); const QUARTER = AT().model.MidiUtils.QuarterTime; // 960 const track = score.tracks[trackIndex] || score.tracks[0]; // ---- tick -> seconds, honoring simple tempo automation (one BPM per bar) ---- let tick = 0, bpm = score.tempo || 120; const segs = []; // [{ startTick, bpm, startSec }] for (const mb of score.masterBars) { const autos = mb.tempoAutomations || []; if (autos.length) bpm = autos[autos.length - 1].value; segs.push({ startTick: tick, bpm }); const num = mb.timeSignatureNumerator, den = mb.timeSignatureDenominator; tick += Math.round(QUARTER * 4 * num / den); } let acc = 0; for (let i = 0; i < segs.length; i++) { if (i > 0) { const p = segs[i - 1]; acc += (segs[i].startTick - p.startTick) * (60 / (p.bpm * QUARTER)); } segs[i].startSec = acc; } const tickToSec = (t) => { let seg = segs[0]; for (let i = 1; i < segs.length; i++) { if (segs[i].startTick <= t) seg = segs[i]; else break; } return seg.startSec + (t - seg.startTick) * (60 / (seg.bpm * QUARTER)); }; // ---- walk tracks > staves > bars > voices > beats > notes ---- const notes = []; let chordId = 0; for (const stave of track.staves) { for (const bar of stave.bars) { for (const voice of bar.voices) { for (const beat of voice.beats) { if (beat.isRest || !beat.notes || beat.notes.length === 0) continue; const onset = beat.absolutePlaybackStart; const t = +tickToSec(onset).toFixed(4); const dur = +Math.max(0.05, tickToSec(onset + beat.playbackDuration) - tickToSec(onset)).toFixed(4); const id = beat.notes.length > 1 ? ++chordId : null; // multi-note beat = a chord for (const note of beat.notes) { if (note.isTieDestination) continue; // sustain of an earlier hit, not a new onset if (note.string == null || note.fret == null) continue; notes.push({ t, string: flipString ? 7 - note.string : note.string, fret: note.fret, dur, chordId: id }); } } } } } notes.sort((a, b) => a.t - b.t || a.string - b.string); const end = notes.length ? notes[notes.length - 1].t + notes[notes.length - 1].dur + 1.2 : 0; return { bpm: score.tempo || 120, notes, end }; } ``` ### web/aruco-poselock.js ```javascript // aruco-poselock.js — lock a three.js Group onto a real ArUco marker in a webcam feed. // // Detect (js-aruco2) -> pose (POS.Posit) -> handedness fix -> one-euro smooth -> // confidence gate (fade to a rendered "ghost" neck when the marker is lost). // Verified against js-aruco2 source (aruco.js, posit1/2.js, samples/debug-posit) and // three.js r160 (ArUco agent, 2026-06-21). // // Requires js-aruco2 globals on the page (cv.js -> aruco.js -> svd.js -> posit2.js), // which attach window.AR / window.CV / window.POS. Print the marker from the // ARUCO_MIP_36h12 dictionary and MEASURE its physical side length (mm). // // Units are MILLIMETRES throughout (MARKER_SIZE_MM seeds the model scale): build the // fretboard geometry in mm, set camera near~10 / far~10000. import * as THREE from 'three'; // adaptive 1-euro filter (low jitter when still, low lag when moving) export class OneEuro { constructor(minCutoff = 1.2, beta = 0.02, dCutoff = 1.0) { Object.assign(this, { minCutoff, beta, dCutoff, xPrev: null, dxPrev: 0, tPrev: null }); } _a(cutoff, dt) { const r = 2 * Math.PI * cutoff * dt; return r / (r + 1); } filter(x, t) { if (this.xPrev === null) { this.xPrev = x; this.tPrev = t; return x; } const dt = Math.max(1e-3, t - this.tPrev); const dx = (x - this.xPrev) / dt; const aD = this._a(this.dCutoff, dt); const dxHat = aD * dx + (1 - aD) * this.dxPrev; const a = this._a(this.minCutoff + this.beta * Math.abs(dxHat), dt); const xHat = a * x + (1 - a) * this.xPrev; this.xPrev = xHat; this.dxPrev = dxHat; this.tPrev = t; return xHat; } } // Set the PerspectiveCamera fov from the webcam focal length (vertical fov). export function matchCameraToWebcam(camera, videoW, videoH, focalPx) { camera.fov = 2 * Math.atan(videoH / (2 * focalPx)) * 180 / Math.PI; camera.aspect = videoW / videoH; camera.updateProjectionMatrix(); } // Build a per-frame pose updater. Pass the