# Project export: UPIC: Self-Generated Training Data Using Zero-Shot Reasoning This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Instead of training a robot on hundreds of episodes for a specific task, UPIC interacts with its environment and dynamically reasons & learns how to accomplish any task, even those never seen before. - Devpost: https://devpost.com/software/upic-self-generated-training-data-using-zero-shot-reasoning - GitHub: https://github.com/snellogisn/limbic - Video: https://www.youtube.com/embed/vVmj7ADweAk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Claude Opus 4.8 (52 commits), avaheb859 (49 commits), SamyukthN (23 commits), AnAirbusBeluga (13 commits) ## Devpost submission (written by the team) No Devpost description available. ## README (from the GitHub repository) # limbic A **cross-platform** (macOS · Windows · Linux), **LLM-driven** control stack for a tabletop robot arm. Speak an instruction; an LLM perceives the scene, compiles a list of **motion primitives**, and the arm carries it out — and it all runs on a plain laptop with **no physical arm** thanks to a built-in software mock. It is a clean, cross-platform reimagining of a Windows-locked LeRobot SO-101 control stack. See [`ARCHITECTURE.md`](ARCHITECTURE.md) for the full design. --- ## Why it runs anywhere The earlier code only ran on Windows: hard-coded `COM7` serial ports, Windows-only DirectShow cameras, and IK solvers with no macOS binaries. limbic fixes all of that: - **Serial ports auto-detected** on every OS (`$LIMBIC_PORT` to override). - **Cameras** open with the right backend per OS (AVFoundation / DirectShow / V4L2). - **Kinematics are pure Python** — zero binary dependencies. - **A mock backend** simulates the arm, so the entire pipeline runs on a bare machine with nothing plugged in. --- ## Install The core stack needs **nothing** — it's pure Python and runs as-is. Install extras only for the capabilities you want: ```bash pip install -r requirements.txt # serial + camera + LLM brain # or pick à la carte: pip install pyserial # real-arm USB port detection pip install opencv-python # the camera sense pip install anthropic # the runtime LLM brain pip install "lerobot[feetech]" # drive the physical SO-101 arm ``` Requires Python 3.10+. --- ## Quick start ### Drive the arm directly (auto mock ⇄ real) ```python from limbic import RobotArm with RobotArm() as arm: # real arm if one is attached, else the mock arm.go_home() arm.open_gripper() arm.move_to_xyz(180, 0, 60) # table-frame mm: +x forward, +y left, +z up arm.close_gripper() arm.lift_by(80) ``` ### Run a plan (a list of motion primitives) ```bash python -m limbic.primitives.example_plan # pick & place, on the mock arm ``` ### Let an LLM compile and run the plan ```bash export ANTHROPIC_API_KEY=sk-... python examples/run_mock_demo.py # perceive → plan → execute (mock) ``` `run_mock_demo.py` also runs **offline** (no API key) by executing a canned plan, so you can always see the whole pipeline move the arm. --- ## Selecting hardware vs. mock Everything is environment-driven — no code edits: | Variable | Meaning | Example | |---|---|---| | `LIMBIC_BACKEND` | `auto` (default), `real`, or `mock` | `mock` | | `LIMBIC_PORT` | serial port (else auto-detected) | `COM7` · `/dev/cu.usbserial-10` | | `LIMBIC_ROBOT_ID` | robot id for the SDK | `limbic` | | `ANTHROPIC_API_KEY` | needed only for the runtime brain | `sk-...` | `auto` uses the real arm when a serial port is found **and** `lerobot` is installed; otherwise it transparently falls back to the mock and tells you so. > **Safety:** every motion — human, scripted, or LLM-issued — passes through the > workspace clamp and per-joint soft limits in `limbic/control/safety.py` before > any command reaches a motor. An out-of-reach target stops at the nearest > reachable point; it is never sent raw. --- ## Layout ``` limbic/ control/ The Body — movement, gripper, guardrails, mock⇄real backend primitives/ The Skills — one motion primitive per file + the plan runner inputs/ The Senses — motor + camera readings the LLM can query brain/ The Mind — instruction → validated list of primitives → run platform_support.py — the cross-platform seam (ports, cameras, OS) examples/ run_mock_demo.py — end-to-end demo on the mock arm ``` The motion primitives and sensory inputs are **auto-discovered**: add a capability by dropping a single new file in `primitives/library/` or `inputs/library/` — nothing else to wire up. This is also how the LLM invents or revises primitives. ## Detected evidence (automated analysis) Indexed codebase: 86 recognized source files, 607 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (98 of 98) ``` .gitattributes .gitignore ARCHITECTURE.md assets/so101/so101_new_calib.urdf calibration/accuracy_model_STALE.json calibration/accuracy_model.json calibration/accuracy_samples_STALE.csv calibration/accuracy_samples.csv classes.txt CLAUDE.md docs/CALIBRATION_VALUES.md docs/HANDOFF_camera_localization.md docs/HANDOFF_ik_ruler_check.md docs/VISION.md examples/run_mock_demo.py limbic/__init__.py limbic/_core.py limbic/brain/__init__.py limbic/brain/orchestrator.py limbic/brain/system_prompt.py limbic/brain/tools.py limbic/control/__init__.py limbic/control/_prep_ik_wrapper.py limbic/control/_prep_planar_ik.py limbic/control/accuracy_model.py limbic/control/arm.py limbic/control/backends.py limbic/control/calibration.py limbic/control/config.py limbic/control/ik_chain.py limbic/control/kinematics.py limbic/control/localization.py limbic/control/mink_ik.py limbic/control/safety.py limbic/inputs/__init__.py limbic/inputs/base.py limbic/inputs/library/__init__.py limbic/inputs/library/camera.py limbic/inputs/library/gripper_state.py limbic/inputs/library/joint_state.py limbic/inputs/library/object_detections.py limbic/inputs/library/tip_position.py limbic/inputs/registry.py limbic/platform_support.py limbic/primitives/__init__.py limbic/primitives/authoring.py limbic/primitives/base.py limbic/primitives/example_plan.py limbic/primitives/library/__init__.py limbic/primitives/library/aligned_pick.py limbic/primitives/library/close_hand.py limbic/primitives/library/descend_to.py limbic/primitives/library/home.py limbic/primitives/library/lift.py limbic/primitives/library/move_to.py limbic/primitives/library/open_hand.py limbic/primitives/library/pick.py limbic/primitives/library/place.py limbic/primitives/library/push.py limbic/primitives/library/reach_above.py limbic/primitives/library/reposition_for_pick.py limbic/primitives/library/throw_forward.py limbic/primitives/registry.py limbic/primitives/run_sequence.py limbic/runlog.py limbic/vision/__init__.py limbic/vision/detector.py limbic/vision/dino.py limbic/vision/sizing.py limbic/vision/visual_align.py limbic/vision/workspace.py pyproject.toml README.md requirements.txt scripts/arm_connect_check.py scripts/calibrate_accuracy.py scripts/click_localize.py scripts/describe_machine.py scripts/extrinsics_live.py scripts/go_home.py scripts/stage2_calibration_check.py scripts/stage2_ruler_check.py scripts/stage3_extrinsics_robust.py scripts/stage3_extrinsics.py scripts/stage3_intrinsics.py scripts/vision_detect_demo.py scripts/vision_detect_dual.py scripts/workspace_view.py web/pipeline.py web/README.md web/server.py web/static/app.js web/static/index.html web/static/runs.html web/static/style.css WEBSITE_SETUP.md weights/clip/ViT-B-32.pt weights/yolov8s-world.pt ``` ### Dependencies - pyproject.toml: anthropic@>=0.40, anthropic@>=0.40, opencv-python@>=4.8, opencv-python@>=4.8, pygrabber@>=0.2, pyserial@>=3.5, pyserial@>=3.5, torch@>=2.0, torch@>=2.0, ultralytics@>=8.1, ultralytics@>=8.1 - requirements.txt: anthropic@>=0.40, opencv-python@>=4.8, pyserial@>=3.5 ### Recent commits (newest first) - ik(mink): wrist tilt down extreme is +90, not -90 - ik(mink): pin wrist TILT at the -90 down extreme (top-down) [REVERTABLE] - Revert "Iterative checking with strict straight down no offset movement." - Merge remote-tracking branch 'origin/main' - Iterative checking with strict straight down no offset movement. - feat(brain): prioritize top-down (90deg) grasps; reposition far objects - docs(brain): claw closes on centre — prompt rule matches the removed offset - tune(grasp): remove the claw lateral offset (-5mm -> 0) - Added pushing adjustment and movements. - feat(brain): measured before/after error so retries actually learn - Merge remote-tracking branch 'origin/main' - Added wrist tilt calculation with adjustment. - tune(grasp): reduce claw lateral offset by 0.5cm (-10mm -> -5mm) - tune(grasp): drop from 1.5cm above + account for cube height in support - revert IK wrist_roll change; top-down rule is about wrist TILT, not roll - ik(mink): lock wrist_roll at 90deg for top-down picks [REVERTABLE] - feat(grasp): drop-from-above height + top-down pick preference (physical rules) - docs(brain): point throw guidance at the existing throw_forward primitive - revert(vision): drop the camera->arm x offset; use raw cam coordinates - throwing. ## Key source files (fetched from GitHub, selected and truncated for size) ### WEBSITE_SETUP.md ```markdown # Website / Demo-Box Setup — pip installs Everything you need to `pip install` to run the limbic **web interface** on the **x64 demo computer** with the **mink (MuJoCo) IK solver**, the **Claude brain**, and the **real SO-101 arm**. > ⚠️ x64 only. `mujoco`/`mink` and `torch` have **no ARM64-Windows wheels**, so this > setup does **not** run on the Snapdragon/ARM64 dev box — that machine falls back > to the planar IK solver. Run the website on the x64 box. > Use **Python 3.13 (64-bit)** — the verified `mujoco`/`mink` wheels are `cp313 win_amd64`. --- ## 1. Create + activate a venv ```powershell python -m venv venv .\venv\Scripts\Activate.ps1 python -m pip install --upgrade pip ``` ## 2. The pip installs ```powershell # --- Inverse kinematics ---------------------------------------------------- # numpy + ikpy: the FK chain (always imported, both IK engines build off it) pip install numpy ikpy # mink (MuJoCo) reaching IK — the NEW default solver the website uses pip install mujoco mink # mink solves the QP with the "daqp" backend; install it explicitly pip install qpsolvers daqp # --- The Claude brain (natural-language planning) -------------------------- pip install "anthropic>=0.40" # --- The real arm ---------------------------------------------------------- pip install "pyserial>=3.5" # COM-port detection pip install "lerobot[feetech]" # drives the physical SO-101 (Feetech servos) ``` ### One-liner ```powershell pip install numpy ikpy mujoco mink qpsolvers daqp "anthropic>=0.40" "pyserial>=3.5" "lerobot[feetech]" ``` --- ## 3. (Optional) Camera vision — only for `detect_objects` Needed **only** if the demo uses live object detection (Part B) instead of typed `(x, y)` coordinates. Heavy; skip it if you're driving by coordinates. ```powershell pip install "opencv-python>=4.8" pygrabber # capture + by-name camera enum (Windows) pip install "torch>=2.0" transformers # Grounding DINO detector ``` --- ## 4. Run it ```powershell $env:ANTHROPIC_API_KEY = 'sk-ant-...' # inline ONLY — never write the key to a file $env:LIMBIC_BACKEND = 'real' # drive the physical arm $env:LIMBIC_PORT = 'COM5' # the COM port on THIS machine (Device Manager > Ports) python web\server.py # then open http://localhost:8765 ``` `LIMBIC_IK` already defaults to `mink`, so the website uses the new solver with no extra flag. ## 5. Confirm it's actually using mink (not the fallback) Watch the server startup log: - ✅ Planner line says **`planner: CLAUDE`** (key was picked up). - ✅ **No** warning like `mink IK unavailable (...); falling back to the closed-form planar solver`. If you see that fallback warning, `mujoco`/`mink`/`daqp` didn't import — re-check step 2 in the **same venv** you're launching `web/server.py` from. --- ## Why each one | Package | Why it's needed | |---|---| | `numpy` | array math under the IK chain + mink | | `ikpy` | builds the SO-101 FK chain from th [truncated — 611 more characters] ``` ### ARCHITECTURE.md ```markdown # limbic — Architecture `limbic` is a control stack for a tabletop robot arm that a person drives in plain language: you give an instruction, an LLM perceives the scene and compiles a list of **motion primitives**, and the arm executes them. It is built to run on **macOS, Windows and Linux**, with or without physical hardware. ## The three layers (plus the mind) ``` instruction ("pick up the block and put it on the left") │ ▼ ┌──────────────┐ browses catalogs, queries senses, emits an ordered plan │ THE MIND │ limbic/brain/ (Claude API: perceive → plan → run) └──────┬───────┘ │ list[ {primitive, args} ] ▼ ┌──────────────┐ reusable arm skills, one file each, LLM-authorable │ THE SKILLS │ limbic/primitives/ (home, move_to, pick, place, push, …) └──────┬───────┘ │ RobotArm method calls (already safety-clamped + smoothed) ▼ ┌──────────────┐ movement, gripper, GUARDRAILS — the only thing that │ THE BODY │ limbic/control/ touches motors; auto mock⇄real backend └──────┬───────┘ │ joint commands (degrees / 0..100 gripper) ▼ MockBackend (software sim) or RealBackend (LeRobot SO-101 over USB) ┌──────────────┐ read-only perceptions the mind can query while planning │ THE SENSES │ limbic/inputs/ (joint_state, tip_position, camera, …) └──────────────┘ ``` ### The Body — `limbic/control/` The only layer that commands motors. Everything else goes through its `RobotArm` class, so the safety and smoothing logic is written once and shared. | File | Role | |---|---| | `arm.py` | `RobotArm` — the stable tool surface: `move_to_xyz`, `reach_above`, `descend_to`, `lift_by`, `set_joint`, `go_home`, `open_gripper`/`close_gripper`/`set_gripper`, `current_xyz`, `read_joints`. Every move is workspace-clamped, soft-limit-clamped, and streamed with an ease-in/ease-out velocity profile. | | `safety.py` | The **single source of truth for guardrails**: per-joint soft limits + the Cartesian workspace dome. Targets outside the safe region are clamped to the nearest reachable point, never sent raw. | | `kinematics.py` | Pure-Python closed-form IK/FK (table-mm ⇄ joint-degrees). No numpy/ikpy/placo, so it runs identically on every OS — this is what unblocks macOS, where the reference project's solvers had no binaries. | | `backends.py` | The hardware seam: `HardwareBackend` interface, `MockBackend` (software sim), `RealBackend` (LeRobot SO-101). `make_backend()` auto-picks: real if a serial port is found and `lerobot` is installed, else mock. | | `config.py` | Env-driven connection + motion config (`$LIMBIC_PORT`, `$LIMBIC_BACKEND`, …). | ### The Skills — `limbic/primitives/` A folder of motion primitives, **one per file**, each a `Primitive` subclass declaring `name`, `summary`, `parameters`, and a `run(arm, **kwargs)` that calls only the `RobotArm` tool surface (so it inherits all safety). The `registry` auto-discovers every file, so the catalog the [truncated — 3495 more characters] ``` ### pyproject.toml ``` [build-system] requires = ["setuptools>=61"] build-backend = "setuptools.build_meta" [project] name = "limbic" version = "0.1.0" description = "Cross-platform, LLM-driven control stack for a tabletop robot arm." readme = "README.md" requires-python = ">=3.10" # The core stack is pure-Python with zero required dependencies (it runs on the # mock backend out of the box). Optional features live in [project.optional-dependencies]. dependencies = [] [project.optional-dependencies] serial = ["pyserial>=3.5"] # real-arm USB serial port detection camera = ["opencv-python>=4.8", "pygrabber>=0.2; sys_platform == 'win32'"] # camera sense (pygrabber = by-name enumeration on Windows) brain = ["anthropic>=0.40"] # the runtime LLM orchestrator # Vision (Part B). x64 / macOS / Linux only — torch has no ARM64-Windows wheel # (§0.4); kept OUT of base so non-x64 installs don't break. vision = ["torch>=2.0", "ultralytics>=8.1"] # Physical SO-101 hardware: install separately with `pip install "lerobot[feetech]"`. all = ["pyserial>=3.5", "opencv-python>=4.8", "anthropic>=0.40", "torch>=2.0", "ultralytics>=8.1"] [tool.setuptools.packages.find] include = ["limbic*"] ``` ### requirements.txt ``` # limbic dependencies. # # The CORE stack (control layer, kinematics, primitives, inputs registry, the # mock backend) is PURE PYTHON and needs nothing here — it imports and runs on a # bare macOS / Windows / Linux machine. Everything below is optional and only # needed for specific capabilities. # --- Serial port auto-detection (replaces the hard-coded COM7) --------------- # Needed to find/drive the real arm's USB port on any OS. Not needed for the mock. pyserial>=3.5 # --- Cameras (the inputs/camera sense) --------------------------------------- # Cross-platform capture (AVFoundation on macOS, DirectShow on Windows, V4L2 on # Linux) is selected automatically in platform_support.py. Not needed for the mock. opencv-python>=4.8 # --- The LLM brain ----------------------------------------------------------- # Only needed to run the runtime Claude orchestrator (brain/). Plans can also be # authored by Claude Code and run directly via the sequence runner with no SDK. anthropic>=0.40 # --- The real robot arm ------------------------------------------------------ # Only needed for the RealBackend (driving physical SO-101 hardware). Install # with the feetech extra: pip install "lerobot[feetech]" # lerobot[feetech] ``` ### web/server.py ```python """A tiny local web server for driving the limbic arm and browsing run logs. Pure Python standard library — no Flask, no build step, no npm. That's deliberate: it starts with one command on any machine, which is what makes it easy to run and to drive from Claude Code. python web/server.py # then open http://localhost:8765 python web/server.py --port 9000 Pages: GET / the "Ask" page — type a task (or click a test button) -> run GET /runs the "Logs" page — a scrollable list of every past run JSON API (what the pages call, and what Claude Code can call directly): POST /api/run body {"task": "...", "mode": "auto|claude|offline"} -> runs the pipeline, returns the structured result GET /api/runs -> [summary, ...] of every run, newest first GET /api/runs/ -> full detail of one run (all three log streams) """ from __future__ import annotations import argparse import json import os from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from urllib.parse import parse_qs, urlparse import pipeline # local module (same folder) _STATIC_DIR = Path(__file__).resolve().parent / "static" _CONTENT_TYPES = { ".html": "text/html; charset=utf-8", ".css": "text/css; charset=utf-8", ".js": "text/javascript; charset=utf-8", ".json": "application/json; charset=utf-8", } class Handler(BaseHTTPRequestHandler): """Routes static pages and the small JSON API.""" # Quieter, friendlier request log line. def log_message(self, fmt: str, *args) -> None: print(f"[web] {self.address_string()} {fmt % args}") # ----- helpers ------------------------------------------------------- # def _send_json(self, payload, status: int = 200) -> None: body = json.dumps(payload, default=str).encode("utf-8") self.send_response(status) self.send_header("Content-Type", "application/json; charset=utf-8") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) def _send_file(self, path: Path) -> None: if not path.is_file(): self._send_json({"error": "not found"}, status=404) return body = path.read_bytes() self.send_response(200) self.send_header("Content-Type", _CONTENT_TYPES.get(path.suffix, "application/octet-stream")) self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) # ----- GET ----------------------------------------------------------- # def do_GET(self) -> None: route = urlparse(self.path).path if route == "/": self._send_file(_STATIC_DIR / "index.html") elif route == "/runs": self._send_file(_STATIC_DIR / "runs.html") elif route == "/api/runs": self._send_json({"runs": pipeline.list_runs()}) elif route == "/api/run/live": # Poll the live thought/movement stream of an in-flight (or finished) # run started via POST /api/run/start. ?run_id=...&since=. query = parse_qs(urlparse(self.path).query) run_id = (query.get("run_id") or [""])[0] try: since = int((query.get("since") or ["0"])[0]) except ValueError: since = 0 if not run_id: self._send_json({"error": "missing run_id"}, status=400) else: self._send_json(pipeline.live(run_id, since)) elif route.startswith("/api/runs/"): run_id = route[len("/api/runs/"):] detail = pipeline.get_run(run_id) if detail is None: self._send_json({"error": f"no such run: {run_id}"}, status=404) else: self._send_json(detail) elif route.startswith("/static/"): # Resolve safely under the static dir (no path traversal). target = (_STATIC_DIR / route[len("/static/"):]).resolve() if _STATIC_DIR.resolve() in target.parents: self._send_file(target) else: self._send_json({"error": "forbidden"}, status=403) else: self._send_json({"error": "not found"}, status=404) # ----- POST ---------------------------------------------------------- # def do_POST(self) -> None: route = urlparse(self.path).path # Emergency stop: signal the in-flight run to freeze the arm. Handled on a # SEPARATE thread (ThreadingHTTPServer) from the blocked /api/run request, # so it gets through while a run is mid-motion. if route == "/api/stop": self._send_json(pipeline.request_stop()) return if route not in ("/api/run", "/api/run/start"): self._send_json({"error": "not found"}, status=404) return length = int(self.headers.get("Content-Length", "0") or "0") try: payload = json.loads(self.rfile.read(length) or b"{}") except json.JSONDecodeError: self._send_json({"error": "invalid JSON body"}, status=400) return task = (payload.get("task") or "").strip() mode = payload.get("mode", "auto") if not task: self._send_json({"error": "missing 'task'"}, status=400) return # Non-blocking start: returns a run_id at once so the page can stream the # live reasoning via GET /api/run/live. The blocking /api/run is kept for # direct/programmatic callers (e.g. Claude Code) that want the final result. if route == "/api/run/start": self._send_json(pipeline.start_run_async(task, mode=mode)) return try: result = pipeline.run_task(task, mode=mode) except Exception as exc: # never crash the server on a bad run [truncated — 4807 more characters] ``` ### web/static/app.js ```javascript // limbic web console — vanilla JS, no framework. Drives both pages. function esc(s) { return String(s == null ? "" : s) .replaceAll("&", "&").replaceAll("<", "<").replaceAll(">", ">"); } const sleep = (ms) => new Promise(r => setTimeout(r, ms)); function badge(status) { const known = ["completed", "cannot_complete", "error", "stopped"]; const cls = known.includes(status) ? status : "unknown"; const label = (status || "unknown").replace("_", " "); return `${esc(label)}`; } function planHtml(plan) { if (!plan || !plan.length) return "

(no steps)

"; const items = plan.map(s => { const args = Object.entries(s.args || {}).map(([k, v]) => `${k}=${v}`).join(", "); return `
  • ${esc(s.primitive)}(${esc(args)})
    • `; }).join(""); return `
      ${items}
    `; } // A live reasoning event -> a friendly one-line (or block) entry. Covers the // brain's thought phases (orchestrator.py) and the movement stream, so the box // reads like a running narration of what the arm is thinking and doing. const THOUGHT_STYLE = { instruction: { icon: "📋", label: "Task" }, model_choice: { icon: "🧠", label: "Planner" }, verify_disabled:{ icon: "ℹ️", label: "Note" }, attempt: { icon: "🔄", label: "Attempt" }, plan: { icon: "✅", label: "Plan" }, execute: { icon: "⚙️", label: "Executing" }, reasoning: { icon: "💭", label: "Thinking", block: true }, message: { icon: "🗣️", label: "Says", block: true }, perceive: { icon: "👁️", label: "Senses" }, authoring: { icon: "🛠️", label: "New skill" }, plan_validated: { icon: "✅", label: "Plan ready" }, plan_rejected: { icon: "⚠️", label: "Plan rejected" }, refused: { icon: "🛑", label: "Refused" }, verify: { icon: "🔎", label: "Verify" }, stopped: { icon: "✋", label: "Stopped" }, cannot_complete:{ icon: "⚠️", label: "Cannot complete" }, error: { icon: "❌", label: "Error" }, }; function eventHtml(ev) { if (ev.channel === "movements") { const args = ev.requested || ev.target || {}; const detail = Object.entries(args) .map(([k, v]) => `${k}=${typeof v === "number" ? Math.round(v * 100) / 100 : v}`) .join(", "); return `
    🦾` + `Moves ${esc(ev.action || "move")}(${esc(detail)})
    `; } const style = THOUGHT_STYLE[ev.phase] || { icon: "•", label: ev.phase || "" }; let msg = ev.message || ""; // perceive: summarise the sensed objects rather than dumping the raw payload. if (ev.phase === "perceive" && ev.result && ev.result.reading) { const r = ev.result.reading; const objs = (r.objects || r || []); if (Array.isArray(objs) && objs.length) { msg += " → " + objs.map(o => o.label || JSON.stringify(o)).join(", "); } } const bodyCls = style.block ? "t-body t-block" : "t-body"; return `
    ${style.icon}` + `${esc(style.label)} ${esc(msg)}
    `; } // Append one live event to the feed. Streamed reasoning/text arrives as deltas // sharing a `stream_id`: the first delta creates a "typing" bubble, each later one // appends into it (so the user watches the thought form), and `partial:false` // finalises it (caret removed). Everything else renders as a single line. function appendEvent(log, ev) { const sid = ev.stream_id; if (sid && ev.channel !== "movements") { let el = log.querySelector(`.thought[data-stream="${sid}"]`); if (!el) { const style = THOUGHT_STYLE[ev.phase] || { icon: "💭", label: ev.phase || "" }; log.insertAdjacentHTML("beforeend", `
    ` + `${style.icon}` + `${esc(style.label)} ` + `
    `); el = log.querySelector(`.thought[data-stream="${sid}"]`); } const textEl = el.querySelector(".t-text"); if (textEl) textEl.textContent += (ev.message || ""); if (ev.partial === false) { el.classList.remove("streaming"); const caret = el.querySelector(".t-caret"); if (caret) caret.remove(); } return; } log.insertAdjacentHTML("beforeend", eventHtml(ev)); } // ----- Ask page ------------------------------------------------------------- function initAskPage() { const taskEl = document.getElementById("task"); const runBtn = document.getElementById("run"); const stopBtn = document.getElementById("stop"); const busy = document.getElementById("busy"); const resultEl = document.getElementById("result"); const thinkEl = document.getElementById("thinking"); const thinkLog = document.getElementById("thinking-log"); const thinkLive = document.getElementById("think-live"); const thinkMeta = document.getElementById("think-meta"); let polling = false; function resetThinking() { thinkLog.innerHTML = ""; thinkLive.classList.remove("hidden"); thinkMeta.textContent = ""; thinkEl.classList.remove("hidden"); } // Poll the live stream until the run is done, appending new events as they // arrive. Returns the final result object the server hands back on completion. async function streamLive(runId) { let since = 0; while (polling) { let data; try { data = await (await fetch(`/api/run/live?run_id=${encodeURIComponent(runId)}&since=${since}`)).json(); } catch (e) { await sleep(400); continue; } for (const ev of (data.events || [])) { appendEvent(thinkLog, ev); } if (data.events && data.events.length) { since = data.last_seq; thinkLog.scrollTop = th [truncated — 5701 more characters] ``` ### scripts/describe_machine.py ```python """Print a full profile of THIS machine: arch, arm serial port, cameras-by-name, and which capabilities (real arm / vision / kinematics) are installed here. Run this whenever the rig moves to a new computer — it's the "plug in, profile, go" check. Equivalent to `python -m limbic.platform_support`. python scripts/describe_machine.py """ from __future__ import annotations import pathlib import sys sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent)) from limbic.platform_support import format_profile if __name__ == "__main__": print(format_profile()) ``` ### limbic/__init__.py ```python """limbic — a cross-platform, LLM-driven control stack for a tabletop robot arm. Three layers, each its own subpackage: control/ The Body -- movement, gripper, guardrails (RobotArm) primitives/ The Skills -- reusable motion primitives the LLM chains inputs/ The Senses -- motor + camera readings the LLM can query brain/ The Mind -- turns an instruction into a list of primitive calls Everything runs on macOS, Windows and Linux, with no physical arm required: the control layer auto-falls-back to a software mock so the whole pipeline is develop-and-test-anywhere. Quick start: from limbic import RobotArm with RobotArm() as arm: # auto: real arm if present, else mock arm.go_home() arm.move_to_xyz(180, 0, 60) arm.close_gripper() """ from . import runlog from .control import RobotArm, load_config __all__ = ["RobotArm", "load_config", "runlog"] __version__ = "0.1.0" ``` ### scripts/go_home.py ```python """Move the arm to HOME — every motor centred (all joints 0 deg, gripper halfway). This is what "go home" means on this rig: each motor at the middle of its range, a known neutral pose (handy before powering down, tightening hardware, or re-zeroing). Connects the real SO-101 (bronny, calibrate=False), drives there smoothly, holds (torque engaged), and disconnects without going limp. Safety: BARREL-JACK power only. The arm sweeps up/forward to the centred pose — keep the workspace clear. python scripts/go_home.py """ from __future__ import annotations import os import pathlib import sys sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent)) def calibration_path(robot_id: str) -> pathlib.Path: root = os.environ.get("HF_LEROBOT_HOME") or ( pathlib.Path.home() / ".cache" / "huggingface" / "lerobot" ) return pathlib.Path(root) / "calibration" / "robots" / "so_follower" / f"{robot_id}.json" def main() -> None: from limbic.control.arm import RobotArm from limbic.control.backends import RealBackend from limbic.control.config import load_config robot_id = os.environ.get("LIMBIC_ROBOT_ID", "bronny") cal = calibration_path(robot_id) if not cal.exists(): raise SystemExit( f"No calibration file for id {robot_id!r} at {cal}. Aborting " "(connecting without it would launch interactive calibration)." ) cfg = load_config() if cfg.port is None: raise SystemExit("No serial port found. Plug in the arm or set $LIMBIC_PORT.") print(f"Connecting real SO-101 on {cfg.port} as {robot_id!r}; moving to HOME " "(all motors centred)...") arm = RobotArm(config=cfg, backend=RealBackend(cfg), verbose=True) arm.connect() try: joints = arm.go_home() print("\nAt HOME. Joint readings (deg; gripper 0..100):") for name, val in joints.items(): print(f" {name:14s} {val:8.2f}") finally: arm.disconnect() print("Disconnected (torque left engaged; arm holds its pose).") if __name__ == "__main__": main() ``` ### scripts/workspace_view.py ```python """Side-by-side workspace check for both rig cameras (Part B). Shows CAM_B (LEFT) and CAM_A (RIGHT) live, each with the gray-mat WORKSPACE highlighted: the mat is tinted green + outlined yellow, everything off it is dimmed. This is the visual confirmation that what we consider the workspace matches the real mat in both views before we start filtering detections to it. The black arm occluding the mat is fine — black isn't "gray", and the mat fill covers such holes, so the workspace stays whole. Usage: python scripts/workspace_view.py Hotkeys: ESC / q : quit """ from __future__ import annotations import pathlib import sys import cv2 as cv import numpy as np sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent)) from limbic.control import calibration from limbic.platform_support import open_camera from limbic.vision.workspace import gray_mat_mask, highlight PANEL_W, PANEL_H = 640, 360 FULL_W, FULL_H = 1280, 720 ORDER = ["B", "A"] # left panel = LEFT cam (B), right = RIGHT cam (A) def main() -> None: caps = {} for r in ORDER: caps[r] = open_camera(calibration.CAMERAS[r]["name"], width=FULL_W, height=FULL_H) win = "Workspace (gray mat) | ESC quit" cv.namedWindow(win, cv.WINDOW_NORMAL) print("Showing the detected workspace on both cameras. ESC to quit.") try: while True: canvas = np.zeros((PANEL_H, PANEL_W * 2, 3), np.uint8) for i, role in enumerate(ORDER): ok, frame = caps[role].read() if not ok or frame is None: frame = np.zeros((FULL_H, FULL_W, 3), np.uint8) cv.putText(frame, f"CAM_{role}: no frame", (40, 80), cv.FONT_HERSHEY_SIMPLEX, 1.5, (0, 0, 255), 3) else: mask, contour = gray_mat_mask(frame) frame = highlight(frame, mask, contour) disp = cv.resize(frame, (PANEL_W, PANEL_H)) canvas[0:PANEL_H, i * PANEL_W:(i + 1) * PANEL_W] = disp side = calibration.CAMERAS[role]["side"] label = f"CAM_{role} {side}" cv.putText(canvas, label, (i * PANEL_W + 12, 28), cv.FONT_HERSHEY_SIMPLEX, 0.8, (0, 0, 0), 4) cv.putText(canvas, label, (i * PANEL_W + 12, 28), cv.FONT_HERSHEY_SIMPLEX, 0.8, (255, 255, 0), 2) cv.line(canvas, (PANEL_W, 0), (PANEL_W, PANEL_H), (60, 60, 60), 1) cv.imshow(win, canvas) if (cv.waitKey(1) & 0xFF) in (27, ord("q")): break finally: for c in caps.values(): c.release() cv.destroyAllWindows() if __name__ == "__main__": main() ``` [76 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]