# Project export: Robosurge This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Cursor for surgery. - Devpost: https://devpost.com/software/we-use-nix - GitHub: https://github.com/quantum9Innovation/treehacks2026 - Video: https://www.youtube.com/embed/hmrBUtiqueg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Treehacks Grand Prize (2nd)) - Team: 3 GitHub contributor(s) — Ananth (42 commits), Edward Li (24 commits), Claude Opus 4.6 (9 commits) ## Devpost submission (written by the team) ### Inspiration The United States is facing a growing shortage of surgical specialists, projected to reach nearly 20,000 by 2036. Additionally, operating rooms cost roughly $2,200 per hour per surgeon. Today’s robotic surgical robots greatly improve precision by scaling and filtering human motion. But they do not automate surgical tasks. This is due to a lack of reliable AI spatial vision understanding and AI robot communication. Instead of making surgeries more accessible, current robots only make them more complex and costly. What Robosurge does Robosurge is a remotely controlled integrated surgery platform. It allows for a single surgeon to perform entire complex procedures autonomously from anywhere in the world. In traditional surgeries, an attending surgeon commands a large team of assistant surgeons. The attending surgeon syncs to Robosurge’s camera arm via a Meta Quest 3 VR. Then the surgeon can command an array of magnetically controlled robotic arms using natural language. This architecture reduces required personnel from 10 -> 1, and distance to patient from meters to anywhere around the world. ### How we built it We built the entire system from the ground up, from the basic kinematics systems to the frontend web control panel. Inverse Kinematic Control System: We built a custom inverse kinematic control system for the robot arms, bypassing limitations of the provided SDK and improving speed while adding safety guarantees. Custom Depth and Coordinate Mapping: We built our own coordinate resolution system using an Intel RealSense Depth Camera and ArUCo tags for consistent motion across all four arms. We then perform an affine transformation between the coordinate space of the camera with the coordinate space of the robot arm to give precise control to the robots. Multilayer AI Computation: We use both Meta’s Segment Anything Model (SAM) and YOLOv11 on edge and an ASUS Ascent GX10 to identify and segment objects. We also perform LLM fusion between GPT-5.2 and Gemini Robotics-ER 1.5 for two levels of visual understanding as well as individual move orchestration. Realtime VR Position Mapping: Using real-time orientation data from a Meta Quest 3, we are able to have a live head track with a physical moving camera on a robot arm. Built with a websocket server, a real-time MJPEG video stream, and a custom rotation translation algorithm Dynamic Frontend: A TanStack and FastAPI powered web app supplies real-time video feeds using WebRTC, with WebSocket and REST API for real-time diagnostics and controls. Reproducible Builds: By using Nix in both our production and development environments, we are able to guarantee reproducible builds of our robot control systems using a purely functional package manager which operates the same regardless of system state. Theorem Proving and Formal Verification: We used dependent type theory in Lean to prove that trajectories are within safe physical bounds while tracing out the shape of a lemniscate, sinusoidal longitudinal wave, vertical spring (evolving according to Hooke’s law), and helix. We also created a custom solver for systems of autonomous differential equations using the Runge-Kutta method, used to trace trajectories in the Lorenz and Rössler attractors. (btw this entire portion was written with NO AI) Challenges We Ran Into Our primary challenge was navigating a high-intensity 36-hour learning curve; despite having no prior robotics background, our team successfully mastered multi-arm kinematics and hardware integration. Lacking specialized 3D scanning hardware, we engineered a precise coordinate positioning stack using ArUco markers and custom affine transformations to map the Intel RealSense 3D feed directly to the robotic joint-space. We also addressed the inherent 3D spatial reasoning limitations of current LLMs by developing robust grounding harnesses to translate high-level natural language intent into reliable physical actions. We also had to deal with precision issues when attempting to trace out famously chaotic systems, which required us to implement Runge-Kutta (RK4) numerical methods within Lean. The varied backgrounds of our team and our different approaches to tackling these issues led to the three-part development of our codebase, with a primary focus on agents, a secondary focus on augmented reality, and a tertiary focus on formal verification and theorem proving. ### Accomplishments we're proud of As a team of first-time hardware engineers, we are really proud of what we have built. In 36 hours, we built a full inverse kinematic system, 3D depth mapping, and agentic robot control. We successfully designed, cut, and assembled a custom robotic frame and 3D CAD model to house our four-arm station. This is also very likely one of the first times the Lean theorem prover has been used so extensively in a project of this scale at any hackathon. (oh yeah, did we mention we use Nix, btw?) ### What we learned We learned how to work extensively with computer vision, formal verification, and hardware-software integration, all of which were incredibly challenging yet rewarding. We discovered that integrating VLMs with physical hardware involves a steep learning curve, especially in building a 12-DOF coordinate system that allows an AI to see a pixel and touch a target perfectly. ### What's next Firstly, we want to integrate haptic feedback into the VR controllers so surgeons can feel another layer of information, e.g. actually feel tissue resistance as they move the robotic arms. Secondly, we plan to expand the system to support a multiple-control copilot-based model where a single surgeon can oversee multiple operations, where we can gain further efficiency by parallelizing the entire process. Finally, we hope to move to high-fidelity medical simulations and complex anatomical models. When robots start performing the bulk of the grunt work actually involved in an operation, procedures can be simulated rapidly and accurately, decreasing the failure rates of operations through better training. ## README (from the GitHub repository) # TreeHacks 2026 Agentic surgery on a grape, among other things ## Detected evidence (automated analysis) Indexed codebase: 105 recognized source files, 492 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers - AI coding agent: Cursor — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 157) ``` .env.example .gitignore .pre-commit-config.yaml .python-version agent/__init__.py agent/agent_wrapper.py agent/agent.py agent/calibration_data_ARM1.json agent/calibration_data_ARM2.json agent/calibration_data_ARM3.json agent/calibration_data.json agent/calibration.py agent/camera.py agent/coordinate_transform.py agent/gemini_vision.py agent/llm.py agent/models.py agent/prompts.py agent/run.py agent/tools.py agent/vision/__init__.py agent/vision/base.py agent/vision/mock_backend.py agent/vision/sam2_backend.py agent/vision/yolo_backend.py CLAUDE.md control/.envrc control/ctltest.py control/flake.lock control/flake.nix control/main.py control/read.py control/realsense_test.py control/sockets.py control/trajectories.py control/trajectories/helix.json control/trajectories/lemniscate.json control/trajectories/lorenz.json control/trajectories/rossler.json control/trajectories/spring.json control/trajectories/springShift.json control/trajectories/springShift1.json control/trajectories/springShift2.json control/trajectories/springShift3.json control/trajectories/wave.json flake.lock flake.nix lean/.envrc lean/flake.lock lean/flake.nix lean/lakefile.lean lean/lakefile.toml lean/lean-toolchain lean/Main.lean lean/Math.lean lean/Trajectories.lean lean/Types.lean lean/Utils.lean LICENSE motion_controller/__init__.py motion_controller/.gitignore motion_controller/.python-version motion_controller/CLAUDE.md motion_controller/heightmap.py motion_controller/main.py motion_controller/motion.py motion_controller/pyproject.toml motion_controller/README.md motion_controller/testing_scripts/diagonal_sweep.py motion_controller/testing_scripts/random_points.py motion_controller/uv.lock MULTIAGENT_PLAN.md pyproject.toml pyrightconfig.json README.md ruff.toml server/__init__.py server/app.py server/config.py server/events.py server/hardware.py server/routers/__init__.py server/routers/agent.py server/routers/arm.py server/routers/calibration.py server/routers/camera.py server/routers/vision.py server/streams.py uv.lock vr_control/__init__.py vr_control/arm_client.py web/.cta.json web/.cursorrules web/.gitignore web/.vscode/settings.json web/components.json web/index.html web/package.json web/public/manifest.json web/public/robots.txt web/README.md web/src/components/calibration/calibration-wizard.tsx web/src/components/camera/camera-feed.tsx web/src/components/camera/camera-toolbar.tsx web/src/components/camera/grid-overlay.tsx web/src/components/chat/chat-container.tsx web/src/components/chat/chat-message.tsx web/src/components/control/arm-selector.tsx web/src/components/control/connect-all.tsx web/src/components/control/emergency-stop.tsx web/src/components/control/gripper-control.tsx web/src/components/control/movement-grid.tsx web/src/components/control/position-readout.tsx web/src/components/control/probe-ground.tsx web/src/components/control/step-size-selector.tsx web/src/components/display/video-panel.tsx web/src/components/layout/sidebar-nav.tsx web/src/components/layout/status-bar.tsx web/src/components/ui/alert.tsx web/src/components/ui/badge.tsx [37 more files omitted for size] ``` ### Dependencies - motion_controller/pyproject.toml: roarm-sdk@>=0.1.0 - pyproject.toml: aiortc@>=1.12.0, av@>=14.0.0, fastapi@>=0.115.0, google-genai@>=1.63.0, huggingface-hub@>=0.20, numpy@>=2.4.2, openai@>=2.21.0, opencv-python@>=4.13.0.92, pillow@>=12.1.1, pydantic-settings@>=2.0, pyrealsense2@>=2.56.5.9235, python-dotenv@>=1.2.1, roarm-sdk@>=0.1.0, sam2@>=1.0, torch@>=2.0, torchvision@>=0.15, uvicorn[standard]@>=0.34.0 - web/package.json: @tailwindcss/vite@^4.1.18, @tanstack/react-router@^1.132.0, @tanstack/react-router-devtools@^1.132.0, @tanstack/router-plugin@^1.132.0, @testing-library/dom@^10.4.0, @testing-library/react@^16.2.0, @types/node@^22.10.2, @types/react@^19.2.0, @types/react-dom@^19.2.0, @vitejs/plugin-react@^5.0.4, class-variance-authority@^0.7.1, clsx@^2.1.1, jsdom@^27.0.0, lucide-react@^0.561.0, radix-ui@^1.4.3, react@^19.2.0, react-dom@^19.2.0, tailwind-merge@^3.0.2, tailwindcss@^4.1.18, tw-animate-css@^1.3.6, typescript@^5.7.2, vite@^7.1.7, vite-tsconfig-paths@^5.1.4, vitest@^3.0.5 ### Recent commits (newest first) - Merge remote-tracking branch 'origin/vr-controller' - VR Controller code - Fixed agent server code' - FEATgit status! Multiarm Callibration - feat: add display view with 2x2 info grid (color + depth feeds) - fix: wire camera color/depth toggle to WebRTC track switching - Add multi-arm device selector and per-arm ground probe state - fix: allow zeroing over ssh in vr - feat: go! forever ... - feat: finally finish trajectory planning! - fix: rossler wasnt working bc im dumb - Improve webcam MJPEG streaming performance - udp stream and stuff - docs(readme): revise - refactor(style): desloppify (partially) - Replace manual look() tool with auto-injected camera frames per turn - feat: speed up some more - feat: speed up sine wave - feat: add FastAPI server and React web frontend for robot control - fix: trajectories again ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Context7 Always use Context7 MCP when I need library/API documentation, code generation, setup or configuration steps without me having to explicitly ask. ## Self-Improvement When you make a mistake or learn something important during a session, update CLAUDE.md with the lesson so you don't repeat it in future sessions. ## Project Overview Vision-driven robot arm control system for the **RoArm M2** (4-DOF). Multiple LLM agent versions orchestrate a perception-action loop: camera captures scene, vision model detects/segments objects, LLM reasons about actions, motion controller executes movements. ## Commands ```bash uv run vlm-agent # Run agent (multi-provider: OpenAI or Gemini) uv run vlm-agent --calibrate # Run camera-to-arm calibration uv run vlm-agent --provider gemini # Use Gemini ER provider uv add # Add dependency ``` **Linting/formatting**: ```bash uv run ruff check . # Lint uv run ruff format . # Auto-format uv run pyright # Type check ``` Ruff ignores: F403, F405, E731. Pyright ignores missing imports. ## Architecture ### Agent (`agent/`) Consolidated agent package. Multi-provider LLM orchestrator (OpenAI GPT or Gemini ER) with dual vision (SAM2 + Gemini ER). The agent calls tools (`detect`, `segment`, `goto_pixel`, `move_to_xyz`, `gripper_ctrl`, etc.) via structured function calling. Auto-injects camera frames each turn. Key files: - **`agent.py`**: CLI orchestrator (`AgentV3` class) — hardware init, agent loop, interactive REPL - **`agent_wrapper.py`**: Web orchestrator (`WebAgentV3`) — wraps agent for web UI with EventBus - **`llm.py`**: LLM provider abstraction (`OpenAIProvider`, `GeminiProvider`), `create_openai_client`, `ConfirmationHandler` - **`tools.py`**: Tool declarations (provider-agnostic) - **`prompts.py`**: System/task prompt generation - **`camera.py`**: `RealSenseCamera` wrapper - **`gemini_vision.py`**: Gemini ER vision (detect, point, segment) - **`run.py`**: CLI entry point (`uv run vlm-agent`) ### Vision Pipeline `camera → SAM2/Gemini ER detection → depth enrichment (RealSense) → coordinate transform → arm coordinates` - **Vision backends** (`agent/vision/`): Abstract `VisionBackend` interface with SAM2 (default), YOLO, and mock implementations. - **Coordinate transform** (`agent/coordinate_transform.py`): Pixel (x,y) + depth → camera 3D → arm 3D via calibration matrix. - **Calibration** (`agent/calibration.py`): Interactive 8-point procedure using least-squares affine solver. Saved to `calibration_data.json`. RMSE < 20mm is good. - **Data model** (`agent/models.py`): `DetectedObject`, `SceneState`, `Point3D` flow through the pipeline. ### Motion Controller (`motion_controller/`) Ground-relative coordinate system (Z=0 at detected ground, positive up). Key classes/methods in `motion_controller/motion. [truncated — 7169 more characters] ``` ### MULTIAGENT_PLAN.md ```markdown # Multi-Agent System for 3-Arm Collaborative Manipulation ## Context The robot has 4 physical arms (`/dev/ARM0`-`ARM3`), a shared Intel RealSense D435 camera, and an existing single-arm agent (`agent_v3/`). The server already supports multiple simultaneous arm connections with per-device Motion instances and asyncio locks (`server/hardware.py`), but the **agent is a singleton** and the **single-threaded `_hw_executor` prevents true parallel arm motion**. We need a hierarchical multi-agent system where a coordinator LLM plans collaborative tasks and dispatches subtasks to 3 per-arm worker agents that execute in parallel. ## Architecture ``` ┌─────────────────────────┐ │ Coordinator Agent │ │ (scene understanding, │ │ task planning, sync) │ └───┬───────┬───────┬──────┘ │ │ │ ┌─────▼──┐ ┌──▼────┐ ┌▼──────┐ │Worker 0│ │Worker 1│ │Worker 2│ │ (ARM0) │ │ (ARM1) │ │ (ARM2) │ └───┬────┘ └───┬────┘ └───┬────┘ │ │ │ ┌───▼────┐ ┌───▼────┐ ┌───▼────┐ │Motion 0│ │Motion 1│ │Motion 2│ └────────┘ └────────┘ └────────┘ ``` - **Coordinator**: LLM agent (GPT/Gemini via existing `LLMProvider`) that sees the camera feed, detects objects, and assigns subtasks to arms. It orchestrates via phases — parallel subtasks within a phase, barriers between phases. - **Workers**: Lightweight LLM agents (one per arm) that execute a single subtask using motion/vision tools. Reuse `WebAgentV3`'s tool executor methods but bound to a specific device. ## Implementation Plan ### Step 1: Enable parallel arm motion in `server/hardware.py` Currently all arm commands serialize through one thread (`_hw_executor`). Add per-device executors so arms can move simultaneously. **Changes**: - Add `_arm_executors: dict[str, ThreadPoolExecutor]` to `HardwareManager` - Create a per-device executor in `_connect_arm_sync()`, shut it down in `_disconnect_arm_sync()` - Add `run_in_arm_thread(device, fn, *args)` method - Keep `run_in_hw_thread` for non-arm operations (camera init, etc.) ### Step 2: Per-device coordinate transforms in `server/hardware.py` Currently `self.ct` is shared and `load_calibration_for_device()` overwrites the single matrix. For parallel operation, each arm needs its own `CoordinateTransform`. **Changes**: - Add `_per_device_ct: dict[str, CoordinateTransform]` to `HardwareManager` - In `_connect_arm_sync()`, create a new `CoordinateTransform` per device with device-specific calibration - Add `get_ct(device)` accessor method - Keep `self.ct` as the active-device shortcut for backward compat ### Step 3: Create `agent_multiarm/` package with data models New package: `agent_multiarm/` **`agent_multiarm/models.py`** — Core data structures: - `CoordinatorState` [truncated — 6795 more characters] ``` ### pyproject.toml ``` [project] name = "treehacks2026" version = "0.1.0" description = "TreeHacks 2026 project" readme = "README.md" license = "MIT" requires-python = ">=3.12" dependencies = [ "google-genai>=1.63.0", "huggingface-hub>=0.20", "numpy>=2.4.2", "openai>=2.21.0", "opencv-python>=4.13.0.92", "pillow>=12.1.1", "pyrealsense2>=2.56.5.9235", "python-dotenv>=1.2.1", "roarm-sdk>=0.1.0", "sam2>=1.0", "torch>=2.0", "torchvision>=0.15", "fastapi>=0.115.0", "uvicorn[standard]>=0.34.0", "aiortc>=1.12.0", "av>=14.0.0", "pydantic-settings>=2.0", ] [project.scripts] treehacks2026 = "main:main" vlm-agent = "agent.run:main" roarm-server = "server.app:main" [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src", "agent", "motion_controller", "server"] [[tool.uv.index]] name = "local-wheels" url = "./wheels" format = "flat" [dependency-groups] dev = [ "mypy>=1.0", "pyright>=1.1.408", "pytest>=8.0", "ruff>=0.15.1", ] #no-binary-package = ["pyrealsense2"] ``` ### motion_controller/pyproject.toml ``` [project] name = "trees" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.11" dependencies = [ "roarm-sdk>=0.1.0", ] ``` ### web/package.json ``` { "name": "web", "private": true, "type": "module", "scripts": { "dev": "vite dev --port 3000 --host", "build": "vite build", "preview": "vite preview", "test": "vitest run" }, "dependencies": { "@tailwindcss/vite": "^4.1.18", "@tanstack/react-router": "^1.132.0", "@tanstack/react-router-devtools": "^1.132.0", "@tanstack/router-plugin": "^1.132.0", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.561.0", "radix-ui": "^1.4.3", "react": "^19.2.0", "react-dom": "^19.2.0", "tailwind-merge": "^3.0.2", "tailwindcss": "^4.1.18", "tw-animate-css": "^1.3.6", "vite-tsconfig-paths": "^5.1.4" }, "devDependencies": { "@testing-library/dom": "^10.4.0", "@testing-library/react": "^16.2.0", "@types/node": "^22.10.2", "@types/react": "^19.2.0", "@types/react-dom": "^19.2.0", "@vitejs/plugin-react": "^5.0.4", "jsdom": "^27.0.0", "typescript": "^5.7.2", "vite": "^7.1.7", "vitest": "^3.0.5" } } ``` ### motion_controller/main.py ```python def main(): print("Hello from trees!") if __name__ == "__main__": main() ``` ### server/app.py ```python """FastAPI application for robot arm web control.""" import logging import signal from contextlib import asynccontextmanager import uvicorn from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from .config import Settings from .events import EventBus from .hardware import HardwareManager logger = logging.getLogger("server") @asynccontextmanager async def lifespan(app: FastAPI): """Initialize hardware on startup, clean up on shutdown.""" settings: Settings = app.state.settings bus = EventBus() hw = HardwareManager( arm_devices=settings.arm_devices, sam2_model=settings.sam2_model, sam2_device=settings.sam2_device, calibration_path=settings.calibration_path, mock=settings.mock_hardware, enable_sam2=settings.enable_sam2, enable_gemini_vision=settings.enable_gemini_vision, google_api_key=settings.google_api_key, ) app.state.hardware = hw app.state.event_bus = bus await hw.start() logger.info("Hardware initialized") yield # Close WebRTC peer connections first from .routers.camera import _pcs for pc in list(_pcs): try: await pc.close() except Exception: pass _pcs.clear() await hw.shutdown() logger.info("Hardware shut down") def create_app(settings: Settings | None = None) -> FastAPI: """Create and configure the FastAPI application.""" if settings is None: settings = Settings() app = FastAPI( title="RoArm Control Server", version="0.1.0", lifespan=lifespan, ) app.state.settings = settings # CORS app.add_middleware( CORSMiddleware, allow_origins=settings.cors_origins, allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Mount routers from .routers.agent import router as agent_router from .routers.arm import router as arm_router from .routers.calibration import router as calibration_router from .routers.camera import router as camera_router from .routers.vision import router as vision_router app.include_router(arm_router) app.include_router(camera_router) app.include_router(vision_router) app.include_router(agent_router) app.include_router(calibration_router) @app.get("/api/health") async def health(): return {"status": "ok"} return app def main(): """Entry point for `uv run roarm-server`.""" import argparse parser = argparse.ArgumentParser(description="RoArm Control Server") parser.add_argument("--mock", action="store_true", help="Run with mock hardware") args = parser.parse_args() logging.basicConfig( level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s", ) logging.getLogger("aiortc").setLevel(logging.WARNING) logging.getLogger("aioice").setLevel(logging.WARNING) settings = Settings() if args.mock: settings.mock_hardware = True app = create_app(settings) # Force exit if graceful shutdown hangs (e.g. blocked hardware threads) import os import threading _original_sigint = signal.getsignal(signal.SIGINT) _shutting_down = False def _force_exit(signum, frame): nonlocal _shutting_down if _shutting_down: logger.warning("Second Ctrl-C, forcing exit") os._exit(1) _shutting_down = True # Watchdog: force-kill after 5s if graceful shutdown hangs def _watchdog(): import time time.sleep(5) logger.warning("Shutdown timed out after 5s, forcing exit") os._exit(1) threading.Thread(target=_watchdog, daemon=True).start() # Forward first SIGINT to uvicorn for graceful shutdown if callable(_original_sigint): _original_sigint(signum, frame) signal.signal(signal.SIGINT, _force_exit) uvicorn.run(app, host=settings.host, port=settings.port) ``` ### control/main.py ```python import curses import glob import math import sys import time from typing import Callable, NewType from roarm_sdk.roarm import roarm from read import load SERIAL_GLOB_PATTERNS = [ "/dev/cu.usbserial-*", "/dev/cu.usbmodem*", "/dev/ttyUSB*", "/dev/ttyACM*", "/dev/ARM*", ] speed: float = 50 turning: float = 10 skip: bool = False loop: bool = True # trajectories: list[str] = sorted(glob.glob("trajectories/*.json")) trajectories: list[str] = ["trajectories/wave.json"] Pose = NewType("Pose", tuple[float, float, float, float]) clip: Callable[[float, float, float], float] = lambda x, a, b: max(min(x, b), a) def move_up(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x, y, z + speed, t)) def move_down(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x, y, z - speed, t)) def move_left(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x, y + speed, z, t)) def move_right(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x, y - speed, z, t)) def move_forward(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x + speed, y, z, t)) def move_backward(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x - speed, y, z, t)) def open(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x, y, z, t + turning)) def close(pose: Pose) -> Pose: x, y, z, t = pose return Pose((x, y, z, t - turning)) def correct(pose: Pose) -> Pose: x, y, z, t = pose x = clip(x, -490, 490) y = clip(y, -490, 490) z = clip(z, 0, 490) t = clip(t, 0, 90) if math.hypot(x, y, z) > 490: x = 250 y = 0 z = 250 return Pose((x, y, z, t)) def detect_serial_port(): devices = [] for pattern in SERIAL_GLOB_PATTERNS: devices.extend(glob.glob(pattern)) if len(devices) == 1: return devices[0] elif len(devices) > 1: print(f"Multiple serial devices found: {devices}") print("Please specify one with --port") else: return None def main(stdscr): stdscr.nodelay(True) # ports = ["/dev/ttyUSB0", "/dev/ttyUSB1", "/dev/ttyUSB2", "/dev/ttyUSB3"] ports = ["/dev/ttyUSB0", "/dev/ttyUSB1"] # ports = [detect_serial_port()] if len(ports) == 0: print("Error: No USB serial device found. Connect the arm or use --port.") sys.exit(1) print(f"Connecting to RoArm-M2 on {ports}...") arms = [roarm(roarm_type="roarm_m2", port=port, baudrate=115200) for port in ports] for arm in arms: arm.echo_set(0) arm.torque_set(1) print("Moving to home position...") for arm in arms: arm.move_init() time.sleep(1) pose = Pose((250, 0, 250, 0)) delay: float = 0.02 try: if loop: while True: stream = load(trajectories[0])["trajectory"] local_delay = load(trajectories[0])["delay"] for x, y, z, t in stream: pose = Pose((x, y, z, t)) for arm in arms: arm.pose_ctrl([x, y, z, t]) time.sleep(local_delay) if not skip: for trajectory in trajectories: stream = load(trajectory)["trajectory"] local_delay = load(trajectory)["delay"] for x, y, z, t in stream: pose = Pose((x, y, z, t)) for arm in arms: arm.pose_ctrl([x, y, z, t]) time.sleep(local_delay) pose = Pose((250, 0, 250, 0)) while True: x, y, z, t = pose for arm in arms: arm.pose_ctrl([x, y, z, t]) time.sleep(delay) try: key = stdscr.getkey() except curses.error: continue if key == "w": pose = move_up(pose) if key == "a": pose = move_left(pose) if key == "s": pose = move_down(pose) if key == "d": pose = move_right(pose) if key == "q": pose = move_backward(pose) if key == "e": pose = move_forward(pose) if key == "r": pose = open(pose) if key == "f": pose = close(pose) pose = correct(pose) except KeyboardInterrupt: for arm in arms: arm.move_init() print("Keyboard interrupt") curses.wrapper(main) ``` ### web/src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import { RouterProvider } from '@tanstack/react-router' import { router } from './router' createRoot(document.getElementById('root')!).render( , ) ``` ### web/src/routes/index.tsx ```typescript import { createFileRoute } from '@tanstack/react-router' import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card' import { CameraFeed } from '@/components/camera/camera-feed' import { MovementGrid } from '@/components/control/movement-grid' import { GripperControl } from '@/components/control/gripper-control' import { PositionReadout } from '@/components/control/position-readout' import { StepSizeSelector } from '@/components/control/step-size-selector' import { ProbeGround } from '@/components/control/probe-ground' import { ArmSelector } from '@/components/control/arm-selector' import { ConnectAll } from '@/components/control/connect-all' import { useKeyboardControls } from '@/lib/hooks/use-keyboard-controls' import { useWebSocket } from '@/lib/hooks/use-websocket' import { useState } from 'react' export const Route = createFileRoute('/')({ component: ControlPage, }) function ControlPage() { useWebSocket() useKeyboardControls(true) const [clickInfo, setClickInfo] = useState<{ pixel_x: number; pixel_y: number } | null>(null) const [segResult, setSegResult] = useState<{ arm_coordinates: { x: number; y: number; z: number } | null score: number } | null>(null) return (
{/* Camera panel */}
{/* Controls panel */}
Arm Position
Movement
Gripper {/* Debug info */} {(clickInfo || segResult) && ( Debug {clickInfo && (

Click: ({clickInfo.pixel_x}, {clickInfo.pixel_y})

)} {segResult && ( <>

Score: {segResult.score.toFixed(3)}

{segResult.arm_coordinates && (

Arm: ({segResult.arm_coordinates.x.toFixed(0)},{' '} {segResult.arm_coordinates.y.toFixed(0)},{' '} {segResult.arm_coordinates.z.toFixed(0)})

)} )} )}
) } ``` [94 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]