# Project export: Hardware Context Protocol (MCP for Hardware) This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Giving AI a physical embodiment for the world. Control multiple hardware platforms cooperatively via LLM prompts. Deployed with a custom voice-activated autonomous chef robot. - Devpost: https://devpost.com/software/hardware-context-protocol - GitHub: https://github.com/danielzyy/calhacks2025 - Video: https://www.youtube.com/embed/Zj-G-9aPG8w?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Crater: Play-Do Prize; Cal Hacks: Best Hardware Hack) - Team: 4 GitHub contributor(s) — Sahil Kale (53 commits), SuperMK15 (25 commits), Penguronik (10 commits), Daniel Ye (7 commits) ## Devpost submission (written by the team) ### Overview Model Context Protocol (MCP) revolutionized the capabilities of LLMs by offering a radically new way of creating software for our everyday lives. Missing from this was meaningful capability to interact with the real world beyond a screen, and inspired us to develop our Hardware Context Protocol (HCP). It enables LLMs to control hardware components via well-defined exposed interfaces, allowing AI to interact with the world through reading sensor data, commanding actuators, and bringing them all together with context. The problem that inspired us was household serving (or rather, the lack of an intelligent machine to do so) - think making salads, finding snacks. We proved that our HCP SDK is a fast and iterable way to bring intelligence into the homes of everyday individuals by integrating its functionality into a robot arm. ### What it does We created an SDK that allows the user to create a minimal JSON-based definition of a certain hardware node and what actions it exposes to the LLM. This then bootstraps a custom TCP networking layer deployable on the hardware node compliant with our novel HCP interaction layer. The LLM goes through an initial discovery period where any HCP-bootstrapped TCP nodes are able to subscribe to it and pass relevant context in the HCP-defined schema. Sensors, like a camera, can detect the locations of objects and items of interest within their sensing frame, exposing them to the model; these can then be used to inform control actions orchestrated by the HCP. ### Challenges we ran into Architecting HCP to be a scalable architecture that would cleanly integrate a fully in-software LLM to real-life hardware. Exposing generic hardware interfaces on actuator interfaces (such as the demo'd SOARM101) to be controllable by the LLM, developing inverse kinematics and smooth path planning for safe operation of the robot arm as well. ### Accomplishments we're proud of Creating a fully-fledged SDK that we were able to use to bootstrap demo hardware nodes at near-zero ramp-up or integration cost. Creating a fully custom TCP networking stack under a publish/subscribe model that was HCP-compatible and allowed seamless multi-node orchestration. Creating a seamless agentic loop between pure-software LLMs and distributed hardware nodes, both for sensing and environment manipulation. ### What we learned Building HCP taught us that bridging the digital reasoning power of LLMs with the physical world requires both a solid networking backbone and a thoughtful interface abstraction. We learned how critical it is to design schemas that balance simplicity with extensibility. This enables both small embedded devices and complex robotic systems to integrate seamlessly. We also realized that context management is the heart of intelligent automation: giving LLMs structured, real-time situational awareness unlocks emergent problem-solving behaviours that feel almost human. The goal of the SOARM-101 integration is to provide an actuation platform that can be controlled by the HCP SDK. While the HCP SDK provides an authoritative interface for exposing a hardware node's capabilities, integrating a robotic arm required significant effort. The robotic arm provides control functionality to move to a desired position and grab and move objects. LLM interprets the user’s prompt about what they want to eat and moves the arm to pick up the desired ingredients to make a suitable dish. 3DOF End-Effector Commanding In robotic manipulation challenges, the goal is typically to track an end-effector position in the world. However, the SOARM-100 only exposed an API to command joint angles, requiring us to develop our own single-link-chain inverse kinematics model. The solver can solve for multiple manipulator configurations (elbow up/down). The advantage of this approach is that it generalizes the HCP's interaction with the arm (fully in world-frame space), thereby enhancing the control system's generalizability. The development of the model required implementing a custom simulator to validate that null-space solutions are rejected, as well as to provide a sanity check that the forward and inverse kinematics models yield sane results. A figure of the simulator is shown below and has proved to be helpful in debugging and iterating off-target. Camera and Vision Detection A camera was used to identify the locations of each object for the robot to interact with, using AprilTags and OpenCV. It exposes each object's type and position to the HCP layer so the LLM can guide the robot to interact with them. This approach of publishing generic data can be easily expanded to any type of sensor in the future, to help the model gain a better understanding of its environment. ### What's next for Hardware Context Protocol Our next step is to expand the HCP SDK into a fully modular ecosystem that supports standardized drivers, cloud-based orchestration, and real-time safety supervision layers. We hope to release open-source tools, including the ones developed at this very hackathon, for rapidly bootstrapping HCP-compatible nodes across a variety of microcontrollers. In the long term, we envision HCP serving as the foundation for a universal hardware abstraction layer for AI agents, enabling LLMs to intuitively control everything from IoT devices to full-scale industrial robotics, bringing physical intelligence to any environment. ## README (from the GitHub repository) # Hardware Context Protocol (calhacks2025) image ## Setup 1. Create a virtual environment (`python3 -m venv .venv`) 2. Activate the virtual environment (`source .venv/bin/activate` on MacOS/Linux, `.venv\Scripts\activate` on Windows) 3. Install system dependencies (`sudo apt-get install cmake build-essential pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libswresample-dev libavfilter-dev pkg-config`) 4. Install Python dependencies (`pip install -r requirements.txt`) ## Good resources - [LeRobot Installation](https://huggingface.co/docs/lerobot/installation#installation-from-pypi) ## Detected evidence (automated analysis) Indexed codebase: 33 recognized source files, 119 KB. - FastAPI (technology) — detected in the code - Hugging Face (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (42 of 42) ``` .gitignore actuator/__init__.py actuator/.gitignore actuator/actuator_layer.py actuator/config/robot_arm.json actuator/gen/SOARM100_ROBOT_ARM_hcp_support.py actuator/kinematics/__init__.py actuator/kinematics/arm_kinematics.py actuator/kinematics/constants.py actuator/kinematics/dh_table.py actuator/kinematics/kinematics.md actuator/kinematics/test_actuator.py actuator/main.py actuator/utils/basic_bus_check.py actuator/utils/detect_serial.py actuator/visualizer.py examples/learms_setup.py examples/teleop_learms.py hcp_client/asi1client.py hcp_client/chat_loop_voice.py hcp_client/chat_loop.py hcp_client/hcp_executor.py hcp_client/main.py hcp_client/test_hcp.py hcp_sdk/.gitignore hcp_sdk/examples/opencv_camera.json hcp_sdk/examples/robot_arm.json hcp_sdk/hcp_sdk_gen.py hcp_sdk/hcp_sdk_schema.json hcp_sdk/OPENCV_CAMERA_hcp_support.py README.md requirements.txt scripts/regen.sh ui/hcp_ui.py vision/calib.py vision/config/opencv_camera.json vision/detect_cameras.py vision/gen/OPENCV_CAMERA_hcp_support.py vision/main.py vision/tag_detections.py whimsy/cam_client.py whimsy/win_server.py ``` ### Dependencies - requirements.txt: absl-py@==2.3.1, accelerate@==1.11.0, aiohappyeyeballs@==2.6.1, aiohttp@==3.13.1, aiosignal@==1.4.0, annotated-doc@==0.0.3, annotated-types@==0.7.0, anyio@==4.11.0, asttokens@==3.0.0, attrs@==25.4.0, av@==15.1.0, certifi@==2025.10.5, cffi@==2.0.0, cfgv@==3.4.0, charset-normalizer@==3.4.4, click@==8.3.0, cloudpickle@==3.1.1, cmake@==4.1.2, cmeel@==0.57.3, cmeel-assimp@==5.4.3.1, cmeel-boost@==1.87.0.1, cmeel-console-bridge@==1.0.2.3, cmeel-octomap@==1.10.0, cmeel-qhull@==8.0.2.1, cmeel-tinyxml2@==10.0.0, cmeel-urdfdom@==4.0.1, cmeel-zlib@==1.3.1, coal-library@==3.0.1, contourpy@==1.3.3, coverage@==7.11.0, cycler@==0.12.1, datasets@==4.1.1, debugpy@==1.8.17, decorator@==5.2.1, deepdiff@==8.6.1, diffusers@==0.35.2, dill@==0.4.0, distlib@==0.4.0, dm_control@==1.0.34, dm-env@==1.6, dm-tree@==0.1.9, docopt@==0.6.2, draccus@==0.10.0, dynamixel-sdk@==3.8.4, eigenpy@==3.10.3, einops@==0.8.1, eiquadprog@==1.2.9, etils@==1.13.0, evdev@==1.9.2, executing@==2.2.1, Farama-Notifications@==0.0.4, fastapi@==0.120.0, feetech-servo-sdk@==1.0.0, ffmpeg@==1.4, filelock@==3.20.0, fonttools@==4.60.1, frozenlist@==1.8.0, fsspec@==2025.9.0, gitdb@==4.0.12, GitPython@==3.1.45, glfw@==2.10.0, grpcio@==1.73.1, grpcio-tools@==1.73.1, gym-aloha@==0.1.3, gym-hil@==0.1.13, gym-pusht@==0.1.6, gymnasium@==1.2.1, h11@==0.16.0, hebi-py@==2.11.0, hf_transfer@==0.1.9, hf-xet@==1.2.0, hidapi@==0.14.0.post4, httptools@==0.7.1, huggingface-hub@==0.35.3, identify@==2.6.15, idna@==3.11, imageio@==2.37.0, imageio-ffmpeg@==0.6.0, importlib_metadata@==8.7.0, importlib_resources@==6.5.2, iniconfig@==2.3.0, inquirerpy@==0.3.4, ipython@==9.6.0, ipython_pygments_lexers@==1.1.1, ischedule@==1.2.7, jedi@==0.19.2, Jinja2@==3.1.6, jsonlines@==4.0.0, kiwisolver@==1.4.9, labmaze@==1.0.6, lazy_loader@==0.4, lerobot@==0.4.0, lxml@==6.0.2, MarkupSafe@==3.0.3, matplotlib@==3.10.7, matplotlib-inline@==0.2.1, mergedeep@==1.3.4, meshcat@==0.3.2, metaworld@==3.0.0, mock-serial@==0.0.1, mpmath@==1.3.0, mujoco@==3.3.7, multidict@==6.7.0, multiprocess@==0.70.16, mypy_extensions@==1.1.0, networkx@==3.5, nodeenv@==1.9.1, num2words@==0.5.14, numpy@==2.2.6, nvidia-cublas-cu12@==12.6.4.1, nvidia-cuda-cupti-cu12@==12.6.80, nvidia-cuda-nvrtc-cu12@==12.6.77, nvidia-cuda-runtime-cu12@==12.6.77, nvidia-cudnn-cu12@==9.5.1.17, nvidia-cufft-cu12@==11.3.0.4, nvidia-cufile-cu12@==1.11.1.6, nvidia-curand-cu12@==10.3.7.77, nvidia-cusolver-cu12@==11.7.1.2, nvidia-cusparse-cu12@==12.5.4.2, nvidia-cusparselt-cu12@==0.6.3, nvidia-nccl-cu12@==2.26.2, nvidia-nvjitlink-cu12@==12.6.85, nvidia-nvtx-cu12@==12.6.77, opencv-python@==4.12.0.88, opencv-python-headless@==4.12.0.88, orderly-set@==5.5.0, packaging@==25.0, pandas@==2.3.3, parso@==0.8.5, pexpect@==4.9.0, pfzy@==0.3.4, pillow@==12.0.0, pin@==3.4.0, placo@==0.9.14, platformdirs@==4.5.0, pluggy@==1.6.0, pre_commit@==4.3.0, prompt_toolkit@==3.0.52, propcache@==0.4.1, protobuf@==6.31.0, psutil@==7.1.1, ptyprocess@==0.7.0, pure_eval@==0.2.3, pyarrow@==22.0.0, pycparser@==2.23, pydantic@==2.12.3, pydantic_core@==2.41.4, pygame@==2.6.1, Pygments@==2.19.2, pymunk@==6.11.1, pyngrok@==7.4.1, pynput@==1.8.1, PyOpenGL@==3.1.10, pyparsing@==3.2.5, pyquaternion@==0.9.9, pyrealsense2@==2.56.5.9235, pyserial@==3.5, pytest@==8.4.2, pytest-cov@==7.0.0, pytest-timeout@==2.4.0, python-dateutil@==2.9.0.post0, python-dotenv@==1.1.1, python-xlib@==0.33, pytz@==2025.2, PyYAML@==6.0.3, pyyaml-include@==1.4.1, pyzmq@==27.1.0, reachy2_sdk_api@==1.0.21, reachy2-sdk@==1.0.14, regex@==2025.10.23, requests@==2.32.5, rerun-sdk@==0.26.1, rhoban-cmeel-jsoncpp@==1.9.4.9, safetensors@==0.6.2, scikit-image@==0.25.2, scipy@==1.16.2, sentry-sdk@==2.42.1, setuptools@==80.9.0, shapely@==2.1.2, six@==1.17.0, smmap@==5.0.2, sniffio@==1.3.1, stack-data@==0.6.3, starlette@==0.48.0, sympy@==1.14.0, teleop@==0.1.2, termcolor@==3.1.0, tifffile@==2025.10.16, tokenizers@==0.22.1, toml@==0.10.2, torch@==2.7.1, torchcodec@==0.5, torchvision@==0.22.1, tornado@==6.5.2, tqdm@==4.67.1, traitlets@==5.14.3, transformers@==4.57.1, transforms3d@==0.4.2, triton@==3.3.1, typing_extensions@==4.15.0, typing-inspect@==0.9.0, typing-inspection@==0.4.2, tzdata@==2025.2, u-msgpack-python@==2.8.0, urllib3@==2.5.0, uvicorn@==0.38.0, uvloop@==0.22.1, virtualenv@==20.35.3, wandb@==0.21.4, watchfiles@==1.1.1, wcwidth@==0.2.14, websocket-client@==1.9.0, websockets@==15.0.1, wrapt@==2.0.0, xxhash@==3.6.0, yarl@==1.22.0, zipp@==3.23.0 ### Recent commits (newest first) - final commits - vision cleanup - final prompt - Misc actuator tweaks - misc constants - more stuff - add in whimsy camera test file - update ui - Merge branch 'main' of https://github.com/danielzyy/calhacks2025 - add UI support - actuator state machine - vision folder cleanup - Merge branch 'main' of https://github.com/danielzyy/calhacks2025 - add voice support to hcp_client - is actuater close to target now break - more prompting - only update ai if we completed action - add wrist roll multiplier - Merge branch 'main' of https://github.com/danielzyy/calhacks2025 - vision integ ## Key source files (fetched from GitHub, selected and truncated for size) ### actuator/kinematics/kinematics.md ```markdown # SOARM-100 Kinematics A key requirement of being able to make a hardware-context-protocol (HCP) compliant device is creating an abstraction that makes it easy to command the device's physical behaviour. For a robot arm like the SOARM-101, this means exposing an API by which the end effector position can be controlled. The SOARM-100 library offers us only the ability to do joint control, and as a result, required us to analytically develop the inverse kinematic solution to be able to command arbitrary end effector position. ## DH Parameters The axis convention and D-H parameters are identical to those defined in [this repository](https://github.com/Argo-Robot/controls/tree/main). ## Forward Kinematics Modelling The forward kinematics model works by propagating the D-H parameters across all of the joints to determine the end effector rotation and translation matrix. The forward kinematics model is primarily used as a sanity check on the inverse kinematic model to ensure that the joint angles lead to the same solution, as the forward kinematics are easy to experimentally verify and visualize. The algorithm is summarized as follows: 1) Define the D-H parameters for each joint in the robot arm. 2) For each joint, compute the individual transformation matrix using the D-H parameters. 3) Multiply the individual transformation matrices together to get the overall transformation matrix from the base to the end effector. 4) Extract the position and orientation of the end effector from the overall transformation matrix. ## Inverse Kinematics Solution Development The inverse kinematics solution was analytically derived via a geometric approach. Let $x_t, y_t, z_t$ be the target end effector parameters. ### Base Link Since the robot is axially constrained about the +Z world frame axis, the first link's solution can be determined by finding the angle of incidence from the target end effector position to the base. $\theta_1 = atan2(y_t, x_t)$ ### Radial Links In order to simplify the kinematic analysis, the scope of the inverse kinematic problem is limited to find solutions which support a wrist approach angle (in laymans terms, the angle at which the wrist approaches a point). As a result, the inverse kinematics problem breaks apart into 2 steps 1) Solve for the required elbow position given the target end effector position and desired wrist approach angle ($\theta_{wrist-approach}$). We can solve for the required height of the elbow as a function of the wrist length ($L_5$) and wrist approach angle as follows: $z_t = z_{elbow} + L_5\sin(\theta_5)$ We can also solve for the required radial length (how far out the arm needs to be) at the elbow as a function of the above-mentioned wrist parameters. Let $r_{target} = (x_t ^ 2 + y_t ^ 2)$ $r_{elbow} = r_{target} - L_5\cos(\theta_5) = r_{target} - r_{wrist}$ This allows us to define - $x_{elbow} = x_{target} - r_{wrist}\cos{\theta_1}$ - $y_{elbow} = y_{target} - r_{wrist}\sin{\theta_1}$ 2) Solve [truncated — 1347 more characters] ``` ### requirements.txt ``` absl-py==2.3.1 accelerate==1.11.0 aiohappyeyeballs==2.6.1 aiohttp==3.13.1 aiosignal==1.4.0 annotated-doc==0.0.3 annotated-types==0.7.0 anyio==4.11.0 asttokens==3.0.0 attrs==25.4.0 av==15.1.0 certifi==2025.10.5 cffi==2.0.0 cfgv==3.4.0 charset-normalizer==3.4.4 click==8.3.0 cloudpickle==3.1.1 cmake==4.1.2 cmeel==0.57.3 cmeel-assimp==5.4.3.1 cmeel-boost==1.87.0.1 cmeel-console-bridge==1.0.2.3 cmeel-octomap==1.10.0 cmeel-qhull==8.0.2.1 cmeel-tinyxml2==10.0.0 cmeel-urdfdom==4.0.1 cmeel-zlib==1.3.1 coal-library==3.0.1 contourpy==1.3.3 coverage==7.11.0 cycler==0.12.1 datasets==4.1.1 debugpy==1.8.17 decorator==5.2.1 deepdiff==8.6.1 diffusers==0.35.2 dill==0.4.0 distlib==0.4.0 dm-env==1.6 dm-tree==0.1.9 dm_control==1.0.34 docopt==0.6.2 draccus==0.10.0 dynamixel-sdk==3.8.4 eigenpy==3.10.3 einops==0.8.1 eiquadprog==1.2.9 etils==1.13.0 evdev==1.9.2 executing==2.2.1 Farama-Notifications==0.0.4 fastapi==0.120.0 feetech-servo-sdk==1.0.0 ffmpeg==1.4 filelock==3.20.0 fonttools==4.60.1 frozenlist==1.8.0 fsspec==2025.9.0 gitdb==4.0.12 GitPython==3.1.45 glfw==2.10.0 grpcio==1.73.1 grpcio-tools==1.73.1 gym-aloha==0.1.3 gym-hil==0.1.13 gym-pusht==0.1.6 gymnasium==1.2.1 h11==0.16.0 hebi-py==2.11.0 hf-xet==1.2.0 hf_transfer==0.1.9 hidapi==0.14.0.post4 httptools==0.7.1 huggingface-hub==0.35.3 identify==2.6.15 idna==3.11 imageio==2.37.0 imageio-ffmpeg==0.6.0 importlib_metadata==8.7.0 importlib_resources==6.5.2 iniconfig==2.3.0 inquirerpy==0.3.4 ipython==9.6.0 ipython_pygments_lexers==1.1.1 ischedule==1.2.7 jedi==0.19.2 Jinja2==3.1.6 jsonlines==4.0.0 kiwisolver==1.4.9 labmaze==1.0.6 lazy_loader==0.4 lerobot==0.4.0 lxml==6.0.2 MarkupSafe==3.0.3 matplotlib==3.10.7 matplotlib-inline==0.2.1 mergedeep==1.3.4 meshcat==0.3.2 metaworld==3.0.0 mock-serial==0.0.1 mpmath==1.3.0 mujoco==3.3.7 multidict==6.7.0 multiprocess==0.70.16 mypy_extensions==1.1.0 networkx==3.5 nodeenv==1.9.1 num2words==0.5.14 numpy==2.2.6 nvidia-cublas-cu12==12.6.4.1 nvidia-cuda-cupti-cu12==12.6.80 nvidia-cuda-nvrtc-cu12==12.6.77 nvidia-cuda-runtime-cu12==12.6.77 nvidia-cudnn-cu12==9.5.1.17 nvidia-cufft-cu12==11.3.0.4 nvidia-cufile-cu12==1.11.1.6 nvidia-curand-cu12==10.3.7.77 nvidia-cusolver-cu12==11.7.1.2 nvidia-cusparse-cu12==12.5.4.2 nvidia-cusparselt-cu12==0.6.3 nvidia-nccl-cu12==2.26.2 nvidia-nvjitlink-cu12==12.6.85 nvidia-nvtx-cu12==12.6.77 opencv-python==4.12.0.88 opencv-python-headless==4.12.0.88 orderly-set==5.5.0 packaging==25.0 pandas==2.3.3 parso==0.8.5 pexpect==4.9.0 pfzy==0.3.4 pillow==12.0.0 pin==3.4.0 placo==0.9.14 platformdirs==4.5.0 pluggy==1.6.0 pre_commit==4.3.0 prompt_toolkit==3.0.52 propcache==0.4.1 protobuf==6.31.0 psutil==7.1.1 ptyprocess==0.7.0 pure_eval==0.2.3 pyarrow==22.0.0 pycparser==2.23 pydantic==2.12.3 pydantic_core==2.41.4 pygame==2.6.1 Pygments==2.19.2 pymunk==6.11.1 pyngrok==7.4.1 pynput==1.8.1 PyOpenGL==3.1.10 pyparsing==3.2.5 pyquaternion==0.9.9 pyrealsense2==2.56.5.9235 pyserial==3.5 pytest==8.4.2 pytest-cov==7.0.0 pytest-timeout==2.4.0 python-dateutil==2.9.0.post0 python-dotenv==1.1.1 python-xlib==0.33 pytz==2025.2 PyYAML==6.0.3 pyyaml-include==1.4.1 pyzmq==27.1.0 reachy2-sdk==1.0.14 reachy2_sdk_api==1.0.21 regex==2025.10.23 requests==2.32.5 rerun-sdk==0.26.1 rhoban-cmeel-jsoncpp==1.9.4.9 safetensors==0.6.2 scikit-image==0.25.2 scipy==1.16.2 sentry-sdk==2.42.1 setuptools==80.9.0 shapely==2.1.2 six==1.17.0 smmap==5.0.2 sniffio==1.3.1 stack-data==0.6.3 starlette==0.48.0 sympy==1.14.0 teleop==0.1.2 termcolor==3.1.0 tifffile==2025.10.16 tokenizers==0.22.1 toml==0.10.2 torch==2.7.1 torchcodec==0.5 torchvision==0.22.1 tornado==6.5.2 tqdm==4.67.1 traitlets==5.14.3 transformers==4.57.1 transforms3d==0.4.2 triton==3.3.1 typing-inspect==0.9.0 typing-inspection==0.4.2 typing_extensions==4.15.0 tzdata==2025.2 u-msgpack-python==2.8.0 urllib3==2.5.0 uvicorn==0.38.0 uvloop==0.22.1 virtualenv==20.35.3 wandb==0.21.4 watchfiles==1.1.1 wcwidth==0.2.14 websocket-client==1.9.0 websockets==15.0.1 wrapt==2.0.0 xxhash==3.6.0 yarl==1.22.0 zipp==3.23.0 ``` ### vision/main.py ```python from gen.OPENCV_CAMERA_hcp_support import HCPClient import time import queue import tag_detections client = HCPClient() client.start() tag_detections.camera_setup() while True: tag_detections.camera_run() try: # Non-blocking check for new HCP commands action, payload = client.events.get_nowait() print(f"[EVENT] {action}: {payload}") if (action == "get_tags"): # handle the command # tag_detections.getItemPositions() result = {"status": "ok", "result": tag_detections.getItemPositions()} # send the response back to HCP print(result) client.send_response(action, result) print("after send") except queue.Empty: pass # other main loop tasks time.sleep(0.1) ``` ### actuator/main.py ```python from gen.SOARM100_ROBOT_ARM_hcp_support import HCPClient import time import queue import numpy as np from copy import deepcopy from actuator_layer import ActuatorLayer, Mode, ActuatorLayerRequest client = HCPClient() client.start() requestActive = False actuator_layer = ActuatorLayer(Mode.AUTONOMOUS, use_visualizer=True, dry_run=False, virtual=False) while True: actuator_layer.step() try: # Non-blocking check for new HCP commands action, payload = client.events.get_nowait() print(f"[EVENT] {action}: {payload}") prev_request = deepcopy(actuator_layer.request) request = prev_request if (action == "move_arm"): request.x_m = payload.get("x", 0)/1000 request.y_m = payload.get("y", 0)/1000 request.z_m = payload.get("z", 0)/1000 actuator_layer.request_position(request) elif (action == "control_grip"): closed = payload.get("closed", False) gripper_cmd = 0.0 if closed else 1.0 request = prev_request request.gripper_cmd = gripper_cmd actuator_layer.request_position(request) elif (action == "set_wrist_angle"): angle = payload.get("angle", 0.0) request.wrist_angle_rad = np.deg2rad(angle) actuator_layer.request_position(request) # handle the command result = {"status": "ok", "message": f"Handled {action}"} requestActive = True except queue.Empty: pass if requestActive: is_actuator_close_to_target_now = actuator_layer.is_close_to_target() print(f"Is actuator close to target? {is_actuator_close_to_target_now}") # send the response back to HCP if is_actuator_close_to_target_now: print("[EVENT] reached_target") client.send_response(action, result) requestActive = False # other main loop tasks time.sleep(0.01) ``` ### hcp_client/main.py ```python import socket import threading import queue import time from enum import Enum, auto from dataclasses import dataclass, field from typing import Callable import hcp_executor from hcp_executor import Client import json from asi1client import ASI1Client, ASI1ClientError import re import argparse import requests import uuid UI_URL = "http://127.0.0.1:5000" # UI base address (change if needed) USE_UI = False # will be set by argparse # ========================= # Voice support (optional) # ========================= USE_VOICE = False recognizer = None mic = None try: parser = argparse.ArgumentParser() parser.add_argument("--voice", action="store_true", help="Use voice input instead of keyboard") parser.add_argument("--ui", action="store_true", help="Enable UI updates to Flask dashboard") args = parser.parse_args() USE_VOICE = args.voice USE_UI = args.ui if USE_VOICE: import speech_recognition as sr import keyboard recognizer = sr.Recognizer() mic = sr.Microphone() except Exception as e: print(f"[!] Voice support disabled: {e}") def listen_to_speech() -> str: if not USE_VOICE or not recognizer or not mic: return "" print("\n🎙️ Hold SPACE to talk... (release when done)") keyboard.wait("space") with mic as source: recognizer.adjust_for_ambient_noise(source, duration=0.3) print("🎧 Listening... (release space when finished)") audio = recognizer.listen(source) while keyboard.is_pressed("space"): time.sleep(0.05) print("🧠 Processing speech...") try: text = recognizer.recognize_google(audio) print(f"You said: {text}") return text except sr.UnknownValueError: print("❌ Could not understand audio.") except sr.RequestError: print("⚠️ Speech recognition service unavailable.") return "" hcp = hcp_executor.HCPExecutor() try: client = ASI1Client() except ASI1ClientError as e: print(f"Error initializing ASI1Client: {e}") messages = [] HOST = '127.0.0.1' PORT = 9000 MAX_MALFORMED_MESSAGE_RETRY = 3 def extract_dashed_section(text): """ Looks for a section delimited by five dashes (-----) at the start and end. Returns: inside: content between the dashes (exclusive) outside: everything else found: boolean indicating if section existed """ # Regex to match content between two sets of 5+ dashes pattern = r"-----\s*(.*?)\s*-----" match = re.search(pattern, text, re.DOTALL) if match: inside = match.group(1).strip() # everything before + after the section outside = (text[:match.start()] + text[match.end():]).strip() return inside, outside, True else: return None, text.strip(), False def extract_main_json_with_context(text): """ Extract the first JSON object (dict) from a string. Returns a tuple: (parsed_json or None, text_outside_json, json_exists_bool) """ brace_count = 0 current_json = "" in_json = False outside_text = "" json_found = False for char in text: if char == '{': if not in_json: in_json = True current_json = "" brace_count += 1 if in_json: current_json += char else: outside_text += char if char == '}': if in_json: brace_count -= 1 if brace_count == 0: # Attempt to parse JSON try: parsed_json = json.loads(current_json) json_found = True return parsed_json, outside_text, True except json.JSONDecodeError: return None, text, False # Invalid JSON # No JSON found return None, text, False def bytes_to_json(byte_string): """ Convert a byte string to a Python dictionary (parsed JSON). Handles both UTF-8 decoding and JSON parsing errors. """ try: # Step 1: Decode bytes to string decoded_str = byte_string.decode('utf-8') # Step 2: Clean up if there are stray characters (optional) decoded_str = decoded_str.strip() # Step 3: Parse JSON data = json.loads(decoded_str) return data except UnicodeDecodeError: print("Error: Could not decode bytes to UTF-8 string.") except json.JSONDecodeError as e: print(f"Error: Invalid JSON data — {e}") def convert_command(device_id, command_name, command_data): # Map string types to Python equivalents type_map = { "int": int, "float": float, "bool": bool, "str": str } # Extract the description description = command_data.get("freetext_desc", "") # Build the parameter list as tuples (name, python_type) params = [] for p in command_data.get("params", []): for k, v in p.items(): params.append((k, type_map.get(v, str))) # Return your target structure return ( device_id, # fixed platform name command_name, # command name description, # human description params # typed parameter list ) # ========================= # Internal plumbing # ========================= @dataclass class ClientEvent: kind: str # 'connect', 'data', 'disconnect', 'error' addr: tuple payload: bytes | None = None error: Exception | None = None class State(Enum): STARTUP = auto() CONNECTING = auto() RUNNING = auto() def handle_client(conn: socket.socket, addr: tuple, event_q: queue.Queue): event_q.put(ClientEvent('connect', addr)) try: with conn: while True: data = conn.recv(4096) if not data: event_q.put(ClientEvent('disconnect', ad [truncated — 11128 more characters] ``` ### vision/detect_cameras.py ```python import cv2 for i in range(10): cap = cv2.VideoCapture(i) if cap.isOpened(): print(f"✅ Camera found at index {i}") cap.release() else: print(f"❌ No camera at index {i}") ``` ### hcp_client/test_hcp.py ```python import hcp_executor hcp = hcp_executor.HCPExecutor() hcp.register_device("SoarM100", "6-DOF arm controller", port=5001) hcp.register_action( "SoarM100", "move_to", "Move arm to XYZ coordinates", [("x", float), ("y", float), ("z", float)], ) hcp.register_action( "SoarM100", "pincer", "Open or close the pincer", [("open", bool)], ) hcp.execute_action("SoarM100", "move_to", {"x": 1.0, "y": 2.0, "z": 3.0}) ``` ### scripts/regen.sh ```shell cd hcp_sdk python hcp_sdk_gen.py --input examples/robot_arm.json --output ./out --host 127.0.0.1 --port 9000 python hcp_sdk_gen.py --input ../actuator/config/robot_arm.json --output ../actuator/gen --host 172.20.10.4 --port 9000 python hcp_sdk_gen.py --input examples/opencv_camera.json --output ./out --host 127.0.0.1 --port 9000 python hcp_sdk_gen.py --input ../vision/config/opencv_camera.json --output ../vision/gen --host 172.20.10.4 --port 9000 cd .. ``` ### examples/learms_setup.py ```python from lerobot.teleoperators.so101_leader import SO101LeaderConfig, SO101Leader from lerobot.robots.so101_follower import SO101FollowerConfig, SO101Follower # Add the parent directory to the Python path import sys import os sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) from actuator.kinematics.dh_table import * from actuator.kinematics.arm_kinematics import * from actuator.kinematics.constants import * from actuator.utils.detect_serial import detect_so101_ports ports = detect_so101_ports() robot_config = SO101FollowerConfig( port=ports["follower_port"], id="follower_arm5", ) robot = SO101Follower(robot_config) robot.setup_motors() ``` ### whimsy/cam_client.py ```python # client.py import cv2 import sys import time # Replace with your host IP if needed. If WSL can reach localhost, use localhost. # Example: url = "http://172.24.0.1:8000/video_feed" #url = "http://{HOST_OR_IP}:8000/video_feed".replace("{HOST_OR_IP}", "localhost") # edit if needed url = "http://172.24.118.106:8000/video_feed" cap = cv2.VideoCapture(url) if not cap.isOpened(): print(f"cv2.VideoCapture couldn't open {url}. Try editing HOST_OR_IP to your Windows host IP.") sys.exit(1) print("Client: opened stream, press 'q' to quit.") while True: ret, frame = cap.read() if not ret: # If stream breaks, wait and retry time.sleep(0.1) continue cv2.imshow("WSL client view", frame) if cv2.waitKey(1) & 0xFF == ord('q'): break cap.release() cv2.destroyAllWindows() ``` [21 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]