spot.party Spot Party is a multiplayer site for people to make Spot (the Boston Dynamics robot dog) dance. You can do this through: Pressing buttons (up/down/left/right/twerk) – this can be done by anyone, with built-in proper consensus resolution and queueing implementation. We added new input methods that allow for a massive simplification of Spot control: Dancing to your webcam and Spot following along live (pose imitation) Gesture control — make certain gestures and Spot will dance along and navigate the land Natural language — you can give commands like “Hey Spot, {play dead, dance, slide to the left}” Earlier, people would have to use the Python SDK, which would need 251 lines of code for a simple stand and sit operation. Now, commanding Spot is as easy as using an HTTP API (programmatically), simple terminal commands, posing the way you want spot to pose, controlling through gestures and natural language.

How we built it

We used the Boston Dynamics python SDK to control Spot, and connected this to our server to sync user actions on the frontend (built in HTML + Vite + React) to Spot. We used LiveBlocks to make our frontend “multiplayer” and used PoseNet to track user position and changes in position. We used HandsfreeJS to embed the model onto the web via WebAssembly for both our pose and gesture clients for easy accessibility. We used OpenAI GPT-4 + Whisper to do transcription, reasoning, and output for certain commands. On the server, we further implemented rate limiting, authentication, queue control, and anti-DDoS via Cloudflare in order to prevent API abuse (and abuse of our wallets).

Challenges we ran into

Get PoseNet to track a user’s dance through the webcam — we switched from Spot directly copying users to user poses/positions translating to a movement for Spot Getting OpenAI to work on Spot was a nightmare Access to Spot was limited, Spot SDK was difficult to use, so we did a lot of guesswork and rigorous code review. Networking challenges - Spot is controlled through the Ubuntu server on its back: Spot Core. We need to send our code onto Spot Core, and only then can we connect to Spot and see its local IP. Finally, to stream instructions from the web, we parse in a cURL'ed list of commands through Railway

What's next

We’re hoping to record videos of Spot dancing along to more songs, and add support for Spot copying a user dancing. We love spot.