# Project export: Spot Party

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: What if everyone could use spot (the robot dog) and it could dance and understand us?
- Devpost: https://devpost.com/software/spotty-spot-spot
- GitHub: https://github.com/aidenybai/spot
- Demo: https://vimeo.com/879175780?share=copy
- Video: https://player.vimeo.com/video/879175780?byline=0&portrait=0&title=0#t=
- Team: 3 GitHub contributor(s) — Aiden Bai (29 commits), Pranav Teegavarapu (5 commits), Ellen Xu (2 commits)

## Devpost submission (written by the team)

### Overview

spot.party Spot Party is a multiplayer site for people to make Spot (the Boston Dynamics robot dog) dance. You can do this through: Pressing buttons (up/down/left/right/twerk) – this can be done by anyone, with built-in proper consensus resolution and queueing implementation. We added new input methods that allow for a massive simplification of Spot control: Dancing to your webcam and Spot following along live (pose imitation) Gesture control — make certain gestures and Spot will dance along and navigate the land Natural language — you can give commands like “Hey Spot, {play dead, dance, slide to the left}” Earlier, people would have to use the Python SDK, which would need 251 lines of code for a simple stand and sit operation. Now, commanding Spot is as easy as using an HTTP API (programmatically), simple terminal commands, posing the way you want spot to pose, controlling through gestures and natural language.

### How we built it

We used the Boston Dynamics python SDK to control Spot, and connected this to our server to sync user actions on the frontend (built in HTML + Vite + React) to Spot. We used LiveBlocks to make our frontend “multiplayer” and used PoseNet to track user position and changes in position. We used HandsfreeJS to embed the model onto the web via WebAssembly for both our pose and gesture clients for easy accessibility. We used OpenAI GPT-4 + Whisper to do transcription, reasoning, and output for certain commands. On the server, we further implemented rate limiting, authentication, queue control, and anti-DDoS via Cloudflare in order to prevent API abuse (and abuse of our wallets).

### Challenges we ran into

Get PoseNet to track a user’s dance through the webcam — we switched from Spot directly copying users to user poses/positions translating to a movement for Spot Getting OpenAI to work on Spot was a nightmare Access to Spot was limited, Spot SDK was difficult to use, so we did a lot of guesswork and rigorous code review. Networking challenges - Spot is controlled through the Ubuntu server on its back: Spot Core. We need to send our code onto Spot Core, and only then can we connect to Spot and see its local IP. Finally, to stream instructions from the web, we parse in a cURL'ed list of commands through Railway

### What's next

We’re hoping to record videos of Spot dancing along to more songs, and add support for Spot copying a user dancing. We love spot.

## README (from the GitHub repository)

> ⚠️ **Disclaimer**: We are not affiliated, associated, authorized, endorsed by, or in any way officially connected with Boston Dynamics, or any of its subsidiaries or its affiliates. The official Boston Dynamics website can be found at https://www.bostondynamics.com.

> But thank you for letting us make your dog twerk.

# [spot.party](https://spot.party)


// temp

## Detected evidence (automated analysis)

Indexed codebase: 18 recognized source files, 72 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (20 of 20)

```
.gitignore
pose.html
README.md
robot/new.py
robot/README.md
server/index.js
server/package.json
server/README.md
web/gesture.html
web/index.html
web/package.json
web/pose.html
web/README.md
web/src/app.jsx
web/src/gesture.jsx
web/src/http.ts
web/src/live.jsx
web/src/pose.jsx
web/src/style.css
web/vite.config.js
```

### Dependencies

- server/package.json: cors@^2.8.5, express@^4.18.2, express-rate-limit@^7.1.3
- web/package.json: @liveblocks/client@^1.5.2, @vitejs/plugin-react@^4.1.0, handsfree@^8.5.1, million@^2.6.4, react@^18.2.0, react-dom@^18.2.0, vite@^4.5.0

### Recent commits (newest first)

- Merge branch 'main' of https://github.com/aidenybai/spot
- yes yes yes ppap
- hey spot do smth weird
- Merge branch 'main' of https://github.com/aidenybai/spot
- fuck me in the ass
- new2
- Merge branch 'main' of https://github.com/aidenybai/spot
- fuck me in the ass
- fuck me in the ass
- sendhttp
- Merge branch 'main' of https://github.com/aidenybai/spot
- fuck me in the ass
- oop
- pose
- revampy vamp vamp
- revampy vamp vamp
- revampy vamp vamp
- revampy vamp vamp
- revampy vamp vamp
- revampy vamp vamp

## Key source files (fetched from GitHub, selected and truncated for size)

### server/package.json

```
{
  "scripts": {
    "start": "node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "dependencies": {
    "cors": "^2.8.5",
    "express": "^4.18.2",
    "express-rate-limit": "^7.1.3"
  }
}

```

### web/package.json

```
{
  "scripts": {
    "dev": "vite",
    "build": "vite build"
  },
  "dependencies": {
    "@liveblocks/client": "^1.5.2",
    "handsfree": "^8.5.1",
    "million": "^2.6.4",
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "devDependencies": {
    "@vitejs/plugin-react": "^4.1.0",
    "vite": "^4.5.0"
  }
}

```

### server/index.js

```javascript
const express = require('express');
const { rateLimit } = require('express-rate-limit');
const cors = require('cors');
const app = express();

const port = process.env.PORT || 5111;
let clients = [];
let actions = [];
const MAX_N = 50;
const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const limiter = rateLimit({
  windowMs: 10000,
  limit: 500, // 50 req/s
  standardHeaders: 'draft-7',
  legacyHeaders: false,
});

app.use(limiter);
app.use(cors());
app.use(express.json());

app.post('/action', (req, res) => {
  const action = req.body.action;
  const whitelist = ['W', 'A', 'S', 'D', 'Q', 'E', 'T'];

  if (!whitelist.includes(action)) {
    return res.status(400).json({ error: 'Invalid action' });
  }

  // 5% chance to add a T (twerk)
  if (action === 'T') {
    if (Math.random() < 0.05) {
      actions.push('T');
      return res.status(200).json({ status: 'Action added' });
    } else {
      return res.status(400).json({ error: 'Action not added' });
    }
  }

  if (actions.length < MAX_N) {
    actions.push(action);
    return res.status(200).json({ status: 'Action added' });
  } else {
    return res.status(400).json({ error: 'Action queue is full' });
  }
});

app.get('/kill', (req, res) => {
  if (req.query.key !== process.env.MAIN_KEY) return res.status(403).send('no');
  clients = [];
  actions = [];
  res.send('ok');
});

app.get('/info', (_req, res) => {
  const used = process.memoryUsage().heapUsed / 1024 / 1024;
  res.json({
    memoryUsage: `${Math.round(used * 100) / 100} MB`,
    clientsCount: clients.length,
    actionsCount: actions.length,
    clients: clients.map((c) => c.id),
    actions,
  });
});

app.get('/actions', (req, res) => {
  if (req.query.key !== process.env.MAIN_KEY) return res.status(403).send('no');

  res.setHeader('Content-Type', 'text/plain');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.flushHeaders();

  const client = {
    id: Date.now(),
    res,
    ready: false,
  };

  clients.push(client);
  res.write('');

  (async () => {
    await wait(500);
    res.write(' \n');
    await wait(500);
    res.write('P\n');
    await wait(10000);
    res.write('f\n');
    await wait(1500);
    client.ready = true;

    setInterval(() => {
      if (!actions.length) return;
      clients.forEach((client) => {
        if (!client.ready) return;
        client.res.write(`${actions.shift().toLowerCase()}\n`);
      });
    }, 10);
  })();

  req.on('close', () => {
    clients = clients.filter((c) => {
      c.res.write('\t');
      return client.id !== c.id;
    });
  });
});

app.listen(port, () => {
  console.log(`Server is running on port ${port}`);
});

```

### web/src/app.jsx

```javascript
import { createRoot } from 'react-dom/client';
import React, { useState, useEffect } from 'react';
import { sendHttp } from './http';

function App() {
  const [needsCooldown, setNeedsCooldown] = useState(false);

  useEffect(() => {
    let timeout;
    if (needsCooldown) {
      timeout = setTimeout(() => setNeedsCooldown(false), 200);
    }
    return () => {
      if (timeout) clearTimeout(timeout);
    };
  }, [needsCooldown]);

  return (
    <div>
      <ArrowKeysLayout>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('rotate-left');
            setNeedsCooldown(true);
          }}
          name="rotate-left"
        >
          ⤴️
        </Control>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('up');
            setNeedsCooldown(true);
          }}
          name="up"
        >
          🔼
        </Control>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('rotate-right');
            setNeedsCooldown(true);
          }}
          name="rotate-right"
        >
          ⤵️
        </Control>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('left');
            setNeedsCooldown(true);
          }}
          name="left"
        >
          ◀️
        </Control>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('down');
            setNeedsCooldown(true);
          }}
          name="down"
        >
          🔽
        </Control>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('right');
            setNeedsCooldown(true);
          }}
          name="right"
        >
          ▶️
        </Control>
      </ArrowKeysLayout>
      <center>
        <Control
          disabled={needsCooldown}
          action={() => {
            sendHttp('twerk');
            setNeedsCooldown(true);
          }}
          name="twerk"
        >
          🍑
        </Control>
      </center>
    </div>
  );
}

function ArrowKeysLayout({ children }) {
  return (
    <div
      style={{
        display: 'grid',
        columnGap: '1rem',
        gridTemplateColumns: '1fr 1fr 1fr',
      }}
    >
      {children}
    </div>
  );
}

function Control({ children, action, name, disabled }) {
  return (
    <button
      id={name}
      style={{
        padding: '0rem',
        fontSize: '4rem',
        transform: name.startsWith('rotate')
          ? `scale(0.5) rotate(270deg)`
          : null,
      }}
      onClick={action}
      disabled={disabled}
    >
      {children}
    </button>
  );
}

createRoot(document.getElementById('root')).render(<App />);

```

### pose.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" href="/src/style.css" />
    <script
      src="https://unpkg.com/twemoji@latest/dist/twemoji.min.js"
      crossorigin="anonymous"
    ></script>
    <script src="https://unpkg.com/ml5@latest/dist/ml5.min.js"></script>
     <title>spotty spot spot</title>
  </head>
  <body>
    <video id="video"></video>

    <script defer>
        console.log('page loaded')
        const video = document.getElementById('video');
        console.log(video)
            const poseNet = ml5.poseNet(video, modelLoaded);  
    
            function modelLoaded() {  
                console.log('Model Loaded!'); 
                // Call your loading function here
                myLoadingFunction();
            }
    
            poseNet.on('pose', (results) => {   poses = results; });
    
            function myLoadingFunction() {
                console.log('Loading code executed!');
            }
        </script>    
  </body>
</html>

```

### web/pose.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>pose detection with handsfree</title>
  </head>
  <body>
    <div id="root">initializing...</div>
    <script src="./src/pose.jsx" type="module"></script>
  </body>
</html>

```

### web/gesture.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>gesture detection with handsfree</title>
  </head>
  <body>
    <div id="root">initializing...</div>
    <script src="./src/gesture.jsx" type="module"></script>
  </body>
</html>

```

### web/vite.config.js

```javascript
import million from 'million/compiler';
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';
import { resolve } from 'path';

export default defineConfig({
  build: {
    rollupOptions: {
      input: {
        index: resolve(__dirname, 'index.html'),
        pose: resolve(__dirname, 'pose.html'),
        gesture: resolve(__dirname, 'gesture.html'),
      },
    },
  },
  plugins: [million.vite({ auto: true }), react()],
});

```

### web/index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" href="/src/style.css" />
    <script
      src="https://unpkg.com/twemoji@latest/dist/twemoji.min.js"
      crossorigin="anonymous"
    ></script>
    <title>spot.party</title>
  </head>
  <body>
    <svg
      class="cursor"
      id="cursor-template"
      width="24"
      height="36"
      viewBox="0 0 24 36"
      fill="transparent"
      xmlns="http://www.w3.org/2000/svg"
    >
      <path
        d="M5.65376 12.3673H5.46026L5.31717 12.4976L0.500002 16.8829L0.500002 1.19841L11.7841 12.3673H5.65376Z"
        stroke="white"
        stroke-width="1"
      />
    </svg>

    <div id="root">initializing...</div>
    <div id="cursors-container"></div>
    <script src="./src/live.jsx" type="module"></script>
    <script src="./src/app.jsx" type="module"></script>
  </body>
</html>

```

### robot/new.py

```python
import io
import logging
import math
import os
import signal
import sys
import threading
import time
from collections import OrderedDict

import openai
import os

openai.api_key = os.environ['OPENAI_API_KEY']

import bosdyn.api.basic_command_pb2 as basic_command_pb2
import bosdyn.api.power_pb2 as PowerServiceProto
# import bosdyn.api.robot_command_pb2 as robot_command_pb2
import bosdyn.api.robot_state_pb2 as robot_state_proto
import bosdyn.api.spot.robot_command_pb2 as spot_command_pb2
import bosdyn.client.util
from bosdyn.api import geometry_pb2
from bosdyn.client import ResponseError, RpcError, create_standard_sdk
from bosdyn.client.async_tasks import AsyncGRPCTask, AsyncPeriodicQuery, AsyncTasks
from bosdyn.client.estop import EstopClient, EstopEndpoint, EstopKeepAlive
from bosdyn.client.frame_helpers import ODOM_FRAME_NAME
from bosdyn.client.lease import Error as LeaseBaseError
from bosdyn.client.lease import LeaseClient, LeaseKeepAlive
from bosdyn.client.power import PowerClient
from bosdyn.client.robot_command import RobotCommandBuilder, RobotCommandClient
from bosdyn.client.robot_state import RobotStateClient
from bosdyn.client.time_sync import TimeSyncError
from bosdyn.util import duration_str, format_metric, secs_to_hms

LOGGER = logging.getLogger()

VELOCITY_BASE_SPEED = 1 # 0.5  # m/s
VELOCITY_BASE_ANGULAR = 0.8  # rad/sec
VELOCITY_CMD_DURATION = 0.5  # seconds
# COMMAND_INPUT_RATE = 0.05


def _grpc_or_log(desc, thunk):
  try:
    return thunk()
  except (ResponseError, RpcError) as err:
    LOGGER.error('Failed %s: %s', desc, err)


class ExitCheck(object):
  """A class to help exiting a loop, also capturing SIGTERM to exit the loop."""

  def __init__(self):
    self._kill_now = False
    signal.signal(signal.SIGTERM, self._sigterm_handler)
    signal.signal(signal.SIGINT, self._sigterm_handler)

  def __enter__(self):
    return self

  def __exit__(self, _type, _value, _traceback):
    return False

  def _sigterm_handler(self, _signum, _frame):
    self._kill_now = True

  def request_exit(self):
    """Manually trigger an exit (rather than sigterm/sigint)."""
    self._kill_now = True

  @property
  def kill_now(self):
    """Return the status of the exit checker indicating if it should exit."""
    return self._kill_now


class AsyncRobotState(AsyncPeriodicQuery):
  """Grab robot state."""

  def __init__(self, robot_state_client):
    super(AsyncRobotState, self).__init__('robot_state',
                                          robot_state_client,
                                          LOGGER,
                                          period_sec=0.2)

  def _start_query(self):
    return self._client.get_robot_state_async()


class WasdInterface(object):
  """A curses interface for driving the robot."""

  def __init__(self, robot):
    self._robot = robot
    # Create clients -- do not use the for communication yet.
    self._lease_client = robot.ensure_client(LeaseClient.default_service_name)
    try:
      self._estop_client = self._robot.ensure_client(
          EstopClient.default_service_name)
      self._estop_endpoint = EstopEndpoint(self._estop_client, 'GNClient', 9.0)
    except:
      # Not the estop.
      self._estop_client = None
      self._estop_endpoint = None
    self._power_client = robot.ensure_client(PowerClient.default_service_name)
    self._robot_state_client = robot.ensure_client(
        RobotStateClient.default_service_name)
    self._robot_command_client = robot.ensure_client(
        RobotCommandClient.default_service_name)
    self._robot_state_task = AsyncRobotState(self._robot_state_client)
    self._async_tasks = AsyncTasks([self._robot_state_task])
    self._lock = threading.Lock()
    self._command_dictionary = {
        27: self._stop,  # ESC key
        ord('\t'): self._quit_program,
        ord('T'): self._toggle_time_sync,
        ord(' '): self._toggle_estop,
        ord('c'): self._circle_move,
        ord('r'): self._self_right,
        ord('P'): self._toggle_power,
        ord('p'): self._toggle_power,
        ord('v'): self._sit,
        ord('b'): self._battery_change_pose,
        ord('f'): self._stand,
        ord('w'): self._move_forward,
        ord('s'): self._move_backward,
        ord('a'): self._strafe_left,
        ord('d'): self._strafe_right,
        ord('q'): self._turn_left,
        ord('e'): self._turn_right,
        ord('u'): self._unstow,
        ord('j'): self._stow,
        ord('l'): self._toggle_lease,
        ord('g'): self._desmos,
    }
    self._locked_messages = ['', '', '']  # string: displayed message for user
    self._estop_keepalive = None
    self._exit_check = None

    # Stuff that is set in start()
    self._robot_id = None
    self._lease_keepalive = None

  def start(self):
    """Begin communication with the robot."""
    # Construct our lease keep-alive object, which begins RetainLease calls in a thread.
    self._lease_keepalive = LeaseKeepAlive(self._lease_client,
                                           must_acquire=True,
                                           return_at_exit=True)

    self._robot_id = self._robot.get_id()
    if self._estop_endpoint is not None:
      self._estop_endpoint.force_simple_setup(
      )  # Set this endpoint as the robot's sole estop.

  def shutdown(self):
    """Release control of robot as gracefully as possible."""
    LOGGER.info('Shutting down WasdInterface.')
    if self._estop_keepalive:
      # This stops the check-in thread but does not stop the robot.
      self._estop_keepalive.shutdown()
    if self._lease_keepalive:
      self._lease_keepalive.shutdown()

  def flush_and_estop_buffer(self, stdscr):
    """Manually flush the curses input buffer but trigger any estop requests (space)"""
    key = ''
    while key != -1:
      key = stdscr.getch()
      if key == ord(' '):
        self._toggle_estop()

  def add_message(self, msg_text):
    print(msg_text)
    # with self._lock:
    #     self._locked_messages = [msg_text] + self._locked
[truncated — 14829 more characters]
```

[5 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]