# Project export: SpotAssist

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: A cutting-edge robotic companion designed to support users with health challenges and guide them effortlessly to objects within their surroundings.
- Devpost: https://devpost.com/software/spotassist
- GitHub: https://github.com/AnthonyYao7/TreeHacksSpot
- Video: https://www.youtube.com/embed/2cggQ0wLVO8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — AnthonyYao7 (56 commits), Andrew Kim (19 commits), Adrish Kar (2 commits)

## Devpost submission (written by the team)

### Inspiration

In response to the challenges faced by the elderly and disabled who rely on traditional, costly service dogs, our team drew inspiration to create SpotAssist. Traditional service dogs demand substantial resources, extensive training, and are limited to one user at a time. SpotAssist leverages Boston Dynamics’ Spot to overcome these challenges, offering advanced functionalities through technologies like machine learning, computer vision, and natural language processing. This innovative robotic companion not only surpasses traditional service dogs but also holds the potential for mass deployment in settings like senior care homes, addressing financial and logistical constraints. SpotAssist embodies our commitment to making a significant impact by combining technology, compassion, and scalability to enhance the lives of those in need.

### What it does

/

### How we built it

SpotAssist is an assistive robot meant to achieve better functionality than a regular service dog. First, Spot will take an image of its surroundings and generate a list of objects detected along with their corresponding bounding boxes (using YOLO). Then, Spot will wait for the user to make a verbal query including a target landmark to approach; an example of such a command would be “Take me to the nearest chair.” Finally, Spot will gently guide the user to their desired destination through noting the difference between the center x-coordinate of the bounding box and the actual center of the image. Spot will utilize this quantity in order to align its heading angle to be in line with the desired object and subsequently move forward, stopping at a reasonable distance from the object in question. Furthermore, we built health-first integrations from the ground-up for Spot. Using our custom ML and analytical methods and Terra API integrations, we are able to use Spot as an alert for a medical event, such as abnormal heart rhythms and/or heart rate.

### Challenges we ran into

There were a number of technical challenges we encountered throughout the development process. Firstly, a major hurdle was the restrictions placed on the compute module attached to Spot; we were not allowed to perform any commands directly to Spot itself (we instead had to communicate through the module). Furthermore, we were not allowed to SSH into the module (nor were we given elevated privileges), so we had to build our own version of a remote communication system between our remote servers and Spot. This led to many creative avenues being explored for development.

### Accomplishments we're proud of

We’re especially proud of building upon a system/environment that was entirely new to the whole team, especially without any documentation on the compute module we built upon. Furthermore, none of the team knew each other prior to the hackathon, so we’re proud of the teamwork skills we were able to utilize to accomplish our success this weekend.

### What's next

Mobile integration!

## README (from the GitHub repository)

**This image contains basic commands to work with camera, microphone and speaker at Spot's computer**

## Image link
Image link could be acquired in **packages** section of the main repository page

## How to use
The container for this image should be runned with flags `--device /dev/video0` and `--device /dev/snd`.
Also, environmental varialbes `-e SDL_AUDIODRIVER='alsa'`, `-e AUDIODEV='hw:1,0'`, `-e AUDIO_INPUT_DEVICE='hw:2,0'` shoud be added.

Basic example contained in _main.py_ 


## Detected evidence (automated analysis)

Indexed codebase: 16 recognized source files, 39 KB.
- Python (language) — detected in the code
- AWS (technology) — claimed on Devpost, not found in the code
- Flask (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (21 of 21)

```
.github/workflows/asr_build.yml
.github/workflows/build.yml
.github/workflows/ftp_build.yml
.github/workflows/image_processing.yml
.gitignore
AllLibrariesSpot.ipynb
aws-listener.py
aws-sender.py
Dockerfile
healthrec.py
local_socket_testing/client.py
local_socket_testing/server.py
main.py
README.md
remote-client.py
remote.py
requirements.txt
rootkey.csv
spot_controller.py
startup.sh
wait.py
```

### Dependencies

- requirements.txt: bosdyn-choreography-client, bosdyn-client, bosdyn-mission, boto3, numpy, opencv-python

### Recent commits (newest first)

- Update server.py
- Add files via upload
- Update server.py
- Update server.py
- built mock for audio
- built mock for audio
- built mock for audio
- finished movement towards point
- did
- did
- added features
- Merge branch 'master' into image-processing
- fixed bugs
- fixed bugs
- added modularity again
- added modularity again
- added modularity
- added asr handler
- added a lot of features
- changed architecture

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
bosdyn-client
bosdyn-mission
bosdyn-choreography-client
opencv-python
numpy
boto3
```

### Dockerfile

```
FROM ghcr.io/merklebot/hackathon-arm-image:master as build

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETOS
ARG TARGETARCH

ARG Version
ARG GitCommit
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM"


COPY requirements.txt requirements.txt
RUN python3.8 -m pip install -r requirements.txt
COPY . .

CMD ["python3.8", "main.py"]

```

### main.py

```python
import os
import time

from bosdyn.api import image_pb2
from bosdyn.client.image import build_image_request
from spot_controller import SpotController
import socket
import numpy as np
import cv2

ROBOT_IP = "10.0.0.3"
SPOT_USERNAME = "admin"
SPOT_PASSWORD = "2zqa8dgw7lor"

HOST_PORT = 8080
HOST_ADDRESS = os.environ['HOST_ADDRESS']


def send_file(file, sock):
    # send 4 byte integer representing number of characters in the filename.
    # send filename.
    # send 4 byte integer representing number of bytes in file.
    # send file contents.

    sock.sendall(len(file).to_bytes(4, 'little'))
    sock.sendall(file.encode('utf-8'))

    with open(file, 'rb') as f:
        bt = f.read()
        sock.sendall(len(bt).to_bytes(4, 'little'))
        sock.sendall(bt)


def pixel_format_string_to_enum(enum_string):
    return dict(image_pb2.Image.PixelFormat.items()).get(enum_string)


def take_image_handler(spot, sock, command=None):
    # sources = ['back_fisheye_image', 'frontleft_fisheye_image', 'frontright_fisheye_image', 'left_fisheye_image',
    #            'right_fisheye_image']

    sources = ['frontleft_fisheye_image']

    pixel_format = pixel_format_string_to_enum('PIXEL_FORMAT_GREYSCALE_U8')

    image_request = [
        build_image_request(source, pixel_format=pixel_format)
        for source in sources
    ]

    image_responses = spot.get_images(image_request)

    for image in image_responses:
        num_bytes = 1
        dtype = np.uint8
        extension = '.jpg'

        img = np.frombuffer(image.shot.image.data, dtype=dtype)
        if image.shot.image.format == image_pb2.Image.FORMAT_RAW:
            try:
                img = img.reshape((image.shot.image.rows, image.shot.image.cols, num_bytes))
            except ValueError:
                img = cv2.imdecode(img, -1)
        else:
            img = cv2.imdecode(img, -1)

        image_saved_path = str(int(time.time() * 1000)) + '_' + image.source.name + extension
        cv2.imwrite(image_saved_path, img)

        send_file(image_saved_path, sock)


def remove_non_numeric(s):
    # Using filter and lambda to remove non-numeric characters
    filtered = filter(lambda x: x.isdigit(), s)
    # Joining the filtered characters back into a string
    return ''.join(filtered)


def move_towards_point_handler(spot, sock, command):
    print(command)
    point = command[len('move_towards_point'):]
    x, w = point.split(',')
    x = int(remove_non_numeric(x))
    w = int(remove_non_numeric(w))

    spot.move_by_velocity_control(v_x=0, v_y=0, v_rot=(x - w // 2) / 6000, cmd_duration=1)
    time.sleep(4)


def asr_handler(spot, sock, command):
    print("Start recording audio")
    dest_file = "spotQuery.wav"
    cmd = f'arecord -vv --format=cd --device={os.environ["AUDIO_INPUT_DEVICE"]} -r 48000 --duration=10 -c 1 {dest_file}'
    try:
        # Execute the audio recording command
        # subprocess.run(cmd, shell=True, check=True)
        # Send the recorded audio file to the local machine
        os.system(cmd)
        send_file(dest_file, sock)
    except Exception as e:
        print(f"Error during audio recording: {e}")
    finally:
        # Clean up: remove the temporary audio file on Spot
        os.remove(dest_file)
    print(cmd)


COMMAND_HANDLERS = {'take_image': take_image_handler,
                    'move_towards_point': move_towards_point_handler,
                    'start_asr': asr_handler}


def main():
    # print("Start recording audio")
    # sample_name = "aaaa.wav"
    # cmd = f'arecord -vv --format=cd --device={os.environ["AUDIO_INPUT_DEVICE"]} -r 48000 --duration=10 -c 1 {sample_name}'
    # print(cmd)
    # os.system(cmd)
    # print("Playing sound")
    # os.system(f"ffplay -nodisp -autoexit -loglevel quiet {sample_name}")

    with SpotController(username=SPOT_USERNAME, password=SPOT_PASSWORD, robot_ip=ROBOT_IP) as spot:

        time.sleep(1)

        spot.move_head_in_points(yaws=[0.2, 0],
                                 pitches=[0.3, 0],
                                 rolls=[0.4, 0],
                                 sleep_after_point_reached=1)

        time.sleep(1)

        # # Make Spot to move by goal_x meters forward and goal_y meters left
        # spot.move_to_goal(goal_x=0.5, goal_y=0)
        # time.sleep(3)
        #
        # # Control Spot by velocity in m/s (or in rad/s for rotation)
        # spot.move_by_velocity_control(v_x=-0.3, v_y=0, v_rot=0, cmd_duration=2)
        # time.sleep(3)

        if spot is None:
            print("Failed to initialize Spot")
            return

        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

        try:
            s.connect((HOST_ADDRESS, HOST_PORT))
            print(f"Successfully connected to {HOST_ADDRESS}:{HOST_PORT}")

        except Exception as e:
            print(f"Failed to connect to {HOST_ADDRESS}:{HOST_PORT}")
            print(f"Error: {e}")
            s.close()
            return

        buffer = ''

        while True:
            data = s.recv(4096)
            if not data:
                print("Disconnected from the server.")
                break

            buffer += data.decode('utf-8')
            while '\n' in buffer:
                command, buffer = buffer.split('\n', 1)

                for comm, handler in COMMAND_HANDLERS.items():
                    if comm in command:
                        handler(spot, s, command)
                        break

        s.close()


# 10.19.187.105

if __name__ == '__main__':
    main()

```

### local_socket_testing/server.py

```python
import socket
from threading import Thread, Lock
import struct
from enum import Enum
from ultralytics import YOLO
from collections import deque
import time
import cv2
import os

ROTATION_ANGLE = {
    'back_fisheye_image': 0,
    'frontleft_fisheye_image': -78,
    'frontright_fisheye_image': -102,
    'left_fisheye_image': 0,
    'right_fisheye_image': 180
}


mq = deque()


def send_commands(sock):
    global mq
    while True:
        print("Waiting for file")
        while len(mq) == 0: time.sleep(1)

        command = mq.popleft()
        sock.sendall((command + '\n').encode('utf-8'))


class RobotStates(Enum):
    WAITING_FOR_COMMAND = 1  # commands are of the form take me to something
    TARGETING = 2  # back and forth between robot and server with yolo and images
    WALKING = 3  # robot is facing towards the target and will start walking
    # ALARM_MODE = 4


# robot_state = RobotStates.TARGETING
# robot_state_mutex = Lock()

file_handler_threads = []

model = YOLO("yolov8n.pt")

translation = None
most_recent_classes = None
most_recent_boxes = None
target_class = None


def int_from_bytes(bt):
    return struct.unpack('<I', bt)[0]


def handle_new_file(file):
    global target_class

    ext = file.split('.')[1]

    # with robot_state_mutex:
    if ext == 'jpg':
        # if robot_state != RobotStates.TARGETING:
        #     print("JPG files only relevant in targeting stage")
        #     return

        results = model(file)[0]
        boxes = results.boxes.xyxy.numpy()
        class_names = results.names
        pred = results.boxes.cls.numpy()

        global translation, most_recent_boxes, most_recent_classes
        if translation is None:
            translation = class_names

        most_recent_classes = pred
        most_recent_boxes = boxes

        print(target_class)

        if target_class is None:
            return

        ind = None

        for i, cl in enumerate(pred):
            if cl == target_class:
                ind = i
                break

        if ind is None:
            print("im fucked")

        image = cv2.imread(file)
        w = image.shape[1]

        rel_bbox = most_recent_boxes[ind]
        dims = results.orig_shape
        x_hat = (rel_bbox[0] + rel_bbox[2]) // 20
        mq.append(f"move_towards_point{x_hat},{w}")

        """
        Call yolo on jpg and get bounding boxes and return response to spot
        """

    elif ext == 'wav':
        # if robot_state != RobotStates.WAITING_FOR_COMMAND:
        #     print("WAV files only relevant when waiting for commands")
        #     return
        """
        Send wav file to whisper api for transcription
        """
        from openai import OpenAI
        client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])
        audio_file = open(file, "rb")
        vocal_query = client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file,
            response_format="text"
        )

        obj_arr = [translation[most_recent_classes[i]] for i in range(len(most_recent_classes))]
        # translation: number to object
        prompt = f"{obj_arr} \n YOUR TASK: given the previous array of strings denoting objects detected in an image, isolate only one object that matches the object specified by the following verbal query: \"{vocal_query}\" \n THIS IS VERY IMPORTANT: ONLY RETURN THE NUMERICAL 0-BASED INDEX OF THE FIRST SUCH RELEVANT OBJECT IN THE ARRAY, WITH NO OTHER TEXT IN YOUR OUTPUT"
        response = client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": prompt}]
        )
        resp_ret = response.choices[0].message.content
        # for term in translation:
        try:
            # if translation[term] == translation[most_recent_classes[int(resp_ret)]]:
            target_class = int(most_recent_classes[int(resp_ret)])
            print('aowiefjwiaofj', target_class)
        except ValueError:
            print("errored out")
            print(resp_ret)
    else:
        pass


def read_bytes(sock, n):
    data = b''

    while len(data) < n:
        chunk = sock.recv(n - len(data))
        if not chunk:
            print("Connection closed by the client during data reception.")
            break
        data += chunk

    return data


def read_from_client(client_socket, address, file_name_prefix):
    while True:
        filename_length_bytes = client_socket.recv(4)

        if not filename_length_bytes:
            print("Connection closed by the client.")
            break

        filename_length = int_from_bytes(filename_length_bytes)
        filename = read_bytes(client_socket, filename_length).decode('utf-8')
        file_content_length_bytes = client_socket.recv(4)

        if not filename_length_bytes:
            print("Connection closed by the client.")
            break

        file_content_length = int_from_bytes(file_content_length_bytes)
        file_content = read_bytes(client_socket, file_content_length)
        file_name = f"{file_name_prefix}/{filename}"

        with open(file_name, 'wb') as file:
            file.write(file_content)

        file_handler_threads.append(
            Thread(target=handle_new_file, args=(file_name,)))
        file_handler_threads[-1].start()


def handle_client_connection(client_socket: socket.socket, address, file_name_prefix):
    with client_socket:
        print(f"Connection from {address} has been established.")

        # Create and start the reading thread
        reading_thread = Thread(
            target=read_from_client,
            args=(client_socket, address, file_name_prefix))
        reading_thread.start()

        message_sending_thread = Thread(
            target=send_commands,
            args=(client_socket,)
        )

        message_sending_thread.start()

        while True:
            command = input("Start process:")
[truncated — 1209 more characters]
```

### wait.py

```python
import time

time.sleep(10)
```

### startup.sh

```shell
#!/bin/bash

python3.8 aws-listener.py

```

### aws-sender.py

```python
import boto3

# Read credentials from file
with open("rootkey.csv", "r") as file:
    lines = file.readlines()  # Read all lines into a list

    # Skip the header row (assuming it exists)
    header = lines[0]

    # Access the second row (index 1) and split it into cells
    row_data = lines[1].split(",")

    # Extract cells A2 and B2
    access_key = row_data[0].strip()
    secret_key = row_data[1].strip()

print("Access Key: ", access_key)
print("Secret Key: ", secret_key)

# Initialize SQS client
queue_url = 'https://sqs.us-east-2.amazonaws.com/905418297534/MyQueue.fifo'
session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name="us-east-2")
sqs = session.client('sqs')

def send_message(message):
    try:
        # Send message to the queue
        response = sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=message,
            MessageGroupId='messageGroup1'
        )
        print(f"Message sent: {response['MessageId']}")
    except Exception as e:
        print(f"Error sending message: {e}")

if __name__ == "__main__":
    # Prompt user to enter a message
    message = input("Enter message to send: ")

    # Send the message to the queue
    send_message(message)

```

### remote-client.py

```python
import socket
import json

# Define the IP address and port of the module running on the Spot
MODULE_IP = '0.0.0.0'  # Replace with the actual IP address
MODULE_PORT = 12345  # Replace with the actual port

def send_spot_instructions(instructions):
    try:
        # Create a socket connection
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.connect((MODULE_IP, MODULE_PORT))

            # Convert instructions to JSON format
            instructions_json = json.dumps(instructions)

            # Send instructions to the module
            s.sendall(instructions_json.encode())

            # Receive response from the module if needed
            # response = s.recv(1024)
            # print("Response:", response.decode())

    except Exception as e:
        print("An error occurred while sending instructions to Spot:", e)

# Example instructions to move Spot forward by 0.5 meters
instructions = {
    "action": "move_forward",
    "distance": 0.5
}

# Send instructions to Spot
send_spot_instructions(instructions)

# Listen for more instructions and send them to Spot as needed
while True:
    # User input
    action = input("Enter an action for Spot (e.g., move_forward, turn_left, etc.): ")
    distance = float(input("Enter the distance or angle for the action: "))
    instructions = {
        "action": action,
        "distance": distance
    }

    # Send instructions to Spot
    send_spot_instructions(instructions)


```

### healthrec.py

```python
# terra-python
import logging
import flask
from flask import request
from terra.base_client import Terra

logging.basicConfig(level=logging.INFO)
_LOGGER = logging.getLogger("app")

terra = Terra(api_key='tmrHjwLQuYDgKXwXRcltrBN_cCEcTyhs', dev_id='terraapitest-testing-UbidJIemMU', secret="912dea01818b97b313312af52ed65f6372224d76eaec5889")

app = flask.Flask(__name__)

@app.route("/consumeTerraWebhook", methods=["POST"])
def consume_terra_webhook() -> flask.Response:
    # body_str = str(request.get_data(), 'utf-8')
    body = request.get_json()
    _LOGGER.info(
        "Received webhook for user %s of type %s",
        body.get("user", {}).get("user_id"),
        body["type"])
    verified = terra.check_terra_signature(request.get_data().decode("utf-8"), request.headers['terra-signature'])
    data_inner_dict = body["data"][0]
    potentials_ecg_list = data_inner_dict["heart_data"]["ecg_signal"][0]["raw_signal"] #Emulating the fetching of data from wearable device using Terra API
    #Exponentially weighted moving averages
    disorder = False
    THRESH_VAL = 100
    TOTAL_LEN = len(potentials_ecg_list)
    BATCH_SIZE = 5
    alpha = 2.0 / (BATCH_SIZE + 1)
    curAvg = 0.0
    iterVal = 0
    for initVal in potentials_ecg_list[:BATCH_SIZE]:
       curAvg += abs(initVal["potential_uV"])
    curAvg /= BATCH_SIZE
    for elem in potentials_ecg_list[BATCH_SIZE:]:
      curAvg = (alpha * abs(curAvg)) + ((1 - alpha) * abs(elem["potential_uV"]))
      # Debugging/Testing
      # print(elem["potential_uV"])
      if (curAvg > THRESH_VAL):
         iterVal += 1
      if (iterVal > int(0.1 * TOTAL_LEN)):
         print("Warning! Heart patterns indicate risk of heart arrhythmia!")
         disorder = True
         break
    if (not disorder):
       print("Heart rate appears to be healthy!")
    if verified:
      return flask.Response(status=200)
    else:
      return flask.Response(status=403)
    
    
if __name__ == "__main__":
    app.run(host="localhost", port=8000)
```

### remote.py

```python
import socket
import json
from spot_controller import SpotController

# Define the IP address and port for the server to listen on
SERVER_IP = '0.0.0.0'  # Listen on all available network interfaces
SERVER_PORT = 12345  # Choose a port number for communication

# Initialize SpotController with your credentials and robot IP
SPOT_USERNAME = "admin"
SPOT_PASSWORD = "2zqa8dgw7lor"
ROBOT_IP = "10.0.0.3"

def handle_client_connection(client_socket):
    try:
        # Receive data from the client
        data = client_socket.recv(1024).decode()

        # Parse the received JSON data
        instructions = json.loads(data)

        # Perform the action based on the instructions received
        if instructions["action"] == "move_forward":
            distance = instructions["distance"]
            # Move Spot forward by the specified distance
            # Example: spot.move_to_goal(goal_x=distance, goal_y=0)
            print(f"Moving Spot forward by {distance} meters")

        # You can add more actions here based on your requirements

        # Send a response back to the client if needed
        # client_socket.sendall(b"Instruction received and executed successfully")

    except Exception as e:
        print("An error occurred while processing client instructions:", e)

    finally:
        # Close the client socket
        client_socket.close()

def start_server():
    # Create a socket for the server
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
        # Bind the socket to the specified IP address and port
        server_socket.bind((SERVER_IP, SERVER_PORT))

        # Listen for incoming connections
        server_socket.listen(5)
        print(f"Server listening on {SERVER_IP}:{SERVER_PORT}")

        while True:
            # Accept incoming connections
            client_socket, _ = server_socket.accept()
            print("Client connected")

            # Handle the client connection in a separate thread or process
            handle_client_connection(client_socket)

# Main function
if __name__ == "__main__":
    start_server()

```

[7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]