# Project export: HeySpot

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Empowering independence through AI-guided robotic assistance
- Devpost: https://devpost.com/software/robodog-1uo5rs
- GitHub: https://github.com/purelyKai/Cal-Hacks-11.0/
- Video: https://player.vimeo.com/video/1021486813?byline=0&portrait=0&title=0#t=
- Team: 4 GitHub contributor(s) — Kai Black (47 commits), Taz Larson (19 commits), Taz Larson (2 commits), ka-reem (1 commits)

## Devpost submission (written by the team)

### Overview

Abstract: This project aims to assist visually impaired individuals, including those who are blind, by leveraging the Boston Dynamics Spot robot and AI technology to help them navigate complex environments. Through voice commands and intelligent decision-making, the robot autonomously moves in response to user inputs. The system utilizes Deepgram's speech-to-text API to process voice commands and Groq’s AI API to determine appropriate movement actions. The commands are then executed using the Boston Dynamics Spot SDK. Although network connectivity challenges limited testing, the project lays a solid foundation for further development into a practical assistive technology. Features: Voice Command Input: Users provide voice commands to control the robot’s movements. Speech-to-Text Processing: Audio is converted to text using the Deepgram API. AI-Powered Decision Making: Groq's AI API processes text commands to determine movement sequences. Robot Movement Execution: Commands are sent to Spot using the Boston Dynamics Spot SDK. Docker Integration: The application is containerized using Docker and runs on Spot’s onboard computer. Tech Stack: Hardware: Boston Dynamics Spot robot APIs: Deepgram: Speech-to-text conversion Groq: AI-based decision making for movement Boston Dynamics Spot SDK: For robot control Deepgram: Speech-to-text conversion Groq: AI-based decision making for movement Boston Dynamics Spot SDK: For robot control Programming Language: Python Containerization: Docker Platform: Runs on Spot’s onboard computer About the Project:

### Inspiration

: Our team was inspired by the challenges faced by visually impaired individuals in navigating unfamiliar or complex environments. We aimed to use cutting-edge robotics and AI technology to create a guide dog-like experience that could improve independence and safety for blind and visually impaired users. What We Learned: We learned the complexities of working with robotics hardware in real-world conditions, especially when integrating multiple APIs for voice control and AI-based decision-making. Handling real-time input and making decisions autonomously based on that data was both a rewarding and educational experience. How We Built the Project: We developed a Python-based application that runs through a Docker image on the Spot robot’s onboard computer. The system takes voice command input, converts it to text using Deepgram, and processes the command using Groq’s AI to decide on the best movement action. The Boston Dynamics Spot SDK then executes these movements. Challenges We Faced: Our main challenge was network connectivity with the Spot robot. Due to these issues, we were unable to test our program on the actual robot until late on the second day of the Hackathon. This delayed our progress and limited our ability to fine-tune and optimize the robot’s real-world interactions. Despite this setback, we successfully implemented a functional voice-controlled movement system. Things to Consider: Current version only includes basic voice commands and robot movement. Further development could involve more complex navigation logic, multi-modal feedback (audio/tactile), and integration with Fetch.ai for real-time path planning. Future Work: Enhanced Command Recognition: Add sentiment detection or context for better alignment of verbal inputs with actions. Multi-modal Feedback: Implement audio and tactile feedback to improve user interaction. Path Planning: Integrate Fetch.ai for dynamic navigation and obstacle avoidance.

## README (from the GitHub repository)

# AI-Powered Voice-Controlled Spot Robot (Cal-Hacks-11.0)

![alt text](Spot-Robot-Image.jpg)

## Abstract:

This project explores the use of Boston Dynamics’ Spot robot, integrating AI and voice control to assist in navigating complex environments. The Python-based application processes voice commands through Deepgram's speech-to-text API and utilizes Groq’s AI API to determine the robot's movement actions. The robot executes these actions using the Boston Dynamics Spot SDK. The current implementation showcases the ability to control Spot via voice commands, laying a foundation for future development as an assistive technology.

## Features:

- **Voice Command Input**: Users provide voice commands to control the robot’s movements.
- **Speech-to-Text Processing**: Audio is converted to text using the Deepgram API.
- **AI-Powered Decision Making**: Groq's AI API, provided with context, processes text commands to determine movement sequences.
- **Robot Movement Execution**: Commands are sent to Spot using the Boston Dynamics Spot SDK.
- **Docker Integration**: The application is containerized using Docker and runs on Spot’s onboard computer.

## Tech Stack:

- **Hardware**: Boston Dynamics Spot robot
- **Boston Dynamics Spot SDK**: For robot control
- **APIs**:
  - **Deepgram**: Speech-to-text conversion
  - **Groq**: AI-semantic decision making for movement
- **Programming Language**: Python
- **Containerization**: Docker
- **Platform**: Runs on Spot’s onboard computer

## Things to Consider:

- Current version only includes voice commands and robot movement.
- Further development could involve more complex navigation logic, multi-modal feedback (audio/tactile), and integration with Fetch.ai for real-time path planning.

## Future Work:

- **Enhanced Command Recognition**: Add sentiment detection or context for better alignment of verbal inputs with actions.
- **Multi-modal Feedback**: Implement audio and tactile feedback to improve user interaction.
- **Path Planning**: Integrate Fetch.ai agents for dynamic navigation and obstacle avoidance on a large scale (i.e. the city or surrounding blocks).


## Detected evidence (automated analysis)

Indexed codebase: 14 recognized source files, 24 KB.
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (23 of 23)

```
.env.example
.github/workflows/build.yaml
.gitignore
job.json
LICENSE
README.md
requirements.txt
Server/__init__.py
Server/deepgram.py
Server/fetch_ai.py
Server/requirements.txt
Server/server.py
Server/stt.py
Spot/.gitignore
Spot/commands.py
Spot/config.py
Spot/Dockerfile
Spot/groq_ai.py
Spot/main.py
Spot/README.md
Spot/requirements.txt
Spot/spot_controller.py
Spot/stt.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Fixed deployment
- Add support for more movement commands
- Add image to README + Documentation
- Adjust environment and job
- Update README.md
- .
- please work
- fix?
- bug fix
- notify whene getting voice input
- mic issues fixed
- kill arecord
- better prompt
- better prompt
- code cleanup
- placeholder
- final test?
- audio env var
- fix typo
- .

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
��b o s d y n - a p i = = 4 . 1 . 0  
 b o s d y n - c h o r e o g r a p h y - c l i e n t = = 4 . 1 . 0  
 b o s d y n - c l i e n t = = 4 . 1 . 0  
 b o s d y n - c o r e = = 4 . 1 . 0  
 b o s d y n - m i s s i o n = = 4 . 1 . 0  
 b o s d y n - o r b i t = = 4 . 1 . 0  
 c e r t i f i = = 2 0 2 4 . 8 . 3 0  
 c h a r s e t - n o r m a l i z e r = = 3 . 4 . 0  
 D e p r e c a t e d = = 1 . 2 . 1 4  
 g r p c i o = = 1 . 6 7 . 0  
 i d n a = = 3 . 1 0  
 n u m p y = = 2 . 1 . 2  
 p r o t o b u f = = 5 . 2 8 . 2  
 P y J W T = = 2 . 9 . 0  
 p y n m e a 2 = = 1 . 1 9 . 0  
 r e q u e s t s = = 2 . 3 2 . 3  
 u r l l i b 3 = = 2 . 2 . 3  
 w r a p t = = 1 . 1 6 . 0  
 
```

### Spot/Dockerfile

```

FROM ghcr.io/merklebot/hackathon-amd-image:master as build


ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETOS
ARG TARGETARCH

ARG Version
ARG GitCommit
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" 


COPY requirements.txt requirements.txt
RUN python3.8 -m pip install -r requirements.txt
COPY . .

CMD ["python3.8", "main.py"]

```

### Spot/requirements.txt

```
��b o s d y n - a p i  
 b o s d y n - c h o r e o g r a p h y - c l i e n t  
 b o s d y n - c l i e n t  
 b o s d y n - c o r e  
 b o s d y n - m i s s i o n  
 b o s d y n - o r b i t  
 c e r t i f i  
 c h a r s e t - n o r m a l i z e r  
 D e p r e c a t e d  
 g r p c i o  
 i d n a  
 n u m p y  
 p r o t o b u f  
 P y J W T  
 p y n m e a 2  
 r e q u e s t s  
 u r l l i b 3  
 w r a p t  
 g r o q  
 d e e p g r a m - s d k = = 2 . *  
 p y t h o n _ d o t e n v  
 a s y n c i o  
 
```

### Server/requirements.txt

```
��b l i n k e r = = 1 . 8 . 2  
 b o s d y n - a p i = = 4 . 1 . 0  
 b o s d y n - c h o r e o g r a p h y - c l i e n t = = 4 . 1 . 0  
 b o s d y n - c l i e n t = = 4 . 1 . 0  
 b o s d y n - c o r e = = 4 . 1 . 0  
 b o s d y n - m i s s i o n = = 4 . 1 . 0  
 b o s d y n - o r b i t = = 4 . 1 . 0  
 c e r t i f i = = 2 0 2 4 . 8 . 3 0  
 c h a r s e t - n o r m a l i z e r = = 3 . 4 . 0  
 c l i c k = = 8 . 1 . 7  
 c o l o r a m a = = 0 . 4 . 6  
 D e p r e c a t e d = = 1 . 2 . 1 4  
 F l a s k = = 3 . 0 . 3  
 g r p c i o = = 1 . 6 7 . 0  
 i d n a = = 3 . 1 0  
 i t s d a n g e r o u s = = 2 . 2 . 0  
 J i n j a 2 = = 3 . 1 . 4  
 M a r k u p S a f e = = 3 . 0 . 2  
 n u m p y = = 2 . 1 . 2  
 p r o t o b u f = = 5 . 2 8 . 2  
 P y J W T = = 2 . 9 . 0  
 p y n m e a 2 = = 1 . 1 9 . 0  
 p y t h o n - d o t e n v = = 1 . 0 . 1  
 r e q u e s t s = = 2 . 3 2 . 3  
 u r l l i b 3 = = 2 . 2 . 3  
 W e r k z e u g = = 3 . 0 . 4  
 w r a p t = = 1 . 1 6 . 0  
 
```

### Server/server.py

```python
from flask import Flask, request, jsonify
import deepgram
import fetch_ai
#from spot_control.spot_interface import SpotInterface
from Spot.config import Config

app = Flask(__name__)

# Initialize the Spot interface
#spot = SpotInterface(Config.SPOT_IP, Config.SPOT_PORT)

@app.route('/transcribe', methods=['POST'])
def transcribe_voice():
    """Handle voice input from Spot, send to Deepgram, and process it."""
    # Get voice data from the request
    voice_data = request.files.get('voice')
    if not voice_data:
        return jsonify({'error': 'No voice data received'}), 400

    # Call Deepgram to transcribe the voice input
    transcript = deepgram.transcribe(voice_data)
    
    # Send the transcript to Fetch.ai for processing
    response = fetch_ai.process_text(transcript)
    
    # Return the result
    return jsonify({'response': response})

@app.route('/spot/command', methods=['POST'])
def send_spot_command():
    """Receive a command and send it to the Spot robot."""
    data = request.json
    command = data.get('command')
    
    if not command:
        return jsonify({'error': 'No command provided'}), 400
    
    # Execute the command on the Spot robot
    #result = spot.execute_command(command)
    result = 0
    
    return jsonify({'status': 'success', 'result': result})

if __name__ == '__main__':
    app.run(host=Config.SERVER_HOST, port=Config.SERVER_PORT, debug=Config.DEBUG)

```

### Spot/main.py

```python
import time
from spot_controller import SpotController
from commands import SpotCommands

ROBOT_IP = "192.168.80.3"  # Replace with actual robot IP
SPOT_USERNAME = "admin"    # Replace with Spot username
SPOT_PASSWORD = "2zqa8dgw7lor"  # Replace with Spot password

def main():
    print("Initializing Spot...")
    time.sleep(2)

    with SpotController(username=SPOT_USERNAME, password=SPOT_PASSWORD, robot_ip=ROBOT_IP) as spot:
        spotCommander = SpotCommands(spot)
        spot.power_on_stand_up()
        print("Spot powered on and standing.")

        try:
            while True:
                print("Awaiting commands...")
                commands = spotCommander.getCommands()

                for command in commands:
                    if "forward" in command:
                        spotCommander.forward()
                    elif "back" in command:
                        spotCommander.back()
                    elif "turnLeft" in command:
                        spotCommander.turnLeft()
                    elif "turnRight" in command:
                        spotCommander.turnRight()
                    elif "bow" in command:
                        spotCommander.bow()
                    elif "circleDance" in command:
                        spotCommander.circleDance(radius=1, duration=5)
                    elif "lieDown" in command:
                        spotCommander.lieDown()
                        break
                    elif "sidestep" in command:
                        direction = "left" if "left" in command else "right"
                        spotCommander.sidestep(direction=direction, steps=2)
                    elif "patrol" in command:
                        waypoints = [(1, 0), (2, 2), (0, 2)]  # Example waypoints
                        spotCommander.patrol(waypoints=waypoints)
                    else:
                        print(f"Unknown command: {command}")
                time.sleep(2)

        except KeyboardInterrupt:
            print("Shutting down...")
            spot.power_off_sit_down()
            print("Spot is powered off.")

if __name__ == "__main__":
    main()

```

### Server/__init__.py

```python
from .server import app
```

### Server/stt.py

```python
from flask import Flask, request, jsonify
from deepgram import DeepgramClient, PrerecordedOptions
from Spot.config import Config

app = Flask(__name__)

# Initialize Deepgram client
deepgram = DeepgramClient(Config.DEEPGRAM_API_KEY)

# @app.route('/transcribe', methods=['POST'])
# def transcribe(audio_data):

        

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)  # Update port as necessary

```

### Spot/config.py

```python
import os
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

class Config:
    # Deepgram and Fetch.ai API keys
    DEEPGRAM_API_KEY = os.getenv('DEEPGRAM_API_KEY', 'default_deepgram_key')
    FETCH_AI_API_KEY = os.getenv('FETCH_AI_API_KEY', 'default_fetchai_key')
    Spot_Groq_Api_Key = os.getenv("Spot_Groq_Api_Key")

    # Spot robot configuration
    AGENT_RPC = os.getenv('AGENT_RPC', 'ws://127.0.0.1:8888')
    OWNER_KEY = os.getenv('OWNER_KEY', '')
    USER_KEY_PATH = os.getenv('USER_KEY_PATH', '')

    # Flask settings
    ENV = os.getenv('FLASK_ENV', 'development')
    DEBUG = ENV == 'development'

    # Spot Robot network configuration
    SPOT_IP = os.getenv('SPOT_IP', '127.0.0.1')
    SPOT_PORT = int(os.getenv('SPOT_PORT', '8080'))

    SERVER_HOST = '0.0.0.0'
    SERVER_PORT = 5000

```

### Spot/groq_ai.py

```python
from groq import Groq
import time

def get_commands(text):
    client = Groq(
        api_key = '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',
    )
    chat_completion = client.chat.completions.create(
        messages = [
            {
                "role": "system",
                "content": """Given the MESSAGE, extract any relevant commands to those listed below in the order they appear.
                - forward
                - back
                - turnLeft
                - turnRight

                Respond with only the commands mentioned in the MESSAGE below, list only commands you hear, it's okay to list commands more than once if they are in MESSAGE more than once.
                
                """

            },
            {
            "role":"user",
            "content":f"MESSAGE: {text}",
            
            }
        ],
        model = "llama-3.1-70b-versatile",
    )

    time.sleep(5)

    ret_str = chat_completion.choices[0].message.content

    print(ret_str)
    
    return ret_str

```

[5 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]