# Project export: Pyro Machitis This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Pyro Machitis, built on Boston Dynamics Spot, locates fire victims using vision, communicates in 15 languages, guides them to safety, relays their location to firefighters, and is remotely controlled. - Devpost: https://devpost.com/software/robot-z6y2gp - GitHub: https://github.com/nJasow04/3b1y-dog-client-side - Video: https://www.youtube.com/embed/jHj0dxjJoFg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Ansh Dholakia (5 commits), Jason Wan (5 commits), Sushant Gupta (1 commits) ## Devpost submission (written by the team) ### Inspiration In emergency situations like fires, every second counts, and clear communication is crucial. We were inspired by the potential of Boston Dynamics' Spot to assist in rescue operations by bridging language barriers and navigating hazardous environments, made possible with flame-retardant polyurethane foam used to make Spot. The idea of a multilingual robotic assistant that could locate victims and aid firefighters motivated us to create Pyro Machitis. What We Learned Developing Pyro Machitis taught us the importance of interdisciplinary collaboration. We gained hands-on experience with robotics integration, real-time language processing, and user interface design. We also learned how to use multiple APIs together to make a functioning application. How We Built the Project Hardware Integration: Enhanced the Boston Dynamics Spot robot with high-resolution cameras to detect victims in a fire. Software Development: Utilized the Bosdyn API to program movement software controlled remotely and relay the video and audio to the controller. Multilingual Communication: Leveraged Groq's AI accelerators and Google Cloud Platform to enable real-time translation and communication in over 15 languages. User Interface: Created an intuitive UI using HTML, CSS, and Flask for operators to monitor and control the robot effectively. Backend Systems: Integrated systems using Bosdyn for robot control, Groq for AI processing, and GCP for data management. Challenges We Faced New Technology: Working with Boston Dynamics was a first for all of us, and it required us to read and understand complex documentation and conduct extensive testing to make it work as intended. Accessing Spot's hardware had complex requirements that we had to overcome. Integration Complexity: Combining hardware and software from multiple platforms required extensive testing and problem-solving, especially since the hardware can function remotely without being physically connected to the front end. ## README (from the GitHub repository) # CalHacks 13 monorepo > The spot for our code for running spotter on [Spot](https://bostondynamics.com/products/spot/) **backend**: web server for multilingual, multimodal AI processing - REST API written in Python using Flask - [Poetry](https://python-poetry.org/) for dependency management - [Google Cloud Services](https://cloud.google.com/?hl=en) - [Gemini](https://gemini.google.com/) - [Groq](https://groq.ai/) **Root folder**: frontend that enables easy robot control and shows the robot's camera and data feed - serves data and object recognition from AI backend - makes requests to robot control backend based on keyboard and mouse inputs - [Bun](https://bun.sh/) - [React](https://react.dev/) - [shadcn-ui](https://ui.shadcn.com/) components **robot-control-server**: web server that enables remote control of the robot - REST API/gRPC service written in Python using Flask - Boston Dynamics SDK ---- ## Press - Demo - https://devpost.com/software/robot-z6y2gp ---- ## Running the stack: First, clone this repo. If you don't have access to a Spot, you can still demo the entire frontend and AI part of the app locally, as long as you provide your Groq and GCP Client Json as an environment variables. ### backend 1. install Poetry 2. `poetry install` 3. `poetry run dev` to start the backend ### frontend 1. npm i 2. npm run dev ### robot server 1. connect to Spot's wifi network 2. run `python3 python/examples/wasd_server/app.py` and replace the hostname `192.168.80.3` with your Spot's IP 3. enter the `admin` username & password, or the equivalent credentials for your Spot 4. send requests to control endpoints manually or via the frontend ## Detected evidence (automated analysis) Indexed codebase: 21 recognized source files, 65 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code ## Codebase structure (from repository index) ### Files (37 of 37) ``` .gitignore backend/app.py backend/client_secret.json backend/Dockerfile backend/gcp-yt.json backend/image_stream_pb2_grpc.py backend/image_stream_pb2.py backend/image_stream.proto backend/microservice.py backend/pyproject.toml backend/requirements.txt backend/scratch.py LICENSE MIA_Demo_(Empty).ipynb package.json public/app.js public/index.html public/manifest.json public/style.css README.md robot-controller/Dockerfile robot-controller/job.json robot-controller/LICENSE robot-controller/main.py robot-controller/README.md robot-controller/requirements.txt robot-controller/spot_controller.py robot-controller/user.key robot-controller/voice.py src/App.css src/App.js src/App.test.js src/index.css src/index.js src/reportWebVitals.js src/setupTests.js wasd.py ``` ### Dependencies - backend/pyproject.toml: autoflake@^2.0.0, black@^22.12.0, flake8@^6.0.0, flake8-bugbear@^22.12.6, flask@^3.0.0, google-cloud-texttospeech@2.18.0, groq@0.11.0, gtts@^2.4.0, hume@^0.4.1, isort@^5.11.4, mypy@^0.991, openai@1.23.2, opencv-python@^4.8.1.78, pep8-naming@^0.13.3, pyaudio@^0.2.13, pydub@0.25.1, pygame@2.6.1, pytest@^7.2.0, pytest-cov@^4.0.0, python-dotenv@^1.0.0 - backend/requirements.txt: fastapi@==0.95.2, grpcio@==1.67.0, grpcio-tools@==1.67.0, Pillow@==10.0.0, protobuf@==5.28.2, uvicorn@==0.22.0 - package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, react@^18.3.1, react-dom@^18.3.1, react-scripts@5.0.1, web-vitals@^2.1.4 - robot-controller/requirements.txt: bosdyn-client@>= 3.1 ### Recent commits (newest first) - Created using Colab - Merge branch 'main' of https://github.com/nJasow04/3b1y-dog-client-side - adding robot controller and readme - spot movement - adding voice detection backedn - Merge branch 'main' of https://github.com/nJasow04/3b1y-dog-client-side - adding dockerilfe etc - 90% finished UI - updating code for Sushant - Since we're just doing frontend, we don't actually need to do any complex stuff. Its just one page - Set up react app - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "3b1y-dog-client-side", "version": "0.1.0", "private": true, "dependencies": { "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "react": "^18.3.1", "react-dom": "^18.3.1", "react-scripts": "5.0.1", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### robot-controller/requirements.txt ``` bosdyn-client >= 3.1 ``` ### backend/requirements.txt ``` fastapi==0.95.2 uvicorn==0.22.0 grpcio==1.67.0 grpcio-tools==1.67.0 Pillow==10.0.0 protobuf==5.28.2 ``` ### robot-controller/Dockerfile ``` FROM ghcr.io/merklebot/hackathon-amd-image:master as build ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ARG TARGETPLATFORM ARG BUILDPLATFORM ARG TARGETOS ARG TARGETARCH ARG Version ARG GitCommit RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" COPY requirements.txt requirements.txt RUN python3.8 -m pip install -r requirements.txt COPY . . CMD ["python3.8", "main.py"] ``` ### backend/Dockerfile ``` # Use the official Python image from the Docker Hub FROM python:3.10-slim # Set environment variables ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 # Set work directory WORKDIR /app # Install system dependencies RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ && rm -rf /var/lib/apt/lists/* # Install Python dependencies COPY requirements.txt . RUN pip install --upgrade pip RUN pip install --no-cache-dir -r requirements.txt # Copy project COPY . . # Expose the port that the app runs on # Cloud Run expects the service to listen on the port specified by the PORT environment variable, default 8080 EXPOSE 8080 # Set environment variables for gRPC ENV GRPC_PORT=50051 # Define the default command to run the app using Uvicorn CMD ["uvicorn", "microservice:app", "--host", "0.0.0.0", "--port", "8080"] ``` ### backend/pyproject.toml ``` [tool.poetry] name = "backend" version = "0.1.0" description = "" authors = ["Ansh Dholakia", "Krishan"] readme = "README.md" [tool.poetry.dependencies] python = "^3.10" flask = "^3.0.0" opencv-python = "^4.8.1.78" groq = "0.11.0" python-dotenv = "^1.0.0" pyaudio = "^0.2.13" gtts = "^2.4.0" pygame = "2.6.1" openai = "1.23.2" pydub = "0.25.1" hume = "^0.4.1" [tool.poetry.dev-dependencies] pytest = "^7.2.0" isort = "^5.11.4" flake8 = "^6.0.0" mypy = "^0.991" pytest-cov = "^4.0.0" black = "^22.12.0" autoflake = "^2.0.0" flake8-bugbear = "^22.12.6" pep8-naming = "^0.13.3" google-cloud-texttospeech = "2.18.0" [tool.mypy] python_version = "3.10" disallow_untyped_defs = true check_untyped_defs = true warn_redundant_casts = true warn_unused_ignores = true ignore_missing_imports = true [tool.isort] line_length = 120 multi_line_output = 3 use_parentheses = true include_trailing_comma = true force_grid_wrap = 0 [tool.black] line_length = 120 exclude = ''' /( \.eggs # exclude a few common directories in the | \.git # root of the project | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ ''' [tool.poetry.scripts] lint = "scripts.scripts:lint" l = "scripts.scripts:lint" format = "scripts.scripts:format_code" f = "scripts.scripts:format_code" format-lint = "scripts.scripts:format_and_lint" fl = "scripts.scripts:format_and_lint" test = "scripts.scripts:test" t = "scripts.scripts:test" format-lint-test = "scripts.scripts:format_lint_test" flt = "scripts.scripts:format_lint_test" dev = "scripts.scripts:dev" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ``` ### src/App.js ```javascript import logo from './logo.svg'; import './App.css'; function App() { return (
logo

Edit src/App.js and save to reload.

Learn React
); } export default App; ``` ### src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### robot-controller/main.py ```python import os import time from spot_controller import SpotController import cv2 ROBOT_IP = "192.168.80.3"#os.environ['ROBOT_IP'] SPOT_USERNAME = "admin"#os.environ['SPOT_USERNAME'] SPOT_PASSWORD = "2zqa8dgw7lor"#os.environ['SPOT_PASSWORD'] def capture_image(): camera_capture = cv2.VideoCapture(0) rv, image = camera_capture.read() print(f"Image Dimensions: {image.shape}") camera_capture.release() def main(): #example of using micro and speakers print("Start recording audio") sample_name = "aaaa.wav" cmd = f'arecord -vv --format=cd --device={os.environ["AUDIO_INPUT_DEVICE"]} -r 48000 --duration=10 -c 1 {sample_name}' print(cmd) os.system(cmd) print("Playing sound") os.system(f"ffplay -nodisp -autoexit -loglevel quiet {sample_name}") # # Capture image # Use wrapper in context manager to lease control, turn on E-Stop, power on the robot and stand up at start # and to return lease + sit down at the end with SpotController(username=SPOT_USERNAME, password=SPOT_PASSWORD, robot_ip=ROBOT_IP) as spot: time.sleep(2) capture_image() # Move head to specified positions with intermediate time.sleep spot.move_head_in_points(yaws=[0.2, 0], pitches=[0.3, 0], rolls=[0.4, 0], sleep_after_point_reached=1) capture_image() time.sleep(3) # Make Spot to move by goal_x meters forward and goal_y meters left spot.move_to_goal(goal_x=1, goal_y=0.5) time.sleep(3) capture_image() # Control Spot by velocity in m/s (or in rad/s for rotation) # spot.move_by_velocity_control(v_x=-0.3, v_y=0, v_rot=0, cmd_duration=2) # capture_image() # time.sleep(3) # Go back to the initial position spot.move_to_goal(goal_x=-1, goal_y=-0.5) time.sleep(3) capture_image() if __name__ == '__main__': main() ``` ### backend/app.py ```python import os from pathlib import Path from dotenv import load_dotenv from flask import Flask, jsonify, request, send_from_directory from google.cloud import texttospeech from groq import Groq import uuid # Load environment variables from .env file load_dotenv() # Set Google Application Credentials os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'gcp-yt.json' # Initialize Google Text-to-Speech client tts_client = texttospeech.TextToSpeechClient() # Initialize Groq client client = Groq( # This is the default and can be omitted api_key=os.environ.get("GROQ_API_KEY"), ) app = Flask(__name__) # Ensure the 'static' directory exists for serving audio files os.makedirs('static', exist_ok=True) @app.route("/audio-input", methods=["POST"]) def audio_input(): # Check if the request contains an audio file if 'audio' not in request.files: return jsonify({'error': 'No audio file in request'}), 400 audio_file = request.files['audio'] selected_language = request.form.get('language', 'en') # Default to English US if not provided print("Selected language:", selected_language) # Save the uploaded audio file with a unique filename original_filename = audio_file.filename if original_filename == '': original_filename = 'uploaded_audio' else: original_filename = os.path.splitext(original_filename)[0] uploaded_extension = os.path.splitext(audio_file.filename)[1] or '.webm' uploaded_filename = f"{original_filename}{uploaded_extension}" uploaded_path = os.path.join('static', uploaded_filename) audio_file.save(uploaded_path) # Send the audio file directly to the Whisper API for transcription try: # Send the audio file to Whisper API transcript_response = client.audio.transcriptions.create( model="whisper-large-v3", language=None, file=Path(uploaded_path), response_format="json" ) transcript = transcript_response.text except Exception as e: print(f"Error transcribing audio: {e}") return jsonify({'error': 'Failed to transcribe audio'}), 500 # Translate the transcription into the selected language using GPT model via Groq try: # Use the Chat Completion API for better translation completion = client.chat.completions.create( model="llama3-8b-8192", messages=[ { "role": "system", "content": f"You are a helpful assistant that translates text to {selected_language}. \ Please provide the translation without any additional text." }, { "role": "user", "content": transcript } ], temperature=0.5, ) translated_text = completion.choices[0].message.content.strip() print("Translated Text:", translated_text) except Exception as e: print(f"Error translating text: {e}") return jsonify({'error': 'Failed to translate text'}), 500 # Convert the translated text to speech using Google TTS try: # Set the text input to be synthesized synthesis_input = texttospeech.SynthesisInput(text=translated_text) # Build the voice request, select the language code and the ssml voice gender voice = texttospeech.VoiceSelectionParams( language_code=selected_language, # Select a voice based on language; you can customize this as needed # For example, using the first available voice for the language name=select_voice(selected_language) ) # Select the type of audio file you want returned audio_config = texttospeech.AudioConfig( audio_encoding=texttospeech.AudioEncoding.MP3 ) # Perform the text-to-speech request on the translated text response = tts_client.synthesize_speech( input=synthesis_input, voice=voice, audio_config=audio_config ) # Generate a unique filename for the synthesized audio audio_filename = f"{uuid.uuid4()}.mp3" audio_output_path = os.path.join('static', audio_filename) # Write the response to the output file with open(audio_output_path, 'wb') as out: out.write(response.audio_content) print(f'Audio content written to file {audio_output_path}') except Exception as e: print(f"Error in text-to-speech synthesis: {e}") return jsonify({'error': 'Failed to synthesize speech'}), 500 # Return the translated text and the URL to the synthesized audio return jsonify({ 'message': 'Audio file received, transcribed, translated, and synthesized successfully', 'transcript': transcript, 'translated_text': translated_text, 'language': selected_language, 'audio_url': f"/static/{audio_filename}" }), 200 def select_voice(language_code): """ Selects an appropriate voice based on the chosen language. You can expand this mapping as needed. """ voice_mapping = { 'en': 'en-US-Wavenet-D', 'es': 'es-ES-Wavenet-D', 'fr': 'fr-FR-Wavenet-D', 'de': 'de-DE-Wavenet-D', 'it': 'it-IT-Wavenet-D', # Add more mappings as needed } return voice_mapping.get(language_code, 'en-US-Wavenet-D') # Default to English US if not found # Route to serve static audio files @app.route('/static/') def serve_audio(filename): return send_from_directory('static', filename) if __name__ == "__main__": app.run(host="0.0.0.0", port=5001) ``` [14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]