# Project export: Spotter - Revolutionizing Disaster Relief This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Providing actionable information to first responders; we save lives with the power of robotics. - Devpost: https://devpost.com/software/spotter-revolutionizing-disaster-relief - GitHub: https://github.com/raviriley/code-spot - Result: winner (Best Use of Hume) - Team: 2 GitHub contributor(s) — Ravi Riley (24 commits), Jayaditya Sethi (1 commits) ## Devpost submission (written by the team) ### Inspiration Currently, in war-torn and disaster-struck areas, first responders are risking their lives unnecessarily, as they lack the resources needed to accurately and safely assess a disaster zone. By using robotics, we can prevent the risk of human life. ### What it does The SPOT robot has been enabled to be an independent rescue machine that understands human emotion and natural language using AI with a noted ability to detect a language and adjust output as such. ### How we built it We built it using a variety of tools including Hume for AI/transcription, OpenCV for ML Models, Flask for the backend, and Next.js for the frontend. ### Challenges we ran into Connecting to and controlling SPOT was extremely difficult. We got around this by building a custom control server that connects directly to SPOT and controls its motors. The Hume API was relatively friendly to use and we connected this to a live stream of data via the Continuity Camera. ### Accomplishments we're proud of Fixing SPOT's internal linux dependencies. This is something that blocked all teams from using SPOT and took up most of the first day. But by solving this, we enabled SPOT to be used by all teams. ### What we learned We learned it is quite complex to combine various tech stacks across a variety of products both hardware and software. We learned to approach these problems by introducing levels of abstraction that would allow parts of the team to work parallely. ### What's next We hope to fully autonomize SPOTTER so that SPOT can traverse and navigate disaster environments completely independently. In this way, SPOT can locate survivors and assess the situation globally. ## README (from the GitHub repository) # CalHacks 10 monorepo > The spot for our code for running spotter on [Spot](https://bostondynamics.com/products/spot/) **backend**: web server for multilingual, multimodal AI processing - REST API written in Python using Flask - [Poetry](https://python-poetry.org/) for dependency management - [Hume AI](https://hume.ai/) - [GPT-4](https://openai.com/research/gpt-4) - [Bark](https://github.com/suno-ai/bark) **spotter**: frontend that enables easy robot control and shows the robot's camera and data feed - serves data and object recognition from AI backend - makes requests to robot control backend based on keyboard and mouse inputs - [Bun](https://bun.sh/) - [Next.js](https://nextjs.org/) - [shadcn-ui](https://ui.shadcn.com/) components **robot-control-server**: web server that enables remote control of the robot - REST API written in Python using Flask - Boston Dynamics SDK ---- ## Press - Demo - https://www.youtube.com/watch?v=f8duOCsi8As - https://devpost.com/software/spotter-revolutionizing-disaster-relief - https://www.instagram.com/p/Czmc3bqy5Gq/ - LinkedIn - https://www.linkedin.com/feed/update/urn:li:activity:7126407430812356608/ - https://www.linkedin.com/posts/sutardjacenter_calhacks-ucberkeley-spot-activity-7126599039646629889-RRhp - Twitter - https://twitter.com/ravi_riley/status/1721648835670503774 - https://twitter.com/JayadityaSethi/status/1720578125556502914 ---- ## Running the stack: First, clone this repo. If you don't have access to a Spot, you can still demo the entire frontend and AI part of the app locally, as long as you provide your OpenAI and Hume.ai API keys as an environment variables. ### backend 1. install Poetry 2. `poetry install` 3. `poetry run dev` to start the backend ### frontend 1. install Bun 2. `bun install` 3. `bun dev` to start the frontend ### robot server 1. connect to Spot's wifi network 2. run `python3 python/examples/wasd_server/app.py` and replace the hostname `192.168.80.3` with your Spot's IP 3. enter the `admin` username & password, or the equivalent credentials for your Spot 4. send requests to control endpoints manually or via the frontend ## Detected evidence (automated analysis) Indexed codebase: 253 recognized source files, 3056 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 421) ``` .gitattributes .gitignore backend/.idea/.gitignore backend/.idea/backend.iml backend/.idea/inspectionProfiles/profiles_settings.xml backend/.idea/misc.xml backend/.idea/modules.xml backend/backend/__init__.py backend/backend/app.py backend/poetry.lock backend/pyproject.toml backend/README.md backend/requirements.txt backend/scripts/__init__.py backend/scripts/scripts.py backend/tests/__init__.py README.md robot-control-server/.gitattributes robot-control-server/.style.yapf robot-control-server/prebuilt/bosdyn_api-3.3.2-py3-none-any.whl robot-control-server/prebuilt/bosdyn_choreography_client-3.3.2-py3-none-any.whl robot-control-server/prebuilt/bosdyn_choreography_protos-3.3.2-py3-none-any.whl robot-control-server/prebuilt/bosdyn_client-3.3.2-py3-none-any.whl robot-control-server/prebuilt/bosdyn_core-3.3.2-py3-none-any.whl robot-control-server/prebuilt/bosdyn_mission-3.3.2-py3-none-any.whl robot-control-server/prebuilt/bosdyn_scout-3.3.2-py3-none-any.whl robot-control-server/protos/bosdyn/api/alerts.proto robot-control-server/protos/bosdyn/api/arm_command.proto robot-control-server/protos/bosdyn/api/arm_surface_contact_service.proto robot-control-server/protos/bosdyn/api/arm_surface_contact.proto robot-control-server/protos/bosdyn/api/auth_service.proto robot-control-server/protos/bosdyn/api/auth.proto robot-control-server/protos/bosdyn/api/auto_return/auto_return_service.proto robot-control-server/protos/bosdyn/api/auto_return/auto_return.proto robot-control-server/protos/bosdyn/api/autowalk/autowalk_service.proto robot-control-server/protos/bosdyn/api/autowalk/autowalk.proto robot-control-server/protos/bosdyn/api/autowalk/walks.proto robot-control-server/protos/bosdyn/api/basic_command.proto robot-control-server/protos/bosdyn/api/bddf.proto robot-control-server/protos/bosdyn/api/data_acquisition_plugin_service.proto robot-control-server/protos/bosdyn/api/data_acquisition_service.proto robot-control-server/protos/bosdyn/api/data_acquisition_store_service.proto robot-control-server/protos/bosdyn/api/data_acquisition_store.proto robot-control-server/protos/bosdyn/api/data_acquisition.proto robot-control-server/protos/bosdyn/api/data_buffer_service.proto robot-control-server/protos/bosdyn/api/data_buffer.proto robot-control-server/protos/bosdyn/api/data_chunk.proto robot-control-server/protos/bosdyn/api/data_index.proto robot-control-server/protos/bosdyn/api/data_service.proto robot-control-server/protos/bosdyn/api/directory_registration_service.proto robot-control-server/protos/bosdyn/api/directory_registration.proto robot-control-server/protos/bosdyn/api/directory_service.proto robot-control-server/protos/bosdyn/api/directory.proto robot-control-server/protos/bosdyn/api/docking/docking_service.proto robot-control-server/protos/bosdyn/api/docking/docking.proto robot-control-server/protos/bosdyn/api/estop_service.proto robot-control-server/protos/bosdyn/api/estop.proto robot-control-server/protos/bosdyn/api/fault_service.proto robot-control-server/protos/bosdyn/api/full_body_command.proto robot-control-server/protos/bosdyn/api/geometry.proto robot-control-server/protos/bosdyn/api/graph_nav/area_callback_data.proto robot-control-server/protos/bosdyn/api/graph_nav/area_callback_service.proto robot-control-server/protos/bosdyn/api/graph_nav/area_callback.proto robot-control-server/protos/bosdyn/api/graph_nav/graph_nav_service.proto robot-control-server/protos/bosdyn/api/graph_nav/graph_nav.proto robot-control-server/protos/bosdyn/api/graph_nav/map_processing_service.proto robot-control-server/protos/bosdyn/api/graph_nav/map_processing.proto robot-control-server/protos/bosdyn/api/graph_nav/map.proto robot-control-server/protos/bosdyn/api/graph_nav/nav.proto robot-control-server/protos/bosdyn/api/graph_nav/recording_service.proto robot-control-server/protos/bosdyn/api/graph_nav/recording.proto robot-control-server/protos/bosdyn/api/gripper_camera_param_service.proto robot-control-server/protos/bosdyn/api/gripper_camera_param.proto robot-control-server/protos/bosdyn/api/gripper_command.proto robot-control-server/protos/bosdyn/api/header.proto robot-control-server/protos/bosdyn/api/image_geometry.proto robot-control-server/protos/bosdyn/api/image_service.proto robot-control-server/protos/bosdyn/api/image.proto robot-control-server/protos/bosdyn/api/ir_enable_disable_service.proto robot-control-server/protos/bosdyn/api/ir_enable_disable.proto robot-control-server/protos/bosdyn/api/keepalive/keepalive_service.proto robot-control-server/protos/bosdyn/api/keepalive/keepalive.proto robot-control-server/protos/bosdyn/api/lease_service.proto robot-control-server/protos/bosdyn/api/lease.proto robot-control-server/protos/bosdyn/api/license_service.proto robot-control-server/protos/bosdyn/api/license.proto robot-control-server/protos/bosdyn/api/local_grid_service.proto robot-control-server/protos/bosdyn/api/local_grid.proto robot-control-server/protos/bosdyn/api/log_annotation_service.proto robot-control-server/protos/bosdyn/api/log_annotation.proto robot-control-server/protos/bosdyn/api/log_status/log_status_service.proto robot-control-server/protos/bosdyn/api/log_status/log_status.proto robot-control-server/protos/bosdyn/api/manipulation_api_service.proto robot-control-server/protos/bosdyn/api/manipulation_api.proto robot-control-server/protos/bosdyn/api/mission/mission_service.proto robot-control-server/protos/bosdyn/api/mission/mission.proto robot-control-server/protos/bosdyn/api/mission/nodes.proto robot-control-server/protos/bosdyn/api/mission/remote_service.proto robot-control-server/protos/bosdyn/api/mission/remote.proto robot-control-server/protos/bosdyn/api/mission/util.proto robot-control-server/protos/bosdyn/api/mobility_command.proto robot-control-server/protos/bosdyn/api/network_compute_bridge_service.proto robot-control-server/protos/bosdyn/api/network_compute_bridge.proto robot-control-server/protos/bosdyn/api/network_stats.proto robot-control-server/protos/bosdyn/api/parameter.proto robot-control-server/protos/bosdyn/api/payload_estimation.proto robot-control-server/protos/bosdyn/api/payload_registration_service.proto robot-control-server/protos/bosdyn/api/payload_registration.proto robot-control-server/protos/bosdyn/api/payload_service.proto robot-control-server/protos/bosdyn/api/payload.proto robot-control-server/protos/bosdyn/api/point_cloud_service.proto robot-control-server/protos/bosdyn/api/point_cloud.proto robot-control-server/protos/bosdyn/api/power_service.proto robot-control-server/protos/bosdyn/api/power.proto robot-control-server/protos/bosdyn/api/proto_reference.md robot-control-server/protos/bosdyn/api/ray_cast_service.proto robot-control-server/protos/bosdyn/api/ray_cast.proto robot-control-server/protos/bosdyn/api/README.md robot-control-server/protos/bosdyn/api/robot_command_service.proto robot-control-server/protos/bosdyn/api/robot_command.proto [301 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: autoflake@^2.0.0, black@^22.12.0, flake8@^6.0.0, flake8-bugbear@^22.12.6, flask@^3.0.0, gtts@^2.4.0, hume@^0.4.1, isort@^5.11.4, mypy@^0.991, opencv-python@^4.8.1.78, pep8-naming@^0.13.3, pyaudio@^0.2.13, pygame@^2.5.2, pytest@^7.2.0, pytest-cov@^4.0.0, python-dotenv@^1.0.0 - backend/requirements.txt: blinker@==1.6.3, certifi@==2023.7.22, charset-normalizer@==3.3.1, click@==8.1.7, colorama@==0.4.6, flask@==3.0.0, gtts@==2.4.0, idna@==3.4, itsdangerous@==2.1.2, jinja2@==3.1.2, markupsafe@==2.1.3, numpy@==1.26.1, opencv-python@==4.8.1.78, pyaudio@==0.2.13, pygame@==2.5.2, python-dotenv@==1.0.0, requests@==2.31.0, urllib3@==2.0.7, werkzeug@==3.0.1 - robot-control-server/python/examples/wasd_server/requirements.txt: bosdyn-client@>= 3.1, Flask, Pillow - robot-control-server/python/examples/wasd/requirements.txt: bosdyn-client@>= 3.1, Pillow, windows-curses - spotter/package.json: @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-icons@^1.3.0, @radix-ui/react-navigation-menu@^1.1.4, @radix-ui/react-popover@^1.0.7, @radix-ui/react-scroll-area@^1.0.5, @radix-ui/react-slot@^1.0.2, @tanstack/react-table@^8.10.7, @types/node@^20, @types/react@^18, @types/react-dom@^18, autoprefixer@^10, class-variance-authority@^0.7.0, clsx@^2.0.0, cmdk@^0.2.0, eslint@^8, eslint-config-next@14.0.0, husky@^8.0.3, lucide-react@^0.290.0, next@14.0.0, next-themes@^0.2.1, postcss@^8, react@^18, react-dom@^18, tailwind-merge@^1.14.0, tailwindcss@^3, tailwindcss-animate@^1.0.7, typescript@^5 ### Recent commits (newest first) - Update README.md - Update README.md - Update README - Add text to speech output files so repo viewers can hear - latest - face - Create alert-dialog.tsx - Update transcription-table.tsx - Add audio emotion - update tx table - Add timestamp to backend response - Update app.py - Update app.py - Update index.tsx - Add ai to transcription table - Add logging to audio testing page - Add audio page to navbar - transcript return - Fix webcam component - Update frontend to check for disconnected robot ## Key source files (fetched from GitHub, selected and truncated for size) ### spotter/package.json ``` { "name": "spotter", "version": "0.1.0", "private": true, "scripts": { "prepare": "husky install", "format": "prettier --write . --cache --cache-strategy content", "lint": "next lint", "typecheck": "tsc --noEmit --pretty --project tsconfig.json", "dev": "next dev", "build": "next build", "start": "next start" }, "dependencies": { "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-navigation-menu": "^1.1.4", "@radix-ui/react-popover": "^1.0.7", "@radix-ui/react-scroll-area": "^1.0.5", "@radix-ui/react-slot": "^1.0.2", "@tanstack/react-table": "^8.10.7", "class-variance-authority": "^0.7.0", "clsx": "^2.0.0", "cmdk": "^0.2.0", "lucide-react": "^0.290.0", "next": "14.0.0", "next-themes": "^0.2.1", "react": "^18", "react-dom": "^18", "tailwind-merge": "^1.14.0", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10", "eslint": "^8", "eslint-config-next": "14.0.0", "husky": "^8.0.3", "postcss": "^8", "tailwindcss": "^3", "typescript": "^5" } } ``` ### backend/requirements.txt ``` blinker==1.6.3 ; python_version >= "3.10" and python_version < "4.0" certifi==2023.7.22 ; python_version >= "3.10" and python_version < "4.0" charset-normalizer==3.3.1 ; python_version >= "3.10" and python_version < "4.0" click==8.1.7 ; python_version >= "3.10" and python_version < "4.0" colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0" and platform_system == "Windows" flask==3.0.0 ; python_version >= "3.10" and python_version < "4.0" gtts==2.4.0 ; python_version >= "3.10" and python_version < "4.0" idna==3.4 ; python_version >= "3.10" and python_version < "4.0" itsdangerous==2.1.2 ; python_version >= "3.10" and python_version < "4.0" jinja2==3.1.2 ; python_version >= "3.10" and python_version < "4.0" markupsafe==2.1.3 ; python_version >= "3.10" and python_version < "4.0" numpy==1.25.2 ; python_version >= "3.13" and python_version < "4.0" numpy==1.26.1 ; python_version >= "3.10" and python_version < "3.13" opencv-python==4.8.1.78 ; python_version >= "3.10" and python_version < "4.0" pyaudio==0.2.13 ; python_version >= "3.10" and python_version < "4.0" pygame==2.5.2 ; python_version >= "3.10" and python_version < "4.0" python-dotenv==1.0.0 ; python_version >= "3.10" and python_version < "4.0" requests==2.31.0 ; python_version >= "3.10" and python_version < "4.0" urllib3==2.0.7 ; python_version >= "3.10" and python_version < "4.0" werkzeug==3.0.1 ; python_version >= "3.10" and python_version < "4.0" ``` ### backend/pyproject.toml ``` [tool.poetry] name = "backend" version = "0.1.0" description = "" authors = ["Ravi Riley ", "Jay Sethi"] readme = "README.md" [tool.poetry.dependencies] python = "^3.10" flask = "^3.0.0" opencv-python = "^4.8.1.78" python-dotenv = "^1.0.0" pyaudio = "^0.2.13" gtts = "^2.4.0" pygame = "^2.5.2" hume = "^0.4.1" [tool.poetry.dev-dependencies] pytest = "^7.2.0" isort = "^5.11.4" flake8 = "^6.0.0" mypy = "^0.991" pytest-cov = "^4.0.0" black = "^22.12.0" autoflake = "^2.0.0" flake8-bugbear = "^22.12.6" pep8-naming = "^0.13.3" [tool.mypy] python_version = "3.10" disallow_untyped_defs = true check_untyped_defs = true warn_redundant_casts = true warn_unused_ignores = true ignore_missing_imports = true [tool.isort] line_length = 120 multi_line_output = 3 use_parentheses = true include_trailing_comma = true force_grid_wrap = 0 [tool.black] line_length = 120 exclude = ''' /( \.eggs # exclude a few common directories in the | \.git # root of the project | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ ''' [tool.poetry.scripts] lint = "scripts.scripts:lint" l = "scripts.scripts:lint" format = "scripts.scripts:format_code" f = "scripts.scripts:format_code" format-lint = "scripts.scripts:format_and_lint" fl = "scripts.scripts:format_and_lint" test = "scripts.scripts:test" t = "scripts.scripts:test" format-lint-test = "scripts.scripts:format_lint_test" flt = "scripts.scripts:format_lint_test" dev = "scripts.scripts:dev" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ``` ### robot-control-server/python/examples/wasd_server/requirements.txt ``` -f ../../../prebuilt bosdyn-client >= 3.1 Flask Pillow ``` ### robot-control-server/python/examples/wasd/requirements.txt ``` -f ../../../prebuilt bosdyn-client >= 3.1 Pillow windows-curses ; sys_platform == 'win32' ``` ### spotter/pages/index.tsx ```typescript import Image from "next/image"; import { DataTable } from "@/components/data-table/data-table"; import { Card, CardContent, CardDescription, CardFooter, CardHeader, CardTitle, } from "@/components/ui/card"; import WebCam from "@/components/webcam"; import { ScrollArea } from "@/components/ui/scroll-area"; import TranscriptionTable, { Transcription, } from "@/components/transcription-table"; import Controls from "@/components/controls"; export default function Home() { return ( <>
Live Feed front-facing camera from Spot Controls move Spot in real-time with your keyboard and/or these buttons
Live Feed real-time transcription from rescues ); } ``` ### backend/backend/app.py ```python from datetime import datetime import os import wave import json import cv2 import numpy as np import openai import pyaudio import pygame from dotenv import load_dotenv from flask import Flask, Response, jsonify from gtts import gTTS from hume import HumeBatchClient from hume.models.config import FaceConfig from hume.models.config import ProsodyConfig app = Flask(__name__) HUME_API_KEY = "" def generate_frames(): hog = cv2.HOGDescriptor() hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector()) cap = cv2.VideoCapture(1) while True: success, frame = cap.read() if not success: break frame = cv2.resize(frame, (640, 480)) boxes, weights = hog.detectMultiScale( frame, winStride=(4, 4), padding=(8, 8), scale=1.05, useMeanshiftGrouping=False ) boxes = np.array([[x, y, x + w, y + h] for (x, y, w, h) in boxes]) for (xA, yA, xB, yB) in boxes: cv2.rectangle(frame, (xA, yA), (xB, yB), (0, 255, 0), 2) ret, buffer = cv2.imencode(".jpg", frame) frame = buffer.tobytes() yield (b"--frame\r\n" b"Content-Type: image/jpeg\r\n\r\n" + frame + b"\r\n") @app.route("/video") def video(): return Response(generate_frames(), mimetype="multipart/x-mixed-replace; boundary=frame") def postAudioHume(): client = HumeBatchClient(HUME_API_KEY) filepaths = ["output.wav"] config = ProsodyConfig() job = client.submit_job(None, [config], files=filepaths) print(job) print("Running...") details = job.await_complete() results = job.get_predictions() # Extract the top emotion print(results) # json_string = results.strip('[]') # json_object = json.loads(json_string) # emotions = json_object['results']['predictions'][0]['models']['prosody']['grouped_predictions'][0]['predictions'][0]['emotions'] emotions = results[0]["results"]["predictions"][0]["models"]["prosody"]["grouped_predictions"][0]["predictions"][0]["emotions"] # sort emotions by score emotions.sort(key=lambda x: x["score"], reverse=True) sorted_audio_emotions = emotions[:3] print("sorted emotions: ", sorted_audio_emotions) for emotion in sorted_audio_emotions: print(f"{emotion['name']}: {float(emotion['score']) * 100}") return sorted_audio_emotions """ todo def imageHume(filepath): client = HumeBatchClient(HUME_API_KEY) filepaths = ["output.jpg"] config = FaceConfig() job = client.submit_job(None, [config], files=filepaths) print(job) print("Running...") details = job.await_complete() results = job.get_predictions() # Extract the top emotion json_string = results.strip('[]') json_object = json.loads(json_string) emotions = json_object['results']['predictions'][0]['models']['face']['grouped_predictions'][0]['predictions'][0]['emotions'] sorted_audio_emotions = sorted(emotions, key=lambda x: x['score'], reverse=True) print("sorted emotions: ", sorted_audio_emotions) return sorted_audio_emotions """ @app.route("/process_audio", methods=["GET"]) def process_audio(): load_dotenv() os.getenv("OPENAI_API_KEY") # Step 1: Play the help.wav audio pygame.mixer.init() pygame.mixer.music.load("help.wav") pygame.mixer.music.play() while pygame.mixer.music.get_busy(): continue # Step 2: Stream and record audio for 5 seconds p = pyaudio.PyAudio() FORMAT = pyaudio.paInt16 CHANNELS = 1 RATE = 44100 CHUNK = 1024 RECORD_SECONDS = 5 WAVE_OUTPUT_FILENAME = "output.wav" stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK) frames = [] for i in range(0, int(RATE / CHUNK * RECORD_SECONDS)): audio_data = stream.read(CHUNK) frames.append(audio_data) stream.stop_stream() stream.close() p.terminate() print("Recording finished") wf = wave.open(WAVE_OUTPUT_FILENAME, "wb") wf.setnchannels(CHANNELS) wf.setsampwidth(p.get_sample_size(FORMAT)) wf.setframerate(RATE) wf.writeframes(b"".join(frames)) wf.close() # Step 3: Transcribe the recorded audio audio_file = open("output.wav", "rb") transcript = openai.Audio.transcribe("whisper-1", audio_file)["text"] # Step 4: Generate GPT-3 response pre_prompt = f"""Act as a relief SPOT robot for a human in a disaster scenario Human: {transcript}. reply in the language of the human. reassure the human that help is on its way, and that they are safe. state that images, voice, and geolocation data is being sent in real time to the rescue teams. just return the text to be spoken by the robot in the language of the human. do not return any other data. do not repeat the human input. just reply to it using above instructions """ response = openai.Completion.create(engine="text-davinci-002", prompt=pre_prompt, max_tokens=100) generation = response.choices[0].text.strip() # Step 5: Speak out the GPT-3 response tts = gTTS(text=generation, lang="en") tts.save("gpt_response.mp3") pygame.mixer.music.load("gpt_response.mp3") pygame.mixer.music.play() while pygame.mixer.music.get_busy(): continue try: sorted_audio_emotions = postAudioHume() face_emotion = [{"name": "Stressed", "score": 0.4763}] except: sorted_audio_emotions = [{"name": "Stressed", "score": 0.4763}] face_emotion = [{"name": "Stressed", "score": 0.4763}] urgency_level = 8 summary_prompt = f''' SPOT robots are identifying survivors in war torn areas and are sending visual and auditory data to remote a paramedic team. Act as the summariser for the paramedic team. The data being sent has an audio transcription of the survivor ({transcript}), a facial emotion analysis using HUME and confidences({sorted_audio_emotions}), a auditory emotion analysis using HUME and confidences ({face_emotion}), a [truncated — 880 more characters] ``` ### robot-control-server/python/examples/wasd_server/app.py ```python import atexit from flask import Flask, jsonify import bosdyn.client.util from bosdyn.client import ResponseError, RpcError, create_standard_sdk from bosdyn.client.robot_command import RobotCommandBuilder, RobotCommandClient, blocking_command, blocking_stand, blocking_sit from wasd import WasdInterface # Assuming the original file is named 'wasd.py' app = Flask(__name__) HOSTNAME = "192.168.80.3" # Global variables to hold robot and interface objects robot = None wasd_interface = None def initialize_robot(): global robot, wasd_interface print("initializing robot...") # Create robot object. sdk = create_standard_sdk('WASDClient') robot = sdk.create_robot(HOSTNAME) try: bosdyn.client.util.authenticate(robot) robot.start_time_sync() except RpcError as err: print("Failed to communicate with robot: %s", err) return False wasd_interface = WasdInterface(robot) try: print("starting control interface...") wasd_interface.start() except (ResponseError, RpcError) as err: print("Failed to initialize robot communication: %s", err) return False try: print("estop state:") print(wasd_interface._estop_str()) print("toggling estop") wasd_interface._toggle_estop() print("estop state:") print(wasd_interface._estop_str()) robot.power_on(timeout_sec=20) if robot.is_powered_on(): print("Powered On") # If everything went smooth, Spot face lights should turn green else: # In case of some problems, e.g. somebody stole control over robot print("Power on failed") # idk if this works yet robot.time_sync.wait_for_sync() command_client = robot.ensure_client(RobotCommandClient.default_service_name) print("standing...") blocking_stand(command_client, timeout_sec=10) print("stand command complete") except (ResponseError, RpcError) as err: print("Failed to toggle estop or power: %s", err) return False # Register shutdown method to be called upon Flask app shutdown atexit.register(wasd_interface.shutdown) # push context manually to app with app.app_context(): initialize_robot() @app.route('/move/forward', methods=['GET']) def move_forward(): wasd_interface._move_forward() return jsonify(status="Moving forward") @app.route('/move/backward', methods=['GET']) def move_backward(): wasd_interface._move_backward() return jsonify(status="Moving backward") @app.route('/move/left', methods=['GET']) def move_left(): wasd_interface._strafe_left() return jsonify(status="Moving left") @app.route('/move/right', methods=['GET']) def move_right(): wasd_interface._strafe_right() return jsonify(status="Moving right") @app.route('/turn/left', methods=['GET']) def turn_left(): wasd_interface._turn_left() return jsonify(status="Turning left") @app.route('/turn/right', methods=['GET']) def turn_right(): wasd_interface._turn_right() return jsonify(status="Turning right") @app.route('/sit', methods=['GET']) def sit(): wasd_interface._sit() return jsonify(status="Sitting") @app.route('/stand', methods=['GET']) def stand(): wasd_interface._stand() return jsonify(status="Standing") @app.route('/battery-change', methods=['GET']) def battery_change(): wasd_interface._battery_change_pose() return jsonify(status="Battery change position") @app.route('/battery', methods=['GET']) def battery(): return jsonify(wasd_interface._battery_str()) @app.route('/self-right', methods=['GET']) def self_righ(): wasd_interface._self_right() return jsonify(status="Self correcting...") @app.route('/shutdown', methods=['GET']) def shutdown(): wasd_interface.shutdown() return jsonify(status="Shutdown complete") if __name__ == '__main__': app.run(host='0.0.0.0', port=5000) ``` ### spotter/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### spotter/next.config.js ```javascript /** @type {import('next').NextConfig} */ const nextConfig = { reactStrictMode: true, }; module.exports = nextConfig; ``` [220 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]