# Project export: Hawkeye This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Hawkeye is a real time multimodal conversation and interaction agent for the Boston Dynamics’ Spot robot. - Devpost: https://devpost.com/software/hawkeye-3kp0tc - GitHub: https://github.com/darryltanzil/spot-boston-dynamics - Video: https://www.youtube.com/embed/PDaMZ3OyuGc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Edward Li (52 commits), Darryl Tanzil (30 commits), Sriram (19 commits), Victor Vannara (14 commits) ## Devpost submission (written by the team) ### Overview Hawkeye is a real time multimodal conversation and interaction agent for the Boston Dynamics’ mobile robot Spot. Leveraging OpenAI’s experimental GPT-4 Turbo and Vision AI models, Hawkeye aims to empower everyone, from seniors to healthcare professionals in forming new and unique interactions with the world around them. ### What it does The core of Hawkeye is powered by its conversation-action engine. Using audio and visual inputs from the real world, all decisions and movements made by Hawkeye are generated and inferenced on the fly in near real time. For instance, when faced with a command like "Hey Spot, can you move a little closer?” Hawkeye digests the task on hand to build step-by-step instructions for the robot to follow. This means that Hawkeye is able to dynamically adapt, change, and learn with new environments. It knows when and how to improve its vantage point, orchestrate complex maneuvers, and advance closer to its target. Hawkeye flawlessly navigates through any environment, all while avoiding a reliance on pre-coded movement patterns. ### How we built it Hawkeye is powered by OpenAI's experimental GPT-4 Turbo and Vision AI models, alongside Spot’s movement SDK. On initialization, Hawkeye uses speech-to-text to wait for an interaction from the user, where it then determines the type and intention of the request. Then, relevant information is fed into OpenAI’s GPT4-Turbo and Vision to generate a chain of command for the robot to follow, with the ability to map out its actions and navigate to the goal. Incrementally, Hawkeye reevaluates and replans its movements to eventually reach its intended goal. ### Challenges we ran into Our main challenge with developing on the Spot is finding ways to rapidly build and prototype new features without having to rebuild the entire project on every change. As such, we developed a custom command server to push to push, patch, and deploy new code over websockets without needing a reboot. This rapidly increased our rate of innovation with the Spot robot, giving us the flexibility and maneuverability to make our project a reality. Another main challenge was determining what kind of command a phrase is when it’s first received from the mic. Since there are “movement”, “image processing”, and “general commands”, all with their own functionality, it was difficult coming up with a simple way to classify each command to be one of the three. We resolved this by creating an input delegate which makes an API call to OpenAI, and makes a judgment on what classification it thinks the command is. ### Accomplishments we're proud of Learning how to engineer prompts and rapidly iterate to make it more accurate was a feat that we're proud of! The learning curve for Spot's SDK was quite steep and we're happy to have come out of it with minimal team friction. We also got it to dance in the last hour by just telling it to dance, which was very satisfying. Not breaking the $75k robot was relieving as well! ### What we learned By using Spot's and its SDK, we worked with a lot of hardware which we otherwise would never have access to. We learnt a lot about the physical sensors and I/O onboard. Moreover, it was the first time working with facial and object recognition libraries for a lot of us. Finally, we became much more proficient with GPT APIs. ### What's next Looking ahead, we envision further refining Hawkeye by incorporating dynamic object tracking along with facial recognition tech to improve the breadth of human-assistance tasks. This would allow it to dynamically change its motion to avoid moving obstacles (e.g., people in a moving crowd). Spot has huge potential as a guide dog once combined with AI, being able to talk to and guide its owner — with Hawkeye’s interaction software, we believe this level of robotics collaboration will shape the future. Technologies Used Spot® - The Agile Mobile Robot Spot SDK for control over hardware Websockets for our dynamic code launcher Python as language of choice Docker for easy containerization GPT-4's vision & turbo preview models ## README (from the GitHub repository) # Hawkeye - Multimodal Interaction Robot [![Watch the video](https://github.com/darryltanzil/spot-boston-dynamics/assets/5387769/54f19158-6752-4d41-ad5a-896e53d35a31)](https://www.youtube.com/watch?v=PDaMZ3OyuGc) Hawkeye is a real time multimodal conversation and interaction agent for the Boston Dynamics’ mobile robot Spot. Leveraging OpenAI’s experimental GPT-4 Turbo and Vision AI models, Hawkeye aims to empower everyone, from seniors to healthcare professionals in forming new and unique interactions with the world around them. ## What it does The core of Hawkeye is powered by its conversation-action engine. Using audio and visual inputs from the real world, all decisions and movements made by Hawkeye are generated and inferenced on the fly in near real time. For instance, when faced with a command like "Hey Spot, can you move a little closer?” Hawkeye digests the task on hand to build step-by-step instructions for the robot to follow. This means that Hawkeye is able to dynamically adapt, change, and learn with new environments. It knows when and how to improve its vantage point, orchestrate complex maneuvers, and advance closer to its target. Hawkeye flawlessly navigates through any environment, all while avoiding a reliance on pre-coded movement patterns. ## How we built it Hawkeye is powered by OpenAI's experimental GPT-4 Turbo and Vision AI models, alongside Spot’s movement SDK. On initialization, Hawkeye uses speech-to-text to wait for an interaction from the user, where it then determines the type and intention of the request. Then, relevant information is fed into OpenAI’s GPT4-Turbo and Vision to generate a chain of command for the robot to follow, with the ability to map out its actions and navigate to the goal. Incrementally, Hawkeye reevaluates and replans its movements to eventually reach its intended goal. ## Challenges we ran into Our main challenge with developing on the Spot is finding ways to rapidly build and prototype new features without having to rebuild the entire project on every change. As such, we developed a custom command server to push to push, patch, and deploy new code over websockets without needing a reboot. This rapidly increased our rate of innovation with the Spot robot, giving us the flexibility and maneuverability to make our project a reality. Another main challenge was determining what kind of command a phrase is when it’s first received from the mic. Since there are “movement”, “image processing”, and “general commands”, all with their own functionality, it was difficult coming up with a simple way to classify each command to be one of the three. We resolved this by creating an input delegate which makes an API call to OpenAI, and makes a judgment on what classification it thinks the command is. ## Accomplishments that we're proud of Learning how to engineer prompts and rapidly iterate to make it more accurate was a feat that we're proud of! The learning curve for Spot's SDK was quite steep and we're happy to have come out of it with minimal team friction. We also got it to dance in the last hour by just telling it to dance, which was very satisfying. Not breaking the $75k robot was relieving as well! ## What we learned By using Spot's and its SDK, we worked with a lot of hardware which we otherwise would never have access to. We learnt a lot about the physical sensors and I/O onboard. Moreover, it was the first time working with facial and object recognition libraries for a lot of us. Finally, we became much more proficient with GPT APIs. ## What's next for Hawkeye Looking ahead, we envision further refining Hawkeye by incorporating dynamic object tracking along with facial recognition tech to improve the breadth of human-assistance tasks. This would allow it to dynamically change its motion to avoid moving obstacles (e.g., people in a moving crowd). Spot has huge potential as a guide dog once combined with AI, being able to talk to and guide its owner — with Hawkeye’s interaction software, we believe this level of robotics collaboration will shape the future. ## Technologies Used * Spot® - The Agile Mobile Robot * Spot SDK for control over hardware * Websockets for our dynamic code launcher * Python as language of choice * Docker for easy containerization * GPT-4's vision & turbo preview models ## Gallery image Boston Dynamics Robot with Speakers & Webcam attached ![image](https://github.com/darryltanzil/spot-boston-dynamics/assets/5387769/a18ae56c-429d-48d2-a604-e42640652cad) Initial Sketches for Software ## Detected evidence (automated analysis) Indexed codebase: 19 recognized source files, 61 KB. - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (23 of 23) ``` .github/workflows/build.yml .gitignore ai_pipeline/__init__.py ai_pipeline/bot_handler.py ai_pipeline/image_detection.py ai_pipeline/input_delegator.py ai_pipeline/misc_request.py ai_pipeline/move_process.py ai_pipeline/recognize_speech.py cam.py client_automate_local.py client_automate.py client_control.py Dockerfile main.py payload copy.py README.md requirements.txt responding.py spot_controller.py vid_server.py wasd wormholelite.py ``` ### Dependencies - requirements.txt: gTTS, openai, opencv-python, pyaudio, python-dotenv, SpeechRecognition, websocket-client ### Recent commits (newest first) - Update README.md - added video description - Change subheading - Update README.md - Remove line from README - Edit README - Update README.md - README edits - Merge branch 'master' of github.com:darryltanzil/spot-boston-dynamics - lessen timeout durations - Merge branch 'master' of github.com:darryltanzil/spot-boston-dynamics - Better error reporting - chained command timeout - Merge branch 'master' of github.com:darryltanzil/spot-boston-dynamics - Change turn logic - Merge remote-tracking branch 'origin/master' - command chaining - Fix Camera Bug - Remove Second Webcam - Merge branch 'master' of github.com:darryltanzil/spot-boston-dynamics ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` websocket-client python-dotenv openai opencv-python SpeechRecognition pyaudio gTTS ``` ### Dockerfile ``` FROM ghcr.io/merklebot/hackathon-arm-image:master as build ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ARG TARGETPLATFORM ARG BUILDPLATFORM ARG TARGETOS ARG TARGETARCH ARG Version ARG GitCommit RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" COPY requirements.txt requirements.txt RUN apt-get install -y portaudio19-dev RUN apt-get install -y python3-pyaudio RUN apt-get install -y flac RUN python3.8 -m pip install -r requirements.txt COPY . . CMD ["python3.8", "main.py"] ``` ### main.py ```python import os import time import socket import subprocess import cv2 import base64 import sys import traceback from wormholelite import CameraVideo from websocket import create_connection from ai_pipeline.input_delegator import delegate_input from ai_pipeline.recognize_speech import listen_for_keyword from ai_pipeline.bot_handler import playAudio def start(spot): playAudio(delegate_input(listen_for_keyword(), spot)) ROBOT_IP = "10.0.0.3"#os.environ['ROBOT_IP'] SPOT_USERNAME = "admin"#os.environ['SPOT_USERNAME'] SPOT_PASSWORD = "2zqa8dgw7lor"#os.environ['SPOT_PASSWORD'] """ 4 different states: Two methods to activate Spot: - Use Tensorflow - Use Audio from Webcam (triggered by 'hey spot') Listening 1. use object detection API to walk towards object 2. take snapshot of project, send image to code processing Open AI 2.5 (if extra time) crop image before sending Processing 3. Once response is recieved, return back info in text Responding 4. Turn to person 5. Convert response into Text-to-speech, play through speaker 6. if possible, look at person and point to object Idling """ # def voice_callback(message): # get def main(): # if True: from spot_controller import SpotController with SpotController(username=SPOT_USERNAME, password=SPOT_PASSWORD, robot_ip=ROBOT_IP) as spot: globals()["spot_global"] = spot ws = create_connection("wss://737c-171-66-13-247.ngrok-free.app", ping_timeout=None) ws.send("Hello, World") # cam = CameraVideo(0, max_fps=1, height=360, width=480) while True: try: cmd = ws.recv() # if cmd == "[CAM]": # frame = cam.get_frame() # _, enc = cv2.imencode('.jpg', frame) # jpg_as_text = base64.b64encode(enc) # # print(jpg_as_text) # ws.send(jpg_as_text) # continue if cmd == "[PAYLOAD]": with open("/tmp/payload.py", "w+") as f: f.seek(0) f.truncate() f.write(ws.recv()) try: process = subprocess.Popen(["python3", "/tmp/payload.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) for line in process.stdout: ws.send(line.strip()) for line in process.stderr: ws.send(line.strip()) process.wait() except Exception as e: ws.send(f"Error executing command: {e}") ws.send("[EOL]") continue res = eval(cmd) if res: ws.send(str(res)) else: ws.send("None") except Exception as e: print(traceback.format_exc()) ws.send(str(e)) if __name__ == '__main__': main() import time time.sleep(50) ``` ### responding.py ```python """ TODO: responding """ ``` ### cam.py ```python # # import the opencv library # import cv2 # # define a video capture object # vid = cv2.VideoCapture() # while(True): # # Capture the video frame # # by frame # ret, frame = vid.read() # # Display the resulting frame # cv2.imshow('frame', frame) # # the 'q' button is set as the # # quitting button you may use any # # desired button of your choice # if cv2.waitKey(1) & 0xFF == ord('q'): # break # # After the loop release the cap object # vid.release() # # Destroy all the windows # cv2.destroyAllWindows() ``` ### payload copy.py ```python import asyncio import websockets import cv2 import base64 import time # Function to read frames from video device, encode as base64, and send over WebSocket async def send_frames(): # Open the video capture device (0 for default camera) cap = cv2.VideoCapture(0) # Create WebSocket connection async with websockets.connect("ws://localhost:8765") as websocket: while True: # Read frame from video device ret, frame = cap.read() if not ret: break # Encode frame as JPEG and then as base64 _, buffer = cv2.imencode('.jpg', frame) jpg_as_text = base64.b64encode(buffer) # Send base64-encoded frame over WebSocket await websocket.send(jpg_as_text) time.sleep(1) # Release the video capture device cap.release() # Main function to run the event loop async def main(): await send_frames() # Run the event loop if __name__ == "__main__": asyncio.run(main()) ``` ### vid_server.py ```python import asyncio import websockets import base64 import cv2 import numpy as np # Function to decode base64 image data and display it using OpenCV def display_image(base64_str): # Decode base64 string to bytes img_bytes = base64.b64decode(base64_str) # Convert bytes to numpy array img_np = np.frombuffer(img_bytes, dtype=np.uint8) # Decode image array using OpenCV img = cv2.imdecode(img_np, cv2.IMREAD_COLOR) # Display image cv2.imshow('Frame', img) cv2.waitKey(1) # Adjust according to your refresh rate # WebSocket server handler async def server(websocket, path): while True: try: # Receive base64-encoded JPEG image base64_str = await websocket.recv() # Display the image display_image(base64_str) except websockets.exceptions.ConnectionClosedError: print("Client disconnected") break # Start the WebSocket server start_server = websockets.serve(server, "localhost", 8765) print("WebSockets video server starting", flush=True) asyncio.get_event_loop().run_until_complete(start_server) print("WebSockets video server running", flush=True) asyncio.get_event_loop().run_forever() ``` ### client_control.py ```python #!/usr/bin/env python import asyncio import websockets import os async def echo(websocket, path): msg_buffer = [] async for message in websocket: print(message, flush=True) if msg_buffer: cmd = msg_buffer.pop(0) else: cmd = input("CMD: ") if cmd == "r": with open("exec.py", "r") as f: msg_buffer += f.readlines() cmd = msg_buffer.pop(0) if cmd == "p": with open("payload.py", "r") as f: payload = f.read() await websocket.send("[PAYLOAD]") await websocket.send(payload) while True: output = await websocket.recv() if output == "[EOL]": break print(output, flush=True) await websocket.send("\"DONE\"") continue # print("SEND:", cmd.strip()) await websocket.send(cmd) start_server = websockets.serve(echo, "0.0.0.0", os.environ.get('PORT') or 8080, ping_timeout=None) print("WebSockets echo server starting", flush=True) asyncio.get_event_loop().run_until_complete(start_server) print("WebSockets echo server running", flush=True) asyncio.get_event_loop().run_forever() ``` ### client_automate_local.py ```python #!/usr/bin/env python import asyncio import websockets import os import cv2 import base64 import numpy as np import time async def echo(websocket, path): print("init camera") # Initialize the camera cap = cv2.VideoCapture(0) async def move_right(): print("Moving Right") # Function to move left async def move_left(): print("Moving Left") # Function to move up async def move_up(): print("Moving Up") # Function to move down async def move_down(): print("Moving Down") # Load the pre-trained face detection model face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml') # Function to detect face position and call corresponding movement functions async def detect_face_and_move(frame): # Convert the frame to grayscale gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) # Detect faces in the frame faces = face_cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30)) # Get the center of the frame center_x = frame.shape[1] // 2 center_y = frame.shape[0] // 2 for (x, y, w, h) in faces: # Calculate the center of the detected face face_center_x = x + w // 2 face_center_y = y + h // 2 # Check the position of the face relative to the center of the frame if face_center_x < center_x - 75: # Face too far left await move_left() elif face_center_x > center_x + 75: # Face too far right await move_right() elif face_center_y < center_y - 75: # Face too far up await move_up() elif face_center_y > center_y + 75: # Face too far down await move_down() else: print("OK") cap = cv2.VideoCapture(0) while True: # Read a frame from the camera ret, frame = cap.read() if not ret: print("Error capturing frame") break await detect_face_and_move(frame) cv2.imshow('Frame', frame) if cv2.waitKey(1) & 0xFF == ord('q'): break # # start_server = websockets.serve(echo, "0.0.0.0", os.environ.get('PORT') or 8080) # # print("WebSockets ai server starting", flush=True) # asyncio.get_event_loop().run_until_complete(start_server) # # print("WebSockets ai server running", flush=True) # asyncio.get_event_loop().run_forever() if __name__ == "__main__": asyncio.run(echo(None, None)) ``` ### client_automate.py ```python #!/usr/bin/env python import asyncio import websockets import os import cv2 import base64 import numpy as np import time async def echo(websocket, path): async def move_right(): print("Moving Right") # Function to move left async def move_left(): print("Moving Left") # Function to move up async def move_up(): print("Moving Up") # Function to move down async def move_down(): print("Moving Down") # Load the pre-trained face detection model face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml') # Function to detect face position and call corresponding movement functions async def detect_face_and_move(frame): # Convert the frame to grayscale gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) # Detect faces in the frame faces = face_cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30)) # Get the center of the frame center_x = frame.shape[1] // 2 center_y = frame.shape[0] // 2 for (x, y, w, h) in faces: # Calculate the center of the detected face face_center_x = x + w // 2 face_center_y = y + h // 2 # Check the position of the face relative to the center of the frame if face_center_x < center_x - 75: # Face too far left await move_left() elif face_center_x > center_x + 75: # Face too far right await move_right() elif face_center_y < center_y - 75: # Face too far up await move_up() elif face_center_y > center_y + 75: # Face too far down await move_down() else: print("OK") hello = await websocket.recv() while True: await websocket.send("[CAM]") # Get image back from client as base64 encoded jpeg, and display it using cv2 base64_str = await websocket.recv() # print(base64_str) if base64_str == "[ERROR]": print("Error receiving frame") continue img_bytes = base64.b64decode(base64_str) img_np = np.frombuffer(img_bytes, dtype=np.uint8) img = cv2.imdecode(img_np, cv2.IMREAD_COLOR) # print(img.shape) await detect_face_and_move(img) cv2.imshow('Frame', img) if cv2.waitKey(1) & 0xFF == ord('q'): break start_server = websockets.serve(echo, "0.0.0.0", os.environ.get('PORT') or 8080) print("WebSockets ai server starting", flush=True) asyncio.get_event_loop().run_until_complete(start_server) print("WebSockets ai server running", flush=True) asyncio.get_event_loop().run_forever() ``` [9 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]