# Project export: Hound This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Introducing Hound, Boston Dynamic's archnemesis and Spot's alterego. The Hound not only detects and tracks its targets but also "hunts" them down! The Hound is no fool, though, and flees when pressed. - Devpost: https://devpost.com/software/hound-mgb461 - GitHub: https://github.com/ahmed-k-aly/hackathon-spot - Video: https://player.vimeo.com/video/914182004?byline=0&portrait=0&title=0#t=0 - Team: 0 GitHub contributor(s) — ## Devpost submission (written by the team) ### Inspiration I wrote my college application essays on Boston Dynamics. When I saw Spot, I was a little kid again with a bright mind and an innocent smile. However, I didn't know what I wanted to build exactly, until I noticed the infinite supply of QR Codes around me! ### What it does Spot utilizes a graph search algorithm to find a QR code in its environment. On detection, Spot gallops the entire distance (depth-perception) to the target. If Spot overcommited and got too close to the target, it flees the scene as a dog usually would! ### How we built it I used Merklebot's abstraction and frameworks to run my code on Spot. I utilized a depth-perception algorithm that estimates how far an object is in the environment. The robot would search for the QR code, and once it found it, it would dash at the qr code. If it lost the target, it would commence search again. Search is based on DFS for simplicty. However, it can easily be changed to a more sophisticated algorithm, such as A*, for instance. ### Challenges we ran into It is a difficult task to get the search to work. Additionally, it's very tedious to get live data from the robot for analytics and debugging. Managing Spot necessitated high-quality code, so the robot doesn't damage anything, which was a strenuous task. ### Accomplishments we're proud of I actually worked with Spot and did something!!!!! ### What we learned Murphy's Law exists for a reason. I should be better at estimating the breadth of projects. ### What's next for Untitled Add a particle filtering sampling algorithm that listens on the microphone, and can detect where targets might be based on a sound they exhibit. This can be incorporated in an A* algorithm as its heuristic. Play hide-and-seek! ## README (from the GitHub repository) **This image contains basic commands to work with camera, microphone and speaker at Spot's computer** ## Image link Image link could be acquired in **packages** section of the main repository page ## How to use The container for this image should be runned with flags `--device /dev/video0` and `--device /dev/snd`. Also, environmental varialbes `-e SDL_AUDIODRIVER='alsa'`, `-e AUDIODEV='hw:1,0'`, `-e AUDIO_INPUT_DEVICE='hw:2,0'` shoud be added. Basic example contained in _main.py_ ## Detected evidence (automated analysis) Indexed codebase: 4 recognized source files, 19 KB. - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (7 of 7) ``` .github/workflows/build.yml .gitignore Dockerfile main.py README.md requirements.txt spot_controller.py ``` ### Dependencies - requirements.txt: gTTS@~=2.5.1 ### Recent commits (newest first) - done - fixed coords, final - finall - final - finishing up - added better directions - optimization - faster search - requirements diet - optimizations - reset the code - creates a new file now for audio - added try/catch loop - bugfix - fixed removal bug - fixed a bug with filenames - file problems - added some files - added mic listening and spatial directions - added doNothing functionality ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` gTTS~=2.5.1 ``` ### Dockerfile ``` FROM ghcr.io/merklebot/hackathon-arm-image:master as build ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ARG TARGETPLATFORM ARG BUILDPLATFORM ARG TARGETOS ARG TARGETARCH ARG Version ARG GitCommit RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" COPY requirements.txt requirements.txt RUN python3.8 -m pip install -r requirements.txt COPY . . CMD ["python3.8", "main.py"] ``` ### main.py ```python import os import time from spot_controller import SpotController ROBOT_IP = "10.0.0.3"#os.environ['ROBOT_IP'] SPOT_USERNAME = "admin"#os.environ['SPOT_USERNAME'] SPOT_PASSWORD = "2zqa8dgw7lor"#os.environ['SPOT_PASSWORD'] TIMEOUT_LIMIT = 90 # IN SECONDS MAX_DISTANCE = 18 # IN CM LOOP_TIMEOUT = 30 # in seconds import cv2 import numpy as np from gtts import gTTS def estimate_distance(points, real_qr_size=21, focal_length=50, sensor_size=24, image_size=1080): """ Estimates the distance of a QR code from the camera based on the QR code's pixel coordinates. Parameters: - points: numpy.ndarray representing the coordinates of the four corners of the detected QR Code. - real_qr_size: The real-world length of the QR code's side in centimeters. Default is 21 cm for A4 paper width. - focal_length: The focal length of the camera in millimeters. Default is 50 mm. - sensor_size: The size of the camera sensor in millimeters. Default is 24 mm, assuming a full-frame sensor's height. - image_size: The size of the image in pixels along the dimension that corresponds to the sensor_size. Default is 1080 pixels for the height in a 1920x1080 resolution. Returns: - A list of estimated distances in centimeters for each QR code detected in the image. """ distances = [] for qr in points: # Calculate the width and height of the QR code in pixels width_px = np.linalg.norm(qr[0] - qr[1]) height_px = np.linalg.norm(qr[0] - qr[3]) # Use the average of width and height for a more accurate estimate avg_size_px = (width_px + height_px) / 2 # Convert focal length to centimeters focal_length_cm = focal_length / 10 # Calculate the distance using the formula distance = (focal_length_cm * real_qr_size * image_size) / (avg_size_px * sensor_size) distances.append(distance) return distances def detect_object(image): qcd = cv2.QRCodeDetector() retval, decoded_info, points, straight_qrcode = qcd.detectAndDecodeMulti(image) if retval: print(f"QR code found at {points[0]}") distance = estimate_distance(points) print(f"QR code found at {points[0]} with estimated distance of {distance} cm") return distance[0] else: print("No QR code found") return -1 def try_to_detect(spot): print("Trying to detect QR code...") # search algorithm distance = search(spot) return distance def search(spot): """A depth first search algorithm to search for the object in the environment by ONLY turning the head of the robot. Args: spot (SpotController): an instance of the SpotController class to control the robot's movement. """ # The robot should only rotate its head in increments of 30 degrees to allow continuous scanning of the environment. Angles are in radians. possibleAngles = [0, 0.523599, 1.0472, 1.5708, -0.523599, -1.0472, -1.5708 ] possibleDirections = ["left", "right", "up", "down"] search_after = 0.1 # map angles and directions to yaws and pitches timer = int(time.time()) frontier = [] explored = [] # The robot should start by looking straight ahead frontier.append((0, "doNothing")) say_something("Searching ") while frontier: if (int(time.time()) - timer) > LOOP_TIMEOUT: print("Time out") return -1 camera_capture = cv2.VideoCapture(0) # set camera at full hd #camera_capture.set(cv2.CAP_PROP_FRAME_WIDTH, 1920) #camera_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080) # Pop the last element from the frontier current = frontier.pop() # move the head to the current state if current[1] == "left": # rolls should be in radians based on the angle spot.move_head_in_points(yaws=[current[0]/3.14, current[0]/3.14], pitches=[0.0, 0.0], rolls=[0.15, -0.15], sleep_after_point_reached=search_after) elif current[1] == "right": spot.move_head_in_points(yaws=[current[0]/3.14, current[0]/3.14], pitches=[0.0, 0.0], rolls=[-0.15, 0.15], sleep_after_point_reached=search_after) elif current[1] == "up": spot.move_head_in_points(yaws=[0,0], pitches=[current[0]/3.14,current[0]/3.14], rolls=[0.0, 0.0], sleep_after_point_reached=search_after) elif current[1] == "down": spot.move_head_in_points(yaws=[0,0], pitches=[current[0]/3.14, current[0]/3.14], rolls=[0.0, 0.0], sleep_after_point_reached=search_after) elif current[1] == "doNothing": ## do nothing pass rv, image = camera_capture.read() camera_capture.release() # Check if the current state is the goal state distance = detected_qr_code(image) if distance != -1: print("QR code found") say_something("Target Acquired") return distance while (distance == -1): # keep trying again for five seconds print("Trying to detect QR code...") if (int(time.time()) - timer > 5): print("No QR code found") break camera_capture = cv2.VideoCapture(0) rv, image = camera_capture.read() # add image processing here to detect object distance = detect_object(image) camera_capture.release() # Add the current state to the explore [truncated — 3320 more characters] ``` ### spot_controller.py ```python import time import bosdyn.client from bosdyn.client.robot_command import RobotCommandClient, RobotCommandBuilder, blocking_stand # , blocking_sit from bosdyn.geometry import EulerZXY from bosdyn.api.spot import robot_command_pb2 as spot_command_pb2 from bosdyn.client.frame_helpers import ODOM_FRAME_NAME from bosdyn.api.basic_command_pb2 import RobotCommandFeedbackStatus from bosdyn.client.estop import EstopClient, EstopEndpoint, EstopKeepAlive from bosdyn.client.robot_state import RobotStateClient from bosdyn.client.frame_helpers import ODOM_FRAME_NAME, VISION_FRAME_NAME, BODY_FRAME_NAME, \ GRAV_ALIGNED_BODY_FRAME_NAME, get_se2_a_tform_b from bosdyn.client import math_helpers import traceback VELOCITY_CMD_DURATION = 0.5 class SpotController: def __init__(self, username, password, robot_ip): self.username = username self.password = password self.robot_ip = robot_ip sdk = bosdyn.client.create_standard_sdk('ControllingSDK') self.robot = sdk.create_robot(robot_ip) id_client = self.robot.ensure_client('robot-id') self.robot.authenticate(username, password) self.command_client = self.robot.ensure_client(RobotCommandClient.default_service_name) self.robot.logger.info("Authenticated") self._lease_client = None self._lease = None self._lease_keepalive = None self._estop_client = self.robot.ensure_client(EstopClient.default_service_name) self._estop_endpoint = EstopEndpoint(self._estop_client, 'GNClient', 9.0) self._estop_keepalive = None self.state_client = self.robot.ensure_client(RobotStateClient.default_service_name) def release_estop(self): self._estop_endpoint.force_simple_setup() self._estop_keepalive = EstopKeepAlive(self._estop_endpoint) def set_estop(self): if self._estop_keepalive: try: self._estop_keepalive.stop() except: self.robot.logger.error("Failed to set estop") traceback.print_exc() self._estop_keepalive.shutdown() self._estop_keepalive = None def lease_control(self): self._lease_client = self.robot.ensure_client('lease') self._lease = self._lease_client.take() self._lease_keepalive = bosdyn.client.lease.LeaseKeepAlive(self._lease_client, must_acquire=True) self.robot.logger.info("Lease acquired") def return_lease(self): self._lease_client.return_lease(self._lease) self._lease_keepalive.shutdown() self._lease_keepalive = None def __enter__(self): self.lease_control() self.release_estop() self.power_on_stand_up() return self def __exit__(self, exc_type, exc_val, exc_tb): if exc_type: self.robot.logger.error("Spot powered off with " + exc_val + " exception") self.power_off_sit_down() self.return_lease() self.set_estop() return True if exc_type else False def move_head_in_points(self, yaws, pitches, rolls, body_height=0, sleep_after_point_reached=0, timeout=3): for i in range(len(yaws)): footprint_r_body = EulerZXY(yaw=yaws[i], roll=rolls[i], pitch=pitches[i]) params = RobotCommandBuilder.mobility_params(footprint_R_body=footprint_r_body, body_height=body_height) blocking_stand(self.command_client, timeout_sec=timeout, update_frequency=0.02, params=params) self.robot.logger.info("Moved to yaw={} rolls={} pitch={}".format(yaws[i], rolls[i], pitches[i])) if sleep_after_point_reached: time.sleep(sleep_after_point_reached) def wait_until_action_complete(self, cmd_id, timeout=15): start_time = time.time() while time.time() - start_time < timeout: feedback = self.command_client.robot_command_feedback(cmd_id) mobility_feedback = feedback.feedback.synchronized_feedback.mobility_command_feedback if mobility_feedback.status != RobotCommandFeedbackStatus.STATUS_PROCESSING: print("Failed to reach the goal") return False traj_feedback = mobility_feedback.se2_trajectory_feedback if (traj_feedback.status == traj_feedback.STATUS_AT_GOAL and traj_feedback.body_movement_status == traj_feedback.BODY_STATUS_SETTLED): print("Arrived at the goal.") return True time.sleep(0.5) def move_to_goal(self, goal_x=0, goal_y=0): cmd = RobotCommandBuilder.synchro_trajectory_command_in_body_frame( goal_x_rt_body=goal_x, goal_y_rt_body=goal_y, goal_heading_rt_body=0, frame_tree_snapshot=self.robot.get_frame_tree_snapshot() ) # cmd = RobotCommandBuilder.synchro_se2_trajectory_point_command(goal_x=goal_x, goal_y=goal_y, goal_heading=0, # frame_name=GRAV_ALIGNED_BODY_FRAME_NAME) cmd_id = self.command_client.robot_command(lease=None, command=cmd, end_time_secs=time.time() + 10) self.wait_until_action_complete(cmd_id) self.robot.logger.info("Moved to x={} y={}".format(goal_x, goal_y)) def power_on_stand_up(self): self.robot.power_on(timeout_sec=20) assert self.robot.is_powered_on(), "Not powered on" self.robot.time_sync.wait_for_sync() blocking_stand(self.command_client, timeout_sec=10) def power_off_sit_down(self): self.move_head_in_points(yaws=[0], pitches=[0], rolls=[0]) self.robot.power_off(cut_immediately=False) def make_stance(self, x_offset, y_offset): state = self.state_client.get_robot_state() vo_T_body = get_se2_a_tform_b(state.kinematic_state.transforms_snapshot, VISION_FRAME_NAME, [truncated — 2131 more characters] ``` ### .github/workflows/build.yml ```yaml name: Build image on: push: branches: [ main ] tags: - "*" workflow_dispatch: env: REGISTRY: ghcr.io IMAGE_NAME: ${{ github.repository }} jobs: build: runs-on: ubuntu-latest permissions: contents: read packages: write steps: - uses: actions/checkout@v3 - name: Set up QEMU uses: docker/setup-qemu-action@v2 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v2 - uses: docker/login-action@v2 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Extract metadata (tags, labels) for image builder uses: docker/metadata-action@v4 id: meta with: images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} - name: Build and push container image uses: docker/build-push-action@v3 with: context: . push: true platforms: linux/arm64/v8 tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} ```