# Project export: The Cat
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: Firefighters helped him down from a tree—now he’s here to help them.
- Devpost: https://devpost.com/software/the-cat-ek4py5
- GitHub: https://github.com/bulletchicken/ILOVETREEHACKSSS
- Video: https://www.youtube.com/embed/C_yn3GpcAMQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Jeremy Su (8 commits)
## Devpost submission (written by the team)
### Inspiration
Just a week ago, local news reported on an elderly man who wasn’t found in time by first responders, ultimately leading to his tragic passing. His story hit home—he reminded me of my grandma, someone I always build projects for. With the recent California wildfires adding to the urgency, there had to be a faster way to help in these situations. Why a Cat? Firefighters always save cats from trees—what if a cat could be part of the rescue team? Inspired by the Coraline cat—known simply as The Cat—who appears just when Coraline needs him the most, coming to save the day. What if we could all have The Cat?
### What it does
This 8 jointed cat has the dexterity of well... a cat. This means that no matter what kinds of debris, rubble, or housing, The Cat can overcome. With this ability, it runs around burning houses in search for people. After finding them, it sends a summarized report to emergency services on them including their age and medical health so first responders can act more efficiently. It also has a swinging tail to be more visible in fires, a flashlight to improve the quality of reports it creates through smoke, and a gimbal for a head to stabilize it's camera no matter how much damage it takes. It can also converse with people with low latency, giving them instructions like creating a damp rag. On top of this, it is able to use the internet to search for streets to find the nearest fire hydrants so fire fighters know exactly where to go.
### How we built it
Lots of blood, sweat, and tears went into making this—mainly blood though because it kept scratching me. For the brains of the cat, the processing and summarization report was through using OpenAI's GPT 4o mini vision feature. For the conversational stage, I hacked a way to use ElevenLabs' Conversational AI with python! For finding a nearby fire hydrant, I utilized ScrapyBara to find one by deploying a virtual environment and scrapping the web. For making calls, I used Twilio For detecting people, Yolov8 nano. For many of the robotics parts, I used a mash of different libraries, utilizing sin functions to map out the motion of each leg. The hardware system is built off of 9 metal gear servos, (4 legs, 1 tail), controlled by an Arduino NANO + PWM Servo Driver, powered by x2 14500 batteries ran through a step down transformer. On the side, there is also a gyroscope which helps stabilizes the head turning it into a gimbal. A majority of the materials are made from scraps like eraser heads for the feet!
### Challenges we ran into
Making the cat walk took up more than 12 hours, second only by the design reiterations for the cat. Building without planning proper weight, materials, or knowledge on walking robots, were some of the greatest obstacles I've ever faced at a hackathon. Getting through airport security was also pretty tough with a box full of electronics.
### Accomplishments we're proud of
SOLO. This was my first solo in-person hackathon and created one of my most technically advanced projects. I am proud that the walking looks smooth on the cat and the overall functionality met my expectations.
## README (from the GitHub repository)
No README available.
## Detected evidence (automated analysis)
Indexed codebase: 8 recognized source files, 16 KB.
- HTML (language) — detected in the code
- Python (language) — detected in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (15 of 15)
```
.gitignore
.vscode/settings.json
actualcalling.py
ardu/ardu.ino
baramemory.txt
brain.py
catmovement/catmovement.ino
catwalk/catwalk.ino
heart.py
index.html
leftbrain.py
mainbrain.py
mouth.py
throat.py
yolov8n.pt
```
### Dependencies
No dependency index available.
### Recent commits (newest first)
- polishing
- convo bot is now in
- redesigning the code, adding conversation system
- we did the thing with the catwalk. it walks. and now its time for the ai
- the GYAAATT
- uhh arduino cat is head moving working yuh
- THE BEGINNING -> yolov8 computer vision is IN!
- Initial commit
## Key source files (fetched from GitHub, selected and truncated for size)
### index.html
```html
ElevenLabs Widget Demo
```
### heart.py
```python
import pyautogui
import time
import webbrowser
import os
import random
from serial import Serial
arduino = Serial('/dev/cu.usbmodemFX2348N1', 9600)
def interact_with_elevenlabs():
try:
# Get screen size
screen_width, screen_height = pyautogui.size()
# Move to bottom right corner and click "Go Live" button
pyautogui.moveTo(screen_width - 210, screen_height - 88)
pyautogui.click()
print("Clicked Go Live button")
# Wait briefly then click the widget button that appears
time.sleep(3)
pyautogui.moveTo(screen_width - 150, screen_height - 150)
pyautogui.click()
time.sleep(0.5)
pyautogui.moveTo(screen_width/2, screen_height/2 + 10)
pyautogui.click()
print("Clicked widget button")
# Periodically wag tail after last click
while True:
time.sleep(random.uniform(5, 10)) # Random interval between 5-10 seconds
arduino.write(b'2')
except Exception as e:
print(f"An error occurred: {e}")
if __name__ == "__main__":
interact_with_elevenlabs()
```
### throat.py
```python
import os
from dotenv import load_dotenv
import uuid
from elevenlabs import VoiceSettings
from elevenlabs.client import ElevenLabs
import asyncio
load_dotenv()
ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY")
if not ELEVENLABS_API_KEY:
raise ValueError("ELEVENLABS_API_KEY environment variable is not set")
client = ElevenLabs(api_key=ELEVENLABS_API_KEY)
async def text_to_speech(text: str) -> str:
# Calling the text_to_speech conversion API with detailed parameters
response = client.text_to_speech.convert(
voice_id="nPczCjzI2devNBz1zQrb", # Adam pre-made voice
output_format="mp3_22050_32",
text=text,
model_id="eleven_flash_v2_5", # use the turbo model for low latency
voice_settings=VoiceSettings(
stability=1.0,
similarity_boost=1.0,
style=0.9,
use_speaker_boost=True,
),
)
# uncomment the line below to play the audio back
# play(response)
# Generating a unique file name for the output MP3 file
save_file_path = f"spotify/spotted.mp3"
# Writing the audio to a file
with open(save_file_path, "wb") as f:
for chunk in response:
if chunk:
f.write(chunk)
print(f"{save_file_path}: A new audio file was saved successfully!")
# Return the path of the saved audio file
if __name__ == "__main__":
asyncio.run(text_to_speech("I have spotted someone! Meow meow meow..."))
```
### actualcalling.py
```python
from twilio.rest import Client
from dotenv import load_dotenv
import os
import boto3
import asyncio
# Load environment variables
load_dotenv()
# Get Twilio credentials
account_sid = os.getenv("TWILIO_ACCOUNT_SID")
auth_token = os.getenv("TWILIO_AUTH_TOKEN")
twilio_number = os.getenv("TWILIO_PHONE_NUMBER")
my_phone_number = os.getenv("MY_PHONE_NUMBER")
# Get AWS credentials
aws_access_key = os.getenv("AWS_ACCESS_KEY_ID")
aws_secret_key = os.getenv("AWS_SECRET_ACCESS_KEY")
# Validate that credentials exist
if not all([account_sid, auth_token, twilio_number, aws_access_key, aws_secret_key]):
raise ValueError("Missing required environment variables. Check your .env file.")
# Initialize S3 client
s3_client = boto3.client(
's3',
aws_access_key_id=aws_access_key,
aws_secret_access_key=aws_secret_key
)
# Initialize Twilio client
client = Client(account_sid, auth_token)
async def upload_audio_to_s3():
"""Uploads an audio file to S3 and returns its public URL."""
bucket_name = 'treehacksbucketeer'
file_name = 'output.mp3'
# Ensure the file exists before uploading
if not os.path.exists(file_name):
raise FileNotFoundError(f"Error: {file_name} not found.")
try:
# Upload the file to S3 with public-read ACL
s3_client.upload_file(
file_name,
bucket_name,
file_name,
ExtraArgs={'ACL': 'public-read'}
)
except Exception as e:
print(f"Error uploading to S3: {e}")
return None
async def call_emergency_services(prompt):
try:
call = client.calls.create(
twiml=f'{prompt}',
to=my_phone_number,
from_=twilio_number
)
print(f"Calling emergency services...")
print(f"Call SID: {call.sid}")
except Exception as e:
print(f"Error making the call: {e}")
if __name__ == "__main__":
asyncio.run(call_emergency_services())
```
### mouth.py
```python
import os
import time
from openai import OpenAI
import base64
from dotenv import load_dotenv
from throat import text_to_speech
from actualcalling import call_emergency_services
from actualcalling import upload_audio_to_s3
load_dotenv() # Load environment variables from .env file
# Now you can access variables like this:
api_key = os.getenv('OPENAI_API_KEY')
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode('utf-8')
client = OpenAI(api_key=api_key)
async def analyze_scene(prompt, image_path):
base64_image = encode_image(image_path)
# Create async client
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
# You are an AI assistant helping with emergency rescue operations. Analyze the image and provide detailed information about: 1) Number of people detected 2) Description of each person's position, condition, and clothing 3) Analysis of the room/environment
"role": "system",
"content": prompt
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
}
]
}
],
max_tokens=500
)
print(completion.choices[0].message.content)
# all to send a police report
if image_path == "person_detected.jpg":
with open('baramemory.txt', 'r') as f:
memory_text = f.read()
await text_to_speech(completion.choices[0].message.content + " " + memory_text)
os.system("afplay output.mp3")
#add scrapybara answers to this report.
#if that's not enough, it calls emergency services.
await call_emergency_services(completion.choices[0].message.content + memory_text)
else:
await text_to_speech(completion.choices[0].message.content)
os.system("afplay output.mp3")
return completion.choices[0].message.content
# Example usage:
# scene_analysis = analyze_scene("path_to_captured_frame.jpg")
# print(scene_analysis)
```
### brain.py
```python
import cv2
from ultralytics import YOLO
from mouth import analyze_scene
from serial import Serial
import asyncio
import time
arduino = Serial('/dev/cu.usbmodemFX2348N1', 9600)
time.sleep(1)
#the stand on hind legs first, then drop back down, then walk.
#step 1: Initiate -> stand up -> walk -> look around
arduino.write(b'1')
time.sleep(7)
#step 2: Computer vision AI stuffy
# whenever arduino hears "2", wag tail a bit.
# Load the YOLOv8 model
model = YOLO('yolov8n.pt') # Using YOLOv8 nano model
# Initialize camera
cap = cv2.VideoCapture(0)
async def main():
while True:
# Read frame from camera
ret, frame = cap.read()
if not ret:
break
# Run YOLOv8 inference on the frame
results = model(frame)
# Filter for only human detections (class 0)
for result in results:
boxes = result.boxes
human_boxes = boxes[boxes.cls == 0]
result.boxes = human_boxes
# If humans detected, get coordinates of closest person
if len(human_boxes) > 0:
# Wait for 3 more frames to confirm detection
for _ in range(30):
ret, frame = cap.read()
if not ret:
break
results = model(frame)
boxes = results[0].boxes
if len(boxes[boxes.cls == 0]) == 0:
break
# Display frame to keep UI responsive
annotated_frame = results[0].plot()
cv2.imshow('YOLOv8 Detection', annotated_frame)
cv2.waitKey(1)
else: # Only executes if loop completes without break
cv2.imwrite('person_detected.jpg', frame)
analyze_scene()
arduino.write(b'2') #wags tail
cap.release()
cv2.destroyAllWindows()
return
# Visualize the results on the frame
annotated_frame = results[0].plot()
# Display the frame
cv2.imshow('YOLOv8 Detection', annotated_frame)
# Break loop on 'q' press
if cv2.waitKey(1) & 0xFF == ord('q'):
break
# Release resources
cap.release()
cv2.destroyAllWindows()
arduino.close()
# Run the async main function
if __name__ == "__main__":
asyncio.run(main())
```
### leftbrain.py
```python
import assemblyai as aai
import os
import cv2
from dotenv import load_dotenv
import time
from mouth import analyze_scene
from throat import text_to_speech
def start_listening():
# Load environment variables
load_dotenv()
# Replace with your API key
aai.settings.api_key = os.getenv("ASSEMBLYAI_API_KEY")
# Initialize camera
cap = cv2.VideoCapture(0)
# Track silence duration
last_speech_time = time.time()
SILENCE_THRESHOLD = 3.0 # seconds of silence to consider speech finished
def on_open(session_opened: aai.RealtimeSessionOpened):
"This function is called when the connection has been established."
print("Session ID:", session_opened.session_id)
def on_data(transcript: aai.RealtimeTranscript):
"This function is called when a new transcript has been received."
nonlocal last_speech_time
if not transcript.text:
# Check if enough silence time has passed
if time.time() - last_speech_time > SILENCE_THRESHOLD:
ret, frame = cap.read()
if ret:
cv2.imwrite('memory_artifact.jpg', frame)
print("\nSpeech finished - Photo captured!")
# Await the analyze_scene call
analyze_scene("See if anything in the room can and how it would be used to help in a house fire. Here is what the person said " + transcript.text, "memory_artifact.jpg")
last_speech_time = time.time() # Reset timer
return
# Update the last speech time whenever we receive new text
last_speech_time = time.time()
if isinstance(transcript, aai.RealtimeFinalTranscript):
print(transcript.text, end="\r\n")
else:
print(transcript.text, end="\r")
def on_error(error: aai.RealtimeError):
"This function is called when the connection has been closed."
print("An error occured:", error)
def on_close():
"This function is called when the connection has been closed."
print("Closing Session")
cap.release()
transcriber = aai.RealtimeTranscriber(
on_data=on_data,
on_error=on_error,
sample_rate=44_100,
on_open=on_open, # optional
on_close=on_close, # optional
)
# Start the connection
transcriber.connect()
# Open a microphone stream
microphone_stream = aai.extras.MicrophoneStream()
# Press CTRL+C to abort
transcriber.stream(microphone_stream)
transcriber.close()
```
### mainbrain.py
```python
import os
import cv2
from ultralytics import YOLO
from mouth import analyze_scene
from serial import Serial
import asyncio
import time
from leftbrain import start_listening
from heart import interact_with_elevenlabs
arduino = Serial('/dev/cu.usbmodemFX2348N1', 9600)
time.sleep(2) # Increase delay to 2 seconds
# Clear any pending data in the serial buffer
arduino.reset_input_buffer()
arduino.reset_output_buffer()
#the stand on hind legs first, then drop back down, then walk.
#step 1: Initiate -> stand up -> walk -> look around
arduino.write(b'1')
os.system("afplay spotify/siren.mp3")
arduino.flush() # Ensure the data is written
time.sleep(10)
#step 2: Computer vision AI stuffy
# whenever arduino hears "2", wag tail a bit.
# Load the YOLOv8 model
model = YOLO('yolov8n.pt') # Using YOLOv8 nano model
# Initialize camera
cap = cv2.VideoCapture(0)
async def main():
while True:
# Read frame from camera
ret, frame = cap.read()
if not ret:
break
# Run YOLOv8 inference on the frame
results = model(frame)
# Filter for only human detections (class 0)
for result in results:
boxes = result.boxes
human_boxes = boxes[boxes.cls == 0]
result.boxes = human_boxes
# If humans detected, get coordinates of closest person
if len(human_boxes) > 0:
# Wait for 3 more frames to confirm detection
for _ in range(10):
ret, frame = cap.read()
if not ret:
break
results = model(frame)
boxes = results[0].boxes
if len(boxes[boxes.cls == 0]) == 0:
break
# Display frame to keep UI responsive
annotated_frame = results[0].plot()
cv2.imshow('YOLOv8 Detection', annotated_frame)
cv2.waitKey(1)
else: # Only executes if loop completes without break
cv2.imwrite('person_detected.jpg', frame)
arduino.write(b'2') #wags tail
#sends a report + call all right here
os.system("afplay spotify/spotted.mp3")
await analyze_scene("Quickly summarize the scene in 2-3 sentences as if you are talking to a person. Describe 1) Number of people 2) Any details about the person's age, gender, whether they look injured or distressed, and ignore their clothing 3) A quick one word guess of the room they are in.", 'person_detected.jpg')
arduino.write(b'2') #wags tail
#enter conversational mode for the rest of the time
#start_listening()
cap.release()
cv2.destroyAllWindows()
return
# Visualize the results on the frame
annotated_frame = results[0].plot()
# Display the frame
cv2.imshow('YOLOv8 Detection', annotated_frame)
# Break loop on 'q' press
if cv2.waitKey(1) & 0xFF == ord('q'):
break
# Release resources
cap.release()
cv2.destroyAllWindows()
arduino.close()
# Run the async main function
if __name__ == "__main__":
asyncio.run(main())
#demo delay
time.sleep(10)
interact_with_elevenlabs()
```