# Project export: EmberScout: Autonomous Rescue for when Humans can't This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: EmberScout: An autonomous AI-on-edge powered rover designed to navigate wildfire zones, detect obstacles, and locate survivors, ensuring rescue missions where humans can't safely reach. - Devpost: https://devpost.com/software/quakescout-the-ai-pathfinder-for-saving-lives-in-the-rubble - GitHub: https://github.com/josiexw/treehacks25 - Video: https://www.youtube.com/embed/wIJV6qguReI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Suraj (5 commits), nathanjzhao (4 commits) ## Devpost submission (written by the team) ### Overview The LA Wildfires and Limited Personnel In the face of devastating wildfires that recently engulfed parts of Los Angeles, over 7,500 firefighting and emergency personnel rushed to battle the inferno. The response force swelled as reinforcements arrived from across the nation - Texas alone contributed more than 130 personnel and 45 engines/ambulances, while Oregon and several other states dispatched their bravest. Even international assistance poured in from three countries. Yet despite this massive mobilization, we discovered a haunting reality: the sheer scale of such disasters forces emergency teams to make impossible choices. With the majority of firefighters necessarily focused on containing the spread of fires, fewer resources remain for rescue operations. The tragic consequence? Precious lives are lost to smoke inhalation and burns while trapped in areas that rescue teams simply cannot reach in time. This sobering realization drove us to ask: What if we could multiply the reach of our rescue teams without dividing their resources? What if we could send scouts into dangerous conditions where human rescuers cannot safely go? Autonomous Control for Finding Victims Among the Ember EmberScout is our answer to this critical challenge - an autonomous RC car built specifically for deployment in disaster zones where traditional rescue methods fall short. But what sets it apart isn't just its ability to navigate treacherous terrain autonomously. We designed EmberScout to operate entirely on edge computing, a crucial feature in disaster scenarios where smoke, embers, and infrastructure damage can render cloud-based systems useless. Our prototype can continue searching for survivors (human body parts) by running advanced detection algorithms locally even when network connections fail. This edge-first approach means EmberScout doesn't need to rely on external GPU systems or cloud services that are often unavailable, guaranteeing usefulness in emergencies such as wildfires. How We Built EmberScout EmberScout's hardware foundation starts with a custom 3D-printed PRL chassis housing a TreeHacks-provided Jetson Orin Nano. By integrating NanoOwl and computer vision processing algorithms, we developed autonomous control systems that enable the RC car to navigate independently and identify people trapped in rubble. This runs entirely on edge to maintain functionality in communication-compromised environments. The system is powered by DC brushless motors controlled via ESP-32, while a Next.js interface driven by v0 enables both real-time demonstration capabilities and potential remote teleoperation - crucial for keeping human operators at a safe distance during active disasters. EmberScout seeks to identify the goal of a human body part to "identify" a victim and send a signal to nearby humans to pinpoint the location of a victim. Challenges and What We Learned Hardware development at hackathons presents a unique set of challenges, and our experience at TreeHacks drove this point home. First was printing the RC car itself, being able to adjust for the modularity of the 3D print such that we didn't need to run back and forth. In the end, we realized that we needed to bootstrap much of the hardware with tape and hot glue, especially with a design that revolved around our ever-changing electronics. Power management emerged as our most significant challenge. Fitting sufficient power delivery for robust motors onto a compact RC car base required innovative thinking. Our breakthrough came through implementing a 5V voltage regulator to power the Jetson Orin Nano via GPIO pins (Thanks Sarvesh!). Our next breakthrough was scouring around for a powerful battery that could power our motor systems and carry the weight of our Jetson Orin Nano amongst other devices on the system. Ultimately, we found that the real complexity lies in the interdependence of our software and hardware systems. Building a solution where both elements needed to work in perfect harmony required more than just technical skill - it demanded a crystal-clear vision at every iteration, without losing sight of our overall goal and the time crunch. While working on parallel development pipelines, we needed not only strong product direction but a comprehensive understanding of our complete implementation pipeline before writing a single line of code, ensuring seamless integration between software and hardware components. After this, being able to wire everything correctly and understand hardware that we had never worked with before in a software setting was another difficulty where we parsed through hundreds of documentation pages trying to understand foreign documentation on our electronics. Ultimately, we got everything to integrate together through lots of communication and help from the TreeHacks hardware team and trial-and-error. What's Next Currently, the implemented features include computer vision for object avoidance, path-finding, and manual remote control once the human is found. The EmberScout's target can be set to human body parts such as arms or hands so that the car can detect humans partially hidden under rubble. The next step is to have a response protocol when the RC car finds a person in need of rescue. Our goal is to attach a speaker/microphone model to relay conversation with the victim along with a robust flair/pinpointing system such that rescuers can find the victims. The addition of a thermal camera would also aid in finding people, leading to a more effective disaster response. In conclusion, we'd like to thank the amazing TreeHacks team for being so ready to help us out, the PRL, the amazing sponsors, and all the mentors (especially Sarvesh, who stayed up with us until 5 AM to aid our hacking and solve crucial problems with us). We hope that EmberScout can serve as a rough prototype for AI-on-edge devices for disaster recovery and we will continue to iterate on the idea due to our vision of its potential. Demo https://tinyurl.com/rc-car-treehacks25 ## README (from the GitHub repository) # jetson ssh into jetson: ssh -L 7860:localhost:7860 jetson@10.19.176.210 cd jetson-containers/ docker attach python3 tree_demo16.py --camera 0 --resolution 640x480 ../../data/owl_image_encoder_patch32.engine ctrl p ctrl q to detach properly how to open docker container jetson-containers run --workdir /opt/nanoowl $(autotag nanoowl) ## Detected evidence (automated analysis) Indexed codebase: 34 recognized source files, 297 KB. - C (language) — detected in the code - C++ (language) — detected in the code - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (49 of 49) ``` backend/server.py backend/yolov8n.pt frontend/.gitignore frontend/eslint.config.mjs frontend/jsconfig.json frontend/next.config.mjs frontend/package.json frontend/postcss.config.mjs frontend/public/udp_listener_example.py frontend/README.md frontend/src/app/api/autonomous-control/route.js frontend/src/app/api/control/route.js frontend/src/app/api/motor-control/route.js frontend/src/app/api/parse-task/route.js frontend/src/app/globals.css frontend/src/app/layout.js frontend/src/app/page.js frontend/src/components/ControlButtons.jsx frontend/src/components/PersonDetection.js frontend/tailwind.config.mjs frontend/tsconfig.json jetson/detection.html jetson/index.html jetson/raw.html jetson/run.sh jetson/run2.sh jetson/train_demo_final.py jetson/tree_demo10.py jetson/tree_demo11.py jetson/tree_demo12.py jetson/tree_demo13_original.py jetson/tree_demo13.py jetson/tree_demo14.py jetson/tree_demo15.py jetson/tree_demo16.py jetson/tree_demo7.py jetson/tree_demo8.py jetson/tree_demo9.py jetson/tree_drawing.py README.md treehacks_rc_car/.gitignore treehacks_rc_car/.vscode/extensions.json treehacks_rc_car/include/README treehacks_rc_car/lib/README treehacks_rc_car/platformio.ini treehacks_rc_car/src/main.cpp treehacks_rc_car/src/servo_control.cpp treehacks_rc_car/src/servo_control.h treehacks_rc_car/test/README ``` ### Dependencies - frontend/package.json: @eslint/eslintrc@^3, @tensorflow-models/coco-ssd@^2.2.3, @tensorflow/tfjs@^4.22.0, @types/react@19.0.8, dgram@^1.0.1, eslint@^9, eslint-config-next@15.1.7, next@15.1.7, openai@^4.85.1, postcss@^8, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^3.4.1, typescript@5.7.3 ### Recent commits (newest first) - stuff - everything - beep - revamped ui + UDP + task descriptions - servo - object detection - frontend redo - controlsv2 - Merge branch 'main' of https://github.com/josiexw/treehacks25 - controls - adding frontend (this is nathan) - pushing frontend (this is nathan) - not working bounding boxes - changed font - frontend changes - test - implemented transcript and changed ui - initialized frontend, backend, and platformio project for arduino - initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/package.json ``` { "name": "frontend2", "version": "0.1.0", "private": true, "scripts": { "dev": "rm -rf .next/ && next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@tensorflow-models/coco-ssd": "^2.2.3", "@tensorflow/tfjs": "^4.22.0", "dgram": "^1.0.1", "next": "15.1.7", "openai": "^4.85.1", "react": "^19.0.0", "react-dom": "^19.0.0" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/react": "19.0.8", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "5.7.3" } } ``` ### backend/server.py ```python from flask import Flask, request, Response, jsonify from flask_cors import CORS import requests import cv2 import numpy as np import queue import time import json import torch from ultralytics import YOLO app = Flask(__name__) CORS(app) # Load Pretrained YOLO Model (Person + Face Detection) device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = YOLO("yolov8n.pt").to(device) TARGET_CLASSES = {0: "person"} # Queues for data streaming speech_queue = queue.Queue() bbox_queue = queue.Queue() def detect_people_and_faces(image): """Run YOLO to detect people""" height, width, _ = image.shape # Run inference on the image results = model(image) humans = [] for r in results: for box in r.boxes.data: print("BOXES DETECTED:", box.tolist()) x1, y1, x2, y2, conf, cls = box.tolist() class_id = int(cls) # Only detect persons (class ID 0) if class_id in TARGET_CLASSES and conf > 0.8: humans.append({ "x1": int(x1), "y1": int(y1), "x2": int(x2), "y2": int(y2), "confidence": float(conf), "class": TARGET_CLASSES[class_id] }) return humans @app.route('/video_feed', methods=['POST']) def video_feed(): """Receive video frames, detect people and send bounding boxes""" file = request.files.get('frame') if not file: return "No frame received", 400 nparr = np.frombuffer(file.read(), np.uint8) frame = cv2.imdecode(nparr, cv2.IMREAD_COLOR) humans = detect_people_and_faces(frame) # Store bounding boxes in queue for streaming bbox_queue.put(humans) # Draw bounding boxes on frame (for debugging) for human in humans: label = f"{human['class']} {human['confidence']:.2f}" cv2.rectangle(frame, (human["x1"], human["y1"]), (human["x2"], human["y2"]), (0, 255, 0), 2) cv2.putText(frame, label, (human["x1"], human["y1"] - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) cv2.imshow('YOLO Detection', frame) cv2.waitKey(10) return jsonify({"message": "Frame received", "humans": humans}), 200 @app.route('/control', methods=['POST']) def control_motor(): """Send control commands to Arduino""" data = request.json command = data.get("command") if not command: return jsonify({"error": "No command received"}), 400 try: response = requests.get(f"http:///{command}") return jsonify({"message": f"Sent {command} to Arduino", "arduino_response": response.text}), 200 except requests.exceptions.RequestException as e: return jsonify({"error": f"Failed to send command to Arduino: {e}"}), 500 @app.route('/speech', methods=['POST']) def receive_speech(): """Receive and store speech transcripts""" data = request.json transcript = data.get("transcript") if not transcript: return jsonify({"error": "No transcript received"}), 400 speech_queue.put(transcript) return jsonify({"message": "Speech transcript received", "transcript": transcript}), 200 @app.route('/speech_stream', methods=['GET']) def speech_stream(): """Stream speech transcripts via SSE""" def event_stream(): while True: if not speech_queue.empty(): transcript = speech_queue.get() yield f"data: {transcript}\n\n" time.sleep(0.5) # Prevent excessive CPU usage return Response(event_stream(), mimetype="text/event-stream") @app.route('/bbox_stream', methods=['GET']) def bbox_stream(): """Stream bounding boxes via SSE""" def event_stream(): while True: if not bbox_queue.empty(): boxes = bbox_queue.get() json_data = json.dumps(boxes) print("json_data", json_data) yield f"data: {json_data}\n\n" time.sleep(0.5) # Prevent CPU overload return Response(event_stream(), mimetype="text/event-stream") if __name__ == '__main__': app.run(host='0.0.0.0', port=5000, debug=True, threaded=True) ``` ### frontend/src/app/page.js ```javascript 'use client'; import PersonDetection from '../components/PersonDetection'; export default function Home() { return (
); } ``` ### frontend/src/app/layout.js ```javascript import './globals.css' export const metadata = { title: 'Person Detection', description: 'Real-time person detection using TensorFlow.js', } export default function RootLayout({ children }) { return ( {children} ) } ``` ### frontend/src/app/api/autonomous-control/route.js ```javascript import { NextResponse } from 'next/server'; export async function POST(request) { try { const { enabled } = await request.json(); // Forward the autonomous control state to the Python backend const response = await fetch('http://localhost:7860/autonomous-control', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ enabled }), }); if (!response.ok) { throw new Error('Failed to update autonomous control state'); } const data = await response.json(); return NextResponse.json(data); } catch (error) { console.error('Error in autonomous control API:', error); return NextResponse.json( { success: false, error: error.message }, { status: 500 } ); } } ``` ### frontend/src/app/api/motor-control/route.js ```javascript import { NextResponse } from 'next/server'; export async function POST(request) { try { const { enabled } = await request.json(); // Forward the motor control state to the Python backend const response = await fetch('http://localhost:7860/motor-control', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ enabled }), }); if (!response.ok) { throw new Error('Failed to update remote control state'); } // Get the actual state from the backend response const data = await response.json(); return NextResponse.json(data); // Pass through the backend response directly } catch (error) { console.error('Error in motor control API:', error); return NextResponse.json( { success: false, error: error.message }, { status: 500 } ); } } ``` ### frontend/src/app/api/control/route.js ```javascript import { NextResponse } from 'next/server'; // UDP setup for broadcasting commands const dgram = require('dgram'); const client = dgram.createSocket('udp4'); const BROADCAST_ADDRESS = '255.255.255.255'; const PORT = 8888; // Enable broadcasting client.bind(() => { client.setBroadcast(true); console.log('UDP Client ready for broadcasting on port', PORT); }); // Add error handler client.on('error', (err) => { console.error('UDP Client error:', err); }); export async function POST(request) { try { const { direction } = await request.json(); // Forward the control command to the Python backend const response = await fetch('http://localhost:7860/control', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ direction }), }); if (!response.ok) { throw new Error('Failed to send command to backend'); } return NextResponse.json({ success: true, command: direction }); } catch (error) { console.error('Error in control API:', error); return NextResponse.json( { success: false, error: error.message }, { status: 500 } ); } } ``` ### frontend/src/app/api/parse-task/route.js ```javascript import { NextResponse } from 'next/server'; import OpenAI from 'openai'; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); // COCO-SSD classes that we can detect const COCO_CLASSES = [ 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush' ]; export async function POST(request) { try { const { task } = await request.json(); const response = await openai.chat.completions.create({ model: "gpt-3.5-turbo", messages: [ { role: "system", content: `You are a helper that identifies target objects and potential obstacles from user tasks. Return a JSON object with two fields: 1. "target": The main object to track 2. "obstacles": Array of objects to avoid Keep descriptions simple and clear. No adjectives, add articles of objects (e.g. "a cup"), and add as many redundancies as possible (e.g. for limb, add "a arm", "a hand", "a finger"). Example: {"target": "a cup", "obstacles": ["a chair", "a laptop"]}` }, { role: "user", content: task } ], temperature: 0, max_tokens: 100 }); const parsed = JSON.parse(response.choices[0].message.content); // Send the parsed objects to the Python backend const backendResponse = await fetch('http://localhost:7860/update-prompt', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ prompt: `[${parsed.target}, ${parsed.obstacles.join(', ')}]`, target: parsed.target, obstacles: parsed.obstacles }) }); if (!backendResponse.ok) { throw new Error('Failed to update backend prompt'); } return NextResponse.json(parsed); } catch (error) { console.error('Error parsing task:', error); return NextResponse.json( { error: 'Failed to parse task' }, { status: 500 } ); } } ``` ### jetson/raw.html ```html Raw Video Stream ``` ### jetson/detection.html ```html Detection Feed ``` [23 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]