# Project export: SkySearch This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Software-defined Search-and-Rescue Drones - Devpost: https://devpost.com/software/skysearch-t5ci1v - GitHub: https://github.com/KevinWu098/calhacks24 - Video: https://www.youtube.com/embed/_AtPD2fjSvs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Spike O'Carroll (42 commits), Kevin Wu (36 commits), Bill Zhang (29 commits), Jasmine (9 commits) ## Devpost submission (written by the team) ### Inspiration Hurricane Milton, the most devastating disaster in over 30 years, left more than 3 million people without power and overwhelmed emergency services scrambling to respond. While drone technology now floods us with vast amounts of data, the real challenge lies in making sense of it in rapidly evolving environments—operators are still stuck manually sifting through critical information when time is running out. In moments of chaos, the ability to scale autonomous search-and-rescue missions and intelligently uncover patterns from data becomes essential. SkySearch enables operators to uncover hidden insights on the environment in vast seas of data by integrating real-time data on the environment, telemetry, and previous missions into a single pane of glass (software mission control system). Operators can deploy fleets of drones to investigate regions and collect video feed used to reconstruct the scene, enabling them to drill-down on areas of interest through a semantic search engine. ### What it does Our goal is to enable operators to interact with data and uncover hidden patterns effortlessly. SkySearch is built around the end-to-end search-and-rescue workflow in the following use cases: Search: Drones are deployed through the software by operators and autonomously navigate through terrain to identify objects of interest in real-time. Rescue: Operators can interact with live data to isolate hazards and locate people through a unified search interface. Based on this data, the system then recommends risk-aware, optimized rescue routes for first responders. Core features Environment reconstruction of damaged regions and infrastructure with Gaussian splatting Risk-aware pathfinding for rescue operations and pathfinding Semantic Search through disparate data sources to uncover patterns and recommend actions ### How we built it We designed an embedded architecture that enables software and hardware interfaces to bidirectionally communicate information and commands. Drone SDK used for live video streaming TP-Link Antennas for a local wifi system to create a more robust data pipeline between the drone and the software interface - rather than relying on Satellites and Wifi OpenCV and Apple Depth Pro used to process footage and classify data SingleStore for real-time database management ### Challenges we ran into Accounting for low battery drones Integration between hardware and software interfaces Balancing human judgement with autonomy ### Accomplishments we're proud of Implemented autonomous swarming framework to detect Integrated gaussian splatting Risk-aware map traversal and recommended "safe routes" for emergency responders Dynamic Data Generation to generate and query data dynamically allows for efficient testing and analysis, improving the app's responsiveness and visibility into critical information during rescue missions. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 80 recognized source files, 1206 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AWS (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 128) ``` .gitignore backend/.gitignore backend/.python-version backend/gemini.ipynb backend/pyproject.toml backend/README.md backend/requirements-dev.lock backend/requirements.lock backend/requirements.txt backend/src/backend/__init__.py backend/src/backend/clean_ply.py backend/src/backend/drone_agent.py backend/src/backend/drone.py backend/src/backend/load.ipynb backend/src/backend/main.py backend/src/backend/search.py backend/src/backend/yolo.py client/.env.local.example client/.eslintrc.json client/.gitignore client/bun.lockb client/components.json client/next.config.mjs client/package.json client/postcss.config.mjs client/prettier.config.cjs client/public/flyover.splat client/public/foo.splat client/public/test.ply client/README.md client/src/app/(layout)/dashboard/[slug]/page.tsx client/src/app/(layout)/dashboard/page.tsx client/src/app/(layout)/layout.tsx client/src/app/(layout)/one/page.tsx client/src/app/globals.css client/src/app/layout.tsx client/src/app/page.tsx client/src/app/playground/page.tsx client/src/components/dashboard/ActiveDrones.tsx client/src/components/dashboard/DetectedPersons.tsx client/src/components/dashboard/drone-assets.tsx client/src/components/dashboard/header.tsx client/src/components/dashboard/here/map.tsx client/src/components/dashboard/map/map-overview.tsx client/src/components/dashboard/map/map.tsx client/src/components/dashboard/map/new-map.tsx client/src/components/dashboard/nav.tsx client/src/components/dashboard/rescue/details.tsx client/src/components/dashboard/rescue/nearby-hazards.tsx client/src/components/dashboard/rescue/rescue-workflow.tsx client/src/components/sidebar/sidebar-accordion-radix.tsx client/src/components/sidebar/sidebar-accordion.tsx client/src/components/sidebar/sidebar-constants.ts client/src/components/sidebar/sidebar-link.tsx client/src/components/sidebar/sidebar-nav.tsx client/src/components/sidebar/sidebar-toggle.tsx client/src/components/sidebar/sidebar.tsx client/src/components/sidebar/useOpenItem.ts client/src/components/sidebar/useSidebar.ts client/src/components/splat/splat-viewer.tsx client/src/components/splat/splat.tsx client/src/components/splat/styles.css client/src/components/splat/viewer.tsx client/src/components/ui/badge.tsx client/src/components/ui/button.tsx client/src/components/ui/checkbox.tsx client/src/components/ui/input.tsx client/src/components/ui/progress.tsx client/src/components/ui/separator.tsx client/src/components/ui/skeleton.tsx client/src/components/ui/sonner.tsx client/src/components/ui/switch.tsx client/src/components/ui/toast.tsx client/src/components/ui/toaster.tsx client/src/hooks/use-toast.ts client/src/lib/utils.ts client/tailwind.config.ts client/tsconfig.json experiments/checkpoints/yolo11s_NP.pt experiments/config.toml experiments/datasets/broken-pole/data.yaml experiments/datasets/broken-pole/README.dataset.txt experiments/datasets/broken-pole/README.roboflow.txt experiments/depth-anything.ipynb experiments/gaussian_splat.sh experiments/logger.py experiments/models.py experiments/myconfig.py experiments/pylogger.py experiments/requirements.txt experiments/runs/detect/train/args.yaml experiments/runs/detect/train/results.csv experiments/runs/detect/train2/args.yaml experiments/runs/detect/train2/results.csv experiments/yolo-data.ipynb experiments/yolo-finetune.ipynb experiments/yolo-inference-video.ipynb experiments/yolo-inference.ipynb experiments/yolo11n.pt experiments/yolo11s.pt frontend/.env frontend/.vite/deps_temp_6a0bd87c/chunk-GFWMZNU4.js frontend/.vite/deps_temp_6a0bd87c/chunk-GFWMZNU4.js.map frontend/.vite/deps_temp_6a0bd87c/package.json frontend/.vite/deps_temp_6a0bd87c/react-dom_client.js frontend/.vite/deps_temp_6a0bd87c/react-dom_client.js.map frontend/.vite/deps_temp_6a0bd87c/react.js frontend/.vite/deps_temp_6a0bd87c/react.js.map frontend/bun.lockb frontend/components.json frontend/eslint.config.js frontend/index.html frontend/package.json frontend/postcss.config.js frontend/prettier.config.cjs frontend/README.md frontend/src/App.css frontend/src/App.tsx frontend/src/index.css frontend/src/lib/utils.ts [8 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: djitellopy@>=2.5.0, fastapi@>=0.115.2, netifaces@>=0.11.0, python-dotenv@>=1.0.1, singlestoredb@>=1.7.2, uagents@>=0.16.2, ultralytics@>=8.3.17, uvicorn@>=0.30.1 - backend/requirements.txt: backend@==0.1.0, colorama@==0.4.6, email-validator@==2.2.0, gradio@==4.21.0, grpcio-status@==1.67.0, httptools@==0.6.4, importlib-metadata@==8.5.0, ipykernel@==6.29.5, itsdangerous@==2.2.0, jaraco.collections@==5.1.0, langchain-google-genai@==2.0.1, langgraph@==0.2.39, loguru@==0.7.0, pickleshare@==0.7.5, pip-chill@==1.0.3, pydantic-extra-types@==2.9.0, pydantic-settings@==2.6.0, pyfiglet@==1.0.2, roboflow@==1.1.47, toml@==0.10.2, ujson@==5.10.0, uvloop@==0.21.0, watchfiles@==0.24.0 - client/package.json: @here/maps-api-for-javascript@^1.59.0, @ianvs/prettier-plugin-sort-imports@^4.3.1, @mkkellogg/gaussian-splats-3d@^0.4.5, @radix-ui/react-accordion@^1.2.1, @radix-ui/react-checkbox@^1.1.2, @radix-ui/react-progress@^1.1.0, @radix-ui/react-separator@^1.1.0, @radix-ui/react-slot@^1.1.0, @radix-ui/react-switch@^1.1.1, @radix-ui/react-toast@^1.2.2, @react-google-maps/api@^2.20.3, @react-three/drei@^9.114.6, @react-three/fiber@^8.17.10, @types/google.maps@^3.58.1, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/three@^0.169.0, axios@^1.7.7, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.15, google-maps-react@^2.0.6, lucide-react@^0.453.0, next@14.2.15, next-themes@^0.3.0, postcss@^8, prettier-plugin-tailwindcss@^0.6.8, react@^18, react-dom@^18, sonner@^1.5.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.14, tailwindcss-animate@^1.0.7, three@^0.169.0, typescript@^5 - experiments/requirements.txt: email-validator@==2.2.0, gradio@==4.21.0, httptools@==0.6.4, importlib-metadata@==8.5.0, ipykernel@==6.29.5, itsdangerous@==2.2.0, jaraco.collections@==5.1.0, loguru@==0.7.0, pickleshare@==0.7.5, pip-chill@==1.0.3, pydantic-extra-types@==2.9.0, pydantic-settings@==2.6.0, pyfiglet@==1.0.2, roboflow@==1.1.47, toml@==0.10.2, tomli@==2.0.1, ujson@==5.10.0, ultralytics@==8.3.12, uvloop@==0.21.0, watchfiles@==0.24.0 - frontend/package.json: @eslint/js@^9.13.0, @ianvs/prettier-plugin-sort-imports@^4.3.1, @react-google-maps/api@^2.20.3, @types/eslint__js@^8.42.3, @types/node@^22.7.7, @types/react@^18.3.11, @types/react-dom@^18.3.0, @vitejs/plugin-react-swc@^3.5.0, autoprefixer@^10.4.20, axios@^1.7.7, class-variance-authority@^0.7.0, clsx@^2.1.1, dotenv@^16.4.5, eslint@^9.13.0, eslint-plugin-react-hooks@^5.1.0-rc.0, eslint-plugin-react-refresh@^0.4.12, globals@^15.9.0, lucide-react@^0.453.0, postcss@^8.4.47, prettier-plugin-tailwindcss@^0.6.8, react@^18.3.1, react-dom@^18.3.1, tailwind-merge@^2.5.4, tailwindcss@^3.4.14, tailwindcss-animate@^1.0.7, typescript@^5.6.3, typescript-eslint@^8.10.0, vite@^5.4.8 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/KevinWu098/calhacks24 - add drone - why not - Merge branch 'main' of https://github.com/KevinWu098/calhacks24 - Small ui changes - Add extra person coords for pathfinding - Fix details - Fix /one coords - feat: toast stuff - Fix coords - feat: routing :) - input - Merge pull request #4 from KevinWu098/singlestore - change schem - Merge branch 'main' of https://github.com/KevinWu098/calhacks24 - yay - Merge branch 'singlestore' of https://github.com/KevinWu098/calhacks24 into singlestore - new schema - Pinpoint One demo - Rescue panel is dynamic ## Key source files (fetched from GitHub, selected and truncated for size) ### experiments/requirements.txt ``` email-validator==2.2.0 gradio==4.21.0 httptools==0.6.4 importlib-metadata==8.5.0 ipykernel==6.29.5 itsdangerous==2.2.0 jaraco.collections==5.1.0 loguru==0.7.0 pickleshare==0.7.5 pip-chill==1.0.3 pydantic-extra-types==2.9.0 pydantic-settings==2.6.0 pyfiglet==1.0.2 roboflow==1.1.47 toml==0.10.2 tomli==2.0.1 ujson==5.10.0 ultralytics==8.3.12 uvloop==0.21.0 watchfiles==0.24.0 ``` ### backend/requirements.txt ``` backend==0.1.0 colorama==0.4.6 email-validator==2.2.0 gradio==4.21.0 grpcio-status==1.67.0 httptools==0.6.4 importlib-metadata==8.5.0 ipykernel==6.29.5 itsdangerous==2.2.0 jaraco.collections==5.1.0 langchain-google-genai==2.0.1 langgraph==0.2.39 loguru==0.7.0 pickleshare==0.7.5 pip-chill==1.0.3 pydantic-extra-types==2.9.0 pydantic-settings==2.6.0 pyfiglet==1.0.2 roboflow==1.1.47 toml==0.10.2 ujson==5.10.0 uvloop==0.21.0 watchfiles==0.24.0 ``` ### backend/pyproject.toml ``` [project] name = "backend" version = "0.1.0" description = "Add your description here" authors = [ { name = "Spike O'Carroll", email = "19519553+spikecodes@users.noreply.github.com" } ] dependencies = [ "ultralytics>=8.3.17", "uagents>=0.16.2", "fastapi>=0.115.2", "netifaces>=0.11.0", "singlestoredb>=1.7.2", "uvicorn>=0.30.1", "djitellopy>=2.5.0", "python-dotenv>=1.0.1", ] readme = "README.md" requires-python = ">= 3.8" [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.rye] managed = true dev-dependencies = [] [tool.hatch.metadata] allow-direct-references = true [tool.hatch.build.targets.wheel] packages = ["src/backend"] ``` ### frontend/package.json ``` { "name": "calhacks24", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "@react-google-maps/api": "^2.20.3", "@types/node": "^22.7.7", "axios": "^1.7.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "dotenv": "^16.4.5", "lucide-react": "^0.453.0", "react": "^18.3.1", "react-dom": "^18.3.1", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@eslint/js": "^9.13.0", "@ianvs/prettier-plugin-sort-imports": "^4.3.1", "@types/eslint__js": "^8.42.3", "@types/react": "^18.3.11", "@types/react-dom": "^18.3.0", "@vitejs/plugin-react-swc": "^3.5.0", "autoprefixer": "^10.4.20", "eslint": "^9.13.0", "eslint-plugin-react-hooks": "^5.1.0-rc.0", "eslint-plugin-react-refresh": "^0.4.12", "globals": "^15.9.0", "postcss": "^8.4.47", "prettier-plugin-tailwindcss": "^0.6.8", "tailwindcss": "^3.4.14", "typescript": "^5.6.3", "typescript-eslint": "^8.10.0", "vite": "^5.4.8" } } ``` ### client/package.json ``` { "name": "client", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbo", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@here/maps-api-for-javascript": "^1.59.0", "@mkkellogg/gaussian-splats-3d": "^0.4.5", "@radix-ui/react-accordion": "^1.2.1", "@radix-ui/react-checkbox": "^1.1.2", "@radix-ui/react-progress": "^1.1.0", "@radix-ui/react-separator": "^1.1.0", "@radix-ui/react-slot": "^1.1.0", "@radix-ui/react-switch": "^1.1.1", "@radix-ui/react-toast": "^1.2.2", "@react-google-maps/api": "^2.20.3", "@react-three/drei": "^9.114.6", "@react-three/fiber": "^8.17.10", "axios": "^1.7.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "google-maps-react": "^2.0.6", "lucide-react": "^0.453.0", "next": "14.2.15", "next-themes": "^0.3.0", "react": "^18", "react-dom": "^18", "sonner": "^1.5.0", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7", "three": "^0.169.0" }, "devDependencies": { "@ianvs/prettier-plugin-sort-imports": "^4.3.1", "@types/google.maps": "^3.58.1", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "@types/three": "^0.169.0", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "prettier-plugin-tailwindcss": "^0.6.8", "tailwindcss": "^3.4.14", "typescript": "^5" } } ``` ### frontend/.vite/deps_temp_6a0bd87c/package.json ``` { "type": "module" } ``` ### frontend/src/main.tsx ```typescript import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import App from "./App.tsx"; import "./index.css"; createRoot(document.getElementById("root")!).render( ); ``` ### frontend/src/App.tsx ```typescript import { useEffect, useRef, useState } from "react"; import { Circle, GoogleMap, Marker, useJsApiLoader, } from "@react-google-maps/api"; import { AlertTriangle, BatteryFull, BatteryLow, BatteryMedium, BatteryWarning, ChevronLeft, ChevronRight, Diamond, Flame, MapPin, Plane, User, Wifi, WifiOff, } from "lucide-react"; interface Person { confidence: number; bbox: [number, number, number, number]; image: string; timestamp: string; } interface Drone { name: string; isConnected: boolean; batteryLevel: number; } interface WebSocketData { persons: Person[]; frame: string; droneStatus: Drone; } interface Hazard { type: "warning" | "fire"; location: { lat: number; lng: number }; } function App() { const [persons, setPersons] = useState([]); const [center, setCenter] = useState({ lat: 0, lng: 0 }); const [isRightPanelOpen, setIsRightPanelOpen] = useState(false); const [isConnected, setIsConnected] = useState(false); const [currentLocation, setCurrentLocation] = useState<{ lat: number; lng: number; } | null>(null); const [droneFeed, setDroneFeed] = useState(null); const canvasRef = useRef(null); const [drones, setDrones] = useState([ { name: "Drone 1", isConnected: false, batteryLevel: 0 }, ]); const [hazards, setHazards] = useState([ { type: "warning", location: { lat: 0, lng: 0 } }, { type: "fire", location: { lat: 0, lng: 0 } }, ]); const [selectedHazard, setSelectedHazard] = useState(null); const [focusedItem, setFocusedItem] = useState< "drone" | "hazard" | "person" | null >(null); const [mapRef, setMapRef] = useState(null); const { isLoaded } = useJsApiLoader({ id: "google-map-script", googleMapsApiKey: import.meta.env.VITE_GOOGLE_MAPS_API_KEY, }); useEffect(() => { const socket = new WebSocket("ws://localhost:8000/ws"); socket.onopen = () => { console.log("WebSocket connection established"); setIsConnected(true); }; socket.onmessage = (event) => { const data: WebSocketData = JSON.parse(event.data); setPersons((prevPersons) => { const lastPerson = data.persons[data.persons.length - 1]; if (lastPerson) { return [ { ...lastPerson, image: data.frame, timestamp: new Date().toLocaleTimeString(), }, ]; } return prevPersons; }); setDroneFeed(data.frame); // Update drone status and battery level setDrones((prevDrones) => { return prevDrones.map((drone) => drone.name === data.droneStatus.name ? data.droneStatus : drone ); }); }; socket.onclose = () => { console.log("WebSocket connection closed"); setIsConnected(false); // Disconnect all drones when WebSocket disconnects setDrones((prevDrones) => prevDrones.map((drone) => ({ ...drone, isConnected: false })) ); }; return () => { socket.close(); }; }, []); useEffect(() => { if (navigator.geolocation) { navigator.geolocation.getCurrentPosition( (position) => { const newCenter = { lat: position.coords.latitude, lng: position.coords.longitude, }; setCenter(newCenter); setCurrentLocation(newCenter); }, () => { console.log("Unable to retrieve your location"); } ); } }, []); useEffect(() => { if (droneFeed && canvasRef.current && isRightPanelOpen) { const img = new Image(); img.onload = () => { const canvas = canvasRef.current; if (canvas) { const ctx = canvas.getContext("2d"); if (ctx) { ctx.drawImage(img, 0, 0, canvas.width, canvas.height); } } }; img.src = `data:image/jpeg;base64,${droneFeed}`; } }, [droneFeed, isRightPanelOpen]); useEffect(() => { if (currentLocation) { // Set hazards with random offsets const warningLocation = { lat: currentLocation.lat + (Math.random() - 0.5) * 0.01, lng: currentLocation.lng + (Math.random() - 0.5) * 0.01, }; const fireLocation = { lat: currentLocation.lat + (Math.random() - 0.5) * 0.01, lng: currentLocation.lng + (Math.random() - 0.5) * 0.01, }; setHazards([ { type: "warning", location: warningLocation }, { type: "fire", location: fireLocation }, ]); // Set person with random offset const personLocation = { lat: currentLocation.lat + (Math.random() - 0.5) * 0.005, lng: currentLocation.lng + (Math.random() - 0.5) * 0.005, }; setPersons([ { confidence: 0.95, bbox: [personLocation.lat, personLocation.lng, 0, 0], image: "", timestamp: new Date().toLocaleTimeString(), }, ]); } }, [currentLocation]); [truncated — 14214 more characters] ``` ### client/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import localFont from "next/font/local"; import "./globals.css"; import { Toaster } from "@/components/ui/toaster"; const geistSans = localFont({ src: "./fonts/GeistVF.woff", variable: "--font-geist-sans", weight: "100 900", }); const geistMono = localFont({ src: "./fonts/GeistMonoVF.woff", variable: "--font-geist-mono", weight: "100 900", }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### client/src/app/page.tsx ```typescript import Image from "next/image"; export default function Home() { return (
Next.js logo
  1. Get started by editing{" "} src/app/page.tsx .
  2. Save and see your changes instantly.
); } ``` [70 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]