# Project export: Project Mithril This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Every second counts in earthquake disaster zones. Our solution combines AI, VR, and edge computing to detect structural voids and likely human locations in real time using just a headset. - Devpost: https://devpost.com/software/project-mithril - GitHub: https://github.com/ege-dgny/project_mithril - Team: 1 GitHub contributor(s) — doganay_e (1 commits) ## Devpost submission (written by the team) ### Inspiration We come from a country where earthquakes are not distant headlines — they’re lived realities. We've seen firsthand how chaotic and overwhelming the rescue process can be when buildings collapse and time is running out. That’s why we asked ourselves: How can we use the tools we have — AI, XR, and edge computing — to support the people risking their lives to save others? Project Mithril was born from that urgency: a vision to give rescuers clearer insight, faster decisions, and a better chance at finding those who are still alive — but hidden from view. ### What it does Project Mithril is an AI-powered augmented reality application designed to support search-and-rescue operations in the chaotic aftermath of earthquakes. Rescuers often face dangerous, unstable environments with little to no visibility into where survivors might be trapped. Mithril transforms this experience by equipping teams with AR headsets that scan the interior of collapsed buildings in real time. The system identifies structural elements like slabs, columns, and beams, then analyzes their angles and positions to detect possible survivable voids. Using this information, it highlights safe paths and warns about hazardous areas—all within the rescuer’s field of view. The result is faster, safer, and more focused rescue efforts where every second counts. ### How we built it Mithril was built by combining accessible consumer hardware with optimized artificial intelligence. We use Meta Quest 3 headsets for their passthrough AR capabilities and pair them with the NVIDIA Jetson Orin for real-time, offline computing. Our custom computer vision model is trained to detect patterns common in building collapses, such as inclined slabs forming voids, overhangs that may create shelter pockets, and pancake collapses indicating tight survival spaces. The system also incorporates a behavioral heatmap engine that factors in the time of the earthquake, building type, and typical human behavior to prioritize search zones. Everything is designed to be modular and run offline, so teams can deploy the system instantly, even in areas with zero connectivity. ### Challenges we ran into One of the biggest challenges was running complex computer vision models on-device without sacrificing speed or accuracy. To address this, we optimized our models through pruning and quantization, tailoring them specifically for the Jetson Orin platform. AR tracking in rubble-heavy environments also presented unique difficulties, requiring us to refine our spatial mapping techniques for accuracy under visually confusing conditions. Additionally, access to annotated datasets of real earthquake collapse scenes was limited, so we created a mix of simulations and physics-based renderings to train and validate our models. Perhaps most importantly, working directly with USAR teams helped us recognize how critical usability was; we had to redesign the interface to be effective even under stress, dust, and noise. ### Accomplishments we're proud of We’re proud to have created a fully offline, real-time AR solution that runs on consumer-grade hardware costing under $2,000—well within the budget range of most emergency teams. Our simulations successfully identified likely survivor zones and provided valuable situational awareness within seconds of deployment. More than just a tech demo, Mithrıl is a practical tool ready for field use. It’s designed to fit within the current workflows and training standards of rescue teams, which means it can be adopted quickly without extensive re-training. Most importantly, it offers a new layer of vision and insight to those who put themselves at risk to save others. ### What we learned Throughout the development of Project Mithrıl, we learned that speed and trust are everything in emergency response. Rescuers don’t just want data—they need information that is clear, actionable, and arrives exactly when they need it. We also discovered that transparency matters; users want to understand why a zone was flagged as dangerous or safe. Working with limited hardware proved that innovation doesn’t have to be expensive—clever software can make consumer devices life-saving tools. Finally, we realized that designing for field conditions means thinking beyond the screen, and building for real-world use cases like gloves, dust, noise, and urgency. ### What's next Looking ahead, we’re expanding our training datasets through collaborations with academic institutions and disaster-response organizations to improve model accuracy across diverse building types and collapse patterns. We plan to integrate audio sensors and thermal imaging to increase detection accuracy, especially in low-visibility conditions. Haptic feedback is also in development, so rescuers can receive subtle physical cues alongside visual ones. We’re preparing for broader deployment trials with FEMA and international partners to test and refine Mithrıl in real-world rescue missions. Ultimately, we aim to open the platform to outside developers and NGOs through an API, enabling customization and collaboration that can bring life-saving vision to more people around the world. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 9 recognized source files, 9 KB. - C# (language) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (105 of 105) ``` .DS_Store .gitignore .idea/.idea.project_mithril/.idea/.gitignore .idea/.idea.project_mithril/.idea/encodings.xml .idea/.idea.project_mithril/.idea/indexLayout.xml Assets/Detection.meta Assets/Detection/Bound_Box.prefab Assets/Detection/Bound_Box.prefab.meta Assets/InputSystem_Actions.inputactions Assets/InputSystem_Actions.inputactions.meta Assets/MetaXR.meta Assets/MetaXR/MetaXRProjectSettings.asset Assets/MetaXR/MetaXRProjectSettings.asset.meta Assets/Oculus.meta Assets/Oculus/OculusProjectConfig.asset Assets/Oculus/OculusProjectConfig.asset.meta Assets/Plugins.meta Assets/Plugins/Android.meta Assets/Plugins/Android/AndroidManifest.xml Assets/Plugins/Android/AndroidManifest.xml.meta Assets/Resources.meta Assets/Resources/ImmersiveDebuggerSettings.asset Assets/Resources/ImmersiveDebuggerSettings.asset.meta Assets/Resources/InputActions.asset Assets/Resources/InputActions.asset.meta Assets/Resources/MetaXRAcousticMaterialMapping.asset Assets/Resources/MetaXRAcousticMaterialMapping.asset.meta Assets/Resources/MetaXRAcousticSettings.asset Assets/Resources/MetaXRAcousticSettings.asset.meta Assets/Resources/MetaXRAudioSettings.asset Assets/Resources/MetaXRAudioSettings.asset.meta Assets/Resources/OculusRuntimeSettings.asset Assets/Resources/OculusRuntimeSettings.asset.meta Assets/Resources/OVRBuildConfig.asset Assets/Resources/OVRBuildConfig.asset.meta Assets/Resources/OVRPlatformToolSettings.asset Assets/Resources/OVRPlatformToolSettings.asset.meta Assets/Scenes.meta Assets/Scenes/SampleScene.unity Assets/Scenes/SampleScene.unity.meta Assets/Scripts.meta Assets/Scripts/DetectionListener.cs Assets/Scripts/DetectionListener.cs.meta Assets/Scripts/WebRTCStreamer.cs Assets/Scripts/WebRTCStreamer.cs.meta Assets/StreamingAssets.meta Assets/StreamRT.meta Assets/StreamRT/QuestCamRT.renderTexture Assets/StreamRT/QuestCamRT.renderTexture.meta Assets/XR.meta Assets/XR/Loaders.meta Assets/XR/Loaders/OpenXRLoader.asset Assets/XR/Loaders/OpenXRLoader.asset.meta Assets/XR/Loaders/SimulationLoader.asset Assets/XR/Loaders/SimulationLoader.asset.meta Assets/XR/Resources.meta Assets/XR/Resources/XRSimulationRuntimeSettings.asset Assets/XR/Resources/XRSimulationRuntimeSettings.asset.meta Assets/XR/Settings.meta Assets/XR/Settings/OpenXR Editor Settings.asset Assets/XR/Settings/OpenXR Editor Settings.asset.meta Assets/XR/Settings/OpenXR Package Settings.asset Assets/XR/Settings/OpenXR Package Settings.asset.meta Assets/XR/Settings/XRSimulationSettings.asset Assets/XR/Settings/XRSimulationSettings.asset.meta Assets/XR/UserSimulationSettings.meta Assets/XR/UserSimulationSettings/Resources.meta Assets/XR/UserSimulationSettings/Resources/XRSimulationPreferences.asset Assets/XR/UserSimulationSettings/Resources/XRSimulationPreferences.asset.meta Assets/XR/UserSimulationSettings/SimulationEnvironmentAssetsManager.asset Assets/XR/UserSimulationSettings/SimulationEnvironmentAssetsManager.asset.meta Assets/XR/XRGeneralSettingsPerBuildTarget.asset Assets/XR/XRGeneralSettingsPerBuildTarget.asset.meta edge_mithril/.DS_Store edge_mithril/docker-compose.yml edge_mithril/lt_launch.sh edge_mithril/nano_llm.sh edge_mithril/server.py Packages/manifest.json Packages/packages-lock.json ProjectSettings/AudioManager.asset ProjectSettings/ClusterInputManager.asset ProjectSettings/DynamicsManager.asset ProjectSettings/EditorBuildSettings.asset ProjectSettings/EditorSettings.asset ProjectSettings/GraphicsSettings.asset ProjectSettings/InputManager.asset ProjectSettings/MemorySettings.asset ProjectSettings/MultiplayerManager.asset ProjectSettings/NavMeshAreas.asset ProjectSettings/PackageManagerSettings.asset ProjectSettings/Packages/com.unity.testtools.codecoverage/Settings.json ProjectSettings/Physics2DSettings.asset ProjectSettings/PresetManager.asset ProjectSettings/ProjectSettings.asset ProjectSettings/ProjectVersion.txt ProjectSettings/QualitySettings.asset ProjectSettings/SceneTemplateSettings.json ProjectSettings/TagManager.asset ProjectSettings/TimeManager.asset ProjectSettings/UnityConnectSettings.asset ProjectSettings/VersionControlSettings.asset ProjectSettings/VFXManager.asset ProjectSettings/XRPackageSettings.asset ProjectSettings/XRSettings.asset ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Initial Commit ## Key source files (fetched from GitHub, selected and truncated for size) ### edge_mithril/docker-compose.yml ```yaml version: "3.9" services: vlm: image: $(jetson-containers autotag nano_llm) network_mode: host runtime: nvidia ipc: host command: ./run_nano_llm.sh hub: image: python:3.11-slim network_mode: host volumes: - ./backend:/app working_dir: /app command: python3 server.py ``` ### edge_mithril/server.py ```python from fastapi import FastAPI, WebSocket app, clients = FastAPI(), set() @app.websocket("/ws/detections") async def sock(ws: WebSocket): await ws.accept(); clients.add(ws) try: while True: await ws.receive_text() except: clients.remove(ws) @app.post("/push") async def push(item: dict): dead = [] for c in clients: try: await c.send_json(item) except: dead.append(c) for d in dead: clients.remove(d) ``` ### edge_mithril/nano_llm.sh ```shell #!/usr/bin/env bash exec python3 -m nano_llm.vision.example \ --model llava-v1.6-3b \ --video-input "udpsrc port=9000 \ ! application/x-rtp,encoding-name=H264 \ ! rtph264depay ! h264parse ! nvv4l2decoder \ ! video/x-raw(memory:NVMM),format=NV12 ! videoconvert ! appsink sync=false" \ --prompt "In this frame find entrances to survivable voids (life triangles) \ inside collapsed structures. Return JSON [{x,y,w,h,score}]." \ --ws-out ws://10.42.0.1:8000/ws/detections ``` ### edge_mithril/lt_launch.sh ```shell #!/usr/bin/env bash set -e SSID="RescueNet" PASS="quakeDemo2025" IFACE="wlan0" # adapt if your Jetson Wi-Fi is wlan1 CHANNEL="149" # 5 GHz DFS-free CONN_NAME="LT-Hotspot" # custom name so we don't clash with NM's default # ---------- bring up hotspot if it's not active ------------- if ! nmcli -t -f NAME connection show --active | grep -q "^${CONN_NAME}$"; then if nmcli connection show "${CONN_NAME}" &>/dev/null; then echo "[ltfinder] Activating existing hotspot ${CONN_NAME}" nmcli connection up "${CONN_NAME}" else echo "[ltfinder] Creating hotspot ${CONN_NAME}" nmcli dev wifi hotspot ifname "${IFACE}" ssid "${SSID}" \ band a channel "${CHANNEL}" password "${PASS}" \ connection.id "${CONN_NAME}" fi else echo "[ltfinder] Hotspot already up" fi # ensure 10.42.0.1 is up (NetworkManager default) echo "[ltfinder] Hotspot IP: $(ip -4 addr show "${IFACE}" | grep -oP '(?<=inet\s)\d+(\.\d+){3}')" # ---------- start docker compose ---------------------------- cd /home/jetson/edge_mithril /usr/bin/docker compose up ``` ### Assets/Scripts/WebRTCStreamer.cs ```c# // Assets/Scripts/WebRTCStreamer.cs using UnityEngine; using Unity.WebRTC; using System.Collections; using System.Net.Sockets; using System.Text; public class WebRTCStreamer : MonoBehaviour { [Header("Camera source")] public RenderTexture src; // drag QuestCamRT here [Header("Quick UDP signalling")] public string remoteIP = "192.168.0.42"; // Jetson Orin public int remotePort = 9000; RTCPeerConnection pc; VideoStreamTrack track; IEnumerator Start() { // (No WebRTC.Initialize() needed in 3.x) pc = new RTCPeerConnection(); track = new VideoStreamTrack(src); pc.AddTrack(track); /* -------- create SDP offer -------- */ var offerOp = pc.CreateOffer(); yield return offerOp; // wait until done if (offerOp.IsError) { Debug.LogError($"Offer error: {offerOp.Error.message}"); yield break; } var desc = offerOp.Desc; var setOp = pc.SetLocalDescription(ref desc); yield return setOp; if (setOp.IsError) { Debug.LogError($"SetLocalDescription error: {setOp.Error.message}"); yield break; } /* -------- fire offer to Jetson via one UDP packet -------- */ using (var udp = new UdpClient()) { byte[] bytes = Encoding.UTF8.GetBytes(pc.LocalDescription.sdp); udp.Send(bytes, bytes.Length, remoteIP, remotePort); Debug.Log($"Sent SDP offer ({bytes.Length} bytes) to {remoteIP}:{remotePort}"); } } void OnDestroy() { track?.Dispose(); pc?.Close(); // (No WebRTC.Dispose() in 3.x) } } ``` ### Assets/Scripts/DetectionListener.cs ```c# using UnityEngine; using System.Collections.Generic; using System.Text; using System.Threading.Tasks; using Meta.Net.NativeWebSocket; // namespace in Meta's fork public class DetectionListener : MonoBehaviour { [Header("HUD wiring")] public Canvas hudCanvas; // drag your Canvas here public RectTransform boxPrefab; // drag the lime-outline Image prefab WebSocket ws; readonly List pool = new(); /* ------------------------------------------------------------ */ async void Start() { ws = new WebSocket("ws://10.42.0.1:8000/ws/detections"); // Corrected line: The OnMessage event handler expects a single byte[] argument. ws.OnMessage += (bytes, offset, length) => HandleMsg(bytes, offset, length); await ws.Connect(); } /* ------------------------------------------------------------ */ void HandleMsg(byte[] bytes, int offset, int length) { // Jetson sends: {"detections":[{"x":0.3,"y":0.2,"w":0.1,"h":0.1,"score":0.85}]} var json = Encoding.UTF8.GetString(bytes, offset, length); var wrapper = JsonUtility.FromJson(json); if (wrapper == null || wrapper.detections == null) return; EnsurePool(wrapper.detections.Length); for (int i = 0; i < wrapper.detections.Length; i++) { var d = wrapper.detections[i]; var rt = pool[i]; rt.gameObject.SetActive(true); // flip Y for Unity's anchor space rt.anchorMin = new Vector2(d.x, 1f - (d.y + d.h)); rt.anchorMax = new Vector2(d.x + d.w, 1f - d.y); } // hide unused boxes for (int i = wrapper.detections.Length; i < pool.Count; i++) pool[i].gameObject.SetActive(false); } /* ------------------------------------------------------------ */ void EnsurePool(int n) { while (pool.Count < n) pool.Add(Instantiate(boxPrefab, hudCanvas.transform)); } /* ------------------------------------------------------------ */ async void OnApplicationQuit() { if (ws != null) await ws.Close(); } /* ------------------------------------------------------------ */ [System.Serializable] struct Detection { public float x, y, w, h, score; } // ←— make this a CLASS so it can be null-checked [System.Serializable] class DetectionArray { public Detection[] detections; } } ``` ### .idea/.idea.project_mithril/.idea/indexLayout.xml ```xml ``` ### .idea/.idea.project_mithril/.idea/encodings.xml ```xml ``` ### Assets/Plugins/Android/AndroidManifest.xml ```xml ```