# Project export: RealityRip This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: A camera-based spatial mapping system that converts real-world environments into interactive digital maps in real time, all without requiring expensive LiDAR hardware! - Devpost: https://devpost.com/software/change-qeonbz - GitHub: https://github.com/DPandaman/RealityRip - Video: https://www.youtube.com/embed/-KmVYt7u5AI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 0 GitHub contributor(s) — ## Devpost submission (written by the team) ### Overview Objective A drone flight simulator that combines high-fidelity Gaussian Splatting digital twin environments with a generative AI commentator. Our goal was to train fpv drone pilots in digital twins of real environments they’d fly in, and give them live feedback to keep them engaged. Using an iPhone, RealityRip can reconstruct a room or building into a navigable 3D model within minutes. The generated digital twin can be flown through immediately or modified to simulate obstacles, hazards, or mission scenarios. Motivation In search-and-rescue operations, pilots must navigate unfamiliar, cluttered, and often dangerous environments under extreme time pressure. Preparing for these conditions is hard because of unrealistic and limited map selection, mainly due to the effort required to 3d model a training course. RealityRip was created to fix that. We wanted to build a training system that replicates the spatial complexity of real environments while actively guiding pilots as they learn. By reconstructing real-world spaces using Gaussian Splatting, we transform homes, buildings, and disaster zones into high-fidelity digital twins. Pilots can train in environments that mirror the unpredictability and constraints of real missions without having to fear not having practiced beforehand.. Features Gaussian Splatting to generate a fully 3d, textured, model of your world from just an iphone Both Local and Cloud AI assistance Physics based FPV sim ### Gaussian Splatting Gaussian Splatting is a technique to create a fully textured 3d model of a portion of the world solely off of video. It generates both extremely accurate depth maps and texture maps, but comes with the tradeoff of requiring a large amount of compute. For this project, because of our focus on Edge AI, we decided to augment the Gaussian Splatting with the iPhone's built in lidar camera to focus on reducing latency. We use the iPhone’s built-in LIDAR sensor, a forked version of an iPhone app to transmit data to our ASUS computer, and the raw computing power of the ASUS Ascent GX10. We used SplaTAM, which is a Gaussian Splatting implementation that also takes advantage of the iPhone’s lidar. We decided to include the iPhone’s lidar in our splatting because it helped with stability, especially in darker rooms. It also greatly reduced the compute necessary to generate our splat. We forked SplaTAM and the iPhone app NeRF Capture to get around an issue they had with wireless communication. These 2 projects talk to each other using multicast, which is a P2P networking protocol that is blocked by Eduroam. We forked both of these to add support for sending directly to an IP address, and added support for real time visualization of gaussian splats as well. Local and Cloud AI integrations We use both local (running on an ASUS Ascent GX10) and cloud (GPT-5.2) AI models, including VLMs, LLMs, and a text to speech model. GPT-5.2 is used as a VLM, and is used to help critique a flyers pathing to a given route. For more latency sensitive tasks, like generating audio feedback for the user, we use local models running on the ASUS Ascent GX10. We are using openai’s gpt-oss-120b for our LLM, which we chose due to its relatively light weight, and also native fp4 compute, which takes full advantage of the GX10’s best computational unit. For our text-to-speech (TTS) model, we are going with the newly released Chatterbox from resemble.ai. Our main focus here was to keep the latency down, while still having an extremely expressive and good sounding model. We’re super happy about using open models on the GX10 because of how much they reduce total latency in our system. Physics-Driven FPV Controller We used Unity to create our FPV sim, mainly because of the quality of its documentation. Unity has a plugin that allows it to directly render our .ply and .gltf files that our gaussian splatting creates, and also is a full game engine, which made creating our drone sim relatively easy. One of the most important parts of FPV sims is ensuring that they match the physics of the real world. We leaned on Unity’s RigidBody system, which gave us collision detection along with sensible gravity and acceleration. Setup & Installation Clone the repository. Unity Version: Open in Unity 2022.3+ or Unity 6. API Configuration: Ensure you have your OpenAI API key set as an environment variable (OPENAI_API_KEY) on your system. The AIService.cs script will pull this automatically. Hardware: Connect your RadioMaster Pocket via USB-C (Game Controller mode). Press Play and fly! Team Members: Devanshu Pandya (University of Illinois) Julia Jiang (Stanford University) Koichi Kimoto (Stanford University) Rohan Godha (Georgia-Tech) ## README (from the GitHub repository) # RealityRip | TreeHacks 2026 * Devanshu Pandya (University of Illinois) * Julia Jiang (Stanford University) * Koichi Kimoto (Stanford University) * Rohan Godha (Georgia-Tech) ## Objective A drone flight simulator that combines high-fidelity Gaussian Splatting digital twin environments with a generative AI commentator. Our goal was to train fpv drone pilots in digital twins of real environments they’d fly in, and give them live feedback to keep them engaged. Using an iPhone, RealityRip can reconstruct a room or building into a navigable 3D model within minutes. The generated digital twin can be flown through immediately or modified to simulate obstacles, hazards, or mission scenarios. ## Motivation In search-and-rescue operations, pilots must navigate unfamiliar, cluttered, and often dangerous environments under extreme time pressure. Preparing for these conditions is hard because of unrealistic and limited map selection, mainly due to the effort required to 3d model a training course. RealityRip was created to fix that. We wanted to build a training system that replicates the spatial complexity of real environments while actively guiding pilots as they learn. By reconstructing real-world spaces using Gaussian Splatting, we transform homes, buildings, and disaster zones into high-fidelity digital twins. Pilots can train in environments that mirror the unpredictability and constraints of real missions without having to fear not having practiced beforehand.. ## Features - Gaussian Splatting to generate a fully 3d, textured, model of your world from just an iphone - Both Local and Cloud AI assistance - Physics based FPV sim ### Gaussian Splatting Gaussian Splatting is a technique to create a fully textured 3d model of a portion of the world solely off of video. It generates both extremely accurate depth maps and texture maps, but comes with the tradeoff of requiring a large amount of compute. For this project, because of our focus on Edge AI, we decided to augment the Gaussian Splatting with the iPhone's built in lidar camera to focus on reducing latency. We use the iPhone’s built-in LIDAR sensor, a forked version of an iPhone app to transmit data to our ASUS computer, and the raw computing power of the ASUS Ascent GX10. We used SplaTAM, which is a Gaussian Splatting implementation that also takes advantage of the iPhone’s lidar. We decided to include the iPhone’s lidar in our splatting because it helped with stability, especially in darker rooms. It also greatly reduced the compute necessary to generate our splat. We forked SplaTAM and the iPhone app NeRF Capture to get around an issue they had with wireless communication. These 2 projects talk to each other using multicast, which is a P2P networking protocol that is blocked by Eduroam. We forked both of these to add support for sending directly to an IP address, and added support for real time visualization of gaussian splats as well. ### Local and Cloud AI integrations We use both local (running on an ASUS Ascent GX10) and cloud (GPT-5.2) AI models, including VLMs, LLMs, and a text to speech model. GPT-5.2 is used as a VLM, and is used to help critique a flyers pathing to a given route. For more latency sensitive tasks, like generating audio feedback for the user, we use local models running on the ASUS Ascent GX10. We are using openai’s gpt-oss-120b for our LLM, which we chose due to its relatively light weight, and also native fp4 compute, which takes full advantage of the GX10’s best computational unit. For our text-to-speech (TTS) model, we are going with the newly released Chatterbox from resemble.ai. Our main focus here was to keep the latency down, while still having an extremely expressive and good sounding model. We’re super happy about using open models on the GX10 because of how much they reduce total latency in our system. ### Physics-Driven FPV Controller We used Unity to create our FPV sim, mainly because of the quality of its documentation. Unity has a plugin that allows it to directly render our .ply and .gltf files that our gaussian splatting creates, and also is a full game engine, which made creating our drone sim relatively easy. One of the most important parts of FPV sims is ensuring that they match the physics of the real world. We leaned on Unity’s RigidBody system, which gave us collision detection along with sensible gravity and acceleration. ## Setup & Installation 1. **Clone the repository.** 2. **Unity Version:** Open in Unity 2022.3+ or Unity 6. 3. **API Configuration:** Ensure you have your OpenAI API key set as an environment variable (OPENAI_API_KEY) on your system. The AIService.cs script will pull this automatically. 4. **Hardware:** Connect your RadioMaster Pocket via USB-C (Game Controller mode). 5. **Press Play** and fly! ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 80 KB. - C# (language) — detected in the code - C (language) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 216) ``` .DS_Store .gitmodules DroneSim/.DS_Store DroneSim/.gitignore DroneSim/.vscode/extensions.json DroneSim/.vscode/launch.json DroneSim/.vscode/settings.json DroneSim/Assets/.DS_Store DroneSim/Assets/DronePhysics.physicMaterial DroneSim/Assets/DronePhysics.physicMaterial.meta DroneSim/Assets/FPVCamera.cs DroneSim/Assets/FPVCamera.cs.meta DroneSim/Assets/GaussianAssets.meta DroneSim/Assets/GaussianAssets/poster_col.bytes.meta DroneSim/Assets/GaussianAssets/poster_oth.bytes.meta DroneSim/Assets/GaussianAssets/poster_pos.bytes.meta DroneSim/Assets/GaussianAssets/poster_shs.bytes.meta DroneSim/Assets/GaussianAssets/poster.asset DroneSim/Assets/GaussianAssets/poster.asset.meta DroneSim/Assets/GaussianAssets/room_col.bytes.meta DroneSim/Assets/GaussianAssets/room_oth.bytes.meta DroneSim/Assets/GaussianAssets/room_pos.bytes.meta DroneSim/Assets/GaussianAssets/room_shs.bytes.meta DroneSim/Assets/GaussianAssets/room.asset DroneSim/Assets/GaussianAssets/room.asset.meta DroneSim/Assets/huang_corridor.glb DroneSim/Assets/huang_corridor.glb.meta DroneSim/Assets/huang_steps.glb DroneSim/Assets/huang_steps.glb.meta DroneSim/Assets/huangStairs.glb DroneSim/Assets/huangStairs.glb.meta DroneSim/Assets/InputSystem_Actions.inputactions DroneSim/Assets/InputSystem_Actions.inputactions.meta DroneSim/Assets/NeonRed.mat DroneSim/Assets/NeonRed.mat.meta DroneSim/Assets/Readme.asset DroneSim/Assets/Readme.asset.meta DroneSim/Assets/Scenes.meta DroneSim/Assets/Scenes/SampleScene.unity DroneSim/Assets/Scenes/SampleScene.unity.meta DroneSim/Assets/scripts.meta DroneSim/Assets/scripts/AIService.cs DroneSim/Assets/scripts/AIService.cs.meta DroneSim/Assets/scripts/DroneCommentator.cs DroneSim/Assets/scripts/DroneCommentator.cs.meta DroneSim/Assets/scripts/DroneController.cs DroneSim/Assets/scripts/DroneController.cs.meta DroneSim/Assets/scripts/FlightManager.cs DroneSim/Assets/scripts/FlightManager.cs.meta DroneSim/Assets/scripts/GoalManager.cs DroneSim/Assets/scripts/GoalManager.cs.meta DroneSim/Assets/scripts/GoalMarker.prefab DroneSim/Assets/scripts/GoalMarker.prefab.meta DroneSim/Assets/scripts/MissionController.cs DroneSim/Assets/scripts/MissionController.cs.meta DroneSim/Assets/scripts/PathArchitect.cs DroneSim/Assets/scripts/PathArchitect.cs.meta DroneSim/Assets/scripts/PathGenerator.cs DroneSim/Assets/scripts/PathGenerator.cs.meta DroneSim/Assets/scripts/RedGlow.mat DroneSim/Assets/scripts/RedGlow.mat.meta DroneSim/Assets/scripts/RedRing.prefab DroneSim/Assets/scripts/RedRing.prefab.meta DroneSim/Assets/scripts/RescueGoalGenerator.cs DroneSim/Assets/scripts/RescueGoalGenerator.cs.meta DroneSim/Assets/scripts/UIManager.cs DroneSim/Assets/scripts/UIManager.cs.meta DroneSim/Assets/scripts/VisionBridge.cs DroneSim/Assets/scripts/VisionBridge.cs.meta DroneSim/Assets/scripts/VoiceService.cs DroneSim/Assets/scripts/VoiceService.cs.meta DroneSim/Assets/Settings.meta DroneSim/Assets/Settings/DefaultVolumeProfile.asset DroneSim/Assets/Settings/DefaultVolumeProfile.asset.meta DroneSim/Assets/Settings/Mobile_Renderer.asset DroneSim/Assets/Settings/Mobile_Renderer.asset.meta DroneSim/Assets/Settings/Mobile_RPAsset.asset DroneSim/Assets/Settings/Mobile_RPAsset.asset.meta DroneSim/Assets/Settings/PC_Renderer.asset DroneSim/Assets/Settings/PC_Renderer.asset.meta DroneSim/Assets/Settings/PC_RPAsset.asset DroneSim/Assets/Settings/PC_RPAsset.asset.meta DroneSim/Assets/Settings/SampleSceneProfile.asset DroneSim/Assets/Settings/SampleSceneProfile.asset.meta DroneSim/Assets/Settings/UniversalRenderPipelineGlobalSettings.asset DroneSim/Assets/Settings/UniversalRenderPipelineGlobalSettings.asset.meta DroneSim/Assets/TextMesh Pro.meta DroneSim/Assets/TextMesh Pro/Fonts.meta DroneSim/Assets/TextMesh Pro/Fonts/LiberationSans - OFL.txt DroneSim/Assets/TextMesh Pro/Fonts/LiberationSans - OFL.txt.meta DroneSim/Assets/TextMesh Pro/Fonts/LiberationSans.ttf.meta DroneSim/Assets/TextMesh Pro/Resources.meta DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials.meta DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Drop Shadow.mat DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Drop Shadow.mat.meta DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Fallback.asset DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Fallback.asset.meta DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Outline.mat DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Outline.mat.meta DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF.asset DroneSim/Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF.asset.meta DroneSim/Assets/TextMesh Pro/Resources/LineBreaking Following Characters.txt DroneSim/Assets/TextMesh Pro/Resources/LineBreaking Following Characters.txt.meta DroneSim/Assets/TextMesh Pro/Resources/LineBreaking Leading Characters.txt DroneSim/Assets/TextMesh Pro/Resources/LineBreaking Leading Characters.txt.meta DroneSim/Assets/TextMesh Pro/Resources/Sprite Assets.meta DroneSim/Assets/TextMesh Pro/Resources/Sprite Assets/EmojiOne.asset DroneSim/Assets/TextMesh Pro/Resources/Sprite Assets/EmojiOne.asset.meta DroneSim/Assets/TextMesh Pro/Resources/Style Sheets.meta DroneSim/Assets/TextMesh Pro/Resources/Style Sheets/Default Style Sheet.asset DroneSim/Assets/TextMesh Pro/Resources/Style Sheets/Default Style Sheet.asset.meta DroneSim/Assets/TextMesh Pro/Resources/TMP Settings.asset DroneSim/Assets/TextMesh Pro/Resources/TMP Settings.asset.meta DroneSim/Assets/TextMesh Pro/Shaders.meta DroneSim/Assets/TextMesh Pro/Shaders/SDFFunctions.hlsl DroneSim/Assets/TextMesh Pro/Shaders/SDFFunctions.hlsl.meta DroneSim/Assets/TextMesh Pro/Shaders/TMP_Bitmap-Custom-Atlas.shader DroneSim/Assets/TextMesh Pro/Shaders/TMP_Bitmap-Custom-Atlas.shader.meta DroneSim/Assets/TextMesh Pro/Shaders/TMP_Bitmap-Mobile.shader DroneSim/Assets/TextMesh Pro/Shaders/TMP_Bitmap-Mobile.shader.meta [96 more files omitted for size] ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update README - Restored SplaTAM and NeRFCapture submodules - Added drone sim in Unity ## Key source files (fetched from GitHub, selected and truncated for size) ### scripts/FlightManager.cs ```c# // contains respawn logic using UnityEngine public class FlightManager : MonoBehaviour { public GameObject drone; public Transform startPoint; public DroneCommentator commentator; public void ResetDrone(){ // move drone to start drone.transform.position = startPoint.position; drone.transform.rotation = startPoint.rotation; // kill momentum Rigidbody rb = drone.GetComponent(); rb.linearVelocity = Vector3.zero; rb.angularVelocity = Vector3.zero; // reset goal state commentator.ResetGoalStatus(); Debug.Log("respawned: drone reset to start"); } } ``` ### scripts/PathGenerator.cs ```c# // uses Unity Splines package to handle text -> path request using UnityEngine; using UnityEngine.Splines; // dependency: unity splines package using System.Linq; using System.Collections.Generic; public class PathGenerator : MonoBehaviour { // container for the catmull-rom/bezier spline interpolation public SplineContainer splineContainer; public void BuildRescuePath(List waypoints) { // sanity check for null references if (splineContainer == null) { Debug.LogError("PathGenerator: spline container missing. aborting trajectory generation."); return; } Spline spline = splineContainer.Spline; spline.Clear(); // flushing previous trajectory // Debug.Log($"PathGenerator: interpolating path for {waypoints.Count} nodes."); // NOTE: could add x0 (current drone pos) as the first knot for continuity // spline.Add(new BezierKnot(Vector3.zero)); foreach (Transform t in waypoints) { // defining the control point (knot) in 3D space BezierKnot knot = new BezierKnot(t.position); // enforcing C1 continuity (tangents) // setting tangent vectors manually to ensure smooth curvature through the knot // avoids sharp discontinuities in the derivative (velocity) knot.TangentIn = new Vector3(0, 0, -1f); knot.TangentOut = new Vector3(0, 0, 1f); spline.Add(knot); } // update the spline mesh instantiation // required to visualize the vector field (arrows/line renderer) if(splineContainer.GetComponent()) { splineContainer.GetComponent().UpdateInstances(); } Debug.Log("PathGenerator: trajectory computed."); } } ``` ### scripts/GoalManager.cs ```c# // gps and mission logic: stores list of specific locations in the map // you can choose which location the drone will fly toward using UnityEngine; using System.Collections.Generic; using System.Linq; public class GoalManager : MonoBehaviour { [System.Serializable] public struct Landmark { public string name; // e.g. "kitchen" public Transform location; // spot in 3d space } // connections public List landmarks; // list of room spots public DroneCommentator commentator; // drone ref public void SetActiveGoal(string goalName){ // find spot by name Landmark target = landmarks.FirstOrDefault(l => l.name.ToLower() == goalName.ToLower()); if (target.location != null){ // update commentator target commentator.currentGoal = target.location; // reset goal flag so ai can trigger again commentator.goalReached = false; Debug.Log($"goal set to: {target.name}"); } else{ Debug.LogWarning($"landark not found: {goalName}"); } } public void SetRandomGoal(){ // pick random spot from list if (landmarks.Count > 0){ int rnd = Random.Range(0, landmarks.Count); commentator.currentGoal = landmarks[rnd].location; commentator.goalReached = false; } } // run this to find all landmarks under a specific parent object public void FindAllLandmarks(){ // clear existing list landmarks.Clear(); // look at all children of this manager foreach (Transform child in transform){ Landmark l; l.name = child.name; // uses the gameobject name as the goal name l.location = child; landmarks.Add(l); } Debug.Log($"found {landmarks.Count} landmarks"); } void Start(){ // find them automatically when the game starts FindAllLandmarks(); } } ``` ### scripts/DroneController.cs ```c# // drone controlling logic using UnityEngine; using UnityEngine.InputSystem; [RequireComponent(typeof(Rigidbody))] public class DroneController : MonoBehaviour { [Header("settings")] public float throttlePower = 20f; public float cyclicPower = 5f; public float yawPower = 2f; [Header("inputs")] public InputActionReference throttleAction; public InputActionReference cyclicAction; public InputActionReference yawAction; private Rigidbody rb; private Vector2 cyclicInput; public float throttleInput; private float yawInput; void Awake(){ // init physics rb = GetComponent(); rb.mass = 1.0f; rb.drag = 1.0f; rb.angularDrag = 2.0f; } void Update(){ // read stick values if (cyclicAction != null) cyclicInput = cyclicAction.action.ReadValue(); if (yawAction != null) yawInput = yawAction.action.ReadValue(); // remap throttle -1:1 to 0:1 float rawThrottle = (throttleAction != null) ? throttleAction.action.ReadValue() : -1f; throttleInput = (rawThrottle + 1f) / 2f; } void FixedUpdate(){ // apply forces HandlePropellers(); HandleStabilization(); } void HandlePropellers(){ // apply lift Vector3 liftForce = Vector3.up * (throttleInput * throttlePower); rb.AddRelativeForce(liftForce); // apply rotations rb.AddRelativeTorque(Vector3.right * cyclicInput.y * cyclicPower); // pitch rb.AddRelativeTorque(Vector3.back * cyclicInput.x * cyclicPower); // roll rb.AddRelativeTorque(Vector3.up * yawInput * yawPower); // yaw } void HandleStabilization(){ // auto level if no input if (cyclicInput.magnitude < 0.1f){ Quaternion targetRotation = Quaternion.Euler(0, transform.eulerAngles.y, 0); transform.rotation = Quaternion.Slerp(transform.rotation, targetRotation, Time.fixedDeltaTime * 2.0f); } } } ``` ### scripts/MissionController.cs ```c# // main script // tells VisionBridge to scan room // passes VLM results to goal generator // gets path from path generator // sends final mission to commentator using UnityEngine; public class MissionController : MonoBehaviour { [Header("Subsystem Dependencies")] public VisionBridge vision; // visual perception public RescueGoalGenerator generator; // state estimation public PathGenerator architect; // trajectory planning public DroneCommentator commentator; // human-machine interface (HMI) [Header("Mission Config")] [TextArea] public string missionPrompt = "Scan scene. Identify structural hazards and potential survivor locations (tables, corners). Prioritize tight gaps."; // Trigger via UI Event public void StartMissionGeneration() { Debug.Log("--- MISSION START: INITIALIZING SEQUENCE ---"); // Step 1: Perception Update // asynchronous call to VLM for scene analysis vision.ScanScene(missionPrompt, (aiResponse) => { // callback received: measurement data acquired // Step 2: State Estimation // generate goalposts based on semantic parsing of VLM output generator.GenerateGoalsFromAI(aiResponse, Camera.main.transform); // Step 3: Path Planning // compute optimal trajectory through estimated waypoints architect.BuildRescuePath(generator.activeGoals); // Step 4: User Feedback // construct status report for the HMI string stats = $"{generator.activeGoals.Count} points of interest."; string status = generator.activeGoals.Count > 0 ? "optimal path found." : "convergence failed (fallback used)."; if(commentator != null) { // synthesizing voice response commentator.Announce("Mission Plan Generated", $"{stats} {status}"); } Debug.Log($"MISSION READY: {stats}. pipeline execution complete."); }); } } ``` ### scripts/GoalGenerator.cs ```c# // turns text from VLM into physical 3d object using UnityEngine; using System.Collections.Generic; public class RescueGoalGenerator : MonoBehaviour { [Header("Simulation Parameters")] public GameObject goalPrefab; // visualization marker (torus) public LayerMask splatLayer; // gaussian splat mesh collider layer // maintaining a list of active waypoints to manage scene clutter public List activeGoals = new List(); public void GenerateGoalsFromAI(string aiResponse, Transform droneTransform) { // reset state for the new measurement update ClearGoals(); Debug.Log($"RescueGoalGen: processing measurement: '{aiResponse}'"); // 1. Semantic Parsing (The "Measurement Model") // currently using a naive heuristic (keyword matching) instead of strict json parsing // TODO: implement robust json deserialization for production bool targetIdentified = false; // hypothesis: if 'table' or 'gap' is detected, high probability of survivor/hazard if (aiResponse.ToLower().Contains("table") || aiResponse.ToLower().Contains("gap")) { // estimating position: projecting 2.0m forward vector + slight negative Z bias // this assumes the POI is directly in the camera's FOV center Vector3 predictedState = droneTransform.position + droneTransform.forward * 2.0f + Vector3.down * 0.5f; SpawnGoal(predictedState, "Hazard_Gap"); targetIdentified = true; } // hypothesis: 'corner' or 'wall' implies structural bounds if (aiResponse.ToLower().Contains("corner") || aiResponse.ToLower().Contains("wall")) { // heuristic: offset to the right to simulate peripheral detection SpawnGoal(droneTransform.position + droneTransform.right * 1.5f, "Structure_Ref"); targetIdentified = true; } // 2. State Correction / Fallback // if the measurement update failed (VLM hallucinated or saw nothing), // initialize a default search pattern (covariance is high, so we search wide) if (!targetIdentified || activeGoals.Count == 0) { Debug.LogWarning("RescueGoalGen: measurement invalid. reverting to prior belief (default pattern)."); SpawnGoal(droneTransform.position + Vector3.forward * 3f, "Search_Area_Alpha"); SpawnGoal(droneTransform.position + Vector3.forward * 5f + Vector3.right * 2f, "Search_Area_Beta"); } } void SpawnGoal(Vector3 pos, string name) { // instantiating the visual marker at the estimated coordinates GameObject g = Instantiate(goalPrefab, pos, Quaternion.identity); g.name = name; activeGoals.Add(g.transform); } public void ClearGoals() { // cleaning up the scene graph foreach (var t in activeGoals) Destroy(t.gameObject); activeGoals.Clear(); } } ``` ### scripts/VisionBridge.cs ```c# // captures drone view and sends it to local Ollama instance on GX10 using UnityEngine; using UnityEngine.Networking; using System.Text; using System.Collections; using System; public class VisionBridge : MonoBehaviour { [Header("VLM Config")] // local inference endpoint. running on the GX10 via ollama/localai // ensure port 11434 is exposed and firewall isn't blocking public string localApiUrl = "http://localhost:11434/v1/chat/completions"; public string modelName = "llava"; // using llava/moondream for lower inference latency public Camera droneCamera; public void ScanScene(string prompt, Action callback) { // spin up the async routine to avoid blocking the main thread (rendering) StartCoroutine(ProcessScan(prompt, callback)); } IEnumerator ProcessScan(string prompt, Action callback) { // Measurement Acquisition // creating a temp render texture to grab the current frame buffer // standard 512x512 resolution to balance VLM context window vs. visual fidelity RenderTexture rt = new RenderTexture(512, 512, 24); droneCamera.targetTexture = rt; Texture2D tex = new Texture2D(rt.width, rt.height, TextureFormat.RGB24, false); // force render trigger droneCamera.Render(); RenderTexture.active = rt; // read pixels from GPU to CPU memory // expensive operation, optimized by reusing the rect tex.ReadPixels(new Rect(0, 0, rt.width, rt.height), 0, 0); tex.Apply(); // cleanup to prevent memory leaks in VRAM droneCamera.targetTexture = null; RenderTexture.active = null; Destroy(rt); // 2. Data Marshalling // encoding to base64 jpg. standard protocol for sending image tensors to llms via json // compression quality 50 is a heuristic: good enough for object detection, low bandwidth byte[] bytes = tex.EncodeToJPG(50); string base64Image = Convert.ToBase64String(bytes); Destroy(tex); // yeet the texture // 3. Construct Payload // constructing the JSON body for the POST request // mirroring the openai chat completion schema string json = $@"{{ ""model"": ""{modelName}"", ""messages"": [ {{ ""role"": ""user"", ""content"": ""{prompt} \n[IMG]{base64Image}[/IMG]"" }} ], ""stream"": false }}"; var request = new UnityWebRequest(localApiUrl, "POST"); byte[] bodyRaw = Encoding.UTF8.GetBytes(json); request.uploadHandler = new UploadHandlerRaw(bodyRaw); request.downloadHandler = new DownloadHandlerBuffer(); request.SetRequestHeader("Content-Type", "application/json"); // debug: sending packet // Debug.Log("VisionBridge: propagating state to VLM..."); yield return request.SendWebRequest(); if (request.result == UnityWebRequest.Result.Success) { // measurement update successful string response = request.downloadHandler.text; // passing raw json to callback for parsing callback(response); } else { // connection refused or timeout. // probable cause: ollama service not running or port mismatch Debug.LogError($"VisionBridge: inference failed. error: {request.error}"); } } } ``` ### scripts/VoiceService.cs ```c# // speaks the commentary out loud via local TTS using UnityEngine; using UnityEngine.Networking; using System.Text; using System.Collections; using System.Globalization; public class VoiceService : MonoBehaviour { [Header("TTS Endpoint")] public string ttsUrl = "http://10.32.83.219:8004/tts"; public int requestTimeout = 15; [Header("TTS Parameters")] [Range(0f, 2f)] public float temperature = 0.8f; [Range(0f, 2f)] public float exaggeration = 1.3f; [Range(0f, 2f)] public float cfgWeight = 0.5f; [Range(0.25f, 2f)] public float speedFactor = 1f; public int seed = 3000; public string language = "en"; public string voiceMode = "predefined"; public bool splitText = true; public int chunkSize = 240; public string outputFormat = "wav"; public string predefinedVoiceId = "Michael.wav"; [Header("connections")] public AudioSource voiceSource; public void Speak(string text){ if (string.IsNullOrEmpty(text)) return; StartCoroutine(PostTTS(text)); } IEnumerator PostTTS(string text){ // sanitize for json string safeText = EscapeJson(text); string safeLanguage = EscapeJson(language); string safeVoiceMode = EscapeJson(voiceMode); string safeOutputFormat = EscapeJson(outputFormat); string safeVoiceId = EscapeJson(predefinedVoiceId); string json = $@"{{ \"text\": \"{safeText}\", \"temperature\": {ToInvariant(temperature)}, \"exaggeration\": {ToInvariant(exaggeration)}, \"cfg_weight\": {ToInvariant(cfgWeight)}, \"speed_factor\": {ToInvariant(speedFactor)}, \"seed\": {seed}, \"language\": \"{safeLanguage}\", \"voice_mode\": \"{safeVoiceMode}\", \"split_text\": {(splitText ? "true" : "false")}, \"chunk_size\": {chunkSize}, \"output_format\": \"{safeOutputFormat}\", \"predefined_voice_id\": \"{safeVoiceId}\" }}"; var request = new UnityWebRequest(ttsUrl, "POST"); byte[] bodyRaw = Encoding.UTF8.GetBytes(json); request.uploadHandler = new UploadHandlerRaw(bodyRaw); request.downloadHandler = new DownloadHandlerAudioClip(ttsUrl, GetAudioType(outputFormat)); request.SetRequestHeader("Content-Type", "application/json"); request.timeout = requestTimeout; yield return request.SendWebRequest(); if (request.result == UnityWebRequest.Result.Success){ AudioClip clip = DownloadHandlerAudioClip.GetContent(request); if (clip != null && voiceSource != null){ voiceSource.clip = clip; voiceSource.Play(); } } else{ Debug.LogWarning($"TTS unavailable: {request.error} (is TTS running on {ttsUrl}?)"); } } static string EscapeJson(string input) { if (string.IsNullOrEmpty(input)) return ""; return input.Replace("\\", "\\\\") .Replace("\"", "\\\"") .Replace("\r", "\\r") .Replace("\n", "\\n"); } static string ToInvariant(float value) => value.ToString(CultureInfo.InvariantCulture); static AudioType GetAudioType(string format) { if (string.IsNullOrEmpty(format)) return AudioType.WAV; switch (format.Trim().ToLowerInvariant()) { case "mp3": case "mpeg": return AudioType.MPEG; case "ogg": case "ogg_vorbis": case "vorbis": return AudioType.OGGVORBIS; case "wav": default: return AudioType.WAV; } } } ``` ### scripts/AIService.cs ```c# using UnityEngine; using UnityEngine.Networking; using System.Text; using System.Collections; using System; public class AIService : MonoBehaviour { [Header("Conversation Model Settings")] public string apiUrl = "http://10.32.83.219:8000/v1/completions"; public string modelName = "openai/gpt-oss-20b"; public string healthUrl = ""; public int requestTimeout = 60; public int maxTokens = 256; [Range(0f, 2f)] public float temperature = 0.7f; [Range(0f, 1f)] public float topP = 1f; void Start() { StartCoroutine(CheckConnection()); } IEnumerator CheckConnection() { string pingUrl = !string.IsNullOrEmpty(healthUrl) ? healthUrl : (apiUrl.Contains("/v1/completions") ? apiUrl.Replace("/v1/completions", "/v1/models") : apiUrl); var ping = new UnityWebRequest(pingUrl, "GET"); ping.downloadHandler = new DownloadHandlerBuffer(); ping.timeout = 5; yield return ping.SendWebRequest(); if (ping.result == UnityWebRequest.Result.Success) Debug.Log($"SUCCESS: LLM connected. Model: {modelName}"); else Debug.LogError($"CRITICAL: Cannot reach LLM at {apiUrl}. Is it running?"); } public void SendPrompt(string systemPrompt, string userPrompt, Action callback){ StartCoroutine(PostRequest(systemPrompt, userPrompt, callback)); } IEnumerator PostRequest(string systemRole, string userMessage, Action callback){ // Sanitize inputs to prevent JSON breakage string safeSystem = EscapeJson(systemRole); string safeUser = EscapeJson(userMessage); string prompt = $"System: {safeSystem}\nUser: {safeUser}\nAssistant:"; string safePrompt = EscapeJson(prompt); string json = $@"{{ \"model\": \"{modelName}\", \"prompt\": \"{safePrompt}\", \"max_tokens\": {maxTokens}, \"temperature\": {temperature.ToString(System.Globalization.CultureInfo.InvariantCulture)}, \"top_p\": {topP.ToString(System.Globalization.CultureInfo.InvariantCulture)} }}"; var request = new UnityWebRequest(apiUrl, "POST"); byte[] bodyRaw = Encoding.UTF8.GetBytes(json); request.uploadHandler = new UploadHandlerRaw(bodyRaw); request.downloadHandler = new DownloadHandlerBuffer(); request.SetRequestHeader("Content-Type", "application/json"); request.timeout = requestTimeout; yield return request.SendWebRequest(); if (request.result == UnityWebRequest.Result.Success){ string responseText = request.downloadHandler.text; // Manual JSON parsing — OpenAI-compatible completions response format string content = ParseResponseText(responseText); if (content != null) callback(content); else Debug.LogError($"AIService: failed to parse response: {responseText}"); } else{ Debug.LogError($"AIService: {request.error}\nResponse: {request.downloadHandler.text}"); } } string ParseResponseText(string json) { // find the last "text" key in choices string key = "\"text\":"; int keyIndex = json.LastIndexOf(key); if (keyIndex == -1) return null; int start = keyIndex + key.Length; while (start < json.Length && (json[start] == ' ' || json[start] == '\n' || json[start] == '\r')) start++; if (start >= json.Length || json[start] != '"') return null; start++; // skip opening quote int end = start; while (end < json.Length){ if (json[end] == '"' && json[end - 1] != '\\') break; end++; } if (end >= json.Length) return null; string content = json.Substring(start, end - start); content = content.Replace("\\n", "\n").Replace("\\\"", "\""); return content; } static string EscapeJson(string input) { if (string.IsNullOrEmpty(input)) return ""; return input.Replace("\\", "\\\\") .Replace("\"", "\\\"") .Replace("\r", "\\r") .Replace("\n", "\\n"); } } ``` ### scripts/DroneCommentator.cs ```c# // contains the logic for generative commentary // monitors physics and gives live feedback using UnityEngine; using TMPro; using System.Collections.Generic; using System; public class DroneCommentator : MonoBehaviour { // connections public AIService aiService; public TextMeshProUGUI uiText; public DroneController droneCtrl; // need this for throttle check public Transform currentGoal; // need this for goal check public VoiceService voiceService; // for speaking comments out loud // settings [TextArea(3, 10)] public string personaPrompt = "You are a Gen Z flight commentator. Use slang like 'cooked', 'bet', 'no cap', and 'skill issue'. Keep it short."; // state variables private bool isTalking = false; private float idleTimer = 0f; private Rigidbody rb; private Vector3 lastAngularVelocity; public bool goalReached = false; // prevents goal spam public float jerkThreshold = 15f; private float smoothTurnTimer = 0f; public float smoothTurnMinDuration = 1.5f; // how long to hold the turn public List flightLog = new List(); // stores the full history of comments public void ResetGoalStatus() => goalReached = false; void Start() { // get physics ref rb = GetComponent(); } void Update() { float speed = rb.linearVelocity.magnitude; // check speeding if (speed > 15f && !isTalking){ TriggerCommentary("speeding", "nothing"); } // check if flying unstable if (rb.angularVelocity.magnitude > 5f && !isTalking){ TriggerCommentary("losing control", "gravity"); } // check if jerking turns Vector3 angularAcceleration = (rb.angularVelocity - lastAngularVelocity) / Time.deltaTime; if (angularAcceleration.magnitude > jerkThreshold && !isTalking){ TriggerCommentary("jerking turns", "joystick"); } // check idle if (speed < 0.1f) idleTimer += Time.deltaTime; else idleTimer = 0f; if (idleTimer > 10f && !isTalking){ TriggerCommentary("idle", "nothing"); idleTimer = 0f; } // check near miss if (Physics.Raycast(transform.position, transform.forward, out RaycastHit hit, 1.0f)){ if (speed > 5f && !isTalking){ TriggerCommentary("near miss", hit.collider.name); } } // check if stuck if (droneCtrl != null){ //hitting throttle but not moving if (droneCtrl.throttleInput > 0.8f && speed < 0.1f && !isTalking){ if (Physics.Raycast(transform.position, transform.forward, 1.0f)){ TriggerCommentary("stuck", "wall"); } } } // check if upside down if (Vector3.Dot(transform.up, Vector3.down) > 0.5f && !isTalking){ TriggerCommentary("upside down", "gravity"); } // check if we passed goal if (currentGoal != null){ float distToGoal = Vector3.Distance(transform.position, currentGoal.position); if (distToGoal < 2.0f && !goalReached && !isTalking){ goalReached = true; TriggerCommentary("goal reached", currentGoal.name); } } // check for smooth turn // if we are turning (angular velocity > 1) and not jerking (accel < threshold) if (rb.angularVelocity.magnitude > 1.0f && angularAcceleration.magnitude < (jerkThreshold * 0.5f)){ smoothTurnTimer += Time.deltaTime; } else{ smoothTurnTimer = 0f; // reset if we stop or jerk } // trigger if held long enough if (smoothTurnTimer > smoothTurnMinDuration && !isTalking){ TriggerCommentary("smooth turn", "the air"); smoothTurnTimer = 0f; // reset so it doesn't spam } // check for successful navigation thru difficult obstacle // check for narrow gap navigation bool leftHit = Physics.Raycast(transform.position, -transform.right, 1.5f); bool rightHit = Physics.Raycast(transform.position, transform.right, 1.5f); if (leftHit && rightHit && speed > 5f && !isTalking){ TriggerCommentary("tight gap navigation", "obstacles"); } lastAngularVelocity = rb.angularVelocity; // update spin for next frame } void OnCollisionEnter(Collision collision){ if (collision.relativeVelocity.magnitude > 2f && !isTalking){ TriggerCommentary("crash", collision.gameObject.name); } } public void TriggerCommentary(string eventType, string objectHit){ isTalking = true; uiText.text = "AI Thinking..."; float speedMph = rb.linearVelocity.magnitude * 2.237f; // custom instruction for high-skill moments string skillBonus = ""; if (eventType == "smooth turn" || eventType == "tight gap navigation" || eventType == "goal reached"){ skillBonus = " IMPORTANT: Start your response with 'chat is that rizz'."; } // build prompts string systemPrompt = personaPrompt + skillBonus; string userPrompt = $"I just caused a {eventType} event involving {objectHit} at {speedMph:F1} MPH. React."; aiService.SendPrompt(systemPrompt, userPrompt, (response) => { uiText.text = response; LogCommentary(eventType, response); // log the comment with a timestamp if (voiceService != null) voiceService.Speak(response); Invoke("ResetTalking", 5f); }); } public void LogCommentary(string eventType, string response){ string timestamp = DateTime.Now.ToString("HH:mm:ss"); string logEntry = $"[{timestamp}] {eventType.ToUpper()}: {response}"; // format line flightLog.Add(logEntry); // [truncated — 1183 more characters] ``` [14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]