# Project export: Soca This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Seeing through your ears - Devpost: https://devpost.com/software/soca - GitHub: https://github.com/kdrag0n/spatialdot-treehacks24/ - Demo: https://kdrag0n.dev/thv - Team: 4 GitHub contributor(s) — Danny Lin (16 commits), Huxley Marvit (10 commits), Exr0n (4 commits), Ellen Xu (1 commits) ## Devpost submission (written by the team) ### Inspiration We came across this idea by thinking about underutilized modalities. Albert was on the long bus ride to TreeHacks Friday morning, scrolling through a list of sensors in various Apple devices. Spatial audio stuck out to us as an underexplored mixed-reality interface that could encode a lot of information and create intuitive natural experiences, especially for those who are missing other senses. Vision loss is a really big problem (that’s only getting worse, from first hand experience). Soca is unobtrusive and uses off-the-shelf hardware that we all already happened to have. We’re also really excited to see what other people do with the other sensors hiding in plain sight. Once we began to think about this as a mixed reality interface design problem, we realized there’s so much more we can do. From needfinding, we know that finding stuff is really hard when you can’t see. Instead of groping around, imagine asking “where’s my blue T-shirt” or “where’s the sign for gate C3”? ### What it does Soca transforms visual surroundings into auditory landscapes, to help the visually impaired see the world. We use LiDAR-camera data to find the real-world location of obstacles and points of interest, then point them out with spatial audio. This way, you can navigate around obstacles and towards goals just by listening to the virtual sounds around you. See from your ears, through your phone camera, to the visual world. ### How we built it We use the built-in Apple LiDAR and camera streams to detect obstacles and task objectives, then create localized virtual speakers to orient the user. To perform object detection, we pre-process cluster the point cloud and fine-tuned SegmentEverything. The objects and depth are then converted into xyz world coordinates. Once objects are found, we embed audio cues using spatial localization -- you can “hear” where the objects are located (imagine 3D stereo audio). To construct the scene geometry, we use ARKit to capture a depth map from LiDAR scanning and create the virtual reality environment. We use the AVFoundation audio engine to play spatial audio, with object and point-of-interest detection using LiDAR data clustering and SegmentEverything. We experimented heavily with many parts of the pipeline—fields of view, clustering vs segmentation algorithms, and vision models from classic ssd image recognition to gpt4v (which didn’t work, amazingly enough), and various spatial audio transforms (HRTF, equal power pairing), and sound profiles, to optimize localization. Other than the on-device component, we use Bun for the debug and inference server, for real time analysis and debugging. ### Challenges we ran into Transforming from the depth coordinates, to VR world coordinates, and finally to speaker orientation and Airpods was difficult—we learned loads about camera intrinsics, transform matrices, and the hazard of non-commutative operations :) In addition, spatial audio was also challenging to debug—sometimes “whether it’s working” was so subjective. Finally, we got some good practice getting models running (reinstalled conda twice!) and iterating on prompts as quickly as possible. We were amazed that SegmentAnything ended up being better than GPT4-vision, which we (and all the mentors we talked to) had expected to be the one-shot solution. Accomplishments we're proud of Actually being able to walk around with our eyes closed. We were able to validate that this approach of spatial audio works for navigation, and works well. It was also extremely exciting to explore the potential of this untapped data modality for enhancing how we interface with the world. ### What we learned Intuitive interfaces are hard to build—they can be very powerful if done correctly, but they tap into a low-level part of the brain so if you get one thing slightly wrong from the real physical model, then things feel off. Eg. pure tones and beeps don’t work, but we’re really great at detecting voices and music. Even if a project seems easy, you must get to the iterative debugging stage as quickly as possible. We were very confident in the beginning but ended up not having an MVP until 5am today. Coordinates/pose transforms are hard—one must think carefully! Especially because composing rotations is non-commutative, and we are dealing with { camera, world, AR scene, and listener } coordinate spaces. ### What's next We are iterating with user feedback and implementing additional features for spatial audio navigation. Expanding to more sophisticated capabilities would enable things like: Exploring a city through real-time navigation. Imagine “Take me to the nearest coffee shop” and following non-intrusive audio cues and ambient signals for directions, instead of having to stare at your phone Enhanced object recognition capabilities, such as tracking objects through time + OCR. This will detect and dictate signs, give context for locations and surroundings, and help identify items, enabling the visually impaired to fully navigate the signposted world. Integration with an LLM agent for specific queries and searching in a scene -- “find me the blue T-shirt” and reasoning about “we need to collect more information by going here, or interacting with this object” Plane detection for floors and walls (ARkit), overhead detection for branches / more direct path detection. Denoting sidewalk/floor edges with noise walls, special sound effects for things like stairs, streets, escalators, and elevators ## README (from the GitHub repository) # Soca *Seeing through your ears* ## Inspiration We came across this idea by thinking about underutilized modalities. Albert was on the long bus ride to TreeHacks Friday morning, scrolling through a list of sensors in various Apple devices. Spatial audio stuck out to us as an underexplored mixed-reality interface that could encode a lot of information and create intuitive natural experiences, especially for those who are missing other senses. Vision loss is a really big problem (that’s only getting worse, from first hand experience). Soca is unobtrusive and uses off-the-shelf hardware that we all already happened to have. We’re also really excited to see what other people do with the other sensors hiding in plain sight. Once we began to think about this as a mixed reality interface design problem, we realized there’s so much more we can do. From needfinding, we know that finding stuff is really hard when you can’t see. Instead of groping around, imagine asking “where’s my blue T-shirt” or “where’s the sign for gate C3”? ## What it does Soca transforms visual surroundings into auditory landscapes, to help the visually impaired see the world. We use LiDAR-camera data to find the real-world location of obstacles and points of interest, then point them out with spatial audio. This way, you can navigate around obstacles and towards goals just by listening to the virtual sounds around you. See from your ears, through your phone camera, to the visual world. ## How we built it We use the built-in Apple LiDAR and camera streams to detect obstacles and task objectives, then create localized virtual speakers to orient the user. To perform object detection, we pre-process cluster the point cloud and fine-tuned SegmentEverything. The objects and depth are then converted into xyz world coordinates. Once objects are found, we embed audio cues using spatial localization -- you can “hear” where the objects are located (imagine 3D stereo audio). To construct the scene geometry, we use ARKit to capture a depth map from LiDAR scanning and create the virtual reality environment. We use the AVFoundation audio engine to play spatial audio, with object and point-of-interest detection using LiDAR data clustering and SegmentEverything. We experimented heavily with many parts of the pipeline—fields of view, clustering vs segmentation algorithms, and vision models from classic ssd image recognition to gpt4v (which didn’t work, amazingly enough), and various spatial audio transforms (HRTF, equal power pairing), and sound profiles, to optimize localization. Other than the on-device component, we use Bun for the debug and inference server, for real time analysis and debugging. ## Challenges we ran into Transforming from the depth coordinates, to VR world coordinates, and finally to speaker orientation and Airpods was difficult—we learned loads about camera intrinsics, transform matrices, and the hazard of non-commutative operations :) In addition, spatial audio was also challenging to debug—sometimes “whether it’s working” was so subjective. Finally, we got some good practice getting models running (reinstalled conda twice!) and iterating on prompts as quickly as possible. We were amazed that SegmentAnything ended up being better than GPT4-vision, which we (and all the mentors we talked to) had expected to be the one-shot solution. ## Accomplishments we're proud of Actually being able to walk around with our eyes closed. We were able to validate that this approach of spatial audio works for navigation, and works well. It was also extremely exciting to explore the potential of this untapped data modality for enhancing how we interface with the world. ## What we learned Intuitive interfaces are hard to build—they can be very powerful if done correctly, but they tap into a low-level part of the brain so if you get one thing slightly wrong from the real physical model, then things feel off. Eg. pure tones and beeps don’t work, but we’re really great at detecting voices and music. Even if a project seems easy, you must get to the iterative debugging stage as quickly as possible. We were very confident in the beginning but ended up not having an MVP until 5am today. Coordinates/pose transforms are hard—one must think carefully! Especially because composing rotations is non-commutative, and we are dealing with { camera, world, AR scene, and listener } coordinate spaces. ## What's next for Soca We are iterating with user feedback and implementing additional features for spatial audio navigation. Expanding to more sophisticated capabilities would enable things like: - Exploring a city through real-time navigation. Imagine “Take me to the nearest coffee shop” and following non-intrusive audio cues and ambient signals for directions, instead of having to stare at your phone - Enhanced object recognition capabilities, such as tracking objects through time + OCR. This will detect and dictate signs, give context for locations and surroundings, and help identify items, enabling the visually impaired to fully navigate the signposted world. - Integration with an LLM agent for specific queries and searching in a scene -- “find me the blue T-shirt” and reasoning about “we need to collect more information by going here, or interacting with this object” - Plane detection for floors and walls (ARkit), overhead detection for branches / more direct path detection. Denoting sidewalk/floor edges with noise walls, special sound effects for things like stairs, streets, escalators, and elevators ## Detected evidence (automated analysis) Indexed codebase: 23 recognized source files, 73 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code ## Codebase structure (from repository index) ### Files (49 of 49) ``` .DS_Store agents/.gitignore agents/fetch/.gitignore agents/fetch/gpt4-object-find.py agents/gpt4v.py inference_server/lang_segment_anything.py music1.m4a README.md SpatialDot.xcodeproj/project.pbxproj SpatialDot.xcodeproj/project.xcworkspace/contents.xcworkspacedata SpatialDot.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist SpatialDot.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved SpatialDot.xcodeproj/project.xcworkspace/xcuserdata/huxley.xcuserdatad/UserInterfaceState.xcuserstate SpatialDot.xcodeproj/xcshareddata/xcschemes/SpatialDot.xcscheme SpatialDot.xcodeproj/xcuserdata/dragon.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist SpatialDot.xcodeproj/xcuserdata/dragon.xcuserdatad/xcschemes/xcschememanagement.plist SpatialDot.xcodeproj/xcuserdata/huxley.xcuserdatad/xcschemes/xcschememanagement.plist SpatialDot/AR.swift SpatialDot/Assets.xcassets/AccentColor.colorset/Contents.json SpatialDot/Assets.xcassets/AppIcon.appiconset/Contents.json SpatialDot/Assets.xcassets/Contents.json SpatialDot/ContentView.swift SpatialDot/Preview Content/Preview Assets.xcassets/Contents.json SpatialDot/SpatialDot.entitlements SpatialDot/SpatialDotApp.swift turn.m4a web_app/server/main.py web_app/server/send_dummy_data.py web_app/viz/.gitignore web_app/viz/bun.lockb web_app/viz/package.json web_app/viz/postcss.config.js web_app/viz/public/index.html web_app/viz/public/manifest.json web_app/viz/public/robots.txt web_app/viz/README.md web_app/viz/src/App.css web_app/viz/src/App.js web_app/viz/src/App.test.js web_app/viz/src/assets/pointcloudtest_1.json web_app/viz/src/components/DataReceiver.jsx web_app/viz/src/components/delaunator.js web_app/viz/src/components/PointCloudRenderer.jsx web_app/viz/src/index.css web_app/viz/src/index.js web_app/viz/src/reportWebVitals.js web_app/viz/src/scratch web_app/viz/src/setupTests.js web_app/viz/tailwind.config.js ``` ### Dependencies - web_app/viz/package.json: @jrsmiffy/delaunator@^1.4.5, @react-three/drei@^9.97.6, @react-three/fiber@^8.15.16, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @types/three@^0.161.2, autoprefixer@^10.4.17, delaunator@^5.0.1, postcss@^8.4.35, react@^18.2.0, react-dom@^18.2.0, react-scripts@5.0.1, tailwindcss@^3.4.1, three@^0.161.0, web-vitals@^2.1.4 ### Recent commits (newest first) - add readme from devpost - newurl - music1 - fix merge - merge proj - Merge remote-tracking branch 'origin/swift_sockets' - thres, hrtfhq - Merge branch 'swift_sockets' of github.com:kdrag0n/spatialdot-treehacks24 into swift_sockets - mergin time - Merge branch 'main-staging' into swift_sockets - Create lang_segment_anything.py - nav mode, rel rotation - rotate points with initial device pos - Merge branch 'main' of github.com:kdrag0n/spatialdot-treehacks24 - ewma, rotation - asdaf - init sockets - update - wip spatial audio + hook - Merge branch 'main' of github.com:kdrag0n/spatialdot-treehacks24 ## Key source files (fetched from GitHub, selected and truncated for size) ### web_app/viz/package.json ``` { "name": "viz", "version": "0.1.0", "private": true, "dependencies": { "@jrsmiffy/delaunator": "^1.4.5", "@react-three/drei": "^9.97.6", "@react-three/fiber": "^8.15.16", "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "@types/three": "^0.161.2", "autoprefixer": "^10.4.17", "delaunator": "^5.0.1", "postcss": "^8.4.35", "react": "^18.2.0", "react-dom": "^18.2.0", "react-scripts": "5.0.1", "tailwindcss": "^3.4.1", "three": "^0.161.0", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### web_app/server/main.py ```python from fastapi import FastAPI, WebSocket, BackgroundTasks, WebSocketDisconnect import asyncio import numpy as np import json app = FastAPI() receivers = [] @app.websocket("/ws/get_data") async def ws_send_data(websocket: WebSocket): await websocket.accept() receivers.append(websocket) try: while True: # Keep the connection alive # dummy_data = np.random.rand(10).tolist() # await websocket.send_json({"float_array": dummy_data}) # await asyncio.sleep(0.1) # Send data every second, adjust the sleep time as needed await websocket.receive_text() except Exception as e: receivers.remove(websocket) @app.websocket("/ws/send_data") async def ws_receive_data(websocket: WebSocket): """ receive an array of floats from the client, and send it to all the connected clients """ while True: await websocket.accept() try: while True: # data = await websocket.receive_json() data = await websocket.receive_bytes() # decode the bytes to a list of floats data = {"float_array": (np.frombuffer(data, dtype=np.float16) * 100).tolist()} print(data['float_array'][:3]) for receiver in receivers: await receiver.send_json(data) except WebSocketDisconnect as e: print(e, 1) return pass # except (RuntimeError, ConnectionError) as e: except Exception as e: print(e) pass ``` ### web_app/viz/src/App.js ```javascript import logo from './logo.svg'; import { extend, Canvas, useFrame, useLoader, useThree } from '@react-three/fiber' import './App.css'; import PointCloudRenderer from './components/PointCloudRenderer'; import WebSocketReceiver from './components/DataReceiver'; function App() { return (
{/**/}
); } export default App; ``` ### web_app/viz/src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### SpatialDot/SpatialDotApp.swift ```swift // // SpatialDotApp.swift // SpatialDot // // Created by Danny Lin on 2/17/24. // import SwiftUI @main struct SpatialDotApp: App { var body: some Scene { WindowGroup { ContentView() } } } ``` ### inference_server/lang_segment_anything.py ```python from PIL import Image from lang_sam import LangSAM import numpy as np import cv2 import time model = LangSAM() # image_pil = Image.open("./desk.jpeg").convert("RGB") #image_pil = Image.open("treehacks-2024-filedrop.jepg").convert("RGB") # text_prompt = "laptop" # NEXT STEP: make it a serve from fastapi import FastAPI from pydantic import BaseModel app = FastAPI() class ImageRequest(BaseModel): image: str @app.post("/find/{request}") def read_root(request: str, image_req: ImageRequest): img = Image.open(BytesIO(base64.b64decode(image_req.image))).convert("RGB") text_prompt = request start_time = time.time() masks, boxes, phrases, logits = model.predict(image_pil, text_prompt) print(time.time() - start_time) box = [int(b) for b in boxes[0]] img = img_np[int(box[1]):int(box[3]), int(box[0]):int(box[2])] return { 'x': (box[1]+box[3])/2, 'y': (box[0]+box[2])/2 } # #print(masks) # print(boxes) # # box = boxes[0] # img_np = np.array(image_pil) # img = img_np[int(box[1]):int(box[3]), int(box[0]):int(box[2])] # # Image.fromarray(img).save("filtered_image.jpg") # # #print(masks.sum) # # # #cv2.imshow("masked", img) # #cv2.waitKey(0) ``` ### agents/gpt4v.py ```python from openai import OpenAI import cv2 import base64 IMAGE_SIZE = 512 GRID_SIZE = 32 COLOR = (255, 255, 0) FONT_SCALE = 1.0 SAMPLES_PATH = "./images/examples/{}.jpeg" # todo: DSPy? def text(text: str): return { 'type': 'text', 'text': text } def image(norm_img): b64_image = base64.b64encode(cv2.imencode('.jpg', img)[1]).decode() return { 'type': 'image_url', 'image_url': { 'url': f"data:image/jpeg;base64,{b64_image}"}} def make_sys_prompt(): sample_img = annotate_image(cv2.imread(SAMPLES_PATH.format('desk'))) return [ {'role': 'system', 'content': [ text(""" Help the user find something in the image. If it cannot be found, suggest some possible places to go to look for it. The image is partitioned into cells. You must output the label of the cell that contains the object. Follow these steps: 1. Where in the image is the object? 2. What is the label of one of the cells containing the object? You must follow these examples: """), text("EXAMPLE INPUT:\nWhere is my water bottle?"), image(sample_img), text("EXAMPLE OUTPUT:\n1. The object is in top center of the image.\n2. E8"), text("EXAMPLE INPUT:\nWhere is my pencil?"), image(sample_img), text("EXAMPLE OUTPUT:\nThe object is in the center left of the image.\n2. J1") ] } ] def annotate_image(img): # normalize the image img = cv2.resize(img, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_AREA) for i in range(IMAGE_SIZE//GRID_SIZE): cv2.line(img, (0, i*GRID_SIZE), (IMAGE_SIZE, i*GRID_SIZE), COLOR) cv2.line(img, (i*GRID_SIZE, 0), (i*GRID_SIZE, IMAGE_SIZE), COLOR) # cv2.putText(img, f"{i}", (int(FONT_SCALE*13), (i+1)*GRID_SIZE - 2), cv2.FONT_HERSHEY_PLAIN, 1.0, COLOR) # cv2.putText(img, f"{chr(65 + j)}", (j*GRID_SIZE+2, GRID_SIZE-2), cv2.FONT_HERSHEY_PLAIN, 1.0, COLOR) for j in range(IMAGE_SIZE//GRID_SIZE): cv2.putText(img, f"{chr(65 + j)}", (i*GRID_SIZE+1, (j+1)*GRID_SIZE-2), cv2.FONT_HERSHEY_PLAIN, 1.0, COLOR) cv2.putText(img, f"{i}", (i * GRID_SIZE + int(FONT_SCALE*11), (j+1)*GRID_SIZE - 2), cv2.FONT_HERSHEY_PLAIN, 1.0, COLOR) cv2.imshow('annotated', img) cv2.waitKey(0) return img def make_user_prompt(query: str, img): img = annotate_image(img) # make messages return [ { 'role': 'user', 'content': [ text(query), image(img) ] } ] IMG_FILE = "./images/IMG_2867 Large.jpeg"; QUESTION = "where is the door handle" # IMG_FILE = "./images/IMG_2867 Large.jpeg"; QUESTION = "where is my blue jacket" # IMG_FILE = "./images/examples/desk.jpeg"; QUESTION = "Where is my water bottle" img = cv2.imread(IMG_FILE) sys_msg = make_sys_prompt() user_msg = make_user_prompt(QUESTION, img) client = OpenAI() response = client.chat.completions.create( model="gpt-4-vision-preview", messages=[ *sys_msg, *user_msg ], max_tokens=30, temperature=0, n=1 ) print(response.choices[0]) ``` ### SpatialDot/ContentView.swift ```swift // // ContentView.swift // SpatialDot // // Created by Danny Lin on 2/17/24. // import SwiftUI import VideoToolbox import ARKit import RealityKit import SceneKit class SceneDelegate: NSObject, ARSCNViewDelegate { var audioSource = SCNAudioSource(fileNamed: "pinknoise.wav")! override init() { super.init() audioSource.loops = true audioSource.load() } // add red dot with sound for every anchor //func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? { // print("new node:\(anchor.transform)") // create node let node = SCNNode() node.simdTransform = anchor.transform let sphere = SCNSphere(radius: 0.01) sphere.firstMaterial?.diffuse.contents = UIColor.red let sphereNode = SCNNode(geometry: sphere) node.addChildNode(sphereNode) // let audioPlayer = SCNAudioPlayer(source: audioSource) // sphereNode.addAudioPlayer(audioPlayer) return node } // update red dot position func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { // /Users/dragon/code/hackathon/spatialdot-treehacks24/SpatialDot/ContentView.swift:36:60 Value of type 'simd_float4x4' has no member 'position' //node.childNodes.first?.position = anchor.transform.position node.simdTransform = anchor.transform } // remove red dot func renderer(_ renderer: SCNSceneRenderer, didRemove node: SCNNode, for anchor: ARAnchor) { node.removeFromParentNode() } } struct ARViewRepresentable: UIViewRepresentable { let ar: ARClient @State private var delegate = SceneDelegate() func makeUIView(context: Context) -> ARSCNView { let arView = ar.view arView.delegate = delegate arView.audioEnvironmentNode.distanceAttenuationParameters.distanceAttenuationModel = .exponential arView.audioEnvironmentNode.distanceAttenuationParameters.rolloffFactor = 5 return arView } func updateUIView(_ uiView: ARSCNView, context: Context) { } } struct ScaledBezier: Shape { let bezierPath: CGPath func path(in rect: CGRect) -> Path { let path = Path(bezierPath) // Figure out how much bigger we need to make our path in order for it to fill the available space without clipping. let multiplier = min(rect.width, rect.height) // Create an affine transform that uses the multiplier for both dimensions equally. let transform = CGAffineTransform(scaleX: multiplier, y: multiplier) // Apply that scale and send back the result. return path.applying(transform) } } struct ContentView: View { @StateObject private var ar = ARClient() var body: some View { ScrollView { VStack { Picker("Mode", selection: $ar.mode) { Text("Point") .tag(Mode.point) Text("NAV") .tag(Mode.navigation) } .pickerStyle(.segmented) if let depthBuffer = ar.depthBuffer { if let img = UIImage(pixelBuffer: depthBuffer) { ZStack { Image(uiImage: img) .frame(width: 384, height: 512) if let path = ar.contoursPath { ScaledBezier(bezierPath: path) .stroke(.red, lineWidth: 2) .scaleEffect(x: -1, y: 1) .rotationEffect(.degrees(180)) .frame(width: 384, height: 512) } } .rotationEffect(.degrees(90)) .scaleEffect(2) } } ARViewRepresentable(ar: ar) .frame(width: 500, height: 500) Button("Dump") { print(ar.lastPoint) } } } .onAppear { print("running") UIApplication.shared.isIdleTimerDisabled = true } .padding() } } extension UIImage { public convenience init?(pixelBuffer: CVPixelBuffer) { var cgImage: CGImage? VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &cgImage) guard let cgImage else { return nil } self.init(cgImage: cgImage) } } ``` ### SpatialDot/AR.swift ```swift // // AR.swift // SpatialDot // // Created by Danny Lin on 2/17/24. // import Foundation import SwiftUI import ARKit import PHASE import Starscream import CoreMotion private let depthWidth = 256 private let depthHeight = 192 private let depthDownsample = 4 private let depthDW = depthWidth / depthDownsample private let depthDH = depthHeight / depthDownsample private let nSounds = 1 class EwmaF32 { private var value: Float private let weight: Float init(initial: Float, weight: Float) { self.value = initial self.weight = weight } func update(_ sample: Float) -> Float { value = value*weight + sample*(1.0-weight) return value } } private let ewmaWeight: Float = 0.2 enum Mode { case point case navigation } class ARClient: NSObject, ObservableObject, ARSessionDelegate, URLSessionDelegate, WebSocketDelegate { let view = ARSCNView(frame: .zero) let session: ARSession private var pointCloud = [simd_float3]() private var pointCloudOld = [Float]() @Published var depthBuffer: CVPixelBuffer? = nil @Published var contoursPath: CGPath? = nil private var oldAnchors = [ARAnchor]() @Published var mode = Mode.point let engine = AVAudioEngine() var players = [AVAudioPlayerNode]() let env = AVAudioEnvironmentNode() private let hpMotion = CMHeadphoneMotionManager() private let deviceMotion = CMMotionManager() // var webSocket : URLSessionWebSocketTask? var webSocket: WebSocket let startTime = DispatchTime.now() var lastPoint: (Float, Float, Float) = (0,0,0) private var initialHpAttitude: CMAttitude? = nil private var initialPhoneAttitude: CMAttitude? = nil private var phoneAttitudeDiff: CMAttitude? = nil private let ewmaX = EwmaF32(initial: 0, weight: ewmaWeight) private let ewmaY = EwmaF32(initial: 0, weight: ewmaWeight) private let ewmaZ = EwmaF32(initial: 0, weight: ewmaWeight) override init() { var request = URLRequest(url: URL(string: "ws://192.168.18.240:8000/ws/send_data")!) request.timeoutInterval = 500 webSocket = WebSocket(request: request) print(webSocket) webSocket.connect() session = view.session super.init() session.delegate = self start() print("forward = \(PHASEObject.forward)") print("right = \(PHASEObject.right)") print("up = \(PHASEObject.up)") // env.distanceAttenuationParameters.distanceAttenuationModel = .exponential print("model=\(env.distanceAttenuationParameters.distanceAttenuationModel)") print("referenceDistance=\(env.distanceAttenuationParameters.referenceDistance)") print("referenceDistance=\(env.distanceAttenuationParameters.referenceDistance)") env.distanceAttenuationParameters.referenceDistance = 1 env.renderingAlgorithm = .HRTFHQ engine.attach(env) // load wav let url = Bundle.main.url(forResource: "music1trim", withExtension: "wav")! let audioFile = try! AVAudioFile(forReading: url) let audioBuffer = AVAudioPCMBuffer(pcmFormat: audioFile.processingFormat, frameCapacity: AVAudioFrameCount(audioFile.length))! try! audioFile.read(into: audioBuffer) print("begin inits") players.reserveCapacity(nSounds) for i in 0..