# Project export: Ranger This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Audio is a luxury we take for granted, that deaf folks don't get. Ranger is a wearable AR solution aiming to fix that, visualizing audio around them, and bringing them into the world of *sound*. - Devpost: https://devpost.com/software/ranger-6jkv5s - GitHub: https://github.com/YuanSamuel/Treehacks2025/ - Video: https://www.youtube.com/embed/dUlzwdfdT2U?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (NVIDIA: Best Accelerated Compute (5090s per team member - 1 GPU JHH signed [1st] & 5080s per team member [2nd])) - Team: 4 GitHub contributor(s) — PhoenixPhighter (36 commits), YuanSamuel (34 commits), TylerKerch (16 commits), Charan Sriram (9 commits) ## Devpost submission (written by the team) ### Overview 💭 ### Inspiration Sometimes helpful ideas come from the silliest places. One day, as our team was playing Fortnite instead of doing our homework, we realized how helpful it was to have a "sound ring" in game. Noises like enemy footsteps or gunshots are displayed with a visual indicator to show what direction they came from. Luckily, we aren't trapped in Moisty Mire shotgunning ChugJugs, but we did realize people hard of hearing could benefit from this in real life. Both innocuous situations (dropping your wallet in the street and a parent calling you down to dinner) and intense situations (scary noises that would make anyone want to pick up the pace a bit walking home at night, an ambulance screaming by) are much more difficult to navigate for those who can't process auditory input. Visualizing sound helps you enter and understand conversations easily, avoid dangerous situations, and generally be aware of important goings-on in your surroundings. We wanted to build portable, wearable technology that would bring our vision from our colorful video game screen to the real world. We see our product as a proof-of-concept for something that could truly - cheaply, portably, fashionably - transform the lives of deaf people. 🖥 ### What it does Ranger is an edge-based AR solution for audio visualization that uses a Meta Quest frontend to indicate to the wearer where sounds are coming from. The user wears a hat, which has a microphone array to capture omnidirectional audio and send it to our processing unit, a Jetson Nano. Ranger classifies all different sorts of real-world sounds using a classifier model and displays them as icons on a circular grid, placing markers according to distance and direction. For speech, we live-transcribe conversations using Whisper which allows those hard of hearing to immediately parse what's happening, even if speech comes from behind them. The sound visualization does not interfere with your real-world view, only enhancing the information already available. It's a real-life HUD, enriching the wearer's experience and using edge computing to bring them into the wonderful world of sound. 🛠 ### How we built it Hardware: 1 Meta Quest 3 1 Jetson Orin Nano Mics (ReSpeaker 4-Mic Circular Microphone Array, Boya Bluetooth TX/RX) (Most importantly) A giant cowboy hat Software: Python Unity ML Models (Yamnet, Whisper) Ranger runs completely on the edge! All heavy computation is done on a Jetson Orin Nano; all communication is done with direct wired USB-C connections. For a TL;DR: using the Jetson Orin Nano, we developed an audio processing software that takes in a 4-channel microphone input, triangulates the direction, amplitude, and type of the loudest sound occurring at any time step, transcribes any detected speech, and sends all this information to the Meta Quest 3 using network-over-USB. For those interested in a more in-depth overview: Real-time Voice Transcription We use Whisper Mini running on the Jetson Nano for real-time voice transcription. To accomplish real-time voice transcription, we capture the last 10 seconds of a user’s audio and process it immediately. Although the Jetson Nano has the ability to run larger Whisper versions (up to the recently released turbo one with 800M parameters), our priority was reducing latency. Sound triangulation To determine where sound comes from relative to the user, we use a ReSpeaker microphone array. We set it up to triangulate the audio channels to pinpoint where audio comes from (an example of this technique below). This approach gave us a vector with angle and volume, which allows us to position our classified sounds as relatively positioned icons in our 3D scene. Sound classification Sound classification is done using a convolutional neural network (CNN). The model runs on the Jetson and classifies audio in a probability distribution of up to 500 candidate labels. Using this class along with the latest source of noise from the sound triangulation, we’re able to accurately pinpoint what a noise is and where it comes from relative to the user! 🛑 ### Challenges we ran into This was our first time doing a hardware-based project, and our inexperience showed up immediately. We found ourselves sifting through mountains of cables, walking back and forth to the hardware booth every 20 minutes, and flipping between "ITSSOOVER" and "WERESOBACK" faster than the GPU fan on our Jetson Nano. We all loved Operating Systems, Concurrency, and Computer Architecture in school, but building a project completely from scratch, with very little electric or audio engineering knowledge, through largely un-trekked territory was an uphill battle. We had several significant challenges: 1) Parsing raw input data intelligently We planned this out without a pretty cursory understanding of auditory science, so we had to spend a lot of time understanding hardware synchronization concerns, channel mixing, and in-built audio driver configurations. Spending a lot of time thinking and diagramming though, rather than just spurting out poorly written code, was incredibly helpful later during integration. 2) Interfacing between backend and frontend We originally wanted to use bluetooth to communicate between the Jetson Nano and Meta Quest but ran into a ton of issues with trying to get low fidelity bluetooth communication schemes working. After a lot of tinkering, we decided to connect them together with a USB-C cable and used Android Debug Bridge to treat the wire as a network connection through a server socket. 3) Developing a fully on-the-edge system The Jetson Nano relies on a DC power supply, which we tried to get around with a USB-A adapter and a power bank. Our lack of electric engineering know-how showed here, though, as we didn’t realize the USB-A adapter was inherently capping our voltage. We want to give a huge thank you to Mr. Chitoku Yato at Nvidia for saving us on this with a custom USB-C to DC cord. 🏆 ### Accomplishments we're proud of The thing we’re all the most proud of is that we actually built what we set out to build! For our first hardware hack, with three different, distinct devices, across different frameworks, operating systems, and modalities of energy, this was an incredible feat. In the trenches of every challenge - Unity running slower than molasses on our rundown Intel Macs, bugs with multithreaded audio device access, bizarre audio sampling configurations provoking questions that not a soul on StackOverflow seemed to ask - we pushed past it and got things working. Each of us had a different moment we started jumping for joy: “I started going crazy when we first saw the visualization of the DOA (direction of audio)“ - Tyler “When I got to see a radar - like white dots on the circle - and the dots started to move when I did, I almost teared up“ - Samuel “After I spent 4 hours straight on our second Jetson Mini getting the Whisper model working with Cuda“ - Charan “I will never again feel happiness like I did when I saw the Android Debug logs on the Meta Quest print the first packet we sent“ - Sarvesh 🧠 ### What we learned Hardware is called hardware because it’s hard and you can wear it. Getting through our first hardware hack gave us a lot of confidence both for building on this idea and pursuing new ones. Hardware: Nuances of DC power conversion and portability Mechanics of audio input processing Remote usage and sharing of graphic/audio drivers Software: Supporting machine learning for edge devices Cabled network communication Unity scripting and scene visualization ✏️ ### What's next The Meta Quest is the best affordable AR wearable right now, but eventually, we'd want the lightest-weight solution so people would be happy to use Ranger for long periods. We fiddled around with some AR glasses, but many smaller companies focus on treating the glasses as an external monitor, and the Meta RayBans do not have a display. This year, though, the new Meta RayBans will have a visual display, so we could easily swap our (lovably) bulky Quest for a sleek, non-invasive pair of shades. Our team has been chewing on this idea for a long time, and we want to develop this beyond our 36-hour sprint here at TreeHacks. During our development, we thought of a million insanely cool stretch goals to upgrade our current version, and each could be a project on its own. Special audio software that can perform Single Source Separation would let us transcribe multiple voices at once. A more advanced beam-forming location and tracking algorithm would let us intelligently classify objects over time. Porting over multilingual Voice Language Models to our brave little Jetson could expand this project globally, unlocking a new world of interaction for deaf people. This idea has a remarkable depth that we only scratched the surface of, and our intention in the future is to dive deep. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 401 recognized source files, 3282 KB. - C (language) — detected in the code - C# (language) — detected in the code - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 844) ``` .gitignore python-backend/combinedquest.py python-backend/combinedquest2.py python-backend/combinedquest3.py python-backend/realtime-whisper.py python-backend/sendtoquest.py python-backend/tuning.py python-backend/usb_4_mic_array/.gitignore python-backend/usb_4_mic_array/.gitmodules python-backend/usb_4_mic_array/dfu_windows.py python-backend/usb_4_mic_array/dfu.py python-backend/usb_4_mic_array/LICENSE python-backend/usb_4_mic_array/odas_web/.gitignore python-backend/usb_4_mic_array/odas_web/audio-recorder.js python-backend/usb_4_mic_array/odas_web/configure.js python-backend/usb_4_mic_array/odas_web/LICENSE python-backend/usb_4_mic_array/odas_web/main.js python-backend/usb_4_mic_array/odas_web/odas.js python-backend/usb_4_mic_array/odas_web/package.json python-backend/usb_4_mic_array/odas_web/postfiltered.raw python-backend/usb_4_mic_array/odas_web/README.md python-backend/usb_4_mic_array/odas_web/record.js python-backend/usb_4_mic_array/odas_web/recordings.js python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap_slider.css python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap-theme.css python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap-theme.css.map python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap-theme.min.css python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap-theme.min.css.map python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap-toggle.min.css python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap.css python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap.css.map python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap.min.css python-backend/usb_4_mic_array/odas_web/resources/css/bootstrap.min.css.map python-backend/usb_4_mic_array/odas_web/resources/css/style.css python-backend/usb_4_mic_array/odas_web/resources/fonts/helvetiker_regular.typeface.json python-backend/usb_4_mic_array/odas_web/resources/images/introlab_icon.icns python-backend/usb_4_mic_array/odas_web/resources/js/bootstrap_slider.js python-backend/usb_4_mic_array/odas_web/resources/js/bootstrap-toggle.min.js python-backend/usb_4_mic_array/odas_web/resources/js/bootstrap.js python-backend/usb_4_mic_array/odas_web/resources/js/bootstrap.min.js python-backend/usb_4_mic_array/odas_web/resources/js/Chart.bundle.min.js python-backend/usb_4_mic_array/odas_web/resources/js/configure-model.js python-backend/usb_4_mic_array/odas_web/resources/js/DragControls.js python-backend/usb_4_mic_array/odas_web/resources/js/graph.js python-backend/usb_4_mic_array/odas_web/resources/js/interface.js python-backend/usb_4_mic_array/odas_web/resources/js/jquery.min.js python-backend/usb_4_mic_array/odas_web/resources/js/legal.js python-backend/usb_4_mic_array/odas_web/resources/js/npm.js python-backend/usb_4_mic_array/odas_web/resources/js/odas_launcher.js python-backend/usb_4_mic_array/odas_web/resources/js/OrbitControls.js python-backend/usb_4_mic_array/odas_web/resources/js/point-to-image.js python-backend/usb_4_mic_array/odas_web/resources/js/recordings_model.js python-backend/usb_4_mic_array/odas_web/resources/js/share-model.js python-backend/usb_4_mic_array/odas_web/resources/js/source_sphere.js python-backend/usb_4_mic_array/odas_web/resources/js/speech-to-text.js python-backend/usb_4_mic_array/odas_web/resources/js/stats.min.js python-backend/usb_4_mic_array/odas_web/resources/js/tcp_link.js python-backend/usb_4_mic_array/odas_web/resources/js/three.min.js python-backend/usb_4_mic_array/odas_web/resources/js/TrackballControls.js python-backend/usb_4_mic_array/odas_web/resources/js/vue.js python-backend/usb_4_mic_array/odas_web/separated.raw python-backend/usb_4_mic_array/odas_web/servers.js python-backend/usb_4_mic_array/odas_web/settings.js python-backend/usb_4_mic_array/odas_web/share.js python-backend/usb_4_mic_array/odas_web/stream-to-text.js python-backend/usb_4_mic_array/odas_web/views/configure.html python-backend/usb_4_mic_array/odas_web/views/legal.html python-backend/usb_4_mic_array/odas_web/views/live_data.html python-backend/usb_4_mic_array/odas_web/views/recordings.html python-backend/usb_4_mic_array/odas_web/views/share.html python-backend/usb_4_mic_array/odas.cfg python-backend/usb_4_mic_array/odas/.github/workflows/compilation.yaml python-backend/usb_4_mic_array/odas/.github/workflows/scheduled-stats.yml python-backend/usb_4_mic_array/odas/.gitignore python-backend/usb_4_mic_array/odas/CMakeLists.txt python-backend/usb_4_mic_array/odas/config/odaslive/azimut_cma.cfg python-backend/usb_4_mic_array/odas/config/odaslive/azimut_oma.cfg python-backend/usb_4_mic_array/odas/config/odaslive/beam.cfg python-backend/usb_4_mic_array/odas/config/odaslive/delta1010lt.cfg python-backend/usb_4_mic_array/odas/config/odaslive/matrix_creator.cfg python-backend/usb_4_mic_array/odas/config/odaslive/matrix_voice.cfg python-backend/usb_4_mic_array/odas/config/odaslive/minidsp.cfg python-backend/usb_4_mic_array/odas/config/odaslive/pepper.cfg python-backend/usb_4_mic_array/odas/config/odaslive/pseye.cfg python-backend/usb_4_mic_array/odas/config/odaslive/respeaker_4_mic_array.cfg python-backend/usb_4_mic_array/odas/config/odaslive/respeaker_6_mic_array.cfg python-backend/usb_4_mic_array/odas/config/odaslive/respeaker_usb_4_mic_array.cfg python-backend/usb_4_mic_array/odas/config/odaslive/respeaker.cfg python-backend/usb_4_mic_array/odas/config/odaslive/xmos.cfg python-backend/usb_4_mic_array/odas/config/odasserver/sls.cfg python-backend/usb_4_mic_array/odas/demo/odaslive/configs.c python-backend/usb_4_mic_array/odas/demo/odaslive/configs.h python-backend/usb_4_mic_array/odas/demo/odaslive/main.c python-backend/usb_4_mic_array/odas/demo/odaslive/objects.c python-backend/usb_4_mic_array/odas/demo/odaslive/objects.h python-backend/usb_4_mic_array/odas/demo/odaslive/parameters.c python-backend/usb_4_mic_array/odas/demo/odaslive/parameters.h python-backend/usb_4_mic_array/odas/demo/odaslive/profiler.c python-backend/usb_4_mic_array/odas/demo/odaslive/profiler.h python-backend/usb_4_mic_array/odas/demo/odaslive/threads.c python-backend/usb_4_mic_array/odas/demo/odaslive/threads.h python-backend/usb_4_mic_array/odas/demo/odasserver/main.c python-backend/usb_4_mic_array/odas/demo/tools/debug.c python-backend/usb_4_mic_array/odas/demo/tools/server.c python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_categories.h python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_hops.h python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_pots.h python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_powers.h python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_spectra.h python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_targets.h python-backend/usb_4_mic_array/odas/include/odas/aconnector/acon_tracks.h python-backend/usb_4_mic_array/odas/include/odas/ainjector/ainj_targets.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_categories.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_hops.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_pots.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_powers.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_spectra.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_targets.h python-backend/usb_4_mic_array/odas/include/odas/amessage/amsg_tracks.h python-backend/usb_4_mic_array/odas/include/odas/amodule/amod_classify.h [724 more files omitted for size] ``` ### Dependencies - python-backend/usb_4_mic_array/odas_web/package.json: @google-cloud/speech@^3.2.0, @grpc/grpc-js@^1.12.6, electron@^5.0.3, electron-rebuild@^1.8.5, ip@^1.1.5, mathjs@^6.0.1, node-localstorage@^1.3.1, systeminformation@^4.9.0, wav@^1.0.2, wav-file-info@0.0.8 - python-backend/usb_4_mic_array/requirements.txt: click@==7.0, pyusb@==1.0.2 ### Recent commits (newest first) - add all unity code - direction change - Merge branch 'main' of https://github.com/YuanSamuel/Treehacks2025 - direction change - var amount of mics - added resampling - added print - added multi-audio - Merge branch 'main' of https://github.com/YuanSamuel/Treehacks2025 - closed caption approach to transcript - direction change - shift direction - Normalized volume - Volume print - Transcript log - Math - Fix categories and volume - persistent sock - more categories - good ## Key source files (fetched from GitHub, selected and truncated for size) ### python-backend/usb_4_mic_array/test/REAME.md ```markdown ### requirements ``` pip install https://github.com/voice-engine/voice-engine/archive/master.zip pip install numpy git submodule init && git submodule update ``` + windows pip install pip install https://github.com/respeaker/respeaker_python_library/releases/download/v0.4.1/pocketsphinx-0.0.9-cp27-cp27m-win32.whl + linux / macos pip install pocketsphinx ### Get RMS ``` python rms.py ``` ### Compare recording audio and playing audio ``` python echo.py ``` ``` ### python-backend/usb_4_mic_array/requirements.txt ``` pyusb==1.0.2 click==7.0 ``` ### python-backend/usb_4_mic_array/odas_web/package.json ``` { "name": "odas_studio", "version": "0.3.0", "description": "A desktop GUI for the ODAS library", "main": "main.js", "scripts": { "start": "electron .", "postinstall": "./node_modules/.bin/electron-rebuild" }, "author": "Cedric Godin", "license": "MIT", "devDependencies": { "electron": "^5.0.3", "electron-rebuild": "^1.8.5" }, "dependencies": { "@google-cloud/speech": "^3.2.0", "@grpc/grpc-js": "^1.12.6", "ip": "^1.1.5", "mathjs": "^6.0.1", "node-localstorage": "^1.3.1", "systeminformation": "^4.9.0", "wav": "^1.0.2", "wav-file-info": "0.0.8" }, "repository": { "type": "git", "url": "https://github.com/introlab/odas_web.git" } } ``` ### python-backend/usb_4_mic_array/odas_web/main.js ```javascript const electron = require('electron') // Module to control application life. const app = electron.app app.commandLine.appendSwitch('--ignore-gpu-blacklist'); // Allows Web GL on Ubuntu // Module to create native browser window. const BrowserWindow = electron.BrowserWindow const path = require('path') const url = require('url') // Keep a global reference of the window object, if you don't, the window will // be closed automatically when the JavaScript object is garbage collected. let odasStudio = {} function createWindow () { // Create the browser window. odasStudio.mainWindow = new BrowserWindow({ width: 1200, height: 800, webPreferences: { webgl: true, nodeIntegration : true }, icon: path.join(__dirname, 'resources/images/introlab_icon.png'), show: false }) // and load the index.html of the app. odasStudio.mainWindow.loadURL(url.format({ pathname: path.join(__dirname, 'views/live_data.html'), protocol: 'file:', slashes: true })) // Open the DevTools. //mainWindow.webContents.openDevTools() // Emitted when the window is closed. odasStudio.mainWindow.on('closed', function () { // Dereference the window object, usually you would store windows // in an array if your app supports multi windows, this is the time // when you should delete the corresponding element. odasStudio.mainWindow = null record.quit() app.quit() }) odasStudio.mainWindow.on('ready-to-show', function() { odasStudio.mainWindow.show() }) } // This method will be called when Electron has finished // initialization and is ready to create browser windows. // Some APIs can only be used after this event occurs. app.on('ready', createWindow) // Quit when all windows are closed. app.on('window-all-closed', function () { // On OS X it is common for applications and their menu bar // to stay active until the user quits explicitly with Cmd + Q if (process.platform !== 'darwin') { app.quit() } }) app.on('activate', function () { // On OS X it's common to re-create a window in the app when the // dock icon is clicked and there are no other windows open. if (odasStudio.mainWindow === null) { createWindow() } }) // In this file you can include the rest of your app's specific main process // code. You can also put them in separate files and require them here. const sockets = require('./servers.js') const record = require('./record.js') require('./share.js') require('./configure.js') odasStudio.odas = require('./odas.js') sockets.startTrackingServer(odasStudio) sockets.startPotentialServer(odasStudio) ``` ### python-backend/combinedquest2.py ```python import argparse import threading import numpy as np import sounddevice as sd import whisper import tensorflow as tf import tensorflow_hub as hub import socket import csv import time import torch def list_devices(): devices = {"cpu": "CPU"} if torch.cuda.is_available(): for i in range(torch.cuda.device_count()): device_name = torch.cuda.get_device_name(i) vram = torch.cuda.get_device_properties(i).total_memory / (1024 ** 3) devices[f"cuda:{i}"] = f"CUDA (GPU {i}) - {device_name} ({vram:.2f} GB VRAM)" if torch.backends.mps.is_available(): devices["mps"] = "MPS (Mac Metal)" return devices def select_device(): devices = list_devices() print("Available devices for running Whisper:") for i, (key, name) in enumerate(devices.items()): print(f"[{i}] {name}") while True: try: choice = int(input("Select the device number to use for transcription: ")) if 0 <= choice < len(devices): return list(devices.keys())[choice] else: print("Invalid choice. Please select a valid device number.") except ValueError: print("Invalid input. Please enter a number.") def select_input_device(): devices = sd.query_devices() input_indices = [i for i, dev in enumerate(devices) if dev['max_input_channels'] > 0] if not input_indices: print("No input devices available. Please check your audio setup.") exit(1) print("Available input devices:") for i in input_indices: print(f"[{i}] {devices[i]['name']}") while True: try: choice = int(input("Select the device number to use for transcription: ")) if choice in input_indices: return choice else: print("Invalid choice. Please select a valid input device index.") except ValueError: print("Invalid input. Please enter a number.") def audio_callback(indata, frames, time_info, status, rolling_audio, lock): if status: print("[AUDIO] Status:", status) new_data = indata.flatten() if indata.ndim == 2 else indata print("Audio callback happened") with lock: rolling_audio = np.concatenate([rolling_audio, new_data]) max_samples = int(10.0 * 16000) if rolling_audio.shape[0] > max_samples: rolling_audio = rolling_audio[-max_samples:] def transcription_thread(stop_event, audio_model, device, host, port, rolling_audio, lock): print("[TRANSCRIPTION] Thread started.") while not stop_event.is_set(): print("Inside stop event") with lock: if rolling_audio.shape[0] == 0: continue buffer_copy = rolling_audio.copy() try: transcribe_result = audio_model.transcribe(buffer_copy, fp16=("cuda" in device)) transcript_text = transcribe_result['text'].strip() print("[TRANSCRIPTION]", transcript_text) send_message(host, port, f"Transcript: {transcript_text}") except Exception as e: print("[TRANSCRIPTION] Error:", e) time.sleep(1) def classification_thread(stop_event, yamnet_model, class_names, host, port, rolling_audio, lock): target_fs = 16000 num_class_samples = int(1.0 * target_fs) dev = usb.core.find(idVendor=0x2886, idProduct=0x0018) if not dev: print("[ANGLE] USB device not found!") return mic_tuning = Tuning(dev) print("[CLASSIFICATION] Thread started.") while not stop_event.is_set(): with lock: if rolling_audio.shape[0] < num_class_samples: continue buffer_copy = rolling_audio[-num_class_samples:].copy() waveform = tf.convert_to_tensor(buffer_copy, dtype=tf.float32) scores, _, _ = yamnet_model(waveform) mean_scores = np.mean(scores.numpy(), axis=0) top_index = np.argmax(mean_scores) top_score = mean_scores[top_index] classification = f"{class_names[top_index]}: {top_score:.3f}" print("[CLASSIFICATION]", classification) try: direction = mic_tuning.direction except Exception as e: print(f"[ANGLE] Error: {e}") send_message(host, port, f"Class: {classification} | Angle: {direction} | Volume: {np.max(buffer_copy):.3f}") time.sleep(1) def send_message(host, port, message): try: with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: sock.connect((host, port)) sock.sendall(message.encode('utf-8')) except Exception as e: print("[SEND] Error:", e) def main(): parser = argparse.ArgumentParser() parser.add_argument("--model", default="tiny", choices=["tiny", "base", "small", "medium", "large", "turbo"]) parser.add_argument("--metaquest_host", type=str, default="127.0.0.1") parser.add_argument("--metaquest_port", type=int, default=7000) parser.add_argument("--trans_input_device", type=int, default=None) parser.add_argument("--yamnet_csv", type=str, default="./yamnet_local/yamnet_class_map.csv") args = parser.parse_args() stop_event = threading.Event() device = select_device() if args.trans_input_device is None: args.trans_input_device = select_input_device() device_fs = 16000 rolling_audio = np.zeros((0,), dtype=np.float32) lock = threading.Lock() print("[MAIN] Loading models...") audio_model = whisper.load_model(args.model, device=device) yamnet_model = hub.load("./yamnet_local") with open(args.yamnet_csv, 'r') as f: class_names = [row[2] for row in csv.reader(f)][1:] print("[MAIN] Models loaded.") print("[MAIN] Starting audio input stream...") with sd.InputStream(samplerate=device_fs, device=args.trans_input_device, channels=1, dtype="float32", callback=lambda indata, frames, time_info, [truncated — 661 more characters] ``` ### python-backend/sendtoquest.py ```python #!/usr/bin/env python3 import threading import time import csv import numpy as np import sounddevice as sd import tensorflow as tf import tensorflow_hub as hub import usb.core import usb.util import math import socket import argparse from tuning import Tuning # Global variable to hold the current angle from the angle_thread. current_angle = None def load_class_names(csv_path): """ Load the YAMNet class names from a CSV file. The CSV is assumed to have rows of the form: index, mid, display_name. """ class_names = [] skipped = False with open(csv_path, 'r') as f: reader = csv.reader(f) for row in reader: if not skipped: skipped = True continue class_names.append(row[2]) return class_names def angle_thread(stop_event): """ Continuously update the sound direction (angle) and speech detection status. """ global current_angle dev = usb.core.find(idVendor=0x2886, idProduct=0x0018) if not dev: print("[ANGLE] USB device not found!") return mic_tuning = Tuning(dev) print("[ANGLE] Starting angle detection thread.") try: while not stop_event.is_set(): direction = mic_tuning.direction speech_detected = mic_tuning.read('SPEECHDETECTED') current_angle = direction print(f"[ANGLE] Direction: {direction} | SpeechDetected: {speech_detected}") time.sleep(1) except Exception as e: print(f"[ANGLE] Error: {e}") def send_message(host: str, port: int, message: str): """ Connect to the specified host and port, send a message, and print any response from the server. """ try: with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: print(f"[QuestUDP] Connecting to {host}:{port}...") sock.connect((host, port)) print("[QuestUDP] Connected.") print(f"[QuestUDP] Sending message: {message}") sock.sendall(message.encode('utf-8')) response = sock.recv(1024) if response: print("[QuestUDP] Received response:", response.decode('utf-8')) else: print("[QuestUDP] No response received from the server.") except ConnectionRefusedError: print("[QuestUDP] Connection refused. Is the Unity server running on the Quest?") except socket.timeout: print("[QuestUDP] Connection timed out.") except Exception as e: print("[QuestUDP] An error occurred:", e) def sound_classification_thread(stop_event, device_id, metaquest_host, metaquest_port): """ Continuously record audio, compute its RMS (volume), run inference with YAMNet, and send the direction, magnitude, and top prediction to MetaQuest via UDP. Only five classes are considered: Speech, Clapping, Siren, Noise, and Silence. Their respective logits are printed for each inference. """ sample_rate = 16000 # YAMNet expects 16 kHz mono audio. duration = 1.0 # seconds per inference num_samples = int(sample_rate * duration) print("[PREDICTION] Loading YAMNet model...") yamnet_model = hub.load('yamnet_local') class_map_path = "yamnet_local/yamnet_class_map.csv" class_names = load_class_names(class_map_path) print(f"[PREDICTION] Starting audio stream on device {device_id} (sample rate {sample_rate} Hz)...") try: with sd.InputStream(device=device_id, channels=6, samplerate=sample_rate) as stream: while not stop_event.is_set(): audio_data, overflowed = stream.read(num_samples) if overflowed: print("[PREDICTION] Warning: Audio buffer has overflowed!") # Select channel 0 (you could also add logic to choose the highest volume channel) channel_index = 0 mono_audio = audio_data[:, channel_index].astype(np.float32) # Compute RMS (volume) and convert to dB. rms_value = np.sqrt(np.mean(mono_audio**2)) volume_db = 20 * math.log10(rms_value + 1e-9) waveform = tf.convert_to_tensor(mono_audio) # Run inference with YAMNet. with tf.device('/GPU:0'): scores, embeddings, spectrogram = yamnet_model(waveform) mean_scores = np.mean(scores, axis=0) # Define the allowed classes (indices correspond to the rows in the CSV): # 0: Speech, 58: Clapping, 390: Siren, 507: Noise, 494: Silence allowed_indices = [0, 58, 390, 507, 494] # Extract the scores for only the allowed classes. allowed_scores = mean_scores[allowed_indices] # Find the allowed class with the highest score. best_allowed_idx = np.argmax(allowed_scores) predicted_index = allowed_indices[best_allowed_idx] predicted_class = class_names[predicted_index] confidence = allowed_scores[best_allowed_idx] # Optionally, set a confidence threshold. confidence_threshold = 0.5 # adjust threshold as needed if confidence < confidence_threshold: prediction_text = "None (low confidence)" else: prediction_text = f"{predicted_class} (Confidence: {confidence:.3f})" # Create a string with all 5 logits. logits_details = ", ".join(f"{class_names[idx]}: {mean_scores[idx]:.3f}" for idx in allowed_indices) angle_info = current_angle if current_angle is not None else "N/A" # Build the message with direction, magnitude, prediction [truncated — 2302 more characters] ``` ### python-backend/tuning.py ```python # -*- coding: utf-8 -*- import sys import struct import usb.core import usb.util USAGE = """Usage: python {} -h -p show all parameters -r read all parameters NAME get the parameter with the NAME NAME VALUE set the parameter with the NAME and the VALUE """ # parameter list # name: (id, offset, type, max, min , r/w, info) PARAMETERS = { 'AECFREEZEONOFF': (18, 7, 'int', 1, 0, 'rw', 'Adaptive Echo Canceler updates inhibit.', '0 = Adaptation enabled', '1 = Freeze adaptation, filter only'), 'AECNORM': (18, 19, 'float', 16, 0.25, 'rw', 'Limit on norm of AEC filter coefficients'), 'AECPATHCHANGE': (18, 25, 'int', 1, 0, 'ro', 'AEC Path Change Detection.', '0 = false (no path change detected)', '1 = true (path change detected)'), 'RT60': (18, 26, 'float', 0.9, 0.25, 'ro', 'Current RT60 estimate in seconds'), 'HPFONOFF': (18, 27, 'int', 3, 0, 'rw', 'High-pass Filter on microphone signals.', '0 = OFF', '1 = ON - 70 Hz cut-off', '2 = ON - 125 Hz cut-off', '3 = ON - 180 Hz cut-off'), 'RT60ONOFF': (18, 28, 'int', 1, 0, 'rw', 'RT60 Estimation for AES. 0 = OFF 1 = ON'), 'AECSILENCELEVEL': (18, 30, 'float', 1, 1e-09, 'rw', 'Threshold for signal detection in AEC [-inf .. 0] dBov (Default: -80dBov = 10log10(1x10-8))'), 'AECSILENCEMODE': (18, 31, 'int', 1, 0, 'ro', 'AEC far-end silence detection status. ', '0 = false (signal detected) ', '1 = true (silence detected)'), 'AGCONOFF': (19, 0, 'int', 1, 0, 'rw', 'Automatic Gain Control. ', '0 = OFF ', '1 = ON'), 'AGCMAXGAIN': (19, 1, 'float', 1000, 1, 'rw', 'Maximum AGC gain factor. ', '[0 .. 60] dB (default 30dB = 20log10(31.6))'), 'AGCDESIREDLEVEL': (19, 2, 'float', 0.99, 1e-08, 'rw', 'Target power level of the output signal. ', '[-inf .. 0] dBov (default: -23dBov = 10log10(0.005))'), 'AGCGAIN': (19, 3, 'float', 1000, 1, 'rw', 'Current AGC gain factor. ', '[0 .. 60] dB (default: 0.0dB = 20log10(1.0))'), 'AGCTIME': (19, 4, 'float', 1, 0.1, 'rw', 'Ramps-up / down time-constant in seconds.'), 'CNIONOFF': (19, 5, 'int', 1, 0, 'rw', 'Comfort Noise Insertion.', '0 = OFF', '1 = ON'), 'FREEZEONOFF': (19, 6, 'int', 1, 0, 'rw', 'Adaptive beamformer updates.', '0 = Adaptation enabled', '1 = Freeze adaptation, filter only'), 'STATNOISEONOFF': (19, 8, 'int', 1, 0, 'rw', 'Stationary noise suppression.', '0 = OFF', '1 = ON'), 'GAMMA_NS': (19, 9, 'float', 3, 0, 'rw', 'Over-subtraction factor of stationary noise. min .. max attenuation'), 'MIN_NS': (19, 10, 'float', 1, 0, 'rw', 'Gain-floor for stationary noise suppression.', '[-inf .. 0] dB (default: -16dB = 20log10(0.15))'), 'NONSTATNOISEONOFF': (19, 11, 'int', 1, 0, 'rw', 'Non-stationary noise suppression.', '0 = OFF', '1 = ON'), 'GAMMA_NN': (19, 12, 'float', 3, 0, 'rw', 'Over-subtraction factor of non- stationary noise. min .. max attenuation'), 'MIN_NN': (19, 13, 'float', 1, 0, 'rw', 'Gain-floor for non-stationary noise suppression.', '[-inf .. 0] dB (default: -10dB = 20log10(0.3))'), 'ECHOONOFF': (19, 14, 'int', 1, 0, 'rw', 'Echo suppression.', '0 = OFF', '1 = ON'), 'GAMMA_E': (19, 15, 'float', 3, 0, 'rw', 'Over-subtraction factor of echo (direct and early components). min .. max attenuation'), 'GAMMA_ETAIL': (19, 16, 'float', 3, 0, 'rw', 'Over-subtraction factor of echo (tail components). min .. max attenuation'), 'GAMMA_ENL': (19, 17, 'float', 5, 0, 'rw', 'Over-subtraction factor of non-linear echo. min .. max attenuation'), 'NLATTENONOFF': (19, 18, 'int', 1, 0, 'rw', 'Non-Linear echo attenuation.', '0 = OFF', '1 = ON'), 'NLAEC_MODE': (19, 20, 'int', 2, 0, 'rw', 'Non-Linear AEC training mode.', '0 = OFF', '1 = ON - phase 1', '2 = ON - phase 2'), 'SPEECHDETECTED': (19, 22, 'int', 1, 0, 'ro', 'Speech detection status.', '0 = false (no speech detected)', '1 = true (speech detected)'), 'FSBUPDATED': (19, 23, 'int', 1, 0, 'ro', 'FSB Update Decision.', '0 = false (FSB was not updated)', '1 = true (FSB was updated)'), 'FSBPATHCHANGE': (19, 24, 'int', 1, 0, 'ro', 'FSB Path Change Detection.', '0 = false (no path change detected)', '1 = true (path change detected)'), 'TRANSIENTONOFF': (19, 29, 'int', 1, 0, 'rw', 'Transient echo suppression.', '0 = OFF', '1 = ON'), 'VOICEACTIVITY': (19, 32, 'int', 1, 0, 'ro', 'VAD voice activity status.', '0 = false (no voice activity)', '1 = true (voice activity)'), 'STATNOISEONOFF_SR': (19, 33, 'int', 1, 0, 'rw', 'Stationary noise suppression for ASR.', '0 = OFF', '1 = ON'), 'NONSTATNOISEONOFF_SR': (19, 34, 'int', 1, 0, 'rw', 'Non-stationary noise suppression for ASR.', '0 = OFF', '1 = ON'), 'GAMMA_NS_SR': (19, 35, 'float', 3, 0, 'rw', 'Over-subtraction factor of stationary noise for ASR. ', '[0.0 .. 3.0] (default: 1.0)'), 'GAMMA_NN_SR': (19, 36, 'float', 3, 0, 'rw', 'Over-subtraction factor of non-stationary noise for ASR. ', '[0.0 .. 3.0] (default: 1.1)'), 'MIN_NS_SR': (19, 37, 'float', 1, 0, 'rw', 'Gain-floor for stationary noise suppression for ASR.', '[-inf .. 0] dB (default: -16dB = 20log10(0.15))'), 'MIN_NN_SR': (19, 38, 'float', 1, 0, 'rw', 'Gain-floor for non-stationary noise suppression for ASR.', '[-inf .. 0] dB (default: -10dB = 20log10(0.3))'), 'GAMMAVAD_SR': (19, 39, 'float', 1000, 0, 'rw', 'Set the threshold for voice activity detection.', '[-inf .. 60] dB (default: 3.5dB 20log10(1.5))'), # 'KEYWORDDETECT': (20, 0, 'int', 1, 0, 'ro', 'Keyword detected. Current value so needs polling.'), 'DOAANGLE': (21, 0, 'int', 359, 0, 'ro', 'DOA angle. Current value. Orientation depends on build configuration.') } class Tuning: TIMEOUT = 100000 def __init__(self, dev): self.dev = dev def write(self, name, value): try: data = PARAMETERS[name] except KeyError: return if data[5] == 'ro': raise ValueError('{} is read-only'.format(name)) id = data[0] # 4 bytes offset, 4 bytes value, 4 by [truncated — 3389 more characters] ``` ### python-backend/realtime-whisper.py ```python import argparse import os import numpy as np import whisper import torch import sounddevice as sd import scipy.signal # For resampling from time import sleep, time from sys import platform import websocket # Added for websocket communication def list_devices(): devices = {"cpu": "CPU"} if torch.cuda.is_available(): for i in range(torch.cuda.device_count()): device_name = torch.cuda.get_device_name(i) vram = torch.cuda.get_device_properties(i).total_memory / (1024 ** 3) devices[f"cuda:{i}"] = f"CUDA (GPU {i}) - {device_name} ({vram:.2f} GB VRAM)" if torch.backends.mps.is_available(): devices["mps"] = "MPS (Mac Metal)" return devices def select_device(): devices = list_devices() print("Available devices for running Whisper:") for i, (key, name) in enumerate(devices.items()): print(f"[{i}] {name}") while True: try: choice = int(input("Select the device number to use: ")) if 0 <= choice < len(devices): return list(devices.keys())[choice] else: print("Invalid choice. Please select a valid device number.") except ValueError: print("Invalid input. Please enter a number.") def select_input_device(): """ Lists input devices. If PulseAudio is running and has available devices, only those are shown. Otherwise, falls back to listing all input devices. """ devices = sd.query_devices() hostapis = sd.query_hostapis() pulse_api_index = None # Look for a host API that includes "pulse" in its name. for idx, ha in enumerate(hostapis): if "pulse" in ha['name'].lower(): pulse_api_index = idx break # If PulseAudio is found, filter for PulseAudio input devices. if pulse_api_index is not None: input_indices = [i for i, dev in enumerate(devices) if dev['max_input_channels'] > 0 and dev['hostapi'] == pulse_api_index] if input_indices: print("Available PulseAudio input devices:") for i in input_indices: print(f"[{i}] {devices[i]['name']}") while True: try: choice = int(input("Select the device number to use: ")) if choice in input_indices: return choice else: print("Invalid choice. Please select a valid PulseAudio input device index.") except ValueError: print("Invalid input. Please enter a number.") # Fallback: if no PulseAudio devices are found, list all input devices. input_indices = [i for i, dev in enumerate(devices) if dev['max_input_channels'] > 0] if not input_indices: print("No input devices available. Please check your audio setup.") exit(1) print("Available input devices:") for i in input_indices: print(f"[{i}] {devices[i]['name']}") while True: try: choice = int(input("Select the device number to use: ")) if choice in input_indices: return choice else: print("Invalid choice. Please select a valid input device index.") except ValueError: print("Invalid input. Please enter a number.") def main(): parser = argparse.ArgumentParser() parser.add_argument("--model", default="medium", help="Model to use", choices=["tiny", "base", "small", "medium", "large", "turbo"]) parser.add_argument("--non_english", action='store_true', help="Don't use the English model.") parser.add_argument("--energy_threshold", default=500, help="Energy level for mic to detect (in int16 units).", type=int) parser.add_argument("--buffer_duration", default=10, help="Duration (in seconds) of the rolling audio buffer.", type=float) args = parser.parse_args() # GPU/CPU selection for Whisper. device = select_device() # Select input device (preferring PulseAudio devices if available). input_device = select_input_device() sd_device = input_device if input_device is not None else None # Query the device info to get sample rate and host API. device_info = sd.query_devices(sd_device, 'input') hostapi_index = device_info['hostapi'] hostapi_name = sd.query_hostapis()[hostapi_index]['name'] if "pulse" in hostapi_name.lower(): print(f"Using PulseAudio input device: {device_info['name']}") else: print(f"Using non-PulseAudio input device: {device_info['name']}") # Set target sample rate for Whisper. target_fs = 16000 device_fs = int(device_info['default_samplerate']) print(f"Device default sample rate: {device_fs} Hz. Will resample to {target_fs} Hz.") channels = 1 # Adjust model name based on language settings. model_name = args.model if args.model not in ["large", "turbo"] and not args.non_english: model_name = model_name + ".en" audio_model = whisper.load_model(model_name, device=device) print("Model loaded.") # WebSocket setup. use_ws = True ws_url = "ws://localhost:8765" ws = None # Try connecting to the WebSocket server initially. try: ws = websocket.create_connection(ws_url) print(f"Connected to WebSocket server at {ws_url}") except Exception as e: print("Initial connection to WebSocket server failed:", e) ws = None # Prompt for confirmation if WebSocket connection is established. confirm = input(f"WebSocket connection {'was not' if ws is None else 'was'} established. Continue? (y/n): ").strip().lower() if confirm not in ("y", "yes"): print("WebSocket connection will not be used.") ws.close() ws = None use_ws = False # He [truncated — 3972 more characters] ``` ### python-backend/combinedquest.py ```python #!/usr/bin/env python3 import argparse import os import threading import time from time import sleep, time import csv import math import socket import numpy as np import torch import sounddevice as sd import scipy.signal # For resampling import whisper import tensorflow as tf import tensorflow_hub as hub import usb.core import usb.util from tuning import Tuning # Assumes you have this module # ---------------------- # Utility functions for device selection # ---------------------- def list_devices(): devices = {"cpu": "CPU"} if torch.cuda.is_available(): for i in range(torch.cuda.device_count()): device_name = torch.cuda.get_device_name(i) vram = torch.cuda.get_device_properties(i).total_memory / (1024 ** 3) devices[f"cuda:{i}"] = f"CUDA (GPU {i}) - {device_name} ({vram:.2f} GB VRAM)" if torch.backends.mps.is_available(): devices["mps"] = "MPS (Mac Metal)" return devices def select_device(): devices = list_devices() print("Available devices for running Whisper:") for i, (key, name) in enumerate(devices.items()): print(f"[{i}] {name}") while True: try: choice = int(input("Select the device number to use for transcription: ")) if 0 <= choice < len(devices): return list(devices.keys())[choice] else: print("Invalid choice. Please select a valid device number.") except ValueError: print("Invalid input. Please enter a number.") def select_input_device(): devices = sd.query_devices() hostapis = sd.query_hostapis() pulse_api_index = None for idx, ha in enumerate(hostapis): if "pulse" in ha['name'].lower(): pulse_api_index = idx break if pulse_api_index is not None: input_indices = [i for i, dev in enumerate(devices) if dev['max_input_channels'] > 0 and dev['hostapi'] == pulse_api_index] if input_indices: print("Available PulseAudio input devices:") for i in input_indices: print(f"[{i}] {devices[i]['name']}") while True: try: choice = int(input("Select the device number to use for transcription: ")) if choice in input_indices: return choice else: print("Invalid choice. Please select a valid PulseAudio input device index.") except ValueError: print("Invalid input. Please enter a number.") input_indices = [i for i, dev in enumerate(devices) if dev['max_input_channels'] > 0] if not input_indices: print("No input devices available. Please check your audio setup.") exit(1) print("Available input devices:") for i in input_indices: print(f"[{i}] {devices[i]['name']}") while True: try: choice = int(input("Select the device number to use for transcription: ")) if choice in input_indices: return choice else: print("Invalid choice. Please select a valid input device index.") except ValueError: print("Invalid input. Please enter a number.") # ---------------------- # Combined processing: transcription and classification using a single input stream # ---------------------- def combined_processing_loop(stop_event, args, device, audio_model, device_fs, target_fs, yamnet_model, class_names): # Local variables for rolling audio and latest transcription rolling_audio = np.zeros((0,), dtype=np.float32) latest_transcription_local = "" # Classification uses the last 1 second of audio classification_duration = 1.0 # seconds num_class_samples = int(target_fs * classification_duration) def callback(indata, frames, time_info, status): nonlocal rolling_audio if status: print("[COMBINED] Audio stream status:", status) # Append new samples (flatten if needed) new_data = indata.flatten() if indata.ndim == 2 else indata rolling_audio = np.concatenate([rolling_audio, new_data]) iterations = 0 last_prediction = "No prediction: 0" print("[COMBINED] Starting combined processing. Press Ctrl+C to stop.") try: with sd.InputStream(samplerate=device_fs, device=args.trans_input_device, channels=1, dtype="float32", callback=callback): while not stop_event.is_set(): iterations += 1 sleep(0.01) # Allow some audio to accumulate current_buffer = rolling_audio.copy() # Trim buffer to last 'buffer_duration' seconds if needed max_total_samples = int(args.buffer_duration * target_fs) if current_buffer.shape[0] > max_total_samples: current_buffer = current_buffer[-max_total_samples:] rolling_audio = current_buffer.copy() if current_buffer.shape[0] == 0: continue # Transcription: use the entire rolling buffer try: transcribe_result = audio_model.transcribe(current_buffer, fp16=("cuda" in device)) transcript_text = transcribe_result['text'].strip() latest_transcription_local = transcript_text print("[COMBINED] Transcription:", transcript_text) except Exception as e: print("[COMBINED] Transcription error:", e) latest_transcription_local = "" # Classification: use the last num_class_samples if available if current_buffer.shape[0] >= num_class_samples and iterations % 20 == 0: classification_chunk = current_buffer[-num_class_samples:] [truncated — 6092 more characters] ``` ### python-backend/usb_4_mic_array/odas_web/odas.js ```javascript /* * ODAS Control */ exports.odas_process const electron = require('electron') const ipcMain = electron.ipcMain const child_process = require('child_process') ipcMain.on('launch-odas', function(event, core, config) { console.log('received launch command') console.log(core) console.log(config) exports.odas_process = child_process.spawn(core, ['-c', config]) event.sender.send('launched-odas', true) }) ipcMain.on('stop-odas', function(event) { exports.odas_process.kill('SIGINT') exports.odas_process = undefined console.log('received stop command') }) ``` [375 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]