# Project export: Spartan | Edge AI Situational Helmet for First Responders This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: AI-powered helmet that provides real-time situational awareness in high-risk environments. Comes with live world memory, localization, heads-up display, night vision, and physiological monitoring. - Devpost: https://devpost.com/software/spartan-edge-ai-situational-helmet-for-first-responders - GitHub: https://github.com/Ekansh-Mittal1/spartan - Team: 3 GitHub contributor(s) — Sarang Goel (4 commits), Ekansh-Mittal1 (2 commits), Arnav Chakravarthy (1 commits) ## Devpost submission (written by the team) ### Inspiration First responders operate in incredibly high-risk and spontaneous environments where situational awareness can mean the difference between life and death. With the advent of several VLMs and embedded computers, such as the Jetson Orin Nano Super, we saw a potential to supplement first responders' situational awareness. High-risk scenarios mean visibility can be compromised, GPS signals aren't as strong, a risk of stroke, and other critical details may be missed. Many of these first responders work in high-risk environments and already wear helmets, which led to our project: what if we could extend the functionality of these helmets beyond just their physical protectiveness to actively enhance human perception? We came up with the idea of the Spartan helmet as an intelligence layer directly embedded into already used protective equipment. Instead of adding another handheld device or remote dashboard, we designed a system that integrates sensing, computation, and augmented visualization directly into a wearable form factor. Spartan is built to support and not replace human judgment. ### What it does Spartan is a real-time edge AI helmet that fuses multimodal sensing into a live augmented heads-up display (HUD). It integrates dual vision cameras, infrared imaging for night vision, IMU motion tracking, GPS positioning, physiological monitoring (heart rate and temperature), and an on-device inference and contextual world memory. Spartan is powered by an NVIDIA Jetson Orin Nano. It can recognize people and objects in real time, detect these contextual events as and/or hazards, and track what has been seen over time (world memory) thanks to a custom memory system we built; this system leverages real-time VLM reasoning analysis to generate a world context "story" of what the Spartan helmet is seeing as users respond to their respective situations. This contextual memory also stores and monitors responder stress and physiological state, where the helmet is able to track pulse data and temperature data (whether your body is removing or adding heat through the glabrous skin on the forehead). Finally, the Spartan helmet overlays all this critical information directly into a dual-eye HUD. Unlike simple frame-by-frame detection systems, Spartan builds temporal awareness by understanding not just what is visible now, but what has happened recently. ### How we built it Spartan was designed as a fully integrated hardware-software system. We split the building process amongst us into 3 different subsystems. Starting with the hardware and mechanical design, we designed and modeled the helmet enclosure and optical housing using CAD. This allowed us to precisely position the dual-eye display relative to the lenses and optimize spacing for inter-pupillary alignment. We further integrated camera modules and infrared sensors into the helmet shell. We mounted the Jetson Orin Nano Super on the helmet in a manner that allowed it to optimally reach all of its modules. Finally, it took us several attempts, but we managed to route all the sensor wiring as cleanly as possible. Using CAD early in development enabled rapid iteration on ergonomics, weight distribution, and component placement before physical assembly. We then moved on to the embedded hardware stack, which started with the central Jetson Orin Nano Super for the edge inference and compute. The stack included two 12 MP camera modules for depth perception, an infrared camera for night vision, an IMU for motion detection and localization, a GPS to supplement the localization, a heart rate sensor, a temperature sensor to detect body heat, and, finally, a display for the dual-eye HUD display. Lastly, for the software, we tried to make Spartan as modular as possible. The final software stack consisted of Python sensor-level ingestion code and a Python syncing module to sync/packetize all the raw sensor data. Once the data was bundled, we fed it into the real-time inference pipeline using the VILA1.5-3b model. We built a cyclical world memory layer module using the output from the VLM reasoning and the OpenAI API (gpt-5-mini) for an updated data store of what the camera inputs have been capturing over time in a session. The last module was the HUD rendering module, which split the camera feed into two mathematically warped videos with calibration controls to show the user a smooth view of what the glasses are showing them. We purposely separated sensor ingestion, synchronization, inference, memory, and rendering. This modular design allowed us to simulate hardware inputs early and build the HUD and edge AI logic in parallel with mechanical development. ### Challenges we ran into The most unexpected challenge we ran into was actually related to the optics of the dual-eye HUD display. We had to figure out some of the math behind how the optics of similar systems like Google Cardboard VR work and attempt to replicate that in our own software/CAD designs. Proper optical isolation, inter-pupillary alignment, and viewport calibration required a LOT of iterative tuning. Sensor synchronization was also a big challenge, as different sensors operate at different frequencies. The IMU runs at high frequency, GPS updates slowly, and physiological data arrives intermittently. Designing a synchronization system that aligned these streams without blocking the render loop required careful buffer management. Finally, integrating the cameras, figuring out the device tree config, and embedding all the specific drivers for each module took the longest time, as always with hardware builds ;) ### Accomplishments we're proud of A fully modular sensor ingestion and synchronization pipeline A working dual-eye HUD system with live calibration controls On-device inference capable of recognition and contextual tracking Real-time physiological monitoring integrated into the display A CAD-designed enclosure integrating sensing, compute, and optics ### What we learned One of the biggest lessons we took away is that real-time systems require strict separation between computation-heavy tasks and rendering loops. This build required several rendering loops, and the inference for the VLM reasoning was far more computationally heavy than refreshing the HUD; this meant we had to modularize the two processes into two distinct compartments to prevent a visible buffer for the user. We also learned that sensor synchronization is more important than the raw sensor accuracy, where in a system like this, the greater collective of sensor data being synced is far more important than any individual sensor's accuracy. This mattered greatly because minor discrepancies between sensor data severely impair situational awareness, which is crucial in a build like this. Something we learned on both the hardware and software side is that optical systems are as complex as AI systems. We learned the math behind the optics for the dual display and how to combine that with live CV/VLM intelligence. ### What's next The Spartan helmet we built is a very early prototype of a larger vision. The first improvement we would make for obvious comfort is refining the optical calibration and distortion correction to make the focus on the HUD far less strenuous on the eyes. Next, we would expand our basic VLM detection reasoning to more complex hazard classification, which is possible through some of the physical reasoning models like NVIDIA Cosmos. Another improvement, which is a bit harder to do, is improving indoor localization, simply by using a better IMU and more fine-tuned localization sensors to give more accurate information on where the user needs to go. The most exciting next step is to leverage the power of multiple Spartan helmets and have them communicate to help give a larger contextual awareness to a situation (where Person #1 picks up a hazard, and Person #2's Spartan now knows where and what the hazard is before interacting with it). Overall, in the long term, Spartan could evolve into a scalable edge AI platform for firefighting, emergency medical response, disaster recovery, and tactical operations. ## README (from the GitHub repository) # Spartan Helmet HUD with real-time vision (VLM) and world-context pipeline. Runs in browser + Node backend; VLM can be cloud (OpenAI) or local (Qwen on Mac/Jetson). DevPost: https://devpost.com/software/spartan-edge-ai-situational-helmet-for-first-responders ## Project layout | Directory | Description | |-----------|-------------| | **helmet-hud** | Vite + React HUD (2560×1440, camera, corner panels, WebSocket state). | | **helmet-backend** | Node server: WebSocket `/ws/state`, frame ingestion, VLM + world-context, state broadcast. | | **vlm_service** | Python VLM service (FastAPI): local Qwen via MLX (Mac) or TensorRT-LLM (Jetson). | | **docs** | Deployment and hardware docs (e.g. Jetson). | | **scripts** | Shell scripts for starting the edge stack. | ## Edge AI pipeline Two modes: 1. **OpenAI only** (default): Backend uses OpenAI Vision (gpt-4o-mini) for VLM and GPT-5 mini for world context. Set `OPENAI_API_KEY` in `helmet-backend/.env`. 2. **Local Qwen**: Run the Python VLM service (MLX on Mac or TensorRT-LLM on Jetson), then the backend with `VLM_MODE=local` and `VLM_LOCAL_URL` pointing at the service. No OpenAI key needed for VLM; world context still uses OpenAI. ### Local Qwen (Mac) ```bash # Terminal 1: VLM service (MLX) cd vlm_service && pip install -r requirements.txt && python -m vlm_service # Terminal 2: Backend cd helmet-backend && VLM_MODE=local VLM_LOCAL_URL=http://127.0.0.1:5000 npm start # Terminal 3: HUD cd helmet-hud && VITE_WS_URL=http://localhost:8765 npm run dev ``` Open `http://localhost:5173/?source=camera`. ### Jetson Orin Nano Super See **[docs/JETSON_DEPLOY.md](docs/JETSON_DEPLOY.md)** for JetPack, TensorRT-LLM, Qwen2-VL setup, env vars, and run order. ## Quick test (no browser) From `helmet-backend`: `node test-one-frame.mjs` (sends one frame, prints VLM + world; backend and optionally VLM service must be running). ## Detected evidence (automated analysis) Indexed codebase: 44 recognized source files, 166 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (58 of 58) ``` .gitignore docs/JETSON_DEPLOY.md helmet-backend/.env.example helmet-backend/package.json helmet-backend/README.md helmet-backend/server.js helmet-backend/test-one-frame.mjs helmet-backend/world_log.jsonl helmet-hud/.env.example helmet-hud/.gitignore helmet-hud/index.html helmet-hud/package.json helmet-hud/postcss.config.js helmet-hud/README.md helmet-hud/src/App.tsx helmet-hud/src/components/AlertBanner.tsx helmet-hud/src/components/ImuPanel.tsx helmet-hud/src/components/MockVideoCanvas.tsx helmet-hud/src/components/Panel.tsx helmet-hud/src/components/ReasoningPanel.tsx helmet-hud/src/components/StatusPanel.tsx helmet-hud/src/components/VitalsPanel.tsx helmet-hud/src/components/WarpLayer.tsx helmet-hud/src/hooks/useCameraStream.ts helmet-hud/src/hooks/useHudState.ts helmet-hud/src/hooks/useMockState.ts helmet-hud/src/index.css helmet-hud/src/layout.ts helmet-hud/src/lib/warpMesh.ts helmet-hud/src/main.tsx helmet-hud/src/mock/drawFrame.ts helmet-hud/src/types.ts helmet-hud/src/vite-env.d.ts helmet-hud/tailwind.config.js helmet-hud/tsconfig.json helmet-hud/tsconfig.node.json helmet-hud/vite.config.ts qwen_backend/jetson_qwen.py qwen_backend/mps_qwen.py qwen_backend/requirements.txt README.md scripts/camera_feeder.py scripts/requirements-feeder.txt scripts/run-edge-stack.sh scripts/run-vlm-nanollm.sh scripts/run-vlm-service.sh scripts/run-vlm-sglang.sh vlm_service/__init__.py vlm_service/__main__.py vlm_service/backends/__init__.py vlm_service/backends/mlx_backend.py vlm_service/backends/nanollm_backend.py vlm_service/backends/sglang_backend.py vlm_service/backends/trt_backend.py vlm_service/README.md vlm_service/requirements-jetson.txt vlm_service/requirements.txt vlm_service/server.py ``` ### Dependencies - helmet-backend/package.json: dotenv@^16.3.1, openai@^4.52.0, ws@^8.14.2 - helmet-hud/package.json: @types/react@^18.2.43, @types/react-dom@^18.2.17, @vitejs/plugin-react@^4.2.1, autoprefixer@^10.4.16, postcss@^8.4.32, react@^18.2.0, react-dom@^18.2.0, tailwindcss@^3.4.0, typescript@^5.2.2, vite@^5.0.8 - qwen_backend/requirements.txt: mlx-vlm, openai, opencv-python@>=4.8.0, python-dotenv - vlm_service/requirements.txt: fastapi@>=0.104.0, numpy@<2, Pillow@>=9.0.0, python-multipart@>=0.0.6, uvicorn[standard]@>=0.24.0 ### Recent commits (newest first) - Add devpost to README.md - Delete mps_qwen.py - Cleanup - ollama migration - Sglang migration - Camera detection for jetson - Add NanoLLM backend for Jetson - big beautiful merge - qwen requirements - Backend for qwen video processing and world model analysis - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/JETSON_DEPLOY.md ```markdown # Jetson Orin Nano Super Deployment Deploy the edge AI pipeline (local VLM + Node backend + HUD) on Jetson Orin Nano Super. Camera and HUD run in the browser; VLM runs locally via **NanoLLM** (recommended) or TensorRT-LLM. --- ## Option A — NanoLLM (recommended) NanoLLM from the Jetson AI Lab provides optimised inference for vision-language models (VILA, LLaVA, Obsidian, etc.) on Jetson with MLC or TensorRT backends, quantisation, and KV-cache management out of the box. ### Prerequisites - **JetPack 6.x** (Ubuntu 22.04, CUDA 12.x). - **NanoLLM** — install via one of: - **jetson-containers** (easiest): ```bash # Pull the pre-built NanoLLM container jetson-containers run $(autotag nano_llm) ``` - **From source** (if running outside a container): ```bash git clone https://github.com/dusty-nv/NanoLLM cd NanoLLM && pip install -e . ``` - A supported vision model (downloaded automatically on first run): - `Efficient-Large-Model/VILA1.5-3b` (default, good speed/quality) - `Efficient-Large-Model/VILA-2.5-3b` - `liuhaotian/llava-v1.6-vicuna-7b` - Any NanoLLM-compatible VLM ### Environment | Variable | Default | Description | |---------|---------|-------------| | `VLM_BACKEND` | — | **Must be `nanollm`** on Jetson with NanoLLM. | | `VLM_MODEL_PATH` | `Efficient-Large-Model/VILA1.5-3b` | HuggingFace repo ID or local path. | | `NANOLLM_API` | `mlc` | NanoLLM runtime: `mlc` (MLC-LLM) or `trt` (TensorRT). | | `NANOLLM_QUANTIZATION` | `q4f16_ft` | Quantisation preset (see NanoLLM docs). | | `MAX_FRAME_DIM` | `768` | Max pixel dimension; lower = faster inference. | | `VLM_HOST` / `VLM_PORT` | `0.0.0.0` / `5000` | VLM service bind address and port. | | `VLM_PROMPT` | `Describe this image briefly.` | Default prompt. | ### Run order 1. **VLM service** (Python, NanoLLM): ```bash # Quick start with the helper script: ./scripts/run-vlm-nanollm.sh # Or manually: export VLM_BACKEND=nanollm export VLM_MODEL_PATH=Efficient-Large-Model/VILA1.5-3b python -m vlm_service ``` The service listens on `http://0.0.0.0:5000` by default. 2. **Node backend** (orchestrator, world context via OpenAI): ```bash cd helmet-backend export VLM_MODE=local export VLM_LOCAL_URL=http://127.0.0.1:5000 export OPENAI_API_KEY=your_key # for world-context only npm start ``` 3. **HUD** (build and serve; or dev server): ```bash cd helmet-hud export VITE_WS_URL=http://:8765 npm run build && npx serve -s dist -l 5173 # or: npm run dev ``` 4. **Browser**: Open `http://:5173/?source=camera`, allow camera. Frames are sent to the backend → NanoLLM VLM runs locally → world context from OpenAI → HUD displays `vlm_text` and `world_context`. ### One-command start (VLM + Node backend) ```bash export VLM_BACKEND=nanollm export OPENAI_API_KEY=your_key ./scripts/run-edge-stack.sh ``` This starts the NanoLLM VLM service in the background, waits for it to load, then sta [truncated — 3073 more characters] ``` ### qwen_backend/requirements.txt ``` # Webcam VLM test — MLX (Apple Silicon, ~0.6s inference) mlx-vlm opencv-python>=4.8.0 openai python-dotenv ``` ### helmet-backend/package.json ``` { "name": "helmet-backend", "version": "0.1.0", "private": true, "type": "module", "scripts": { "start": "node server.js" }, "dependencies": { "dotenv": "^16.3.1", "openai": "^4.52.0", "ws": "^8.14.2" } } ``` ### helmet-hud/package.json ``` { "name": "helmet-hud", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "preview": "vite preview" }, "dependencies": { "react": "^18.2.0", "react-dom": "^18.2.0" }, "devDependencies": { "@types/react": "^18.2.43", "@types/react-dom": "^18.2.17", "@vitejs/plugin-react": "^4.2.1", "autoprefixer": "^10.4.16", "postcss": "^8.4.32", "tailwindcss": "^3.4.0", "typescript": "^5.2.2", "vite": "^5.0.8" } } ``` ### vlm_service/requirements.txt ``` # VLM service: Mac (MLX), Jetson (NanoLLM / TensorRT-LLM) # Use a dedicated venv to avoid NumPy 1.x vs 2.x conflicts with system scipy/sklearn. fastapi>=0.104.0 uvicorn[standard]>=0.24.0 python-multipart>=0.0.6 # Pin NumPy < 2 so transformers/sklearn/scipy (pulled by mlx-vlm) don't clash with NumPy-1.x-built binaries numpy<2 # ---------- Mac / Apple Silicon only (VLM_BACKEND=mlx) ---------- # mlx-vlm>=0.0.14 # torchvision>=0.20.0 # required by transformers 5.x for Qwen2.5-VL video processor # ---------- Jetson (VLM_BACKEND=nanollm) ---------- # NanoLLM is installed from source or the Jetson AI Lab container. # git clone https://github.com/dusty-nv/NanoLLM && cd NanoLLM && pip install -e . # Or run inside the jetson-containers NanoLLM image (recommended). # Pillow is needed for image decoding: Pillow>=9.0.0 # ---------- Jetson (VLM_BACKEND=trt) ---------- # TensorRT-LLM is platform-specific; install on Jetson per docs/JETSON_DEPLOY.md # tensorrt-llm # uncomment on Jetson when using trt backend ``` ### vlm_service/server.py ```python """FastAPI server for local VLM inference. POST /infer with image + prompt, returns text.""" import base64 import logging import os import traceback from typing import Optional from fastapi import FastAPI, HTTPException from fastapi.responses import JSONResponse from pydantic import BaseModel logging.basicConfig(level=logging.INFO) logger = logging.getLogger("vlm_service") from .backends import get_backend # noqa: E402 app = FastAPI(title="VLM Service", description="Local VLM for edge pipeline (MLX / NanoLLM / TRT)") _backend = None _backend_name: str = "" class InferBody(BaseModel): image_base64: str prompt: Optional[str] = None def _get_backend(): global _backend, _backend_name if _backend is None: _backend_name = (os.environ.get("VLM_BACKEND") or "mlx").lower() _backend = get_backend() _backend.load() return _backend @app.post("/infer") async def infer(body: InferBody): """Run VLM on an image. JSON body: { "image_base64": "", "prompt": "..." (optional) }.""" prompt_val = body.prompt or os.environ.get("VLM_PROMPT", "Describe this image briefly.") try: image_bytes = base64.b64decode(body.image_base64, validate=True) except Exception as e: raise HTTPException(status_code=400, detail=f"Invalid image_base64: {e}") from e try: backend = _get_backend() text, elapsed_s = backend.infer(image_bytes, prompt=prompt_val) return JSONResponse(content={"text": text, "elapsed_s": round(elapsed_s, 4)}) except Exception as e: logger.error("VLM inference failed:\n%s", traceback.format_exc()) return JSONResponse( status_code=500, content={"text": f"[VLM error: {e}]", "elapsed_s": 0.0}, ) @app.get("/health") async def health(): backend_name = (os.environ.get("VLM_BACKEND") or "mlx").lower() return {"status": "ok", "backend": backend_name} ``` ### helmet-backend/server.js ```javascript import "dotenv/config"; import { createServer } from "http"; import { WebSocketServer } from "ws"; import OpenAI from "openai"; import fs from "fs"; const PORT = Number(process.env.PORT) || 8765; const OPENAI_API_KEY = process.env.OPENAI_API_KEY; const OPENAI_MODEL = process.env.OPENAI_WORLD_MODEL ?? "gpt-5-mini"; const VLM_MODE = (process.env.VLM_MODE || "openai").toLowerCase(); const VLM_LOCAL_URL = process.env.VLM_LOCAL_URL?.replace(/\/$/, "") ?? ""; const VLM_PROMPT = process.env.VLM_PROMPT || "Describe this image briefly."; const LOG_FILE = process.env.WORLD_LOG_FILE || "world_log.jsonl"; /** True when using a GPT-5 reasoning model (Responses API: reasoning.effort, max_output_tokens; no temperature/max_tokens). */ const isGpt5 = /^gpt-5(-|$)/.test(OPENAI_MODEL); const WORLD_SYSTEM_PROMPT = `You are a real-time people tracker. Track ONLY people: who is present, what they are wearing, and what they are doing. Ignore objects, furniture, and background. OUTPUT FORMAT (use exactly this, no prose): PEOPLE: - : wearing | doing | status: present/absent | last seen: CHANGES: - <+added / -removed / ~changed> : RULES: 1. Track ONLY people. Do not list objects, animals (unless you treat them as "person" by convention), or scenery. 2. For each person: describe clothing (e.g. "red shirt, dark pants") and current action (e.g. "sitting", "looking at camera", "holding phone"). 3. ADD a new person when first seen. KEEP people not in frame as status: absent. Only REMOVE after many consecutive observations without them. 4. UPDATE clothing or activity when the observation clearly indicates a change. Do not invent or hallucinate changes. 5. If the observation says the image is black, blank, or has no visible content, return the previous registry UNCHANGED. 6. Be terse. No filler. No prose. Only the structured format above. 7. Respond with ONLY the updated registry.`; const defaultHudState = () => ({ bpm: 0, temp_c: 0, quality: 1, heading_deg: 0, mic_level: 0, drawer_open: false, mic_on: true, light_on: false, thermal_on: false, alert_banner: null, mode: "NORM", reasoning_text: "", vlm_text: "", world_context: "", gps_lat: 0, gps_lon: 0, anchor_heading_deg: 0, /** Currently active camera ID (the one being sent to VLM). */ active_camera_id: "", /** All known camera IDs that have sent at least one frame. */ camera_ids: [], }); let hudState = defaultHudState(); let worldUnderstanding = ""; let clients = new Set(); let processing = false; // ── Multi-camera state ────────────────────────────────────────────────────── /** Per-camera latest frame buffer. Key = camera_id (e.g. "jetson-cam-0", "browser"). */ const cameraFrames = new Map(); /** Which camera the VLM pipeline reads from. Empty string = accept any / first seen. */ let activeCameraId = ""; /** Convenience: the frame the pipeline should process next (from the active camera). */ let latestFrame = null; const openai = OPENAI_API_KEY ? new OpenAI({ apiKey: OPENAI_API_KEY }) : null; function broadcast() { const payload = JSON.stringify(hudState); for (const ws of clients) { if (ws.readyState === 1) ws.send(payload); } } async function runVLM(imageBuffer) { const b64 = imageBuffer.toString("base64"); if (VLM_MODE === "local") { if (!VLM_LOCAL_URL) return { text: "[VLM_MODE=local but VLM_LOCAL_URL not set]", elapsed: 0 }; const start = Date.now(); try { const res = await fetch(`${VLM_LOCAL_URL}/infer`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ image_base64: b64, prompt: VLM_PROMPT }), }); const raw = await res.text(); let data; try { data = raw ? JSON.parse(raw) : {}; } catch { console.error("VLM error: response not JSON", res.status, raw?.slice(0, 200)); return { text: `[VLM error: ${res.status} non-JSON]`, elapsed: (Date.now() - start) / 1000 }; } if (!res.ok) { console.error("VLM error:", res.status, data?.text ?? data?.detail ?? raw?.slice(0, 200)); return { text: typeof data?.text === "string" ? data.text : `[VLM ${res.status}]`, elapsed: (Date.now() - start) / 1000, }; } const text = typeof data?.text === "string" ? data.text : String(data?.text ?? ""); const elapsed = typeof data?.elapsed_s === "number" ? data.elapsed_s : (Date.now() - start) / 1000; return { text: text.trim(), elapsed }; } catch (e) { console.error("VLM error:", e.message); return { text: `[VLM error: ${e.message}]`, elapsed: 0 }; } } if (!openai) return { text: "[No OPENAI_API_KEY]", elapsed: 0 }; const start = Date.now(); try { const res = await openai.chat.completions.create({ model: "gpt-4o-mini", max_tokens: 150, messages: [ { role: "user", content: [ { type: "text", text: VLM_PROMPT }, { type: "image_url", image_url: { url: `data:image/jpeg;base64,${b64}` } }, ], }, ], }); const text = (res.choices?.[0]?.message?.content ?? "").trim(); const elapsed = (Date.now() - start) / 1000; return { text, elapsed }; } catch (e) { console.error("VLM error:", e.message); return { text: `[VLM error: ${e.message}]`, elapsed: 0 }; } } function extractResponsesOutputText(res) { if (typeof res.output_text === "string") return res.output_text.trim(); const out = res.output; if (!Array.isArray(out)) return ""; for (const item of out) { if (item.type === "message" && Array.isArray(item.content)) { for (const block of item.content) { if (block?.type === "output_text" && typeof block.text === "string") return block.text.trim(); } } } return ""; } async function runWorld(timestamp, frameSummary, vlmElapsed) { if (!openai) { worldUn [truncated — 5761 more characters] ``` ### helmet-hud/src/main.tsx ```typescript import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; import "./index.css"; ReactDOM.createRoot(document.getElementById("root")!).render( ); ``` ### helmet-hud/src/App.tsx ```typescript import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import { useCameraDevices, useCameraStream } from "./hooks/useCameraStream"; import { useHudState } from "./hooks/useHudState"; import { useMockState } from "./hooks/useMockState"; import { buildWarpMesh } from "./lib/warpMesh"; import { leftViewport, rightViewport } from "./layout"; import { DEFAULT_LAYOUT } from "./types"; import { VitalsPanel } from "./components/VitalsPanel"; import { StatusPanel } from "./components/StatusPanel"; import { ImuPanel } from "./components/ImuPanel"; import { ReasoningPanel } from "./components/ReasoningPanel"; import { AlertBanner } from "./components/AlertBanner"; import { MockVideoCanvas } from "./components/MockVideoCanvas"; import { WarpLayer } from "./components/WarpLayer"; import type { HudState } from "./types"; const PAD = 16; const LAYOUT_W = 2560; const LAYOUT_H = 1440; function useScaleToFit() { const [scale, setScale] = useState(() => Math.min(window.innerWidth / LAYOUT_W, window.innerHeight / LAYOUT_H, 1) ); useEffect(() => { const onResize = () => setScale(Math.min(window.innerWidth / LAYOUT_W, window.innerHeight / LAYOUT_H, 1)); window.addEventListener("resize", onResize); onResize(); return () => window.removeEventListener("resize", onResize); }, []); return scale; } function getWsUrl(): string { const base = import.meta.env.VITE_WS_URL ?? ""; if (base) { const url = new URL("/ws/state", base); url.protocol = url.protocol === "https:" ? "wss:" : "ws:"; return url.toString(); } return ""; } function getStreamBase(): string | null { const base = import.meta.env.VITE_STREAM_BASE ?? ""; return base || null; } function StereoDotViewport() { return (
); } function Viewport({ streamUrl, cameraStream, viewportWidth, viewportHeight, state, connected, stereoDotTest, warpK1, warpK2, separationOffset, zoom, cameraVideoRef, }: { streamUrl: string | null; cameraStream: MediaStream | null; viewportWidth: number; viewportHeight: number; state: HudState; connected: boolean; stereoDotTest: boolean; warpK1: number; warpK2: number; separationOffset: number; zoom: number; /** When set (e.g. left viewport), this ref is assigned to the video element for frame upload to backend. */ cameraVideoRef?: React.RefObject; }) { const videoRef = useRef(null); const imgRef = useRef(null); const setVideoRef = useCallback( (el: HTMLVideoElement | null) => { videoRef.current = el; if (cameraVideoRef != null) (cameraVideoRef as React.MutableRefObject).current = el; }, [cameraVideoRef] ); const mesh = useMemo( () => buildWarpMesh(viewportWidth, viewportHeight, warpK1, warpK2), [viewportWidth, viewportHeight, warpK1, warpK2] ); useEffect(() => { const el = videoRef.current; if (!cameraStream || !el) return; el.srcObject = cameraStream; return () => { if (el) el.srcObject = null; }; }, [cameraStream]); if (stereoDotTest) { return ; } const sourceRef = streamUrl ? imgRef : videoRef; return (
{streamUrl && ( <> } zoom={zoom} uvOffsetX={separationOffset} className="absolute inset-0 w-full h-full" /> )} {cameraStream != null && streamUrl == null && ( <>