# Project export: Sentinel.AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Sentinel predicts hazards before they occur—using world-model embeddings to forecast risk, decide actions, and stop accidents in real time. Enterprise-ready, low-latency predictive safety. - Devpost: https://devpost.com/software/sentinel-ai-ihufb0 - GitHub: https://github.com/timsinashok/ForeSight-Safety - Demo: https://github.com/timsinashok/cosmos-predict2.5 - Video: https://www.youtube.com/embed/-i42Y0hoLRI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Ashok Timsina (5 commits) ## Devpost submission (written by the team) ### Inspiration Three researchers walked into TreeHacks and picked the hardest problem we could find: predict the future. Not metaphorically. Literally. We wanted to build a system that watches the physical world and knows what's about to go wrong — before it happens. Every year, preventable accidents in warehouses, factories, and autonomous systems cost lives and billions of dollars. The technology to prevent them exists — it's just trapped inside massive research models that are too slow to matter. We decided to fix that. ### What it does Sentinel.AI is a predictive safety engine for the physical world. It takes live video from any environment — a warehouse floor, a construction site, an autonomous vehicle's feed — and forecasts hazards seconds into the future. When danger is forming, Gemini-powered agents take real, grounded actions: slowing robots, rerouting vehicles, alerting humans. Current systems react. Sentinel prevents. ### How we built it We forked NVIDIA Cosmos — a state-of-the-art world model — and did something NVIDIA hasn't done yet: we made it fast enough to save lives. The original Cosmos pipeline generates full future video. Beautiful, but 30 minutes per inference on DGX Spark (We call him Sparky✨). Useless for real-time safety. Our breakthrough: we ripped out the video generation head entirely. The model's internal representations — its compressed understanding of physics, motion, and spatial relationships — already contain everything needed to predict danger. We don't need to see the future. We just need to understand it. We attached a lightweight XGBoost classifier directly onto these future-aware embeddings. Then we connected Gemini agents as the decision layer — taking the predicted risk and choosing world-grounded actions: stop, slow, reroute, alert. End-to-end latency: under 1 second. Down from 30 minutes on Sparky. ### Challenges we ran into Cosmos was never designed for real-time inference. Making a research-grade world model run at edge speed meant rethinking the entire pipeline — not fine-tuning it, restructuring it. We also had to prove that embeddings alone (without generated video) retain enough predictive signal to classify risk accurately. THEY DO! Cross-domain generalization was another battle. We trained and tested on automotive collision data and deployed on warehouse scenarios — two very different visual worlds. Getting the representations to transfer required careful temporal feature design. ### Accomplishments we're proud of 30 minutes → under 1 second. That's not optimization. That's a paradigm shift. We proved that world models — the most powerful spatial reasoning systems in AI — can run on edge GPUs for safety-critical decisions. And we showed that Gemini agents can take those predictions and act on them in the real world, not just narrate what they see. ### What we learned The most expensive part of a world model — generating pixels — is also the part you don't need. Representations are the product. This insight applies far beyond safety: any application where you need fast, physics-aware reasoning (robotics, autonomous driving, industrial automation) can benefit from this pattern. We also learned that the gap between a research breakthrough and a deployable system is an engineering problem — and sometimes, the best engineering is knowing what to throw away. ### What's next This isn't a warehouse tool. It's infrastructure for any system that operates in the physical world. Autonomous vehicles. Robotic fleets. Construction sites. Surgical robots. Anywhere humans and machines share space, Sentinel can predict what's about to go wrong and prevent it. The future of safety isn't faster reactions. It's prediction. ## README (from the GitHub repository) Demo is here: https://timsinashok.github.io/Sentinel-AI/ ``` FYI This project was submitted by Ashok, Nils, and Parth for Tree Hacks 2026 (Feb 13 -16, Stanford) Sponsors Used: - NVIDIA - Gemini AI Backend Available at: https://github.com/timsinashok/cosmos-predict2.5 ``` # Sentinel AI ### Predicting hazards before they happen. **Sentinel AI** is a future-aware safety system built on top of **NVIDIA Cosmos**. Instead of labeling the *current frame* as safe/unsafe, it uses **predictive world modeling** to reason about **future states of the environment** and produce an **early risk score** — enabling proactive mitigation (alerts, slowdowns, reroutes, kill-switches) before a near-miss becomes an incident. **Major contribution:** we optimized the NVIDIA Cosmos world-model-based hazard predictor and increased **inference efficiency by ~1800x** compared to the base Cosmos pipeline by operating entirely in representation space and removing pixel-level generation. > Safety shift: **reactive perception → predictive prevention** --- ## What’s in this repo This repository contains an **interactive web demo UI** (Vite + React) that shows the Sentinel AI operator experience: - **Without Sentinel**: the system only “understands” what happened *after* the event (reactive VLM-style explanation). - **With Sentinel**: the system surfaces an **early warning** (time-to-hazard + severity) and a **sequence of mitigation actions** before the collision. The demo is driven by a short clip (`/0.0-14.0.mp4`) plus a lightweight simulation timeline that illustrates the intended end-to-end behavior. --- ## High-level idea (the real system) Most industrial safety stacks today do: - **Vision / VLM** → classify **current** frame as safe/unsafe Sentinel AI instead aims to do: - **World model (Cosmos)** → encode dynamics and forecast **future** states - Extract **future-aware latent representations** - **Early hazard classification** in representation space Conceptual pipeline: ```text video → latent embeddings (future-aware) → classifier → risk score + confidence → mitigation ``` --- ## What we engineered (systems thinking) Sentinel AI is designed as a **systems-level optimization**, not a pixel-generation demo. ### 1) Representation-only inference We keep the predictive signal but skip expensive pixel synthesis: - Encode frames/clips using Cosmos tokenizer / VAE `encode()` - **No diffusion head** - **No future video decoding** ### 2) Fast, reusable embeddings Latents are pooled into compact vectors that can be: - Fed into lightweight classifiers (LogReg / SVM / MLP / XGBoost) - Cached and reused to reduce repeated compute This supports model saving and reuse, and enables near-real-time deployment on edge-class GPUs. ### 3) Temporal signal without full video generation Temporal context comes from short snippets (e.g., last 3–5 seconds at low FPS), aggregated directly in embedding space to learn **risk trajectories**. Tradeoff (intentional): - **Less fidelity** - **Much lower latency** - **More actionable output** ## Why this is systems thinking Sentinel AI makes an explicit engineering tradeoff: we spend compute on **predictive risk scoring** (the part that triggers mitigation) instead of generating pixels. This makes world models **deployable for safety**, not just compelling for demos. --- ## Run the demo locally ### Prerequisites - **Node.js** \(recommended: 18+\) ### Install ```bash npm install ``` ### Start dev server ```bash npm run dev ``` Then open `http://localhost:3000`. ### Production build ```bash npm run build npm run preview ``` --- ## Assets (video + logo) - **Video**: the demo expects the clip to be available at **`/0.0-14.0.mp4`**. - For Vite, the simplest approach is to place it in **`public/0.0-14.0.mp4`** so it’s copied into the build output. - **Logo**: the header loads **`/logo.png`** \(place in `public/logo.png`\). --- ## Repository layout - `App.tsx`: main UI + “With/Without Sentinel” mode logic - `components/ScenePanel.tsx`: video stage + overlays and analysis pause behavior - `components/RiskSummary.tsx`: risk level + time-to-hazard display - `components/AgentActionPanel.tsx`: mitigation action timeline - `constants.ts`: demo timings + initial entities - `types.ts`: shared types and enums --- ## Why this matters Predicting hazards *before* they happen can: - Reduce near-misses and injuries - Improve human–robot / forklift–pedestrian safety - Extend from warehouses to factories, construction sites, and autonomous environments Sentinel AI demonstrates how **world models can be adapted into deployable, safety-critical decision systems** by prioritizing **low-latency risk scoring** over pixel generation. --- ## Acknowledgements - Built for TreeHacks as a prototype UI + systems concept. - Inspired by NVIDIA Cosmos and the broader world-modeling ecosystem. ## Detected evidence (automated analysis) Indexed codebase: 16 recognized source files, 54 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (22 of 22) ``` .github/workflows/deploy-pages.yml .gitignore .gitmodules App.tsx components/AgentActionPanel.tsx components/RiskSummary.tsx components/ScenePanel.tsx components/TimelinePanel.tsx constants.ts index.html index.tsx LICENSE metadata.json package.json public/how-it-works.html public/index.css README.md test-simple.html tsconfig.json types.ts vite-env.d.ts vite.config.ts ``` ### Dependencies - package.json: @types/node@^22.14.0, @vitejs/plugin-react@^5.0.0, lucide-react@^0.564.0, react@^19.2.4, react-dom@^19.2.4, typescript@~5.8.2, vite@^6.2.0 ### Recent commits (newest first) - Update README with demo link and submission info - Revise project details and sponsors in README - added our contribution - added backend submodule so things are visible - remaining changes - Add backend URL to README - addd workflow - updated to vite project - added sponsors and the origin - added how it works and others - changes until now, frontend logo - Add MIT License to the project - added pipeline for with sentinel - without sentinel page fixed and added - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "sentinel:-predictive-safety-system", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview" }, "dependencies": { "lucide-react": "^0.564.0", "react": "^19.2.4", "react-dom": "^19.2.4" }, "devDependencies": { "@types/node": "^22.14.0", "@vitejs/plugin-react": "^5.0.0", "typescript": "~5.8.2", "vite": "^6.2.0" } } ``` ### index.tsx ```typescript import React from 'react'; import ReactDOM from 'react-dom/client'; import App from './App'; const rootElement = document.getElementById('root'); if (!rootElement) throw new Error('Failed to find the root element'); const root = ReactDOM.createRoot(rootElement); root.render( ); ``` ### App.tsx ```typescript import React, { useState, useEffect, useRef } from 'react'; import { Play, RotateCcw, Zap, LayoutDashboard, Settings, Video as VideoIcon } from 'lucide-react'; import ScenePanel from './components/ScenePanel'; import RiskSummary from './components/RiskSummary'; import AgentActionPanel from './components/AgentActionPanel'; import { RiskLevel, MitigationAction, SystemMode } from './types'; import { INITIAL_ENTITIES, SENTINEL_DETECT_AT_SEC, SENTINEL_KILL_SWITCH_AT_SEC, SENTINEL_NOTIFY_SUPERVISOR_AT_SEC, SENTINEL_MITIGATE_AT_SEC, SENTINEL_OBSERVE_OVERLAY_AT_SEC, SENTINEL_VIDEO_START_DELAY_SEC, } from './constants'; const App: React.FC = () => { const baseUrl = import.meta.env.BASE_URL; // --- State --- const [simulationActive, setSimulationActive] = useState(false); // Default to "Without Sentinel" per demo flow const [systemMode, setSystemMode] = useState('PASSIVE'); const [elapsedTime, setElapsedTime] = useState(0); const scenarioDurationSec = systemMode === 'PASSIVE' ? 14 : 6; // UI State const [riskLevel, setRiskLevel] = useState(RiskLevel.SAFE); const [timeToHazard, setTimeToHazard] = useState(null); const [actions, setActions] = useState([]); const startTimeRef = useRef(); const requestRef = useRef(); // --- Logic Loop --- useEffect(() => { if (!simulationActive) { setRiskLevel(RiskLevel.SAFE); setTimeToHazard(null); setActions([]); return; } const videoTimeSec = systemMode === 'ACTIVE' ? Math.max(0, elapsedTime - SENTINEL_VIDEO_START_DELAY_SEC) : elapsedTime; const predictedCollisionTimeSec = systemMode === 'ACTIVE' ? 7.0 : 3.2; const remainingTime = predictedCollisionTimeSec - videoTimeSec; // "Without Sentinel" Logic if (systemMode === 'PASSIVE') { if (remainingTime <= 0.5 && remainingTime > -2.0) { setRiskLevel(RiskLevel.COLLISION); setTimeToHazard(0); } else { setRiskLevel(RiskLevel.SAFE); setTimeToHazard(null); } } // "With Sentinel" Logic else { // Risk becomes meaningful at detection time const detected = videoTimeSec >= SENTINEL_DETECT_AT_SEC && remainingTime > 0; setRiskLevel(detected ? RiskLevel.CRITICAL : RiskLevel.SAFE); setTimeToHazard(detected ? remainingTime : null); // Right panel: ONLY the two final decisions, checked sequentially const next: MitigationAction[] = []; const push = (id: string, timestamp: number, action: string, targetId: string) => { next.push({ id, timestamp, action, targetId, rationale: '', status: 'EXECUTED' }); }; if (videoTimeSec >= SENTINEL_KILL_SWITCH_AT_SEC) { push('d1', SENTINEL_KILL_SWITCH_AT_SEC, 'KILL_SWITCH', 'F-12'); } if (videoTimeSec >= SENTINEL_NOTIFY_SUPERVISOR_AT_SEC) { push('d2', SENTINEL_NOTIFY_SUPERVISOR_AT_SEC, 'NOTIFY_SUPERVISOR', 'SHIFT-LEAD'); } setActions((prev) => { const prevIds = prev.map((a) => a.id).join(','); const nextIds = next.map((a) => a.id).join(','); return prevIds === nextIds ? prev : next; }); } }, [elapsedTime, simulationActive, systemMode]); // --- Animation/Timer Loop --- const animate = (time: number) => { if (startTimeRef.current === undefined) startTimeRef.current = time; const delta = (time - startTimeRef.current) / 1000; // Without Sentinel: play full 14s clip // With Sentinel: keep a tight 6s demo window (no hard pause) if (delta >= scenarioDurationSec) { setElapsedTime(scenarioDurationSec); return; } setElapsedTime(delta); if (simulationActive) requestRef.current = requestAnimationFrame(animate); }; useEffect(() => { if (simulationActive) requestRef.current = requestAnimationFrame(animate); return () => { if (requestRef.current) cancelAnimationFrame(requestRef.current); }; }, [simulationActive]); // --- Handlers --- const handleToggle = (mode: SystemMode) => { setSystemMode(mode); resetSimulation(); }; const runPrediction = () => { setSimulationActive(true); startTimeRef.current = undefined; }; const resetSimulation = () => { setSimulationActive(false); setElapsedTime(0); setActions([]); setRiskLevel(RiskLevel.SAFE); setTimeToHazard(null); }; return (
{/* --- COMMAND BAR (Header) --- */}
{/* Brand */}
Sentinel

Sentinel.ai

Autonomous Safety OS v2.4
{/* Center: How it works */} Understand how it works {/* Global Tools */}
``` ### vite.config.ts ```typescript import path from 'path'; import { defineConfig, loadEnv } from 'vite'; import react from '@vitejs/plugin-react'; export default defineConfig(({ mode }) => { const env = loadEnv(mode, '.', ''); return { // GitHub Pages friendly (relative asset URLs) base: './', server: { port: 3000, host: '0.0.0.0', }, plugins: [react()], define: { 'process.env.API_KEY': JSON.stringify(env.GEMINI_API_KEY), 'process.env.GEMINI_API_KEY': JSON.stringify(env.GEMINI_API_KEY) }, resolve: { alias: { '@': path.resolve(__dirname, '.'), } } }; }); ``` ### test-simple.html ```html Test Video ``` ### types.ts ```typescript export enum RiskLevel { SAFE = 'SAFE', WARNING = 'WARNING', CRITICAL = 'CRITICAL', COLLISION = 'COLLISION' } export enum EntityType { FORKLIFT = 'FORKLIFT', HUMAN = 'HUMAN', ROBOT = 'ROBOT' } export type SystemMode = 'PASSIVE' | 'ACTIVE'; export interface Position { x: number; y: number; } export interface Entity { id: string; type: EntityType; position: Position; velocity: Position; // Vector heading: number; // Degrees path: Position[]; // Future path } export interface MitigationAction { id: string; timestamp: number; action: string; targetId: string; rationale: string; status: 'PENDING' | 'EXECUTED'; } export interface SimulationState { isPlaying: boolean; currentTime: number; // 0 to 100 (percentage of scenario) riskLevel: RiskLevel; timeToHazard: number | null; // Seconds entities: Entity[]; actions: MitigationAction[]; } ``` ### constants.ts ```typescript import { Entity, EntityType } from './types'; // Simulation Constants export const SIMULATION_DURATION_SEC = 10; export const HAZARD_THRESHOLD_SEC = 3.2; export const PREDICTION_WINDOW_SEC = 5; // Demo timing (With Sentinel) export const SENTINEL_VIDEO_START_DELAY_SEC = 0.75; // Story beats (relative to video time, after the start delay) export const SENTINEL_OBSERVE_OVERLAY_AT_SEC = 3.0; export const SENTINEL_DETECT_AT_SEC = 3.6; export const SENTINEL_MITIGATE_AT_SEC = 4.1; // Decision timings (relative to video time) export const SENTINEL_KILL_SWITCH_AT_SEC = 4.5; export const SENTINEL_NOTIFY_SUPERVISOR_AT_SEC = 4.9; // Colors - Redesigned for Light Mode / High Visibility overlay export const COLORS = { bg: '#ffffff', grid: '#e5e7eb', // Light gray grid text: '#111827', // Black text textHighlight: '#111827', // High contrast bounding box colors forklift: '#2563EB', // Bright Blue human: '#ffffff', // White path: '#94a3b8', prediction: '#F59E0B', // Amber hazard: '#DC2626', // Deep Red safe: '#10B981', // Emerald }; // Initial Entities State (Static positions for now as requested) export const INITIAL_ENTITIES: Entity[] = [ { id: 'Trigger Kill Switch MCP', type: EntityType.FORKLIFT, position: { x: 30, y: 55 }, velocity: { x: 1.5, y: 0 }, heading: 90, path: [{ x: 30, y: 55 }, { x: 90, y: 55 }], }, { id: 'H-04', type: EntityType.HUMAN, position: { x: 65, y: 20 }, velocity: { x: 0, y: 1.2 }, heading: 180, path: [{ x: 65, y: 20 }, { x: 65, y: 90 }], } ]; ``` ### index.html ```html Sentinel | Predictive Warehouse Safety
``` ### components/TimelinePanel.tsx ```typescript // File removed - Logic moved to inline timeline in App.tsx for simpler layout ``` [6 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]