# Project export: HorizonX This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: HorizonX is a dual-purpose platform using !!on-device!! VLMs to empower blind people with real-time haptic guider while crowdsourcing anonymized data to help governments improve public safety. - Devpost: https://devpost.com/software/horizonx - GitHub: https://github.com/bsflll/treehack2025 - Video: https://www.youtube.com/embed/dFowTJUub4U?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Tesla: Excellence Prize ($2k Tesla Store Gift Card [1st], $1k Gift Card [2nd], $1k Gift Card [3rd])) - Team: 3 GitHub contributor(s) — Rajan Agarwal (12 commits), zian1117 (3 commits), bsflll (1 commits) ## Devpost submission (written by the team) ### Inspiration GPT 4o sucks!! It is built for the visually impaired, but only with wifi!! Inspired by the potential of Video Language Models (VLMs) , we hope to bridge accessibility gaps. Over 2.2 billion people globally live with vision impairments, we built HorizonX an AI camera, a privacy-first real-time "sight partner" on mobile phone that help the blind individuals navigate crowded places. At the same time, our AI allows people to report hazards (potholes, broken signage) to government simultaneously improving public spaces for everyone, which facilitate government's reconstruction efficiency and transparency. ### What it does HorizonX is a two-part ecosystem: For User - Mobile App for Visually Impaired Users: Narrates surroundings in real time via camera (e.g., “A cyclist approaching from your left”). Guides navigation with obstacle detection and step-by-step audio cues. Reads text aloud (menus, signs, currency) and identifies products. For Governments - Centralized Platform for Public Safety Aggregates anonymized data from users to map hazards (potholes, broken signage) and crowds. Provides dashboards for infrastructure planning and emergency response. ### How we built it Running a VLM on a mobile device is a hefty task. After trying multiple approaches with MLX and WebGPU, we failed over and over to overcome the bandwidth and memory limits of our phones. We found a way to run a Linux emulator on the Galaxy S24 and mount it to local images, then installing core pytorch and tokenization libraries in a python environment to host a streaming API. We have a local endpoint hosted on the network to send the anonymized reported descriptions on a live map. This map also has information sourced from Sonar Research Pro calls to source live citizen complaints and create a centralized view. All of this is brought together with our locally hosted app, which—without connection—can navigate individuals around complex areas. ### Challenges we ran into Extreamly difficulty system integration MLX development of moondream in swift running moondream library: couldn’t install onnxruntime on Termux on Android finding smaller vlms: Moondream2B is by far the smallest model (other than Moondream0.5B) tried translating Moondream0.5B from custom ONNXVL wrapper to ONNX to Torch to serve directly from local safetensor, but would have to rewrite KV caching twice and serve custom KV caching also tried webGPU or running from browser, but don’t have enough RAM on Safari to do inference ### Accomplishments we're proud of Getting a multi-billion sized VLM to run on a mobile device! This is a significant milestone in local-first and privacy-focused AI inference. Since mostly it's impossible to run a 2B model on mobile phones. ### What we learned Vim on a phone is easier than on a laptop ### What's next Speed up model on edge devices by further quantization or better integration with the current software. And provide more humanistic service to the special community and better verification experience for government. Tesla Challenge Also, we participated in Tesla Challenge, here's our statement: 1) inspiration Combine LLMs and CV could combine to provide richer insights than traditional image-processing pipelines alone. By leveraging GPT-based models, we hoped to produce human-like, context-aware answers to driving-related questions, going beyond standard object detection or classification tasks. 2) code https://github.com/bsflll/treehack2025/tree/main/tesla-real-world-video-q-a 3) methodology Data: We sample frames at fixed intervals from short driving videos—here, five frames from a nominal 5-second clip. This approach provides a concise but representative snapshot of each scenario. Prompt: Descriptive Prompt by GPT-4o: Generates high-level contextual awareness of each video. Expert Opinions (Model “o1”): Requests multiple “expert answers” in parallel (e.g., five completions) to encourage diverse perspectives. Consolidation (Model “o1” / Final Step): Summarizes or decides on the best single answer. Minimal Answer Extraction: Ensures only the final letter (choice) is returned for submission or scoring. Asynchronous Execution Failure check at the end: 4) kaggle links Submission by Zian Shi ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 1677 recognized source files, 24843 KB. - C (language) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - PyTorch (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 2075) ``` .gitignore blind/.gitignore blind/app/components/CameraFeed.tsx blind/app/components/TtsService.tsx blind/app/globals.css blind/app/layout.tsx blind/app/page.tsx blind/backend.js blind/eslint.config.mjs blind/next.config.ts blind/package.json blind/postcss.config.mjs blind/README.md blind/tailwind.config.ts blind/tsconfig.json compute.py computed.json frontend/.gitignore frontend/app/api/complaints/route.ts frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/components.json frontend/components/city-map.tsx frontend/components/complaint-sidebar.tsx frontend/components/ui/button.tsx frontend/components/ui/scroll-area.tsx frontend/components/ui/sheet.tsx frontend/data/complaints.json frontend/eslint.config.mjs frontend/lib/utils.ts frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/tailwind.config.ts frontend/tsconfig.json horizonx/bin/activate horizonx/bin/activate.csh horizonx/bin/activate.fish horizonx/bin/Activate.ps1 horizonx/bin/fastapi horizonx/bin/httpx horizonx/bin/pip horizonx/bin/pip3 horizonx/bin/pip3.12 horizonx/bin/pyrsa-decrypt horizonx/bin/pyrsa-encrypt horizonx/bin/pyrsa-keygen horizonx/bin/pyrsa-priv2pub horizonx/bin/pyrsa-sign horizonx/bin/pyrsa-verify horizonx/bin/python horizonx/bin/python3 horizonx/bin/python3.12 horizonx/bin/uvicorn horizonx/database.py horizonx/horizonx.db horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/INSTALLER horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/LICENSE horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/METADATA horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/RECORD horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/REQUESTED horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/top_level.txt horizonx/lib/python3.12/site-packages/aiomysql-0.2.0.dist-info/WHEEL horizonx/lib/python3.12/site-packages/aiomysql/__init__.py horizonx/lib/python3.12/site-packages/aiomysql/_scm_version.pyi horizonx/lib/python3.12/site-packages/aiomysql/_version.py horizonx/lib/python3.12/site-packages/aiomysql/.gitignore horizonx/lib/python3.12/site-packages/aiomysql/connection.py horizonx/lib/python3.12/site-packages/aiomysql/cursors.py horizonx/lib/python3.12/site-packages/aiomysql/log.py horizonx/lib/python3.12/site-packages/aiomysql/pool.py horizonx/lib/python3.12/site-packages/aiomysql/sa/__init__.py horizonx/lib/python3.12/site-packages/aiomysql/sa/connection.py horizonx/lib/python3.12/site-packages/aiomysql/sa/engine.py horizonx/lib/python3.12/site-packages/aiomysql/sa/exc.py horizonx/lib/python3.12/site-packages/aiomysql/sa/result.py horizonx/lib/python3.12/site-packages/aiomysql/sa/transaction.py horizonx/lib/python3.12/site-packages/aiomysql/utils.py horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/INSTALLER horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/LICENSE horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/METADATA horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/RECORD horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/REQUESTED horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/top_level.txt horizonx/lib/python3.12/site-packages/aiopg-1.4.0.dist-info/WHEEL horizonx/lib/python3.12/site-packages/aiopg/__init__.py horizonx/lib/python3.12/site-packages/aiopg/connection.py horizonx/lib/python3.12/site-packages/aiopg/log.py horizonx/lib/python3.12/site-packages/aiopg/pool.py horizonx/lib/python3.12/site-packages/aiopg/py.typed horizonx/lib/python3.12/site-packages/aiopg/sa/__init__.py horizonx/lib/python3.12/site-packages/aiopg/sa/connection.py horizonx/lib/python3.12/site-packages/aiopg/sa/engine.py horizonx/lib/python3.12/site-packages/aiopg/sa/exc.py horizonx/lib/python3.12/site-packages/aiopg/sa/result.py horizonx/lib/python3.12/site-packages/aiopg/sa/transaction.py horizonx/lib/python3.12/site-packages/aiopg/sa/utils.py horizonx/lib/python3.12/site-packages/aiopg/utils.py horizonx/lib/python3.12/site-packages/annotated_types-0.7.0.dist-info/INSTALLER horizonx/lib/python3.12/site-packages/annotated_types-0.7.0.dist-info/licenses/LICENSE horizonx/lib/python3.12/site-packages/annotated_types-0.7.0.dist-info/METADATA horizonx/lib/python3.12/site-packages/annotated_types-0.7.0.dist-info/RECORD horizonx/lib/python3.12/site-packages/annotated_types-0.7.0.dist-info/WHEEL horizonx/lib/python3.12/site-packages/annotated_types/__init__.py horizonx/lib/python3.12/site-packages/annotated_types/py.typed horizonx/lib/python3.12/site-packages/annotated_types/test_cases.py horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/entry_points.txt horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/INSTALLER horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/LICENSE horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/METADATA horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/RECORD horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/top_level.txt horizonx/lib/python3.12/site-packages/anyio-4.8.0.dist-info/WHEEL horizonx/lib/python3.12/site-packages/anyio/__init__.py horizonx/lib/python3.12/site-packages/anyio/_backends/__init__.py horizonx/lib/python3.12/site-packages/anyio/_backends/_asyncio.py horizonx/lib/python3.12/site-packages/anyio/_backends/_trio.py horizonx/lib/python3.12/site-packages/anyio/_core/__init__.py [1955 more files omitted for size] ``` ### Dependencies - blind/package.json: @eslint/eslintrc@^3, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@15.1.7, kokoro-js@^1.2.0, next@15.1.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, react-hotkeys-hook@^4.6.1, tailwindcss@^3.4.1, typescript@^5 - frontend/package.json: @eslint/eslintrc@^3, @radix-ui/react-dialog@^1.1.6, @radix-ui/react-scroll-area@^1.2.3, @radix-ui/react-slot@^1.1.2, @tanstack/react-query@^5.66.0, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@15.1.7, lucide-react@^0.475.0, mapbox-gl@^3.10.0, next@15.1.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, tailwind-merge@^3.0.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 - horizonx/lib/python3.12/site-packages/rust/Cargo.toml: asn1, cfg-if@1, cryptography-cffi, cryptography-keepalive, cryptography-key-parsing, cryptography-openssl, cryptography-x509, cryptography-x509-verification, foreign-types-shared@0.1, once_cell@1, openssl@0.10.68, openssl-sys@0.9.104, pem@3, pyo3, self_cell@1 - horizonx/lib/python3.12/site-packages/rust/cryptography-cffi/Cargo.toml: cc@1.2.1, openssl-sys@0.9.104, pyo3 - horizonx/lib/python3.12/site-packages/rust/cryptography-keepalive/Cargo.toml: pyo3 - horizonx/lib/python3.12/site-packages/rust/cryptography-key-parsing/Cargo.toml: asn1, cfg-if@1, cryptography-x509, openssl@0.10.68, openssl-sys@0.9.104 - horizonx/lib/python3.12/site-packages/rust/cryptography-openssl/Cargo.toml: cfg-if@1, ffi@0.9.101, foreign-types@0.3, foreign-types-shared@0.1, openssl@0.10.68 - horizonx/lib/python3.12/site-packages/rust/cryptography-x509-verification/Cargo.toml: asn1, cryptography-key-parsing, cryptography-x509, once_cell@1, pem@3 - horizonx/lib/python3.12/site-packages/rust/cryptography-x509/Cargo.toml: asn1 - horizonx/requirements.txt: aiomysql@==0.1.1, aiopg@==1.4.0, alembic@==1.13.1, fastapi@==0.109.2, geoalchemy2@==0.10.0, httpx@==0.26.0, passlib@==1.7.4, psycopg2-binary@==2.9.9, pydantic@==2.6.1, python-dotenv@==1.0.1, python-jose@==3.3.0, python-multipart@==0.0.7, sqlalchemy@==2.0.27, uvicorn@==0.27.1, websockets@==12.0 - tesla-real-world-video-q-a/package.json: @mlc-ai/web-llm@^0.2.78 ### Recent commits (newest first) - remove api keys - cleanup - webgpu done - timing - phrasing - wav - starting - latency - new - fix - fix - frontend - Create tts.js - backend - 1 - basic 500m vlm ## Key source files (fetched from GitHub, selected and truncated for size) ### horizonx/lib/python3.12/site-packages/databases-0.9.0.dist-info/LICENSE.md ```markdown Copyright © 2019, [Encode OSS Ltd](https://www.encode.io/). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ``` ### horizonx/lib/python3.12/site-packages/idna-3.10.dist-info/LICENSE.md ```markdown BSD 3-Clause License Copyright (c) 2013-2024, Kim Davies and contributors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ``` ### tesla-real-world-video-q-a/package.json ``` { "dependencies": { "@mlc-ai/web-llm": "^0.2.78" } } ``` ### horizonx/requirements.txt ``` fastapi==0.109.2 uvicorn==0.27.1 websockets==12.0 pydantic==2.6.1 python-multipart==0.0.7 sqlalchemy==2.0.27 psycopg2-binary==2.9.9 alembic==1.13.1 python-jose==3.3.0 passlib==1.7.4 python-dotenv==1.0.1 httpx==0.26.0 aiopg==1.4.0 aiomysql==0.1.1 geoalchemy2==0.10.0 ``` ### blind/package.json ``` { "name": "blind", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "kokoro-js": "^1.2.0", "next": "15.1.7", "react": "^19.0.0", "react-dom": "^19.0.0", "react-hotkeys-hook": "^4.6.1" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@radix-ui/react-dialog": "^1.1.6", "@radix-ui/react-scroll-area": "^1.2.3", "@radix-ui/react-slot": "^1.1.2", "@tanstack/react-query": "^5.66.0", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.475.0", "mapbox-gl": "^3.10.0", "next": "15.1.7", "react": "^19.0.0", "react-dom": "^19.0.0", "tailwind-merge": "^3.0.1", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### horizonx/lib/python3.12/site-packages/rust/Cargo.toml ``` [package] name = "cryptography-rust" version.workspace = true authors.workspace = true edition.workspace = true publish.workspace = true rust-version.workspace = true [dependencies] once_cell = "1" cfg-if = "1" pyo3.workspace = true asn1.workspace = true cryptography-cffi = { path = "cryptography-cffi" } cryptography-keepalive = { path = "cryptography-keepalive" } cryptography-key-parsing = { path = "cryptography-key-parsing" } cryptography-x509 = { path = "cryptography-x509" } cryptography-x509-verification = { path = "cryptography-x509-verification" } cryptography-openssl = { path = "cryptography-openssl" } pem = { version = "3", default-features = false } openssl = "0.10.68" openssl-sys = "0.9.104" foreign-types-shared = "0.1" self_cell = "1" [features] extension-module = ["pyo3/extension-module"] default = ["extension-module"] [lib] name = "cryptography_rust" crate-type = ["cdylib"] [lints.rust] unexpected_cfgs = { level = "warn", check-cfg = ['cfg(CRYPTOGRAPHY_OPENSSL_300_OR_GREATER)', 'cfg(CRYPTOGRAPHY_OPENSSL_309_OR_GREATER)', 'cfg(CRYPTOGRAPHY_OPENSSL_320_OR_GREATER)', 'cfg(CRYPTOGRAPHY_IS_LIBRESSL)', 'cfg(CRYPTOGRAPHY_IS_BORINGSSL)', 'cfg(CRYPTOGRAPHY_OSSLCONF, values("OPENSSL_NO_IDEA", "OPENSSL_NO_CAST", "OPENSSL_NO_BF", "OPENSSL_NO_CAMELLIA", "OPENSSL_NO_SEED", "OPENSSL_NO_SM4"))'] } ``` ### horizonx/lib/python3.12/site-packages/rust/cryptography-keepalive/Cargo.toml ``` [package] name = "cryptography-keepalive" version.workspace = true authors.workspace = true edition.workspace = true publish.workspace = true rust-version.workspace = true [dependencies] pyo3.workspace = true ``` ### horizonx/lib/python3.12/site-packages/rust/cryptography-x509/Cargo.toml ``` [package] name = "cryptography-x509" version = "0.1.0" authors = ["The cryptography developers "] edition = "2021" publish = false # This specifies the MSRV rust-version = "1.65.0" [dependencies] asn1.workspace = true ``` ### horizonx/lib/python3.12/site-packages/rust/cryptography-cffi/Cargo.toml ``` [package] name = "cryptography-cffi" version.workspace = true authors.workspace = true edition.workspace = true publish.workspace = true rust-version.workspace = true [dependencies] pyo3.workspace = true openssl-sys = "0.9.104" [build-dependencies] cc = "1.2.1" [lints.rust] unexpected_cfgs = { level = "warn", check-cfg = ['cfg(python_implementation, values("CPython", "PyPy"))'] } ``` [1518 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]