# Project export: VisionGuard: smart vision for smarter driving

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2025
- Tagline: Our AI-powered system analyzes in-car video footage to transcribe and interpret driver behavior in real time. By detecting safe/unsafe actions, we provide actionable insights to improve road safety.
- Devpost: https://devpost.com/software/visionguard-smart-vision-for-smarter-driving
- GitHub: https://github.com/JayYeung/tesla_kaggle
- Demo: https://docs.google.com/presentation/d/1I6-hDtITWCSTrmS5IM57R79F7E6Rf0M57GbeiiAiLEo/edit?usp=sharing
- Result: winner (Tesla: Excellence Prize ($2k Tesla Store Gift Card [1st], $1k Gift Card [2nd], $1k Gift Card [3rd]); Autonomy Grand Prize (6-months use of a Tesla Model 3 or Y w/ Supervised FSD))
- Team: 2 GitHub contributor(s) — JayYeung (9 commits), TheClassicTechno (2 commits)

## Devpost submission (written by the team)

### Inspiration

Driving in extreme conditions like icy roads and blizzards often leads to accidents due to poor visibility or slippery surfaces. Having personally experienced car accidents caused by vehicles losing control on ice, it’s clear how critical it is to ensure that drivers are cautious and aware. Similarly, my teammates from the crowded Bay Area face challenges with complex road systems and traffic, where crashes are just as frequent. This inspired us to develop VisionGuard, a system that leverages AI to analyze driver behavior and road conditions, ultimately helping drivers make safer decisions. Our goal is to build smarter driving tools that reduce accidents and save lives. What It Does VisionGuard leverages cutting-edge Vision-Language Models (VLMs) to analyze in-car video footage, transcribing and interpreting driver behavior with high precision. By detecting both safe and unsafe actions, VisionGuard generates actionable insights to encourage safer driving practices. Our system identifies critical situations such as: ✅ Distracted driving (e.g., phone use, drowsiness). ✅ Reckless maneuvers (e.g., sudden lane changes, aggressive driving). ✅ Hazardous environmental conditions (e.g., low visibility, icy roads). While we aimed for real-time inference, we found that current VLM models are too computationally expensive for low-latency performance. However, through optimizations, we significantly improved processing efficiency and benchmark accuracy, making this approach feasible for future deployment. Beyond individual drivers, VisionGuard’s insights can benefit: Insurance companies – Enhancing risk assessments and reducing fraud. Fleet managers – Monitoring driver behavior for safety compliance. Autonomous vehicle systems – Providing explainability layers for AI-driven decisions. Our AI-driven approach aims to make roads safer at scale. How We Built It Computer Vision Framework We leveraged state-of-the-art Vision-Language Models (VLMs), including Gemini Pro 2.0 and Pixtral (13B), optimizing them with inference-time techniques such as: Chain of Thought (CoT) Prompting: Inspired by OpenEMMA, guiding multi-step reasoning for complex driving scenarios. Ensembling Methods: Majority Voting (best-performing approach). Weighted Confidence Scores to prioritize high-certainty outputs. Self-Consistency Decoding (multiple runs, selecting the most consistent answer). Majority Voting (best-performing approach). Weighted Confidence Scores to prioritize high-certainty outputs. Self-Consistency Decoding (multiple runs, selecting the most consistent answer). Bias Mitigation for QA Tasks: Detected a bias toward first/last answer choices (A/D). Shuffled response orders and aggregated multiple generations to improve reliability. Detected a bias toward first/last answer choices (A/D). Shuffled response orders and aggregated multiple generations to improve reliability. Temperature Annealing: Dynamically adjusting sampling temperature based on uncertainty. Logit Smoothing: Preventing overconfident but incorrect predictions. Mixture of Depth (MoD): Reducing computation on low-complexity frames to optimize performance. Although these optimizations reduced inference time, achieving true real-time performance on large VLMs remains a challenge. AI-Powered Reasoning We integrated OpenAI’s GPT-4 API to enhance natural language reasoning. VisionGuard translates raw detections into human-readable feedback, such as: "You seem distracted—keep your eyes on the road." "Road conditions are hazardous—reduce your speed." Additionally, we implemented causal reasoning chains, allowing VisionGuard to explain why alerts were triggered, increasing transparency and user trust. Reinforcement Learning for VLM Fine-Tuning We experimented with GRPO (Generalized Reinforcement Policy Optimization) to fine-tune the Qwen VL-2B model, leveraging 4× A100 NVIDIA GPUs. Implementation Details: Used Hugging Face TRL library for RL training. Tensor sharding across GPUs to fit the model in memory. Trained on the NuScenes-QA dataset, similar to Tesla-provided data. Implemented reward shaping for deeper reasoning ability. Used Hugging Face TRL library for RL training. Tensor sharding across GPUs to fit the model in memory. Trained on the NuScenes-QA dataset, similar to Tesla-provided data. Implemented reward shaping for deeper reasoning ability. Despite our efforts, batch size constraints (1) and long training times (6+ hours) prevented meaningful improvements over API-based VLMs, reinforcing the importance of scaling laws for multimodal AI. Inference Pipeline & Bottlenecks We developed a high-throughput processing pipeline, but achieving real-time inference on large VLMs was infeasible due to: High computational cost per frame (especially for complex scenes). Memory bandwidth limitations when streaming video into large models. Latency bottlenecks in API-based reasoning (due to network-dependent processing). To improve performance, we explored: Frame Sampling & Preprocessing: Used event-based sampling to prioritize critical driving moments. Applied contrastive normalization for better video clarity. Used event-based sampling to prioritize critical driving moments. Applied contrastive normalization for better video clarity. Parallelized Inference: Asynchronous execution of multiple VLM instances. ONNX Runtime + TensorRT optimizations for acceleration. Asynchronous execution of multiple VLM instances. ONNX Runtime + TensorRT optimizations for acceleration. Edge Deployment Optimization: Investigated distillation-based lightweight VLMs for future on-device inference. Implemented multi-threaded execution to minimize bottlenecks. Investigated distillation-based lightweight VLMs for future on-device inference. Implemented multi-threaded execution to minimize bottlenecks. Challenges We Ran Into 1. Real-Time Inference Limitations We originally aimed for real-time inference but found that current VLM architectures are computationally too expensive to process full-resolution video at interactive speeds. Even with optimizations like tensor parallelism and ONNX acceleration, the latency was too high for real-time driver feedback. 2. GPU Memory Constraints Frequent OOM (Out-Of-Memory) errors when fine-tuning large VLMs locally. Used DeepSpeed, FP16 precision, and tensor sharding to fit models within 4× A100 GPUs. 3. API Integration Challenges Rate-limiting issues when interfacing with OpenAI and Google Gemini. Latency bottlenecks with cloud-based reasoning models. Accomplishments That We're Proud Of ✅ Functional Prototype: Built a working system that analyzes driving behavior and environmental hazards. ✅ VLM Fine-Tuning Exploration: Pushed the limits of reinforcement learning for multimodal AI. ✅ Pipeline Optimization: Developed low-latency video processing techniques, bringing VLM-powered driving intelligence closer to real-time feasibility. ✅ Cross-Disciplinary Collaboration: Our team bridged AI, reinforcement learning, and computer vision, tackling one of the hardest challenges in multimodal AI. What’s Next for VisionGuard 🚗 Improving Real-Time Feasibility – Exploring smaller, distilled VLMs for on-device deployment. 📊 Data Partnerships – Collaborating with insurance & fleet management to enhance risk prediction. 👁 Multimodal Expansion – Integrating LiDAR, GPS, and radar for enhanced environmental awareness. ⚡ Edge Optimization Research – Experimenting with efficient quantization strategies for mobile and embedded systems. Tech Stack 🔹 Vision-Language Models: Google Gemini, Mixtral AI (Pixtral 13B), OpenAI 🔹 Deep Learning Frameworks: Hugging Face Transformers, DeepSpeed, Torch 🔹 GPU Acceleration: CUDA, ONNX Runtime, TensorRT 🔹 Computer Vision: OpenCV

## README (from the GitHub repository)

## Inspiration

Driving in extreme conditions like icy roads and blizzards often leads to accidents due to poor visibility or slippery surfaces. Having personally experienced car accidents caused by vehicles losing control on ice, it’s clear how critical it is to ensure that drivers are cautious and aware. Similarly, my teammates from the crowded Bay Area face challenges with complex road systems and traffic, where crashes are just as frequent. This inspired us to develop VisionGuard, a system that leverages AI to analyze driver behavior and road conditions in real-time, ultimately helping drivers make safer decisions. Our goal is to build smarter driving tools that reduce accidents and save lives.

## What it does

VisionGuard is an AI-powered system that analyzes in-car video footage in real time to transcribe and interpret driver behavior. By detecting both safe and unsafe actions, the system generates actionable insights to promote safer driving practices. VisionGuard can identify critical situations such as distracted driving, reckless maneuvers, or risky environmental conditions and alert drivers immediately. The insights provided by VisionGuard can also be used by insurance companies, fleet managers, or autonomous vehicle systems to improve road safety.

## How we built it

Computer Vision Framework: Utilized fine-tuned cutting-edge VLM's such as Gemini Pro 2.0, Pixtral with inference-time techniques including Chain of Thought, ensembling, and few-shot learning. For chain of thought, we based our prompts off of those used in OpenEMMA, an open reproduction of Waymo's EMMA project. For ensembling, we utilized a simple majority voting technique which ended up working the best. Additionally, for question-answering, we found that the model tended to be biased towards the first and last options presented (A/D), so we combatted this by shuffling the answer choices and taking the most voted options. Utilizing advanced capabilities of the VLM, we also suppose functionalities like

AI-Powered Reasoning: The system employs OpenAI’s GPT-4 API for natural language interpretation of detected behaviors. By integrating AI reasoning, VisionGuard converts detecteUtilzd patterns into meaningful feedback for the driver, such as “You seem distracted” or “Road conditions require slower speeds.”

Reinforcement Learning: We experiemented with using GRPO to fine-tune our own VLM based off the Qwen VL-2B model, which was the largest we were able to fit on the provided 4xA100 Nvidia GPUs. Our idea was to improve VLM reasoning which is important for understanding many autonomous driving scenes, in a self-reflective manner similar to DeepSeek. We utilized the TRL library from Huggingface Transformers, employing tensor sharding to deploy the VLM across multiple GPUs. We utilized the NuScenes-QA dataset with num_generations = 64 for fine-tuning, which was very similar to the Tesla-provided data. However, we were only able to fit batch_size one and after running for over 6 hours, found that our model, because of it's limited size, performed worse than existing API-based benchmarks.

## Challenges we ran into

OOM Errors: During development, we faced many out-of-memory errors due to limited VRAM when trying to run models locally, as VLMs are large and expensive to train. We ended up optimizing inference via tensor sharding, distributing the weights of a single model amoungst multiple GPUs, using DeepSpeed which barely let us fit the model on device.
Working with APIs: Many team members were beginners when it came to integrating APIs. Debugging and resolving authentication issues with third-party tools took significant effort.
Collaborative Learning Curve: Some team members were new to working with LLMs (Large Language Models), so understanding how to process video reasoning and integrate it into a real-time pipeline required significant collaboration and experimentation.

## Accomplishments that we're proud of

We’re incredibly proud of creating a functional prototype that successfully analyzes video footage to identify risky driving behaviors in real-time. This required seamlessly integrating multiple technologies—computer vision, LLMs, and reinforcement learning—despite being a relatively new domain for our team. Another highlight was how well we collaborated as a team, with everyone stepping out of their comfort zones to learn and contribute to the project. We also managed to build a pipeline that works efficiently with low-latency feedback, ensuring the driver receives timely alerts for unsafe actions.

## What we learned

This project introduced us to working with video reasoning frameworks and applying machine learning models to real-time contexts. We learned how to integrate APIs effectively and leverage AI-powered reasoning to convert raw video data into actionable insights. Additionally, we gained hands-on experience with reinforcement learning techniques, helping us understand how to train and fine-tune vision-language models for specific use cases.

## What’s next for VisionGuard: Smart Vision for Smarter Driving

Real-Time Vehicle Integration: We’d love to integrate VisionGuard into systems like Tesla Autopilot to assist drivers in real-time scenarios.
Expanded Use Cases: Beyond detecting unsafe driving behavior, VisionGuard can be extended to monitor passenger behavior (e.g., ensuring child safety) or analyze external road conditions using dashcams.
Data Partnerships: Collaborating with insurance companies and fleet management systems could help improve their risk assessments and provide discounts to safer drivers.
Advanced Environmental Awareness: Incorporating more sensors (e.g., LiDAR or GPS data) to enhance environmental context understanding, such as icy roads or sudden obstacles.

## Tech Stack:

Google Gemini, Mixtral AI (Pixtral 13B), OpenAI, HuggingFace Transformers, DeepSpeed, Torch, CUDA, OpenCV,


## Detected evidence (automated analysis)

Indexed codebase: 2 recognized source files, 7 KB.
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (26 of 26)

```
.gitignore
baseline.csv
baseline.ipynb
benchmark_gemini_flash.ipynb
benchmark_gemini.ipynb
benchmark_openai.ipynb
combine.py
david.csv
gemini_pro_2.0.csv
inference-techniques copy.ipynb
julibaseline.ipynb
julibaseline2.ipynb
liquidai.ipynb
majority_voted_results.csv
mistral.ipynb
o1_baseline.csv
o4_baseline.csv
openai.ipynb
outputfile.csv
questions (1).csv
README.md
tesla-real-world-video-q-a/questions.csv
tesla-real-world-video-q-a/submission_sample.csv
test.ipynb
updated_questions.csv
zhipuai.ipynb
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- readme
- sdfjk;lj;jsdfjkajks;dfk;alsdfjkl;ajskdfaj;ksdfj;k
- sdfj
- Merge branch 'main' of https://github.com/JayYeung/tesla_kaggle
- fdj;
- mistral
- mistral
- random shufflign
- Merge branch 'main' of https://github.com/JayYeung/tesla_kaggle
- juli changes on mistral AI
- Merge branch 'main' of https://github.com/JayYeung/tesla_kaggle
- pushing openai
- sjdf;k
- Merge branch 'main' of https://github.com/JayYeung/tesla_kaggle
- pushing answers
- sdf
- bot
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### combine.py

```python
import pandas as pd
from collections import Counter

# Load the CSV files
file1 = pd.read_csv("o1_baseline.csv")
file2 = pd.read_csv("o4_baseline.csv")
file3 = pd.read_csv("gemini_pro_2.0.csv")

# Merge the three files on 'id'
merged = file1.merge(file2, on="id", suffixes=("_1", "_2")).merge(file3, on="id")
merged.rename(columns={"answer": "answer_3"}, inplace=True)

# Apply majority voting
def majority_vote(row):
    votes = [row["answer_1"], row["answer_2"], row["answer_3"]]
    return Counter(votes).most_common(1)[0][0]

merged["final_answer"] = merged.apply(majority_vote, axis=1)

# Select only relevant columns
result = merged[["id", "final_answer"]]

# Save the result
result.to_csv("majority_voted_results.csv", index=False)
```