# Project export: BrailleAI : A TwoWay Voice for the Deaf & Blind

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2026
- Tagline: BrailleAI gives DeafBlind people a voice. Type by touch to speak out loud, and feel replies as Braille. It even turns the speaker's emotions into dots you can feel. Affordable and pocket-sized.
- Devpost: https://devpost.com/software/brailleai-a-two-way-voice-for-the-deafblind
- GitHub: https://github.com/lili1501/Refreshable_Braille_Display
- Video: https://www.youtube.com/embed/iX5XIWncCkA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Best Use of Orkes)
- Team: 4 GitHub contributor(s) — Shesadree Priyadarshani (16 commits), Shubhanflash22 (3 commits), SANKALPA HOTA (2 commits), Devin AI (2 commits)

## Devpost submission (written by the team)

### Inspiration

Imagine wanting to say something as simple as "I'm hungry" or "I love you" — but not being able to. For many DeafBlind people, every conversation depends on having another person there to interpret. The tools that could help, called refreshable Braille displays, can cost thousands of dollars. That price tag locks too many people out of something the rest of us take for granted: the freedom to talk. We didn't think that was fair. So we set out to build one small, affordable device that could be a voice, a teacher, and a friend — all controlled by touch.

### What it does

BrailleAI lets a DeafBlind person have a real, two-way conversation with anyone. Talk to anyone: You type with your fingers in Braille, and the device speaks your words out loud. Feel the reply: When someone answers, the device listens, shortens their words with AI, and raises them as Braille dots you can feel. Feel the emotion too: AI senses whether the speaker sounded happy, worried, or excited — and adds a small symbol so you don't just read the words, you feel the feeling behind them. Learn Braille: A patient AI tutor gives practice and instant feedback. Ask an AI: A quiet, touch-only assistant that answers your questions privately.

### How we built it

An ESP32-S3 chip handles voice input and output, while a Raspberry Pi running QNX (a rock solid real time operating system) powers on-device face recognition so the product knows who it's talking with. Two Braille cells are driven by just four small servo motors and 3D-printed sliding parts — the dots stay raised on their own, using almost no power. You type using a simple chord keyboard, like the ones blind students already know. We used AI for everything that makes it feel human. Deepgram/Whisper lets it hear speech, Google's voice lets it speak, and Anthropic's Claude understands tone, summarizes replies, and teaches. For fast, private on-device inference, we ran a neuron model trained on AWS Annapurna (Trainium/Inferentia) silicon from the workshop — so the core intelligence runs right on the product. This was as much an AI-assisted build as it was a hardware build. We used Sai by Simular as our end-to-end automation agent to drive the whole workflow, and Devin as our lead multi-agent partner for hardware–software co-design — helping us shape the circuitry, firmware, and code together instead of in silos. Alongside them we leaned on tools like Cognition, Orkes, Unify, Groq, and ngrok to move fast.

### Challenges we ran into

The hardest part was making Braille affordable. Real Braille displays use tiny, expensive motors. We replaced them with cheap servos and clever 3D printed sliders, which meant a lot of trial, error, and re-printing until each dot rose perfectly. Teaching the AI to turn emotion into a single touchable symbol took many tries to get right.

### What we learned

We learned that good technology isn't about being complicated; it's about removing a wall between people. We also learned a lot about hardware timing, AI speech processing, and how small design choices can make a big difference in someone's daily life.

### What's next

More Braille cells so users can read full sentences at once, support for shorthand Braille, and an offline mode so BrailleAI works anywhere, no internet needed.

## README (from the GitHub repository)

# BrailleAI — Emotion-Aware Braille Communicator

A bidirectional communication aid for **DeafBlind** users that pairs a
**refreshable Braille device** with an **audio-visual emotion model**. When a
hearing person speaks, the device shows a tactile **emotion prefix** + a short
**Braille summary**; the user types back on a Perkins keyboard and is **spoken
aloud**.

This repository is a **monorepo** with two subprojects:

| Folder | What it is |
|--------|-----------|
| [`AV/`](AV/) | **Emotion recognition** — trains the audio+video model (CREMA-D) that produces `ft_best.pt`. |
| [`Braille TTS and STT/`](Braille%20TTS%20and%20STT/) | **The BrailleAI device** — ESP32-S3 firmware + Raspberry Pi service that *uses* that model. |

```
Actual Project/
├── AV/                       # CREMA-D emotion recognition (training + inference)
│   ├── final_1.py            #   train + save the deployable model
│   ├── final_2.py            #   interval inference + ground-truth scoring
│   ├── Crema_run_final.py    #   final pipeline (train + live/video inference)
│   ├── iter1/ iter2/ iter3/  #   the three research iterations (+ result graphs)
│   └── README_CREMA_Emotion.md
│
├── Braille TTS and STT/      # The communicator device
│   ├── esp32_firmware/       #   Arduino sketch for the ESP32-S3
│   ├── raspberry_pi/         #   Python: emotion_inference.py + reference sim
│   └── README.md             #   device build/run guide
│
├── BrailleAI_Project_Outline.pdf
├── README.md                 # (this file)
└── .gitignore
```

---

## How the two halves connect

```
   AV/  (train on a GPU)                 Braille TTS and STT/  (the device)
 ┌─────────────────────────┐          ┌───────────────────────────────────────┐
 │ final_1.py / iter3       │  ft_best │ raspberry_pi/emotion_inference.py      │
 │  AVEmotionNet            │ ───.pt──▶│  loads the model, reads camera+mic     │
 │  (wav2vec2 + ViT)        │          │  └─ "EMOTION:happy:0.62\n" via UART ──▶ │
 └─────────────────────────┘          │ esp32_firmware/  → tactile prefix +     │
                                       │   Braille text on the servo cell        │
                                       └───────────────────────────────────────┘
```

The emotion classes (`anger, disgust, fear, happy, neutral, sad`) map to the
Braille prefixes the firmware flashes before each message.

---

## Quick start

**1. Train / obtain the emotion model** — see [`AV/README_CREMA_Emotion.md`](AV/README_CREMA_Emotion.md)
```bash
cd AV
python final_1.py     # trains and saves the model checkpoint
```
> The Raspberry Pi service can also **auto-download** a prebuilt `ft_best.pt`
> from Google Drive, so you don't have to retrain to run the device.

**2. Build the device** — see [`Braille TTS and STT/README.md`](Braille%20TTS%20and%20STT/README.md)
- Flash `esp32_firmware/` to the ESP32-S3 (Arduino IDE).
- Run `raspberry_pi/emotion_inference.py` on Raspberry Pi OS (Linux).
- Wire **Pi TX → ESP32 RX** + common GND (115200 baud).

---

## What is NOT in this repo (git-ignored)
To keep the repository light and under GitHub's file-size limits, large,
regenerable, or sensitive files are excluded (see `.gitignore`):
- **Model checkpoints / caches:** `*.pt`, `*.joblib`, `*.npy`, `*.npz`
  (e.g. `ft_best.pt`, iter3 audio cache, embeddings).
- **Media / data:** `*.mp4`, `*.wav` (e.g. `camera_test.mp4`, CREMA-D clips).
- **Secrets:** `secrets.h` (ESP32 Wi-Fi + API keys) — use `secrets.example.h`.
- **Logs:** `*.jsonl`, Python `__pycache__/`.

Result graphs (`*.png`) and metrics (`*_results.txt`) **are** kept so the
iteration results are visible.

## 🔐 Security
ESP32 secrets live in `Braille TTS and STT/esp32_firmware/secrets.h`
(git-ignored). The Claude/Deepgram keys and Wi-Fi password that were previously
committed in plaintext are **compromised** — revoke/rotate them and change the
Wi-Fi password before sharing.

## Credits
See `BrailleAI_Project_Outline.pdf` for the full hardware/software design.


## Detected evidence (automated analysis)

Indexed codebase: 34 recognized source files, 267 KB.
- C (language) — detected in the code
- C++ (language) — detected in the code
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (43 of 43)

```
.gitignore
AV/Crema_run_final.py
AV/final_1.py
AV/final_2.py
AV/iter1/iter1_classical_pipeline.py
AV/iter1/iter1_predictions.csv
AV/iter1/iter1_results.txt
AV/iter2/iter2_deep_embeddings.py
AV/iter2/iter2_predictions.csv
AV/iter2/iter2_results.txt
AV/iter3/iter3_clip_predictions.csv
AV/iter3/iter3_clip_results.txt
AV/iter3/iter3_finetune.py
AV/README_CREMA_Emotion.md
Braille TTS and STT/.gitignore
Braille TTS and STT/esp32_firmware/braille_correct.cpp
Braille TTS and STT/esp32_firmware/braille_correct.h
Braille TTS and STT/esp32_firmware/braille_grade2.cpp
Braille TTS and STT/esp32_firmware/braille_grade2.h
Braille TTS and STT/esp32_firmware/braille_input.cpp
Braille TTS and STT/esp32_firmware/braille_input.h
Braille TTS and STT/esp32_firmware/braille_output.cpp
Braille TTS and STT/esp32_firmware/braille_output.h
Braille TTS and STT/esp32_firmware/config.h
Braille TTS and STT/esp32_firmware/emotion_display.cpp
Braille TTS and STT/esp32_firmware/emotion_display.h
Braille TTS and STT/esp32_firmware/esp32_firmware.ino
Braille TTS and STT/esp32_firmware/mic_stt.cpp
Braille TTS and STT/esp32_firmware/mic_stt.h
Braille TTS and STT/esp32_firmware/secrets.example.h
Braille TTS and STT/esp32_firmware/speaker_tts.cpp
Braille TTS and STT/esp32_firmware/speaker_tts.h
Braille TTS and STT/raspberry_pi/_dbg.py
Braille TTS and STT/raspberry_pi/ai_backend.py
Braille TTS and STT/raspberry_pi/braille_core.py
Braille TTS and STT/raspberry_pi/cloud_sync.py
Braille TTS and STT/raspberry_pi/device.py
Braille TTS and STT/raspberry_pi/display_driver.py
Braille TTS and STT/raspberry_pi/emotion_inference.py
Braille TTS and STT/raspberry_pi/input_handler.py
Braille TTS and STT/raspberry_pi/run_tests.py
Braille TTS and STT/README.md
README.md
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Add files via upload
- Delete AV/final_2.py
- Add files via upload
- Delete AV/final_2.py
- Updating
- Merge pull request #1 from lili1501/devin/1782046388-emotion-pi-integration
- Add auto-download of ft_best.pt from Google Drive
- Add Raspberry Pi emotion recognition integration
- Add input_handler.py (Perkins chord input) — fixes missing module
- Remove README-bsny.md
- Add run_tests.py (Cal Hacks)
- Add README-bsny.md (Cal Hacks)
- Add display_driver.py (Cal Hacks)
- Add device.py (Cal Hacks)
- Add cloud_sync.py (Cal Hacks)
- Add braille_core.py (Cal Hacks)
- Add ai_backend.py (Cal Hacks)
- Add _session_log.jsonl (Cal Hacks)
- Add _dbg.py (Cal Hacks)
- Add _device_log.jsonl (Cal Hacks)

## Key source files (fetched from GitHub, selected and truncated for size)

### AV/README_CREMA_Emotion.md

```markdown
# CREMA-D Emotion Recognition — Audio + Video (Three Approaches)

Predicts the **6 emotions** in the
[CREMA-D dataset](https://github.com/CheyneyComputerScience/CREMA-D)
(anger, disgust, fear, happy, neutral, sad) by **fusing audio and video**.
CREMA-D is audio-visual: every clip exists as both `AudioWAV/<id>.wav` and
`VideoFlash/<id>.flv`, and the label is parsed from the filename.

This project is a **progression of three self-contained scripts** (one per
`iterN/` folder), from hand-crafted features to fully fine-tuned deep models:

| # | Folder / script | Approach | Backbones | Test acc | Test macro-F1 |
|---|-----------------|----------|-----------|---------:|--------------:|
| 1 | `iter1/iter1_classical_pipeline.py` | Classical feature-level fusion | hand-crafted (librosa/Parselmouth + MediaPipe) | **0.751** | **0.749** |
| 2 | `iter2/iter2_deep_embeddings.py` | Frozen deep-embedding fusion | wav2vec2 + ViT (frozen) | **0.801** | **0.801** |
| 3 | `iter3/iter3_finetune.py` | **End-to-end fine-tuning** | wav2vec2 + ViT (trainable) | **0.873** | **0.873** |

All numbers above use the **same clip-level stratified split** — 7,442 clips →
**train 5,209 / val 1,116 / test 1,117** (seed 42), with scaling/normalization
fit on **train only**. Chance level = 1/6 ≈ 16.7%. **Macro-F1** is the headline
metric (all six emotions weighted equally).

> **Final progression: 0.749 → 0.801 → 0.873 macro-F1.** Fine-tuning both
> backbones end-to-end (script 3) is the best model, run on a B200 GPU.

A fourth script, **`Crema_run_final.py`** (in the `AV/` root), is the **final
deployable pipeline**: it trains the best (0.873) fine-tuned model *and then
applies it* to brand-new inputs — a **prerecorded video file**, a **live webcam +
microphone** stream, or **both**.

---

## Project layout

Each iteration lives in its **own folder containing both the code and all its
outputs**, so reruns stay self-contained:

```
AV/
├── iter1/                       # Iteration 1 — classical fusion
│   ├── iter1_classical_pipeline.py
│   └── iter1_features.csv, iter1_*.png, iter1_results.txt, iter1_model.joblib, iter1_predictions.csv
├── iter2/                       # Iteration 2 — frozen deep embeddings
│   ├── iter2_deep_embeddings.py
│   └── iter2_embeddings.npz, iter2_*.png, iter2_results.txt, iter2_model.joblib, iter2_predictions.csv
├── iter3/                       # Iteration 3 — end-to-end fine-tuning (BEST)
│   ├── iter3_finetune.py
│   └── iter3_<tag>_model.pt, iter3_<tag>_*.png, iter3_<tag>_results.txt, iter3_*.npy/.npz
├── Crema_run_final.py           # Final deployable pipeline (train + live/video inference)
└── README_CREMA_Emotion.md
```

Each script's `OUT_DIR` already points at its own `iterN/` folder — just run it.

---

## Common setup

### Labels (from the filename)
`ActorID_Sentence_Emotion_Intensity.wav`, e.g. `1001_DFA_ANG_XX.wav`:

| Token | Example | Used as |
|-------|---------|---------|
| ActorID | `1001` | grouping key (for the actor-level split in scr
[truncated — 11043 more characters]
```

### AV/final_1.py

```python
"""
================================================================================
FINAL_1 — TRAIN THE AUDIO+VIDEO EMOTION MODEL AND SAVE IT
================================================================================

This script does ONE job: fine-tune the proven CREMA-D emotion model
(wav2vec2 audio + ViT video, mean-pooled, fused) and save the trained
checkpoint to disk. Use `final_2.py` afterwards to apply it to a recorded
video + audio and score it against a ground-truth file.

    PART 1 — one-time cache of frames + audio
    PART 2 — fine-tune (train/val), save best checkpoint by val macro-F1
    PART 3 — quick TEST metric (sanity check)

The saved checkpoint (`final_output/final_model.pt`) stores the weights AND all
config needed to rebuild the model for inference (classes, backbones, n_frames,
img_size, sample_rate, max_audio_s).

--------------------------------------------------------------------------------
DEPENDENCIES
    pip install torch torchvision transformers pillow opencv-python==4.10.0.84 \
                librosa soundfile scikit-learn joblib pandas "numpy<2"
--------------------------------------------------------------------------------
"""

import os
import re
import glob
import time
import math
import warnings
from pathlib import Path

import numpy as np
import librosa
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

from transformers import AutoModel, AutoFeatureExtractor, AutoImageProcessor
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score

import cv2
warnings.filterwarnings("ignore")


# ================================================================== #
#  CONFIG                                                            #
# ================================================================== #

AUDIO_DIR = r"C:\Users\shubh\Desktop\archive\content\CREMA-D\AudioWAV"
VIDEO_DIR = r"C:\Users\shubh\Desktop\archive\content\CREMA-D\VideoFlash"
OUT_DIR   = r"C:\Users\shubh\Desktop\Hard disk\College(PG)\Non Academic at UCSD\Hackathon\Berkeley June 20-21\Actual Project\AV\final_output"

# Proven config (TEST macro-F1 = 0.873 on the clip-level split).
AUDIO_MODEL = "superb/wav2vec2-base-superb-er"
VIDEO_MODEL = "trpakov/vit-face-expression"

SAMPLE_RATE  = 16000
MAX_AUDIO_S  = 5            # pad/truncate audio to this many seconds
N_FRAMES     = 8           # frames sampled per clip
IMG_SIZE     = 224

EPOCHS       = 20
BATCH        = 48          # lower to 32/24 if you hit OOM
LR_BACKBONE  = 1e-5
LR_HEAD      = 1e-3
WEIGHT_DECAY = 0.01
WARMUP_FRAC  = 0.1
LABEL_SMOOTH = 0.05
GRAD_CLIP    = 1.0
NUM_WORKERS  = 0           # Windows: 0 ; Linux: raise to #cores
FREEZE_AUDIO_CNN = True
PATIENCE     = 5           # early stop on val macro-F1

LIMIT        = 0           # 0 = all clips; e.g. 400 for a quick smoke test
RANDOM_SEED  = 42
TRAIN_FRAC, VAL_FRAC, TEST_FRAC = 0.70, 0.15, 0.15

EMOTION_MAP = {"ANG": "anger", "DIS": "disgust", "FEA": "fear",
               "HAP": "happy", "NEU": "neutral", "SAD": "sad"}

os.makedirs(OUT_DIR, exist_ok=True)
np.random.seed(RANDOM_SEED)
torch.manual_seed(RANDOM_SEED)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
BF16 = DEVICE == "cuda" and torch.cuda.is_bf16_supported()
AMP_DTYPE = torch.bfloat16 if BF16 else torch.float16
MAX_AUDIO_LEN = SAMPLE_RATE * MAX_AUDIO_S
MODEL_PATH = os.path.join(OUT_DIR, "final_model.pt")


# ================================================================== #
#  PRE-PROCESSORS                                                    #
# ================================================================== #

audio_fe = AutoFeatureExtractor.from_pretrained(AUDIO_MODEL)
img_proc = AutoImageProcessor.from_pretrained(VIDEO_MODEL)
IMG_MEAN = torch.tensor(img_proc.image_mean).view(1, 1, 3, 1, 1)
IMG_STD  = torch.tensor(img_proc.image_std).view(1, 1, 3, 1, 1)


def frames_from_video(vpath, n_frames=N_FRAMES):
    frames = np.zeros((n_frames, IMG_SIZE, IMG_SIZE, 3), dtype=np.uint8)
    if not vpath or not os.path.exists(vpath):
        return frames
    cap = cv2.VideoCapture(vpath)
    if not cap.isOpened():
        return frames
    total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 0
    grab = (np.linspace(0, total - 1, n_frames).astype(int)
            if total > 0 else np.arange(n_frames))
    got = []
    for gi in grab:
        cap.set(cv2.CAP_PROP_POS_FRAMES, int(gi))
        ret, fr = cap.read()
        if not ret:
            continue
        fr = cv2.cvtColor(fr, cv2.COLOR_BGR2RGB)
        got.append(cv2.resize(fr, (IMG_SIZE, IMG_SIZE)))
    cap.release()
    for k in range(n_frames):
        if got:
            frames[k] = got[min(k, len(got) - 1)]
    return frames


def audio_from_path(path):
    try:
        y, _ = librosa.load(path, sr=SAMPLE_RATE, mono=True)
    except Exception:
        y = np.zeros(SAMPLE_RATE, dtype=np.float32)
    wav = np.zeros(MAX_AUDIO_LEN, dtype=np.float32)
    L = int(min(len(y), MAX_AUDIO_LEN))
    wav[:L] = y[:L]
    return wav, max(1, L)


# ================================================================== #
#  PART 1 — BUILD/LOAD CACHE                                         #
# ================================================================== #

print("=" * 70)
print("PART 1 — BUILD/LOAD CACHE (frames + audio)")
print("=" * 70)
print(f"Device: {DEVICE} | bf16: {BF16}")

wav_files = sorted(glob.glob(os.path.join(AUDIO_DIR, "*.wav")))
if not wav_files:
    raise FileNotFoundError(f"No .wav files in {AUDIO_DIR}")
if LIMIT and LIMIT > 0:
    wav_files = wav_files[:LIMIT]

video_lookup = {}
for ext in ("*.flv", "*.mp4", "*.avi", "*.wmv"):
    for vp in glob.glob(os.path.join(VIDEO_DIR, ext)):
        video_lookup[Path(vp).stem] = vp

fname_re = re.compile(r"^(\d{4})_([A-Z]{3})_([A-Z]{3})_([A-Z]{2})", re.IGNORECASE)
items = []
for wav_path in wav_files:
    stem = Path(wav_path).stem
    m = fname_re.match(stem)
    if not m:
    
[truncated — 9891 more characters]
```

### AV/final_2.py

```python
"""
================================================================================
FINAL_2 — APPLY THE TRAINED MODEL TO A RECORDED VIDEO + (SEPARATE) AUDIO,
          SCORE IT AGAINST A GROUND-TRUTH FILE (5-second intervals)
================================================================================

Loads the checkpoint saved by `final_1.py`, walks a recorded video in
**5-second intervals**, and for each interval predicts the emotion using
**majority voting** over several overlapping sub-windows. It then compares the
per-interval predictions to a **ground-truth Excel file** and reports accuracy +
macro-F1 + a confusion matrix.

NOTE: video frames are read from VIDEO_PATH and audio is read from a SEPARATE
      AUDIO_PATH (the two were recorded simultaneously on different devices).
      Use AV_OFFSET_S to correct any small start-time misalignment.

HOW MAJORITY VOTING WORKS (per 5-second interval)
    The interval is split into N_SUBCLIPS overlapping sub-windows (each
    SUB_WIN_S seconds). The model predicts on each sub-window (its own audio
    slice + sampled frames). The interval's final label = the most frequent
    prediction across the sub-windows (ties broken by summed class probability).

GROUND-TRUTH FILE FORMAT (Excel .xlsx, 2 columns)
    interval        | Mood
    00:00 - 00:05   | Happy
    00:05 - 00:10   | Fear
    00:10 - 00:15   | Sad
    ...
    `interval` is an MM:SS - MM:SS time range (5 seconds each); `Mood` is one of:
    Anger, Disgust, Fear, Happy, Neutral, Sad (case-insensitive).
    (A dummy file `dummy_ground_truth.xlsx` is provided.)

--------------------------------------------------------------------------------
USAGE
    1) Run final_1.py first so final_output/final_model.pt exists.
    2) Set VIDEO_PATH to your recorded video clip (frames source).
    3) Set AUDIO_PATH to your separately-recorded audio (e.g. .wav/.m4a/.mp4).
    4) (Optional) Set AV_OFFSET_S to align the two recordings.
    5) Set GROUND_TRUTH_XLSX (defaults to the bundled dummy file).
    6) python final_2.py

DEPENDENCIES
    pip install torch transformers opencv-python==4.10.0.84 librosa soundfile \
                scikit-learn pandas openpyxl "numpy<2"   # openpyxl reads .xlsx
--------------------------------------------------------------------------------
"""

import os
import warnings
from collections import Counter

import numpy as np
import pandas as pd
import librosa

import torch
import torch.nn as nn
from transformers import AutoModel, AutoFeatureExtractor, AutoImageProcessor
from sklearn.metrics import accuracy_score, f1_score, confusion_matrix

import cv2
warnings.filterwarnings("ignore")

# Make the ffmpeg binary bundled with imageio-ffmpeg discoverable by
# librosa/audioread (they search PATH for an `ffmpeg` executable). This lets
# librosa decode compressed formats like .m4a/.mp4 without a system ffmpeg.
try:
    import imageio_ffmpeg
    _ffmpeg_exe = imageio_ffmpeg.get_ffmpeg_exe()
    _ffmpeg_dir = os.path.dirname(_ffmpeg_exe)
    os.environ["PATH"] = _ffmpeg_dir + os.pathsep + os.environ.get("PATH", "")
    os.environ.setdefault("IMAGEIO_FFMPEG_EXE", _ffmpeg_exe)
    # audioread looks for an executable literally named "ffmpeg"; the bundled one
    # is named e.g. ffmpeg-win-x86_64-v7.1.exe, so expose a plain "ffmpeg.exe".
    _ffmpeg_alias = os.path.join(_ffmpeg_dir, "ffmpeg.exe")
    if not os.path.exists(_ffmpeg_alias):
        try:
            import shutil
            shutil.copyfile(_ffmpeg_exe, _ffmpeg_alias)
        except Exception:
            pass
except Exception:
    pass


# ================================================================== #
#  CONFIG                                                            #
# ================================================================== #

AV_DIR = r"C:\Users\shubh\Desktop\Hard disk\College(PG)\Non Academic at UCSD\Hackathon\Berkeley June 20-21\Actual Project\AV"
MODEL_PATH       = os.path.join(AV_DIR, "ft_best.pt")
VIDEO_PATH       = os.path.join(AV_DIR, "see.mp4")        # <-- video (frames source)
AUDIO_PATH       = os.path.join(AV_DIR, "Sound.m4a")       # <-- SEPARATE audio recording
GROUND_TRUTH_XLSX = os.path.join(AV_DIR, "dummy_ground_truth.xlsx")  # <-- ground truth (Excel)

# Audio source mode:
#   "separate" (default) -> video frames from VIDEO_PATH, audio from AUDIO_PATH
#                           (use this when the camera has no microphone and you
#                            recorded video + audio simultaneously on 2 devices)
#   "combined"           -> both video frames AND audio come from VIDEO_PATH
#                           (use this when the video file already has an audio track)
AUDIO_MODE = "separate"   # "separate" or "combined"

# If the audio recording started a bit before/after the video, shift it here.
# Positive value => audio started LATER than video (audio is delayed relative to video).
# Negative value => audio started EARLIER than video.
AV_OFFSET_S = 0.0

INTERVAL_S   = 5      # length of each scoring interval (seconds)
SUB_WIN_S    = 3.0    # length of each voting sub-window (seconds)
N_SUBCLIPS   = 3      # number of sub-windows voted per interval

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
BF16 = DEVICE == "cuda" and torch.cuda.is_bf16_supported()
AMP_DTYPE = torch.bfloat16 if BF16 else torch.float16


# ================================================================== #
#  LOAD CHECKPOINT + REBUILD MODEL                                   #
# ================================================================== #

if not os.path.exists(MODEL_PATH):
    raise FileNotFoundError(
        f"Model not found: {MODEL_PATH}\nRun final_1.py first to train and save it.")

ckpt = torch.load(MODEL_PATH, map_location=DEVICE, weights_only=False)
CLASS_LABELS = list(ckpt["classes"])
N_CLASSES    = len(CLASS_LABELS)
AUDIO_MODEL  = ckpt["audio_model"]
VIDEO_MODEL  = ckpt["video_model"]
N_FRAMES     = int(ckpt["n_frames"])
IMG_SIZE     = int(ckpt["img_size"])
SAMPLE_RATE  = 
[truncated — 10246 more characters]
```

### Braille TTS and STT/raspberry_pi/run_tests.py

```python
"""run_tests.py - execute every module self-test in order."""
import runpy, sys
mods = ["braille_core","display_driver","input_handler","ai_backend","cloud_sync","device"]
for m in mods:
    print(f"\n========== {m} ==========")
    runpy.run_module(m, run_name="__main__")
print("\nALL MODULE SELF-TESTS COMPLETED OK")
```

### Braille TTS and STT/esp32_firmware/braille_input.h

```c
#pragma once
#include <Arduino.h>

void initBrailleKeyboard();

// Call this in loop(). Returns:
//   'a'-'z'  = detected Braille letter
//   ' '      = space
//   '\n'     = enter/send
//   '\b'     = backspace
//   'M'      = mode switch
//   0        = no input yet (chord still in progress or idle)
char readBrailleChord();

// Get the raw dot pattern of the last detected chord (bits 0-5 = dots 1-6)
uint8_t getLastChordPattern();

```

### Braille TTS and STT/raspberry_pi/_dbg.py

```python
"""_dbg.py - scratch trace used while debugging the chord state machine."""
from input_handler import ChordInput
from braille_core import char_to_dots, NUMBER_SIGN, CAPITAL_SIGN

emitted = []
ci = ChordInput(on_char=emitted.append)

def chord(dots, t):
    for d in dots:
        ci.key_down(d, t)
    ci.tick(t + ci.window_s)
    print("after", sorted(dots), "buf=", repr(ci.text))

t = 0.0
chord(CAPITAL_SIGN, t); t += 0.1
chord(char_to_dots("h"), t); t += 0.1
chord(char_to_dots("i"), t); t += 0.1
ci.space()
chord(NUMBER_SIGN, t); t += 0.1
chord(char_to_dots("4"), t); t += 0.1
chord(char_to_dots("2"), t); t += 0.1
print("FINAL", repr(ci.text))   # expect: Hi 42
```

### Braille TTS and STT/esp32_firmware/braille_output.h

```c
#pragma once
#include <Arduino.h>

void initBrailleDisplay();
void clearDisplay();

// Display a single character on the cell
void displayChar(char c);

// Display a string scrolling through one character at a time
void displayBrailleString(const char* text, int scrollIndex);

// Display a raw 6-bit dot pattern directly
// pattern bits: bit 0 = dot 1, bit 1 = dot 2, ..., bit 5 = dot 6
void displayPattern(uint8_t pattern);

// Show a "correct" animation (flash all dots up and down)
void displayCorrect();

// Set just one dot up or down (for testing/calibration)
void setDot(int dotNumber, bool up);

// Convert ASCII to 6-bit Braille pattern
uint8_t charToBraillePattern(char c);

// Get servo calibration angles
int getDownAngle();
int getUpAngle();
```

### Braille TTS and STT/esp32_firmware/secrets.example.h

```c
#pragma once
// ============================================================
// BrailleAI — SECRETS TEMPLATE  (safe to commit)
// ============================================================
// Copy this file to  secrets.h  and fill in your own values.
//   cp secrets.example.h secrets.h     (Linux/macOS)
//   copy secrets.example.h secrets.h   (Windows)
// secrets.h is git-ignored so your real keys never get committed.
// ------------------------------------------------------------

// ----- Wi-Fi -----
#define WIFI_SSID  "your-wifi-ssid"
#define WIFI_PASS  "your-wifi-password"

// ----- API Keys -----
#define WIT_AI_TOKEN        "YOUR_WIT_AI_TOKEN"
#define CLAUDE_API_KEY      "YOUR_CLAUDE_API_KEY"
#define GOOGLE_TTS_API_KEY  "YOUR_GOOGLE_TTS_KEY"
#define DEEPGRAM_API_KEY    "YOUR_DEEPGRAM_API_KEY"

```

### Braille TTS and STT/esp32_firmware/mic_stt.h

```c
#pragma once
#include <Arduino.h>

// ============================================================
// Speech-to-Text (INMP441 mic -> Deepgram /v1/listen)
// ------------------------------------------------------------
// Records a few seconds of audio from the INMP441 I2S microphone
// (Port 0) into a PSRAM buffer, POSTs the raw LINEAR16 PCM to
// Deepgram's prerecorded transcription endpoint, and returns the
// recognized text.
//
// Requires WIFI_SSID / WIFI_PASS and a valid DEEPGRAM_API_KEY in
// config.h. Wi-Fi is shared with the Claude corrector; call
// initCorrector() (or recordAndTranscribe(), which connects on
// demand) first.
// ============================================================

// Configure I2S Port 0 for the INMP441 mic. Call once in setup().
void initMic();

// Record RECORD_SECONDS of audio, transcribe it via Deepgram, and
// return the transcript. Blocks for the whole record + upload time.
// Returns "" on capture/network/parse failure.
String recordAndTranscribe();

```

### Braille TTS and STT/esp32_firmware/speaker_tts.h

```c
#pragma once
#include <Arduino.h>

// ============================================================
// Text-to-Speech playback (Deepgram Aura -> MAX98357A amp)
// ------------------------------------------------------------
// Sends a phrase (e.g. the corrected sentence from the Claude
// agent) to Deepgram /v1/speak, requests raw LINEAR16 PCM
// (container=none) at TTS_SAMPLE_RATE, and streams the audio
// straight to the MAX98357A class-D amp over I2S Port 1 — no
// base64, JSON, or MP3 decoding needed.
//
// Requires WIFI_SSID / WIFI_PASS and a valid DEEPGRAM_API_KEY
// in config.h. Wi-Fi is shared with the Claude corrector; call
// initCorrector() (or speak(), which connects on demand) first.
// ============================================================

// Configure I2S Port 1 for the speaker. Call once in setup().
void initSpeaker();

// Synthesize `text` via Deepgram and play it on the speaker.
// Blocks until playback finishes. Returns true if audio played,
// false on empty text, no Wi-Fi, or an API/parse error.
bool speak(const String& text);

```

[22 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]