Project Info

Winner

Best Use of Orkes

BrailleAI : A TwoWay Voice for the Deaf & Blind

Inspiration

Imagine wanting to say something as simple as "I'm hungry" or "I love you" — but not being able to. For many DeafBlind people, every conversation depends on having another person there to interpret. The tools that could help, called refreshable Braille displays, can cost thousands of dollars. That price tag locks too many people out of something the rest of us take for granted: the freedom to talk. We didn't think that was fair. So we set out to build one small, affordable device that could be a voice, a teacher, and a friend — all controlled by touch.

What it does

BrailleAI lets a DeafBlind person have a real, two-way conversation with anyone. Talk to anyone: You type with your fingers in Braille, and the device speaks your words out loud. Feel the reply: When someone answers, the device listens, shortens their words with AI, and raises them as Braille dots you can feel. Feel the emotion too: AI senses whether the speaker sounded happy, worried, or excited — and adds a small symbol so you don't just read the words, you feel the feeling behind them. Learn Braille: A patient AI tutor gives practice and instant feedback. Ask an AI: A quiet, touch-only assistant that answers your questions privately.

How we built it

An ESP32-S3 chip handles voice input and output, while a Raspberry Pi running QNX (a rock solid real time operating system) powers on-device face recognition so the product knows who it's talking with. Two Braille cells are driven by just four small servo motors and 3D-printed sliding parts — the dots stay raised on their own, using almost no power. You type using a simple chord keyboard, like the ones blind students already know. We used AI for everything that makes it feel human. Deepgram/Whisper lets it hear speech, Google's voice lets it speak, and Anthropic's Claude understands tone, summarizes replies, and teaches. For fast, private on-device inference, we ran a neuron model trained on AWS Annapurna (Trainium/Inferentia) silicon from the workshop — so the core intelligence runs right on the product. This was as much an AI-assisted build as it was a hardware build. We used Sai by Simular as our end-to-end automation agent to drive the whole workflow, and Devin as our lead multi-agent partner for hardware–software co-design — helping us shape the circuitry, firmware, and code together instead of in silos. Alongside them we leaned on tools like Cognition, Orkes, Unify, Groq, and ngrok to move fast.

Challenges we ran into

The hardest part was making Braille affordable. Real Braille displays use tiny, expensive motors. We replaced them with cheap servos and clever 3D printed sliders, which meant a lot of trial, error, and re-printing until each dot rose perfectly. Teaching the AI to turn emotion into a single touchable symbol took many tries to get right.

What we learned

We learned that good technology isn't about being complicated; it's about removing a wall between people. We also learned a lot about hardware timing, AI speech processing, and how small design choices can make a big difference in someone's daily life.

What's next

More Braille cells so users can read full sentences at once, support for shorthand Braille, and an offline mode so BrailleAI works anywhere, no internet needed.

BrailleAI — Emotion-Aware Braille Communicator

A bidirectional communication aid for DeafBlind users that pairs a refreshable Braille device with an audio-visual emotion model. When a hearing person speaks, the device shows a tactile emotion prefix + a short Braille summary; the user types back on a Perkins keyboard and is spoken aloud.

This repository is a monorepo with two subprojects:

Folder	What it is
`AV/`	Emotion recognition — trains the audio+video model (CREMA-D) that produces `ft_best.pt`.
`Braille TTS and STT/`	The BrailleAI device — ESP32-S3 firmware + Raspberry Pi service that uses that model.

Actual Project/
├── AV/                       # CREMA-D emotion recognition (training + inference)
│   ├── final_1.py            #   train + save the deployable model
│   ├── final_2.py            #   interval inference + ground-truth scoring
│   ├── Crema_run_final.py    #   final pipeline (train + live/video inference)
│   ├── iter1/ iter2/ iter3/  #   the three research iterations (+ result graphs)
│   └── README_CREMA_Emotion.md
│
├── Braille TTS and STT/      # The communicator device
│   ├── esp32_firmware/       #   Arduino sketch for the ESP32-S3
│   ├── raspberry_pi/         #   Python: emotion_inference.py + reference sim
│   └── README.md             #   device build/run guide
│
├── BrailleAI_Project_Outline.pdf
├── README.md                 # (this file)
└── .gitignore

How the two halves connect

   AV/  (train on a GPU)                 Braille TTS and STT/  (the device)
 ┌─────────────────────────┐          ┌───────────────────────────────────────┐
 │ final_1.py / iter3       │  ft_best │ raspberry_pi/emotion_inference.py      │
 │  AVEmotionNet            │ ───.pt──▶│  loads the model, reads camera+mic     │
 │  (wav2vec2 + ViT)        │          │  └─ "EMOTION:happy:0.62\n" via UART ──▶ │
 └─────────────────────────┘          │ esp32_firmware/  → tactile prefix +     │
                                       │   Braille text on the servo cell        │
                                       └───────────────────────────────────────┘

The emotion classes (anger, disgust, fear, happy, neutral, sad) map to the Braille prefixes the firmware flashes before each message.

Quick start

1. Train / obtain the emotion model — see AV/README_CREMA_Emotion.md

cd AV
python final_1.py     # trains and saves the model checkpoint

The Raspberry Pi service can also auto-download a prebuilt ft_best.pt from Google Drive, so you don't have to retrain to run the device.

2. Build the device — see Braille TTS and STT/README.md

Flash esp32_firmware/ to the ESP32-S3 (Arduino IDE).
Run raspberry_pi/emotion_inference.py on Raspberry Pi OS (Linux).
Wire Pi TX → ESP32 RX + common GND (115200 baud).

What is NOT in this repo (git-ignored)

To keep the repository light and under GitHub's file-size limits, large, regenerable, or sensitive files are excluded (see .gitignore):

Model checkpoints / caches: *.pt, *.joblib, *.npy, *.npz (e.g. ft_best.pt, iter3 audio cache, embeddings).
Media / data: *.mp4, *.wav (e.g. camera_test.mp4, CREMA-D clips).
Secrets: secrets.h (ESP32 Wi-Fi + API keys) — use secrets.example.h.
Logs: *.jsonl, Python __pycache__/.

Result graphs (*.png) and metrics (*_results.txt) are kept so the iteration results are visible.

🔐 Security

ESP32 secrets live in Braille TTS and STT/esp32_firmware/secrets.h (git-ignored). The Claude/Deepgram keys and Wi-Fi password that were previously committed in plaintext are compromised — revoke/rotate them and change the Wi-Fi password before sharing.

Credits

See BrailleAI_Project_Outline.pdf for the full hardware/software design.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CIn code
C++In code
PythonIn code

3 of 3 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

267 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

9 Verified4 Code-supported4 Claimed only1 Blocked

AI correction of Grade 2 Braille typing via ClaudeVerified
Anthropic's Claude understands tone, summarizes replies, and teaches
Claimed on Devposthigh confidence
- Braille TTS and STT/esp32_firmware/braille_correct.cpp:113 — correctSentence() sends the typed sentence to the real Anthropic Claude API (api.anthropic.com/v1/messages) with a system prompt to fix typos/contractions, and parses the response
- Braille TTS and STT/esp32_firmware/config.h:96 — CLAUDE_API_URL and CLAUDE_MODEL constants point to the live Anthropic endpoint
Auto-download of pretrained emotion model (ft_best.pt) from Google DriveVerified
The Raspberry Pi service can also auto-download a prebuilt ft_best.pt from Google Drive, so you don't have to retrain to run the device
Claimed on readmehigh confidence
- Braille TTS and STT/raspberry_pi/emotion_inference.py:66 — download_model_from_gdrive() downloads ft_best.pt via gdown/requests using a hardcoded Google Drive file ID
AV emotion recognition model trained on CREMA-D (audio+video)Verified
AV/ - Emotion recognition, trains the audio+video model (CREMA-D) that produces ft_best.pt
Claimed on readmehigh confidence
- AV/final_1.py:268 — AVEmotionNet combines wav2vec2 audio embeddings and ViT video embeddings, trained/evaluated on CREMA-D with checkpoint saving
- AV/iter3/iter3_finetune.py:1 — Iterative research/finetuning scripts with logged results (iter3_clip_results.txt) present
Braille chord keyboard input (Perkins-style typing)Verified
You type with your fingers in Braille, and the device speaks your words out loud
Claimed on Devposthigh confidence
- Braille TTS and STT/raspberry_pi/input_handler.py:45 — ChordInput class decodes simultaneous dot-key presses into characters using an 80ms debounce window
- Braille TTS and STT/esp32_firmware/esp32_firmware.ino:158 — readBrailleChord() drives the main loop and feeds cells into grade2Word
Braille refreshable display driven by servo motorsVerified
raises them as Braille dots you can feel ... Two Braille cells are driven by just four small servo motors and 3D-printed sliding parts
Claimed on Devposthigh confidence
- Braille TTS and STT/raspberry_pi/braille_core.py:140 — state_to_servo_angle()/cell_to_servo_angles() map dot patterns to servo PWM angles
- Braille TTS and STT/raspberry_pi/display_driver.py:28 — angle_to_duty_12bit() converts angles to PCA9685 PWM duty cycles for physical actuation
- Braille TTS and STT/esp32_firmware/config.h:38 — SERVO_DOT1..6 pin definitions for 6 servos per cell (firmware uses 6 servos/cell, not the 'four small servo motors for two cells' described in the pitch)
Deepgram/Whisper speech-to-textVerified
Deepgram/Whisper lets it hear speech
Claimed on Devposthigh confidence
- Braille TTS and STT/esp32_firmware/mic_stt.cpp:122 — recordAndTranscribe() calls the real Deepgram /v1/listen STT endpoint with an API key; no Whisper integration found anywhere in the repo
Grade 2 Braille contraction expansion on the ESP32 firmwareVerified
device build/run guide describes the firmware; not explicitly in Devpost text but implied by 'Type by touch to speak out loud'
Claimed on readmehigh confidence
- Braille TTS and STT/esp32_firmware/braille_grade2.cpp:1 — Implements letter pattern table and word-level contraction expansion (grade2Word), used in the firmware's main loop
Local interaction logging with cloud sync to Google SheetsVerified
device build/run guide ... remote monitoring
Claimed on readmehigh confidence
- Braille TTS and STT/raspberry_pi/cloud_sync.py:52 — CloudLogger writes JSON-line events to an SD-card log file and syncs unsynced batches to a webhook URL (Google Apps Script), with offline retry logic
Text-to-speech output of typed messageVerified
the device speaks your words out loud
Claimed on Devposthigh confidence
- Braille TTS and STT/esp32_firmware/speaker_tts.cpp:133 — speak() posts typed text to Deepgram TTS and streams the returned PCM audio to the I2S speaker
- Braille TTS and STT/esp32_firmware/esp32_firmware.ino:176 — On ENTER, speak(lineOut) is called with the finished sentence
Braille tutor / learning mode with AI feedbackCode-supported
Learn Braille: A patient AI tutor gives practice and instant feedback
Claimed on Devpostlow confidence
- Braille TTS and STT/raspberry_pi/device.py:74 — tutor() method exists and calls ai.tutor_step(), rendering the reply to the display
- Braille TTS and STT/raspberry_pi/ai_backend.py:50 — tutor_step() only has a MockLLM implementation (simple rule-based stub); no RealLLM/Claude wiring for tutoring exists, and the ESP32 firmware (esp32_firmware.ino) has no tutor mode or mode-cycling logic at all
Emotion detection from speaker's voice/face turned into a tactile symbolCode-supported
AI senses whether the speaker sounded happy, worried, or excited, and adds a small symbol so you don't just read the words, you feel the feeling behind them
Claimed on Devpostmedium confidence
- AV/final_1.py:268 — AVEmotionNet (wav2vec2 audio + ViT video) trains an emotion classifier on CREMA-D, producing ft_best.pt
- Braille TTS and STT/raspberry_pi/emotion_inference.py:215 — predict_emotion() loads the trained model and sends 'EMOTION:<label>:<confidence>' over serial to the ESP32
- Braille TTS and STT/esp32_firmware/emotion_display.cpp:32 — showEmotionPrefix() flashes a single-letter tactile prefix cell for the received emotion label; the trained ft_best.pt checkpoint itself is git-ignored and not present in this clone, so the full pipeline can't be executed/confirmed end-to-end
Speech-to-text: hearing person's speech transcribed and shown/summarizedCode-supported
When someone answers, the device listens, shortens their words with AI, and raises them as Braille dots you can feel
Claimed on Devpostmedium confidence
- Braille TTS and STT/esp32_firmware/mic_stt.cpp:85 — recordAndTranscribe() records mic audio and posts to Deepgram STT, returning a transcript
- Braille TTS and STT/esp32_firmware/esp32_firmware.ino:196 — MODE button transcribes and displays/speaks the heard text, but does not run it through an LLM summarizer; AI summarization ('shortens their words') exists only as MockLLM in the Python reference (ai_backend.py summarize_for_braille), not in the firmware path
Touch-only private AI assistant for questionsCode-supported
Ask an AI: A quiet, touch-only assistant that answers your questions privately
Claimed on Devpostlow confidence
- Braille TTS and STT/raspberry_pi/device.py:81 — assistant() method exists and calls ai.assistant_query(), rendering the answer to the display
- Braille TTS and STT/raspberry_pi/ai_backend.py:219 — assistant_query() only has a MockLLM implementation; no real Claude call is wired for this feature, and the firmware has no assistant mode
Emotion classified from speech via LLM (Claude) understanding toneClaimed only
Anthropic's Claude understands tone, summarizes replies, and teaches
Claimed on Devpostmedium confidence
Google voice for text-to-speechClaimed only
Google's voice lets it speak
Claimed on Devposthigh confidence
On-device neuron model trained on AWS Annapurna (Trainium/Inferentia) for fast private inferenceClaimed only
we ran a neuron model trained on AWS Annapurna (Trainium/Inferentia) silicon ... so the core intelligence runs right on the product
Claimed on Devposthigh confidence
Raspberry Pi running QNX real-time OS for on-device face recognitionClaimed only
a Raspberry Pi running QNX (a rock solid real time operating system) powers on-device face recognition so the product knows who it's talking with
Claimed on Devposthigh confidence
Affordable device using cheap servos and 3D-printed sliders instead of expensive Braille display motorsBlocked
We replaced them with cheap servos and clever 3D printed sliders, which meant a lot of trial, error, and re-printing until each dot rose perfectly
Claimed on Devpostlow confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.