# Project export: Morpheus Ray-Bans: Your ASL Teacher
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: "Silence has a story." Through Meta's Ray-Ban glasses, we can transform sign language into spoken words in real-time. Break free from communication barriers, learn ASL, and connect with 500K+ others!
- Devpost: https://devpost.com/software/morpheus-iq2g1y
- GitHub: https://github.com/stevensegawa/asl-recognition-web.git
- Video: https://www.youtube.com/embed/PgUnRM5JEl4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Steven Segawa (2 commits)
## Devpost submission (written by the team)
### Inspiration
We believe everyone deserves to be understood. While over 500,000 people use American Sign Language (ASL) as their primary language, there's still a huge gap in real-time communication between signers and non-signers. While voice-to-text is everywhere (think Siri or Alexa), ASL translation tech hasn't kept up. We wanted to change that, not just with translation, but by creating a patient, understanding tutor that helps bridge these communication gaps.
### What it does
Morpheus is like having a friendly ASL interpreter and teacher in your browser. It can: Capture signs through your Ray-Ban Metas (or any other device!) in real-time. Capture signs through your Ray-Ban Metas (or any other device!) in real-time. Recognize and categorize finger positions to determine ASL words and letters. Recognize and categorize finger positions to determine ASL words and letters. Give you instant feedback on your signing. Act as a patient tutor, helping you learn and practice ASL. Give you instant feedback on your signing. Act as a patient tutor, helping you learn and practice ASL. Keep the learning experience smooth and stress-free with a clean, simple interface. Keep the learning experience smooth and stress-free with a clean, simple interface.
### How we built it
Think of Morpheus as having three main parts: 😀 The Face (Frontend): Clean, simple web design using HTML/CSS that works on any device Real-time video feedback so you can see your signs Progress tracking that doesn't feel overwhelming 🧠 The Brains (Backend): MediaPipe to track your hand movements A smart AI system we trained to recognize ASL signs A tutoring system powered by OpenAI that adapts to your learning style (Details of the Technical Stack used can be found on the 2nd Slide) ⚡ The Nervous System (Data & Communication): Super fast connections so everything feels instant Careful error handling so things don't break when mistakes happen Smart data management to keep everything running smoothly
### Challenges we ran into
Building Morpheus wasn't easy! We faced some tough challenges: Speed: Our largest challenge was ensuring everything worked in real-time without lag. The Meta Ray-Bans don’t even have an official developer environment, so had to do something similar to a “man in the middle attack”, with a ton of trial and error. We experimented across streaming platforms, video sharing, frame-by-frame capture. In the end, we found using OBS screen capturing of a Facebook Messenger call the fastest approach. Other challenges included: Training a model to understand signs accurately. Creating a teaching style that's helpful but not frustrating. Making sure the interface is welcoming to everyone, regardless of their ASL experience.
### Accomplishments we're proud of
Looking back, we're really proud that we: Created something that actually helps people communicate better. Built a tutor that's patient, understanding, and adapts to each person. Made the whole experience feel natural and friendly. Kept everything running smoothly across different devices. Created a system that's ready to grow and improve. Prizes: Education Track Grand Prize Built an innovative ASL education platform combining real-time sign recognition, adaptive AI tutoring, and progress tracking to make learning more accessible and engaging. Perplexity Hacking With Perplexity: Used Perplexity extensively during development to research ASL teaching methods, study gesture recognition algorithms, and understand best practices in language education. Delve Most Intuitive UX: Created a clean, minimal interface that makes ASL learning approachable through real-time feedback and clear visual cues LumaLabs Reimagining Visual Creation: Enhanced sign language recognition through advanced visual processing and real-time feedback visualization Neo Most Likely to Become a Business: We’re bringing ASL technology to the 20th century––focusing first on young family members and friends of ASL users. Surprising to many, the sign language economy is worth at least $3B, with >$1B spent in just ASL interpreting. Our beachhead / first users would be young, tech-savvy members of Gen Z and Gen Alpha who want to learn sign language and the parents of those young people who want the same thing! OpenAI Most Creative Use: Created an intelligent tutoring system that adapts its teaching style and provides personalized feedback based on each learner's progress Pear VC Best Customer Insights: Built our platform based on extensive research and continuous feedback. We actually had a Zoom call with a potential customer mid-building (we hadn’t even finished our MVP yet!). We fundamentally believe in building fast and iterating faster.
### What we learned
This project taught us about integrating advanced AI and computer vision with wearable technologies, something that none of us had done before. We also learned about the incredible complexity and beauty of ASL, and learned a ton of signs along the way.
### What's next
? Morpheus to the Moon! We're just getting started! Here's what we're excited about: Creating fun, engaging ways to practice ASL Building a community where learners can help each other Making it even better at understanding complex signs and phrases Adding more ways to track your progress Developing games and challenges to make learning more fun Getting feedback from the ASL community to make Morpheus even better We built Morpheus not just as a tool, but as a bridge between communities. While it's not perfect, we believe it's a step toward making communication more accessible for everyone.
## README (from the GitHub repository)
Morpheus - Meta Ray-Bans ASL Tutor
Inspiration
We believe everyone deserves to be understood. While over 500,000 people use American Sign Language (ASL) as their primary language, there's still a huge gap in real-time communication between signers and non-signers. While voice-to-text is everywhere (think Siri or Alexa), ASL translation tech hasn't kept up. We wanted to change that, not just with translation, but by creating a patient, understanding tutor that helps bridge these communication gaps.
What it does
Morpheus is like having a friendly ASL interpreter and teacher in your browser. It can:
Capture signs through your Ray-Ban Metas (or any other device!) in real-time.
Recognize and categorize finger positions to determine ASL words and letters.
Give you instant feedback on your signing. Act as a patient tutor, helping you learn and practice ASL.
Keep the learning experience smooth and stress-free with a clean, simple interface.
How we built it
Think of Morpheus as having three main parts:
😀 The Face (Frontend):
Clean, simple web design using HTML/CSS that works on any device Real-time video feedback so you can see your signs Progress tracking that doesn't feel overwhelming
🧠 The Brains (Backend):
MediaPipe to track your hand movements A smart AI system we trained to recognize ASL signs A tutoring system powered by OpenAI that adapts to your learning style (Details of the Technical Stack used can be found on the 2nd Slide)
⚡ The Nervous System (Data & Communication):
Super fast connections so everything feels instant Careful error handling so things don't break when mistakes happen Smart data management to keep everything running smoothly
Challenges we ran into
Building Morpheus wasn't easy! We faced some tough challenges:
Speed:
Our largest challenge was ensuring everything worked in real-time without lag. The Meta Ray-Bans don’t even have an official developer environment, so had to do something similar to a “man in the middle attack”, with a ton of trial and error. We experimented across streaming platforms, video sharing, frame-by-frame capture. In the end, we found using OBS screen capturing of a Facebook Messenger call the fastest approach.
Other challenges included:
Training a model to understand signs accurately.
Creating a teaching style that's helpful but not frustrating.
Making sure the interface is welcoming to everyone, regardless of their ASL experience.
Accomplishments that we're proud of
Looking back, we're really proud that we:
Created something that actually helps people communicate better.
Built a tutor that's patient, understanding, and adapts to each person.
Made the whole experience feel natural and friendly.
Kept everything running smoothly across different devices.
Created a system that's ready to grow and improve.
Prizes:
Education Track Grand Prize Built an innovative ASL education platform combining real-time sign recognition, adaptive AI tutoring, and progress tracking to make learning more accessible and engaging.
Perplexity Hacking With Perplexity: Used Perplexity extensively during development to research ASL teaching methods, study gesture recognition algorithms, and understand best practices in language education.
Delve Most Intuitive UX: Created a clean, minimal interface that makes ASL learning approachable through real-time feedback and clear visual cues
LumaLabs Reimagining Visual Creation: Enhanced sign language recognition through advanced visual processing and real-time feedback visualization
Neo Most Likely to Become a Business: We’re bringing ASL technology to the 20th century––focusing first on young family members and friends of ASL users. Surprising to many, the sign language economy is worth at least $3B, with >$1B spent in just ASL interpreting. Our beachhead / first users would be young, tech-savvy members of Gen Z and Gen Alpha who want to learn sign language and the parents of those young people who want the same thing!
OpenAI Most Creative Use: Created an intelligent tutoring system that adapts its teaching style and provides personalized feedback based on each learner's progress
Pear VC Best Customer Insights: Built our platform based on extensive research and continuous feedback. We actually had a Zoom call with a potential customer mid-building (we hadn’t even finished our MVP yet!). We fundamentally believe in building fast and iterating faster.
What we learned
This project taught us about integrating advanced AI and computer vision with wearable technologies, something that none of us had done before. We also learned about the incredible complexity and beauty of ASL, and learned a ton of signs along the way.
What's next? Morpheus to the Moon!
We're just getting started! Here's what we're excited about:
Creating fun, engaging ways to practice ASL
Building a community where learners can help each other
Making it even better at understanding complex signs and phrases
Adding more ways to track your progress
Developing games and challenges to make learning more fun
Getting feedback from the ASL community to make Morpheus even better
We built Morpheus not just as a tool, but as a bridge between communities. While it's not perfect, we believe it's a step toward making communication more accessible for everyone.
## Detected evidence (automated analysis)
Indexed codebase: 14 recognized source files, 84 KB.
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- TensorFlow (technology) — detected in the code
- CSS (language) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (21 of 21)
```
.gitattributes
.gitignore
flake.lock
flake.nix
frontend.html
LICENSE
README.md
requirements.txt
src/classifier/retrain_model.py
src/classifier/train_classifier.py
src/context_audio.py
src/data_gen/collect_images.py
src/data_gen/create_dataset.py
src/main.py
src/README.md
src/requirements.txt
src/server.py
src/templates/frontend.html
src/utils.py
src/voice_processor.py
templates/frontend.html
```
### Dependencies
- requirements.txt: elevenlabs@>=2.0.0, flask, flask-cors, mediapipe, numpy, openai, opencv-python, Pillow, pyaudio, pygame, python-dotenv, sounddevice, SpeechRecognition
- src/requirements.txt: autocorrect@==2.6.1, imageio@==2.31.1, mediapipe@==0.10.2, opencv-python@==4.8.0.74, scikit-learn@==1.2.0, tensorflow@==2.12.0
### Recent commits (newest first)
- Update context_audio.py
- Update README.md
- no large model
- Initial commit
## Key source files (fetched from GitHub, selected and truncated for size)
### requirements.txt
```
flask
flask-cors
opencv-python
SpeechRecognition
openai
python-dotenv
Pillow
pygame
sounddevice
numpy
pyaudio
mediapipe
elevenlabs>=2.0.0
```
### src/requirements.txt
```
opencv-python==4.8.0.74
mediapipe==0.10.2
scikit-learn==1.2.0
imageio==2.31.1
autocorrect==2.6.1
tensorflow==2.12.0
```
### src/server.py
```python
from flask import Flask, Response, render_template
from flask_cors import CORS
import cv2
import mediapipe as mp
import queue
import logging
from dotenv import load_dotenv
from utils import recognize_asl_gesture, draw_info_text, draw_landmarks, calc_landmark_list
import traceback
# Configure logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
# Initialize Flask app
app = Flask(__name__)
CORS(app)
# Global variables
stop_threads = False
process_sign_language = True # Set to True by default
voice_triggered = False
current_gesture = ""
def initialize_camera():
"""Initialize camera with fallback options"""
# Try OBS virtual camera first
capture = cv2.VideoCapture(1)
if not capture.isOpened():
logger.warning("Failed to open OBS camera (index 1), trying default camera...")
capture = cv2.VideoCapture(0)
if not capture.isOpened():
logger.error("Failed to open any camera")
return None
logger.info(f"Successfully opened camera")
return capture
def process_hand_landmarks(image, hand_landmarks, process_sign_language):
"""Process hand landmarks and return gesture if enabled"""
global current_gesture
h, w, _ = image.shape
# Calculate landmark points for drawing
landmark_list = calc_landmark_list(image, hand_landmarks)
# Draw the landmarks with our custom drawing function
image = draw_landmarks(image, landmark_list)
if process_sign_language:
try:
# Get ASL gesture using our recognition function
gesture = recognize_asl_gesture(hand_landmarks)
current_gesture = gesture
# Draw bounding box and label
x_values = [lm.x for lm in hand_landmarks.landmark]
y_values = [lm.y for lm in hand_landmarks.landmark]
min_x = int(min(x_values) * w)
max_x = int(max(x_values) * w)
min_y = int(min(y_values) * h)
max_y = int(max(y_values) * h)
# Draw rectangle
cv2.rectangle(
image,
(min_x - 20, min_y - 10),
(max_x + 20, max_y + 10),
(0, 255, 0), # Green color
2
)
# Draw gesture label
image = draw_info_text(
image,
[min_x - 20, min_y - 10, max_x + 20, max_y + 10],
gesture
)
# Draw current gesture in top-left corner
cv2.putText(
image,
f"Current Sign: {gesture}",
(10, 30),
cv2.FONT_HERSHEY_SIMPLEX,
1,
(0, 255, 0), # Green color
2,
cv2.LINE_AA
)
logger.info(f"Recognized gesture: {gesture}")
except Exception as e:
logger.error(f"Error in gesture recognition: {e}")
logger.error(traceback.format_exc())
return image
def generate_frames():
"""Generate processed frames for video streaming"""
global process_sign_language
capture = initialize_camera()
if capture is None:
return
# Initialize MediaPipe
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(
min_detection_confidence=0.6,
min_tracking_confidence=0.5,
max_num_hands=1
)
while not stop_threads:
success, frame = capture.read()
if not success:
logger.warning("Failed to read frame")
continue
try:
# Process frame
image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
image.flags.writeable = False
results = hands.process(image)
image.flags.writeable = True
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
# Draw hand landmarks and process gestures if detected
if results.multi_hand_landmarks:
for hand_landmarks in results.multi_hand_landmarks:
image = process_hand_landmarks(
image,
hand_landmarks,
process_sign_language
)
# Add status indicator
status_text = "RECOGNITION: ON" if process_sign_language else "RECOGNITION: OFF"
status_color = (0, 255, 0) if process_sign_language else (0, 0, 255) # Green for ON, Red for OFF
# Draw status with background
text_size = cv2.getTextSize(status_text, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2)[0]
cv2.rectangle(
image,
(10, image.shape[0] - 40),
(10 + text_size[0] + 20, image.shape[0] - 10),
(0, 0, 0),
-1
)
cv2.putText(
image,
status_text,
(20, image.shape[0] - 20),
cv2.FONT_HERSHEY_SIMPLEX,
0.7,
status_color,
2,
cv2.LINE_AA
)
# Convert to jpg for streaming
ret, buffer = cv2.imencode('.jpg', image)
if not ret:
continue
frame_bytes = buffer.tobytes()
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n\r\n' + frame_bytes + b'\r\n')
except Exception as e:
logger.error(f"Error processing frame: {e}")
continue
capture.release()
@app.route('/')
def index():
return render_template('frontend.html')
@app.route('/video_feed')
def video_feed():
return Response(
generate_frames(),
mimetype='multipart/x-mixed-replace; boundary=frame'
)
@app.route('/toggle_sign_language', meth
[truncated — 669 more characters]
```
### src/main.py
```python
import sys
import time
import cv2
import argparse
import numpy as np
import mediapipe as mp
import sklearn
from autocorrect import Speller
from utils import load_model, save_gif, save_video
from utils import calc_landmark_list, draw_landmarks, draw_info_text
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands
# Autocorrect Word
spell = Speller(lang="en")
# Colors RGB Format
BLACK = (0, 0, 0)
RED = (255, 0, 0)
GREEN = (0, 255, 0)
BLUE = (0, 0, 255)
YELLOW = (0, 255, 255)
WHITE = (255, 255, 255)
# Constants
FONT = cv2.FONT_HERSHEY_SIMPLEX
MAX_HANDS = 1
min_detection_confidence = 0.6
min_tracking_confidence = 0.5
MODEL_PATH = "./classifier"
model_letter_path = f"{MODEL_PATH}/classify_letter_model.p"
print(f"scikit-learn version: {sklearn.__version__}")
# Customize your input
def parse_opt():
parser = argparse.ArgumentParser()
parser.add_argument(
"-s", "--source", type=str, default=None, help="Video Path/0 for Webcam"
)
parser.add_argument(
"-a", "--autocorrect", action="store_true", help="Autocorrect Misspelled Word"
)
parser.add_argument("-g", "--gif", action="store_true", help="Save GIF Result")
parser.add_argument("-v", "--video", action="store_true", help="Save Video Result")
parser.add_argument("-t", "--timing", type=int, default=8, help="Timing Threshold")
parser.add_argument("-wi", "--width", type=int, default=800, help="Webcam Width")
parser.add_argument("-he", "--height", type=int, default=600, help="Webcam Height")
parser.add_argument("-f", "--fps", type=int, default=30, help="Webcam FPS")
opt = parser.parse_args()
return opt
def get_output(idx, _output, output, autocorrect, TIMING):
key = []
for i in range(len(_output[idx])):
character = _output[idx][i]
counts = _output[idx].count(character)
# Add character to key if it exceeds 'TIMING THRESHOLD'
if (character not in key) or (character != key[-1]):
if counts > TIMING:
key.append(character)
# Add key character to output text
text = ""
for character in key:
if character == "?":
continue
text += str(character).lower()
# Autocorrect Misspelled Word
text = spell(text) if autocorrect else text
# Add word to output list
if text != "":
_output[idx] = []
output.append(text.title())
return None
def recognize_gesture(
image,
results,
model_letter_path,
mp_drawing,
current_hand,
_output,
output,
autocorrect,
TIMING,
):
multi_hand_landmarks = results.multi_hand_landmarks
multi_handedness = results.multi_handedness
try:
# Get our ASL recognition function
recognize_asl = load_model(model_letter_path)
_gesture = []
# Draw landmarks and recognize gestures
if results.multi_hand_landmarks:
h, w, _ = image.shape
for idx in reversed(range(len(multi_hand_landmarks))):
current_select_hand = multi_hand_landmarks[idx]
handness = multi_handedness[idx].classification[0].label
# Always draw landmarks
mp_drawing.draw_landmarks(image, current_select_hand, mp_hands.HAND_CONNECTIONS)
landmark_list = calc_landmark_list(image, current_select_hand)
image = draw_landmarks(image, landmark_list)
try:
# Recognize the gesture
gesture = recognize_asl(current_select_hand)
# Draw bounding box and label
x_values = [lm.x for lm in current_select_hand.landmark]
y_values = [lm.y for lm in current_select_hand.landmark]
min_x = int(min(x_values) * w)
max_x = int(max(x_values) * w)
min_y = int(min(y_values) * h)
max_y = int(max(y_values) * h)
cv2.rectangle(
image, (min_x - 20, min_y - 10), (max_x + 20, max_y + 10), BLACK, 4
)
image = draw_info_text(
image, [min_x - 20, min_y - 10, max_x + 20, max_y + 10], gesture
)
_gesture.append(gesture)
except Exception as e:
print(f"Error in gesture recognition: {e}")
continue
# Handle output
if len(_gesture) > 0:
_output[0].append(_gesture[0])
if results.multi_hand_landmarks:
current_hand = len(multi_hand_landmarks)
else:
current_hand = 0
return current_hand, image
except Exception as e:
print(f"Error in recognize_gesture: {e}")
return current_hand, image
def recognize_signs(capture):
current_hand = 0
autocorrect = False
TIMING = 8
output = []
_output = [[], []]
with mp_hands.Hands(
min_detection_confidence=min_detection_confidence,
min_tracking_confidence=min_tracking_confidence,
max_num_hands=MAX_HANDS,
) as hands:
while capture.isOpened():
success, image = capture.read()
if not success:
print("Failed to read frame.")
break
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# To improve performance, optionally mark the image as not writeable to pass by reference
image.flags.writeable = False
results = hands.process(image)
# Draw the hand annotations on the image
image.flags.writeable = True
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
try:
current_hand, image = recognize_gesture(
image,
results,
model_letter_path,
mp_drawing,
current_
[truncated — 1747 more characters]
```
### frontend.html
```html
Sign Language Recognition
Upload Video
Start Recording
```
### templates/frontend.html
```html
Sign Language Recognition
Upload Video
Start Recording
```
### src/voice_processor.py
```python
from openai import OpenAI
import os
from dotenv import load_dotenv
import logging
from datetime import datetime
import pygame
import io
# Load environment variables
load_dotenv()
class VoiceProcessor:
def __init__(self):
# Setup OpenAI API
self.api_key = os.getenv('OPENAI_API_KEY')
if not self.api_key:
raise ValueError("Missing OPENAI_API_KEY in environment variables")
self.client = OpenAI(api_key=self.api_key)
# Setup logging
logging.basicConfig(
filename=f'voice_processing_{datetime.now().strftime("%Y%m%d_%H%M%S")}.log',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
self.logger = logging.getLogger(__name__)
def text_to_speech(self, text):
"""
Convert text to speech using OpenAI's TTS API and play it directly
"""
try:
# Use with_streaming_response for the audio generation
with self.client.audio.speech.with_streaming_response.create(
model="tts-1",
voice="nova", # Using echo voice for clear and friendly tone
input=text
) as response:
# Initialize pygame mixer if not already done
if not pygame.mixer.get_init():
pygame.mixer.init()
# Create a temporary file in memory
with io.BytesIO() as audio_buffer:
# Write the streaming response to the buffer
for chunk in response.iter_bytes():
audio_buffer.write(chunk)
# Reset buffer position
audio_buffer.seek(0)
# Load and play the audio
pygame.mixer.music.load(audio_buffer)
pygame.mixer.music.play()
# Wait for the audio to finish playing
while pygame.mixer.music.get_busy():
pygame.time.Clock().tick(10)
return True, text
except Exception as e:
self.logger.error(f"Error converting to speech: {str(e)}")
print(f"Error: {str(e)}")
return False, None
def correct_sign_output(self, text):
"""
Correct and improve the text output from the sign language model using GPT-4,
focusing primarily on spelling corrections and common word substitutions
"""
try:
# Correct spelling errors and improve natural flow
response = self.client.chat.completions.create(
model="gpt-4o", # Updated to GPT-4o
messages=[
{
"role": "system",
"content": "given a phrase you correct the spelling, make sure to strip any random letters as these are ASL errors. your output must be only the phrase and nothing else."
},
{
"role": "user",
"content": text
}
],
temperature=0.2, # Lower temperature for more consistent corrections
max_tokens=100
)
corrected_text = response.choices[0].message.content.strip()
# Log the correction
self.logger.info(f"Original model output: {text}")
self.logger.info(f"Corrected output: {corrected_text}")
return corrected_text
except Exception as e:
self.logger.error(f"Error correcting text: {str(e)}")
print(f"Error: {str(e)}")
return None
def process_and_speak_sign(self, text):
"""
Process sign language output and convert it to speech
"""
try:
# First correct the text
corrected_text = self.correct_sign_output(text)
if corrected_text:
# Then convert to speech
success, natural_text = self.text_to_speech(corrected_text)
if success:
print(f"\nProcessing complete!")
print(f"Original text: {text}")
print(f"Corrected text: {corrected_text}")
return True
return False
except Exception as e:
self.logger.error(f"Error in process_and_speak_sign: {str(e)}")
print(f"Error: {str(e)}")
return False
def main():
processor = VoiceProcessor()
processor.text_to_speech("I love you")
return
# Extended example outputs from sign language model with spelling mistakes
example_outputs = [
# Common daily activities
"I nede to go to the stoer", # need, store
"The wether is beautifull today", # weather, beautiful
"Can you help me with my homewrok", # homework
"She is reding a bok in the librery", # reading, book, library
# Food and dining
"Im going to the resturant", # I'm, restaurant
"The caffe has good sandwitches", # cafe, sandwiches
"Let's get sum piza for diner", # some, pizza, dinner
"I want a hamberger and fris", # hamburger, fries
# Education
"The techer explained the problm", # teacher, problem
"My mathmatiks test is tomorow", # mathematics, tomorrow
"I forgot my notbok at skool", # notebook, school
"The chemestry experimet failed", # chemistry, experiment
# Technology
"The computr is not workin", # computer, working
"My phon batery is ded", # phone, battery, dead
"I cant conect to the intrnet",
[truncated — 1674 more characters]
```
### src/utils.py
```python
import cv2
import pickle
import string
import imageio
import numpy as np
import mediapipe as mp
# 26 Labels and Unknown Gesture
ascii_string = string.ascii_lowercase.upper() + "?"
labels_dict = {idx: value for idx, value in enumerate(ascii_string)}
# Colors RGB Format
BLACK = (0, 0, 0)
RED = (255, 0, 0)
GREEN = (0, 255, 0)
BLUE = (0, 0, 255)
YELLOW = (0, 255, 255)
WHITE = (255, 255, 255)
def draw_info_text(image, pos, hand_sign_text):
"""Draw the recognized gesture text on the image"""
# Draw background rectangle
cv2.rectangle(image,
(pos[0] - 2, pos[1] - 30),
(pos[2] + 2, pos[1] - 2),
(0, 0, 0),
-1)
# Draw text
cv2.putText(
image,
f"Sign: {hand_sign_text}",
(pos[0] + 5, pos[1] - 10),
cv2.FONT_HERSHEY_SIMPLEX,
0.7,
(255, 255, 255),
2,
cv2.LINE_AA
)
return image
def draw_landmarks(image, landmark_point):
if len(landmark_point) > 0:
# Thumb
cv2.line(image, tuple(landmark_point[2]), tuple(landmark_point[3]), BLACK, 6)
cv2.line(image, tuple(landmark_point[2]), tuple(landmark_point[3]), WHITE, 2)
cv2.line(image, tuple(landmark_point[3]), tuple(landmark_point[4]), BLACK, 6)
cv2.line(image, tuple(landmark_point[3]), tuple(landmark_point[4]), WHITE, 2)
# Index finger
cv2.line(image, tuple(landmark_point[5]), tuple(landmark_point[6]), BLACK, 6)
cv2.line(image, tuple(landmark_point[5]), tuple(landmark_point[6]), WHITE, 2)
cv2.line(image, tuple(landmark_point[6]), tuple(landmark_point[7]), BLACK, 6)
cv2.line(image, tuple(landmark_point[6]), tuple(landmark_point[7]), WHITE, 2)
cv2.line(image, tuple(landmark_point[7]), tuple(landmark_point[8]), BLACK, 6)
cv2.line(image, tuple(landmark_point[7]), tuple(landmark_point[8]), WHITE, 2)
# Middle finger
cv2.line(image, tuple(landmark_point[9]), tuple(landmark_point[10]), BLACK, 6)
cv2.line(image, tuple(landmark_point[9]), tuple(landmark_point[10]), WHITE, 2)
cv2.line(image, tuple(landmark_point[10]), tuple(landmark_point[11]), BLACK, 6)
cv2.line(image, tuple(landmark_point[10]), tuple(landmark_point[11]), WHITE, 2)
cv2.line(image, tuple(landmark_point[11]), tuple(landmark_point[12]), BLACK, 6)
cv2.line(image, tuple(landmark_point[11]), tuple(landmark_point[12]), WHITE, 2)
# Ring finger
cv2.line(image, tuple(landmark_point[13]), tuple(landmark_point[14]), BLACK, 6)
cv2.line(image, tuple(landmark_point[13]), tuple(landmark_point[14]), WHITE, 2)
cv2.line(image, tuple(landmark_point[14]), tuple(landmark_point[15]), BLACK, 6)
cv2.line(image, tuple(landmark_point[14]), tuple(landmark_point[15]), WHITE, 2)
cv2.line(image, tuple(landmark_point[15]), tuple(landmark_point[16]), BLACK, 6)
cv2.line(image, tuple(landmark_point[15]), tuple(landmark_point[16]), WHITE, 2)
# Little finger
cv2.line(image, tuple(landmark_point[17]), tuple(landmark_point[18]), BLACK, 6)
cv2.line(image, tuple(landmark_point[17]), tuple(landmark_point[18]), WHITE, 2)
cv2.line(image, tuple(landmark_point[18]), tuple(landmark_point[19]), BLACK, 6)
cv2.line(image, tuple(landmark_point[18]), tuple(landmark_point[19]), WHITE, 2)
cv2.line(image, tuple(landmark_point[19]), tuple(landmark_point[20]), BLACK, 6)
cv2.line(image, tuple(landmark_point[19]), tuple(landmark_point[20]), WHITE, 2)
# Palm
cv2.line(image, tuple(landmark_point[0]), tuple(landmark_point[1]), BLACK, 6)
cv2.line(image, tuple(landmark_point[0]), tuple(landmark_point[1]), WHITE, 2)
cv2.line(image, tuple(landmark_point[1]), tuple(landmark_point[2]), BLACK, 6)
cv2.line(image, tuple(landmark_point[1]), tuple(landmark_point[2]), WHITE, 2)
cv2.line(image, tuple(landmark_point[2]), tuple(landmark_point[5]), BLACK, 6)
cv2.line(image, tuple(landmark_point[2]), tuple(landmark_point[5]), WHITE, 2)
cv2.line(image, tuple(landmark_point[5]), tuple(landmark_point[9]), BLACK, 6)
cv2.line(image, tuple(landmark_point[5]), tuple(landmark_point[9]), WHITE, 2)
cv2.line(image, tuple(landmark_point[9]), tuple(landmark_point[13]), BLACK, 6)
cv2.line(image, tuple(landmark_point[9]), tuple(landmark_point[13]), WHITE, 2)
cv2.line(image, tuple(landmark_point[13]), tuple(landmark_point[17]), BLACK, 6)
cv2.line(image, tuple(landmark_point[13]), tuple(landmark_point[17]), WHITE, 2)
cv2.line(image, tuple(landmark_point[17]), tuple(landmark_point[0]), BLACK, 6)
cv2.line(image, tuple(landmark_point[17]), tuple(landmark_point[0]), WHITE, 2)
# Key Points
for index, landmark in enumerate(landmark_point):
if index == 0:
cv2.circle(image, (landmark[0], landmark[1]), 5, WHITE, -1)
cv2.circle(image, (landmark[0], landmark[1]), 5, BLACK, 1)
if index == 1:
cv2.circle(image, (landmark[0], landmark[1]), 5, WHITE, -1)
cv2.circle(image, (landmark[0], landmark[1]), 5, BLACK, 1)
if index == 2:
cv2.circle(image, (landmark[0], landmark[1]), 5, WHITE, -1)
cv2.circle(image, (landmark[0], landmark[1]), 5, BLACK, 1)
if index == 3:
cv2.circle(image, (landmark[0], landmark[1]), 5, WHITE, -1)
cv2.circle(image, (landmark[0], landmark[1]), 5, BLACK, 1)
if index == 4:
cv2.circle(image, (landmark[0], landmark[1]), 8, WHITE, -1)
cv2.circle(image, (landmark[0], landmark[1]), 8, BLACK, 1)
if index == 5:
cv2.circle(image, (landmark[0], landmark[1]), 5, WHITE, -1)
cv2.circle(image, (landmark[0], landmark[1]), 5, BLACK, 1)
if index == 6:
cv2.circle(image, (landmark[0], landmark[1]), 5, WHITE, -1)
cv2.circle(imag
[truncated — 6122 more characters]
```
### src/context_audio.py
```python
import cv2
import speech_recognition as sr
import threading
import queue
from datetime import datetime, timedelta
from openai import OpenAI
import base64
import io
from PIL import Image
import time
import os
from dotenv import load_dotenv
import pygame
import tempfile
import sounddevice as sd
import numpy as np
import traceback
import logging
import mediapipe as mp
from utils import recognize_asl_gesture, draw_info_text
import pandas as pd
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
# Modify the import to include error handling
try:
logger.debug("Attempting to import recognize_signs...")
from main import recognize_signs
logger.debug("Successfully imported recognize_signs")
except Exception as e:
logger.error(f"Error importing recognize_signs: {e}")
logger.error(traceback.format_exc())
from voice_processor import VoiceProcessor
import sys
import platform
load_dotenv()
client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
text_queue = queue.Queue()
vision_queue = queue.Queue()
last_text = ""
last_vision_text = ""
text_timestamp = datetime.now()
vision_timestamp = datetime.now()
TEXT_DISPLAY_DURATION = timedelta(seconds=10)
voice_triggered = False
sign_text = ""
process_sign_language = False
voice_processor = VoiceProcessor()
# Modify the pygame initialization to handle errors gracefully
try:
pygame.mixer.quit() # First quit any existing mixer
pygame.mixer.init(frequency=24000) # Initialize with correct frequency for TTS
except Exception as e:
logger.warning(f"Could not initialize pygame mixer: {e}")
def check_environment():
logger.debug(f"Python version: {sys.version}")
logger.debug(f"Operating system: {platform.system()} {platform.release()}")
logger.debug(f"Pygame version: {pygame.version.ver}")
logger.debug(f"OpenAI API key present: {'OPENAI_API_KEY' in os.environ}")
# Add this line after load_dotenv()
check_environment()
def encode_image_to_base64(frame):
# Convert CV2 frame to PIL Image
img = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
# Convert to base64
buffered = io.BytesIO()
img.save(buffered, format="JPEG")
return base64.b64encode(buffered.getvalue()).decode('utf-8')
def analyze_image(frame):
logger.debug("analyze_image")
try:
base64_image = encode_image_to_base64(frame)
response = client.chat.completions.create(
model="gpt-4-turbo",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "As a friendly companion, describe what's immediately around us in 1-2 short sentences. Focus on the most important things: any nearby obstacles, people, or immediate safety concerns a visually impaired person should know about. Don't say more than 3 sentences."
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}",
"detail": "low"
}
}
],
}
],
max_tokens=100 # Reduced token limit to ensure shorter responses
)
vision_queue.put(response.choices[0].message.content)
except Exception as e:
logger.error(f"Error in image analysis: {e}")
# Modify the audio device listing to handle errors
def list_all_audio_devices():
try:
logger.debug("\nAvailable Audio Input Devices:")
logger.debug("------------------------------")
devices = sr.Microphone.list_microphone_names()
for index, name in enumerate(devices):
logger.debug(f"Index {index}: {name}")
logger.debug("------------------------------")
except Exception as e:
logger.warning(f"Could not list audio devices: {e}")
# Add this line after pygame.mixer.init()
list_all_audio_devices()
def audio_processing():
recognizer = sr.Recognizer()
mic_index = 0 #0 for MAC MIC, 3 for MAC SPEAKER TO MIC
while True:
try:
with sr.Microphone(device_index=mic_index) as source:
recognizer.adjust_for_ambient_noise(source)
audio = recognizer.listen(source)
try:
text = recognizer.recognize_google(audio)
logger.debug("received text")
# Check for trigger phrases
trigger_phrases = ["describe surround", "describe my surroundings", "describe surrounding", "describe surroundings", "what's around", "whats around", "describe the room"]
if "sign language" in text.lower():
logger.debug("recognized sign language phrase")
text_queue.put("Recognizing sign language...")
global sign_text, process_sign_language
process_sign_language = True # New flag to trigger sign language processing
elif any(phrase in text.lower() for phrase in trigger_phrases):
text_queue.put("Analyzing surroundings...")
global voice_triggered
voice_triggered = True
else:
text_queue.put(f"Speech: {text}")
except sr.UnknownValueError:
pass
except sr.RequestError:
text_queue.put("Speech recognition service unavailable")
except Exception as e:
logger.error(f"Error in audio processing: {e}")
# Start audio processing thread
audio_thread = threading.Thread(target=audio_processing, daemon=True)
audio_thread.start()
# Initialize video capture
cap = cv2.VideoCapture(1) #0 for MAC CAMERA
[truncated — 7088 more characters]
```
### src/classifier/retrain_model.py
```python
import pickle
import numpy as np
from sklearn.ensemble import RandomForestClassifier
import string
# Create labels
ascii_string = string.ascii_lowercase.upper() + "?"
labels_dict = {idx: value for idx, value in enumerate(ascii_string)}
def train_new_model():
try:
# Load the data
with open("../data/data.pickle", "rb") as f:
data_dict = pickle.load(f)
# Convert data to numpy arrays
X = np.array(data_dict["data"])
y = np.array(data_dict["labels"])
# Train a new model with current scikit-learn version
model = RandomForestClassifier(
n_estimators=100,
max_depth=20,
random_state=42
)
model.fit(X, y)
# Save the model
with open("classify_letter_model.p", "wb") as f:
pickle.dump({"model": model}, f, protocol=4)
print("Successfully trained and saved new model!")
return True
except Exception as e:
print(f"Error training model: {e}")
return False
if __name__ == "__main__":
train_new_model()
```
[4 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]