# Project export: Sign-Speak: No voice unheard This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Lightweight AR glasses that translate ASL signs into spoken audio in real-time. - Devpost: https://devpost.com/software/sign-sight-no-voice-unheard - GitHub: https://github.com/vkodithala/sign-speak - Demo: https://huggingface.co/mehulrao2/POV-ASL-Classification - Video: https://www.youtube.com/embed/ef1BUPAwq9w?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — varoon kodithala (10 commits), Vineeth Sendilraj (1 commits) ## Devpost submission (written by the team) ### Inspiration The four of us had several shared experiences with the deaf community during grade school and were alarmed by the barriers they face in everyday conversation. Deaf people are often relegated to communicating via ASL, which very few outside of their close friends and family understand. In many situations, this confines their conversations to text, rather than the natural back-and-forth dialogue we all appreciate. What’s particularly concerning is that most commercially available ASL translation devices aren't practical for everyday conversation. Most are worn by readers rather than signers, suggesting that the average person has a means of translating ASL to text when interacting with a signer. Others are expensive and bulky ASL detection gloves that interfere with the daily activities of signers. We wondered if we could do better. We asked ourselves how we could leverage recent advancements in computer vision, hand landmark detection, deep learning, and edge computing to develop a wearable device for the deaf that translates ASL signs into spoken audio in real-time, breaking down the barriers to communication that currently exist between the deaf community and the rest of the world. ### What it does Sign-Speak is a pair of smartglasses that allow deaf users to translate their ASL signs into spoken audio in real-time. Most existing solutions focus on translation from the reader's perspective, and to our knowledge there isn't anyone working on translating ASL from the signer's perspective. Evidence for this is the lack of labelled data for back-of-hand ASL signs, which is a challenge that we overcame early in the project (see "Challenges we ran into"). Built with MediaPipe's hand-tracking neural network and hosted on NVIDIA's Jetson Nano, the glasses extract precise hand landmarks, process them through a custom neural network trained on our custom ASL dataset focused only on backside hand movements, and deliver instant voice translations via an attached speaker. Our CV and speech-to-text models are hosted completely on device, making Sign-Speak a true Edge AI product pushing the frontiers of accessibility tech. Our goal is to empower the deaf community to communicate freely and create a world where every conversation transcends the barriers of sound. ### How we built it There weren't any available datasets for back-of-hand ASL signs, so the first step in training our model was to create one. To do so, we developed some scripts that took and labelled nearly 6,000 pictures of our own hands while we were signing the 26 letters of the ASL alphabet (along with some useful stopwords). We then used Google's MediaPipe models to translate these pictures into 21 coordinates representing the joints in the model's hands ("landmarks"). Our last step was to apply random scaling, rotation, and Gaussian noise to the extracted landmarks and train a feed-forward neural network that classifies ASL signs from inputted hand landmark data. The brains of our operation rely on the small but powerful NVIDIA Jetson Orin Nano upon which our computer vision model runs, allowing our users to use SignSight wherever they want. We packaged our image classification model into a .onnx file and uploaded it onto our Jetson to run inference. The Jetson itself is connected to a webcam that captures input, passes it through the model to detect what letters the user is signing in succession, and then uses Piper for on-device text-to-speech translation, sending the audio to a miniature Bluetooth speaker connected to the frames. We 3D-printed our glasses based on custom CAD designs to allow for the mounting of a webcam in the temple area, and fashioned an elastic band around the back of the frames with a Bluetooth speaker to enable the sound output of the TTS translation. Users are meant to carry the Jetson Nano (which is housed in a custom 3D-printed case) in their pocket, allowing them to use Sign-Speak's technology anywhere on Earth. ### Challenges we ran into No existing model or dataset currently focuses on first-person American Sign Language (ASL). Most available ASL datasets and computer vision models concentrate solely on capturing signs from the front, as seen by an observer, rather than from the perspective of the signer. Initially, we considered utilizing synthetic data from vision models. However, we discovered that these models had not been trained on sufficient footage of back-of-hand ASL signs, which hindered the generation of high-quality synthetic data for our needs. As a result, we faced the challenge of developing our dataset and model from scratch. This involved taking thousands of photographs of ASL signs from a back-of-hand perspective. To do this, we implemented MediaLabs' Hand Landmark Detection model, which was effective at estimating finger landmarks, even when joints were obscured from the camera's view. We used this model to convert our images into hand landmarks and used these as inputs to our model. Model overfitting and confusion among similar letters. One significant challenge in developing a robust dataset for back-of-hand ASL detection lies in the complexity of the task. Many signs appear remarkably similar from this perspective, leading to confusion and decreased confidence in our model for certain words. To address this, we utilized hand landmarks instead of relying solely on raw image pixels. This approach allowed MediaPipe to effectively infer the positions of obscured fingers, enhancing our model’s accuracy. Additionally, we faced issues with overfitting due to the custom dataset we created. To mitigate this, we carefully managed the number of training epochs and introduced random noise to improve our model's generalization capabilities. ### Accomplishments we're proud of We created the first ever dataset and computer vision model for the the ASL alphabet from the speaker's perspective. We developed and trained a custom feed-forward neural network that performed well on classifying the 26 letters of the ASL alphabet. We created and 3D-printed custom CAD designs for a pair of smartglasses built to mount a webcam and a case for our Jetson Nano. We configured an OLED display to show the current word that the user is spelling before it's outputted from text-to-speech. ### What we learned While creating SignSight, we learned about developing deep learning models on constrained resources and the tradeoffs associated with developing on small computers. While developing a seamless user experience, we realized that effective assistive technology must blend into daily life without adding complexity. It took thoughtful design choices to decide how a Jetson Nano, speaker, and camera would fit into the form factor of wearable glasses. ### What's next for Sign-Speak Two-way communication. That is, in addition to a hearing person being able to hear a deaf person, a deaf person should be able to understand a non-signing person by translating speech to text. The text would be displayed in a small OLED display on the lens of SignSight in real time as the microphone picks up on speech. During TreeHacks, we spent time trying to enable two-way communication using these glasses. However, due to the constraints of the specific OLED display we got on our hands on, we weren't able to find any libraries that were compatible for our use case. We even tried to create our own library and modify existing ones—work that will continue past TreeHacks. We aim to finish SignSight's speech to text capability to truly ensure no voice is unheard. Adding ASL word recognition. ASL word recognition is a uniquely challenging task because the number of things that you can sign grows far beyond just 26 letters and there are an infinite amount of configurations that you can produce, some of which are in the signer's line of sight and others that aren't. Though we couldn't implement it during this hackathon, our project would be greatly enhanced by training a model capable of recognizing entire words from the ASL vocabulary via a combination of vision models and additional sensors. License 📜 Copyright 2025 ©Varoon Kodithala, Mehul Rao, Arnav Patidar, Vineeth Sendilraj Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ## README (from the GitHub repository) # Sign-Speak ![image](https://github.com/user-attachments/assets/338b1dca-9bc8-48a8-a3a4-5959751999a4) ## Inspiration The four of us had several shared experiences with the deaf community during grade school and were alarmed by the barriers they face in everyday conversation. Deaf people are often relegated to communicating via ASL, which very few outside of their close friends and family understand. In many situations, this confines their conversations to text, rather than the natural back-and-forth dialogue we all appreciate. What’s particularly concerning is that most commercially available ASL translation devices aren't practical for everyday conversation. Most are worn by readers rather than signers, suggesting that the average person has a means of translating ASL to text when interacting with a signer. Others are expensive and bulky ASL detection gloves that interfere with the daily activities of signers. We wondered if we could do better. We asked ourselves how we could leverage recent advancements in computer vision, [hand landmark detection](https://ai.google.dev/edge/mediapipe/solutions/vision/gesture_recognizer), deep learning, and [edge computing](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit/) to develop a wearable device for the deaf that translates ASL signs into spoken audio in real-time, breaking down the barriers to communication that currently exist between the deaf community and the rest of the world. ## What it does Sign-Speak is a pair of smartglasses that allow deaf users to translate their ASL signs into spoken audio in real-time. Most existing solutions focus on translation from the reader's perspective, and to our knowledge there isn't anyone working on translating ASL from the signer's perspective. Evidence for this is the lack of labelled data for back-of-hand ASL signs, which is a challenge that we overcame early in the project (see "Challenges we ran into"). Built with MediaPipe's hand-tracking neural network and hosted on NVIDIA's Jetson Nano, the glasses extract precise hand landmarks, process them through a custom neural network trained on our custom ASL dataset focused only on backside hand movements, and deliver instant voice translations via an attached speaker. Our CV and speech-to-text models are hosted completely **on device**, making Sign-Speak a true Edge AI product pushing the frontiers of accessibility tech. Our goal is to empower the deaf community to communicate freely and create a world where every conversation transcends the barriers of sound. ## How we built it There weren't any available datasets for back-of-hand ASL signs, so the first step in training our model was to create one. To do so, we developed some [scripts](https://github.com/vkodithala/sign-speak/tree/main/data-collection) that took and labelled nearly 6,000 pictures of our own hands while we were signing the 26 letters of the ASL alphabet (along with some useful stopwords). We then used Google's MediaPipe models to translate these pictures into 21 coordinates representing the joints in the model's hands ("landmarks"). Our last step was to apply random scaling, rotation, and Gaussian noise to the extracted landmarks and train a feed-forward neural network that classifies ASL signs from inputted hand landmark data. The brains of our operation rely on the small but powerful NVIDIA Jetson Orin Nano upon which our computer vision model runs, allowing our users to use SignSight wherever they want. We packaged our image classification model into a .onnx file and uploaded it onto our Jetson to run inference. The Jetson itself is connected to a webcam that captures input, passes it through the model to detect what letters the user is signing in succession, and then uses [Piper](https://github.com/rhasspy/piper) for on-device text-to-speech translation, sending the audio to a miniature Bluetooth speaker connected to the frames. We 3D-printed our glasses based on custom CAD designs to allow for the mounting of a webcam in the temple area, and fashioned an elastic band around the back of the frames with a Bluetooth speaker to enable the sound output of the TTS translation. Users are meant to carry the Jetson Nano (which is housed in a custom 3D-printed case) in their pocket, allowing them to use Sign-Speak's technology anywhere on Earth. ## Challenges we ran into *No existing model or dataset currently focuses on first-person American Sign Language (ASL).* Most available ASL datasets and computer vision models concentrate solely on capturing signs from the front, as seen by an observer, rather than from the perspective of the signer. Initially, we considered utilizing synthetic data from vision models. However, we discovered that these models had not been trained on sufficient footage of back-of-hand ASL signs, which hindered the generation of high-quality synthetic data for our needs. As a result, we faced the challenge of developing our dataset and model from scratch. This involved taking thousands of photographs of ASL signs from a back-of-hand perspective. To do this, we implemented MediaLabs' Hand Landmark Detection model, which was effective at estimating finger landmarks, even when joints were obscured from the camera's view. We used this model to convert our images into hand landmarks and used these as inputs to our model. *Model overfitting and confusion among similar letters.* One significant challenge in developing a robust dataset for back-of-hand ASL detection lies in the complexity of the task. Many signs appear remarkably similar from this perspective, leading to confusion and decreased confidence in our model for certain words. To address this, we utilized hand landmarks instead of relying solely on raw image pixels. This approach allowed MediaPipe to effectively infer the positions of obscured fingers, enhancing our model’s accuracy. Additionally, we faced issues with overfitting due to the custom dataset we created. To mitigate this, we carefully managed the number of training epochs and introduced random noise to improve our model's generalization capabilities. ## Accomplishments that we're proud of - We created the first ever dataset and computer vision model for the the ASL alphabet from the speaker's perspective. - We developed and trained a custom feed-forward neural network that performed well on classifying the 26 letters of the ASL alphabet. - We created and 3D-printed custom CAD designs for a pair of smartglasses built to mount a webcam and a case for our Jetson Nano. - We configured an OLED display to show the current word that the user is spelling before it's outputted from text-to-speech. ## What we learned While creating SignSight, we learned about developing deep learning models on constrained resources and the tradeoffs associated with developing on small computers. While developing a seamless user experience, we realized that effective assistive technology must blend into daily life without adding complexity. It took thoughtful design choices to decide how a Jetson Nano, speaker, and camera would fit into the form factor of wearable glasses. ## What's next for Sign-Speak - **Two-way communication.** That is, in addition to a hearing person being able to hear a deaf person, a deaf person should be able to understand a non-signing person by translating speech to text. The text would be displayed in a small OLED display on the lens of SignSight in real time as the microphone picks up on speech. During TreeHacks, we spent time trying to enable two-way communication using these glasses. However, due to the constraints of the specific OLED display we got on our hands on, we weren't able to find any libraries that were compatible for our use case. We even tried to create our own library and modify existing ones—work that will continue past TreeHacks. We aim to finish SignSight's speech to text capability to truly ensure no voice is unheard. - [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 5 recognized source files, 29 KB. - Python (language) — detected in the code - TensorFlow (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (10 of 10) ``` asl_classifier.onnx data-collection/collect_data.py data-collection/collected_asl_data.pkl data-collection/fix_pickle.py glasses_left.stl glasses_main.stl glasses_right.stl README.md speak_stack.py train.py ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Merge pull request #2 from vkodithala/varoon-dev - Update speak_stack.py - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - added CAD files - Merge pull request #1 from vkodithala/varoon-dev - committing all code! - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### speak_stack.py ```python import os os.environ["BLINKA_JETSON_I2C_BUS"] = "7" os.environ["BLINKA_FORCEBOARD"] = "JETSON_ORIN_NX" os.environ["QT_QPA_PLATFORM"] = "offscreen" import sys, select, tty, termios import cv2 import mediapipe as mp import numpy as np import onnxruntime as ort import time from yapper import Yapper, PiperSpeaker, PiperVoiceUS, PiperQuality # For OLED display try: import board import busio import adafruit_ssd1306 except Exception as e: print("OLED library import error:", e) # ------------------------- # Configuration Flags # ------------------------- ENABLE_OLED = True # ------------------------- # Set up OLED Display (if enabled) # ------------------------- if ENABLE_OLED: try: i2c = busio.I2C(board.SCL, board.SDA) WIDTH = 64 HEIGHT = 48 display = adafruit_ssd1306.SSD1306_I2C(WIDTH, HEIGHT, i2c, addr=0x3d) display.fill(0) display.show() except Exception as e: print("OLED initialization error:", e) ENABLE_OLED = False # ------------------------- # TTS Setup # ------------------------- piper = PiperSpeaker(voice=PiperVoiceUS.BRYCE) piper.say("hello") tts_engine = Yapper(speaker=piper) # ------------------------- # Define class names (order must match training) # ------------------------- class_names = [ "A", "B", "C", "D", "delete", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "space", "T", "U", "V", "W", "X", "Y", "Z" ] # ------------------------- # Load the ONNX model # ------------------------- onnx_model_path = "asl_classifier.onnx" ort_session = ort.InferenceSession(onnx_model_path) # ------------------------- # Setup MediaPipe Hands # ------------------------- mp_hands = mp.solutions.hands # ------------------------- # Video Capture (no visualization) # ------------------------- cap = cv2.VideoCapture(0) if not cap.isOpened(): print("Cannot open camera") exit() def preprocess_landmarks(hand_landmarks): coords = [(lm.x, lm.y) for lm in hand_landmarks.landmark] flat = np.array(coords, dtype=np.float32).flatten() return flat.reshape(1, -1) threshold = 0.98 # Confidence threshold word_buffer = "" debounce_interval = 2 # seconds after each commit last_commit_time = 0 paused = False # Pause toggle def update_display(buffer_text, paused_status): """Update the OLED display with the current word buffer and status, if enabled. The text is drawn inverted (from the bottom) so that if the display is mounted upside down, the text appears right-side up.""" if not ENABLE_OLED: return try: display.fill(0) # Draw text such that it appears right-side up when the display is mounted upside down. display.text("Buffer:", 0, 18, 1) disp_text = buffer_text if len(buffer_text) <= 10 else buffer_text[-10:] display.text(disp_text, 0, 8, 1) status_text = "PAUSED" if paused_status else "RUNNING" display.text(status_text, 0, 0, 1) display.show() except Exception as e: print("OLED display error:", e) def get_keypress(timeout=0.01): """Return a single character from stdin if available, else None.""" fd = sys.stdin.fileno() old_settings = termios.tcgetattr(fd) try: tty.setcbreak(fd) rlist, _, _ = select.select([sys.stdin], [], [], timeout) if rlist: return sys.stdin.read(1) finally: termios.tcsetattr(fd, termios.TCSADRAIN, old_settings) return None print("Starting inference (no visualization).") print("Press space to toggle pause; press ESC to quit.") with mp_hands.Hands( static_image_mode=False, max_num_hands=2, min_detection_confidence=0.5, min_tracking_confidence=0.5 ) as hands: while True: # Check for key press via stdin key = get_keypress() if key: if key == "\x1b": # ESC key break elif key == " ": paused = not paused if paused: print("Paused.") else: print("Resumed.") time.sleep(0.3) # debounce delay ret, frame = cap.read() if not ret: print("Failed to grab frame") break # Process frame for hand landmarks (no GUI visualization) image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) image.flags.writeable = False results = hands.process(image) image.flags.writeable = True # Not used for display if not paused: if results.multi_hand_landmarks: hand_landmarks = results.multi_hand_landmarks[0] input_tensor = preprocess_landmarks(hand_landmarks) ort_inputs = {ort_session.get_inputs()[0].name: input_tensor} ort_outs = ort_session.run(None, ort_inputs) logits = ort_outs[0] # Compute softmax probabilities if logits.ndim == 1: probabilities = np.exp(logits) / np.sum(np.exp(logits)) else: probabilities = np.exp(logits) / np.sum(np.exp(logits), axis=1, keepdims=True) probabilities = probabilities[0] pred_idx = int(np.argmax(probabilities)) predicted_letter = class_names[pred_idx] max_conf = probabilities[pred_idx] if max_conf >= threshold: current_time = time.time() if current_time - last_commit_time >= debounce_interval: if predicted_letter == "space": if word_buffer: print("Speaking word:", word_buffer) tts_engine.yap(word_buffer, plain=True) word_buffer = "" else: [truncated — 835 more characters] ``` ### train.py ```python import pickle import numpy as np import torch import torch.nn as nn import torch.optim as optim from torch.utils.data import Dataset, DataLoader from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score import onnxruntime as ort # ------------------------- # Device Selection: prefer CUDA, else CPU # ------------------------- if torch.cuda.is_available(): device = torch.device("cuda") print("Using CUDA") else: device = torch.device("cpu") print("Using CPU") # ------------------------- # 1. Load and Prepare the Data # ------------------------- with open("collected_asl_data.pkl", "rb") as f: data_dict = pickle.load(f) # Create a sorted mapping from class names to integer labels. class_names = sorted(data_dict.keys()) class_to_idx = {class_name: idx for idx, class_name in enumerate(class_names)} print("Class mapping:", class_to_idx) # Prepare the dataset: flatten each landmark sample (assumed to be a list of (x, y) pairs) data_list = [] labels_list = [] expected_length = None for class_name, samples in data_dict.items(): for landmarks in samples: flat_sample = np.array(landmarks, dtype=np.float32).flatten() if expected_length is None: expected_length = flat_sample.shape[0] print("Expected feature length:", expected_length) if flat_sample.shape[0] == expected_length: data_list.append(flat_sample) labels_list.append(class_to_idx[class_name]) else: print(f"Skipping sample from {class_name} with shape {flat_sample.shape}") data_array = np.array(data_list) labels_array = np.array(labels_list) print("Total samples used:", data_array.shape[0]) print("Observation shape:", data_array[0].shape) num_classes = len(class_names) # ------------------------- # 2. Define a Data Augmentation Transform for Landmarks # ------------------------- def augment_landmarks(sample): """ Given a 1D torch tensor (flattened landmarks), reshape to (num_points, 2), then apply random rotation (±10°), scaling (0.9–1.1×), and additive Gaussian noise. Return the augmented flattened tensor. """ # Reshape to (num_points, 2) num_points = sample.shape[0] // 2 landmarks = sample.view(num_points, 2).clone().cpu().numpy() # Compute the center of the landmarks. center = np.mean(landmarks, axis=0) # Random rotation angle (in radians, ±10°) angle = np.deg2rad(np.random.uniform(-10, 10)) cos_val, sin_val = np.cos(angle), np.sin(angle) R = np.array([[cos_val, -sin_val], [sin_val, cos_val]]) # Random scaling factor between 0.9 and 1.1. scale = np.random.uniform(0.9, 1.1) # Subtract center, apply rotation and scaling, then add center back. transformed = (landmarks - center) @ R.T * scale + center # Add Gaussian noise (mean=0, std=0.01) noise = np.random.normal(0, 0.01, size=transformed.shape) transformed += noise # Flatten back to 1D and convert to tensor. augmented = torch.tensor(transformed.flatten(), dtype=torch.float32) return augmented.to(sample.device) # ------------------------- # 3. Create a PyTorch Dataset # ------------------------- class ASLDataset(Dataset): def __init__(self, data, labels, transform=None): self.data = data # numpy array of shape (N, input_dim) self.labels = labels # numpy array of shape (N,) self.transform = transform def __len__(self): return len(self.data) def __getitem__(self, idx): sample = torch.tensor(self.data[idx], dtype=torch.float32) if self.transform: sample = self.transform(sample) label = torch.tensor(self.labels[idx], dtype=torch.long) return sample, label # ------------------------- # 4. Split into Training and Testing Sets # ------------------------- num_samples = data_array.shape[0] indices = np.random.permutation(num_samples) test_ratio = 0.2 num_test = int(test_ratio * num_samples) num_train = num_samples - num_test train_indices = indices[:num_train] test_indices = indices[num_train:] train_data = data_array[train_indices] train_labels = labels_array[train_indices] test_data = data_array[test_indices] test_labels = labels_array[test_indices] # For training, apply augmentation; for testing, no augmentation. train_dataset = ASLDataset(train_data, train_labels, transform=augment_landmarks) test_dataset = ASLDataset(test_data, test_labels, transform=None) print("Training samples:", len(train_dataset)) print("Testing samples:", len(test_dataset)) batch_size = 64 train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True) # ------------------------- # 5. Define the Model # ------------------------- class SimpleClassifier(nn.Module): def __init__(self, input_dim, num_classes): super(SimpleClassifier, self).__init__() self.net = nn.Sequential( nn.Linear(input_dim, 256), nn.ReLU(), nn.Dropout(0.5), nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.5), nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, num_classes), ) def forward(self, x): return self.net(x) input_dim = expected_length model = SimpleClassifier(input_dim, num_classes) model = model.to(device) print(model) # ------------------------- # 6. Training Setup # ------------------------- criterion = nn.CrossEntropyLoss() optimizer = optim.Adam(model.parameters(), lr=1e-3) num_epochs = 125 # ------------------------- # 7. Training Loop # ------------------------- for epoch in range(num_epochs): model.train() running_loss = 0.0 for inputs, targets in train_loader: inputs, targets = inputs.to(device), targets.to(device) optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, targets) loss.backward() optimizer.step() running_loss += loss.item() * inputs.size(0 [truncated — 1950 more characters] ``` ### data-collection/fix_pickle.py ```python import pickle # Specify the pickle file name. pickle_file = "collected_asl_data.pkl" # Load the existing data. with open(pickle_file, "rb") as f: data = pickle.load(f) # Print available keys. print("Available classes in the data:") for key in data.keys(): print(key, end=" ") print("\n") # Prompt user for the letter to delete. letter = ( input("Enter the letter (or 'space') for which you want to delete the data: ") .strip() .lower() ) # Confirm deletion. confirm = ( input(f"Are you sure you want to delete all data for '{letter}'? (y/n): ") .strip() .lower() ) if confirm == "y": # Option 1: Remove the key entirely: # del data[letter] # Option 2: Clear the list for that key: data[letter] = [] print(f"Data for '{letter}' has been deleted.") else: print("No changes made.") # Save the updated data back to the pickle file. with open(pickle_file, "wb") as f: pickle.dump(data, f) print(f"Updated data has been saved to {pickle_file}.") ``` ### data-collection/collect_data.py ```python import cv2 import mediapipe as mp import time import pickle import os # Define the classes we want to collect data for. classes_to_collect = [ "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "space", "delete" ] num_samples = 300 # Define the output file name. output_file = "collected_asl_data.pkl" if os.path.exists(output_file): with open(output_file, "rb") as f: collected_data = pickle.load(f) # Ensure all classes are present in the dictionary. for cls in classes_to_collect: if cls not in collected_data: collected_data[cls] = [] print("Existing data loaded. Sample counts per class:") for cls in classes_to_collect: count = len(collected_data.get(cls, [])) print(f" {cls}: {count} samples") else: collected_data = {cls: [] for cls in classes_to_collect} print("No existing data found. Starting fresh.") # Setup MediaPipe Hands and drawing utilities. mp_hands = mp.solutions.hands mp_drawing = mp.solutions.drawing_utils hands = mp_hands.Hands( static_image_mode=False, max_num_hands=1, # We only need one hand min_detection_confidence=0.5, min_tracking_confidence=0.5, ) # Start webcam capture. cap = cv2.VideoCapture(0) if not cap.isOpened(): print("Cannot open camera") exit() def get_landmarks(frame): """ Process the frame with MediaPipe Hands and return the landmarks (x, y) for the first hand. Returns None if no hand is detected. """ rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) rgb.flags.writeable = False results = hands.process(rgb) if results.multi_hand_landmarks: # Return the first detected hand's landmarks. return results.multi_hand_landmarks[0], results else: return None, results print("Starting data collection. Press Ctrl+C to quit.") try: while True: # Prompt the user to enter a class to record data for. target_class = ( input(f"Enter class to record {classes_to_collect} (or 'quit' to quit): ") .strip() .lower() ) if target_class == "quit": break if target_class not in classes_to_collect: print("Invalid class. Allowed keys:", classes_to_collect) continue print(f"Recording data for class '{target_class}'.") print(f"Collecting {num_samples} samples at approx 10 samples per second...") samples_collected = 0 while samples_collected < num_samples: ret, frame = cap.read() if not ret: print("Failed to capture frame.") continue # Get landmarks and results from MediaPipe. hand_landmarks, results = get_landmarks(frame) # If landmarks detected, draw them on the frame. if hand_landmarks: mp_drawing.draw_landmarks( frame, hand_landmarks, mp_hands.HAND_CONNECTIONS ) # Extract (x,y) coordinates. landmarks = [(lm.x, lm.y) for lm in hand_landmarks.landmark] collected_data[target_class].append(landmarks) samples_collected += 1 # Overlay progress information on the frame. progress_text = ( f"Class: '{target_class}' | Sample: {samples_collected}/{num_samples}" ) cv2.putText( frame, progress_text, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA, ) # Display the frame. cv2.imshow("Data Collection", frame) # Wait a little: roughly 10 frames per second. key = cv2.waitKey(1) & 0xFF if key == 27: # ESC key to break early from current recording break time.sleep(0.05) print(f"Finished collecting data for class '{target_class}'.\n") except KeyboardInterrupt: print("\nData collection interrupted by user.") # Cleanup. cap.release() cv2.destroyAllWindows() hands.close() # Save the collected data to a pickle file. with open(output_file, "wb") as f: pickle.dump(collected_data, f) print(f"Data saved to {output_file}") ```