Project Info
Inspiration
The inspiration behind the ASL Live Translator came from a desire to bridge communication gaps between the hearing and deaf communities. Many people struggle with language barriers, especially when it comes to sign language. This website aims to make communication more accessible and inclusive.
What it does
The ASL Live Translator uses a webcam to capture American Sign Language (ASL) signs, which are then converted into live captions. This allows users to communicate seamlessly with individuals who are deaf or hard of hearing by translating ASL into text.
How we built it
We built the website using Next.js, incorporating: TensorFlow.js and the Handpose model for recognizing ASL signs. Fingerpose for gesture estimation. Webcam feed integration using the react-webcam library. Text-to-Speech integration for real-time speech conversion. The project involves custom components like WebcamFeed, LiveCaptions, and SpeechSynthesis.
Challenges we ran into
Webcam integration was a bit tricky, especially with handling different camera states and permissions. ASL recognition accuracy was challenging, as the model had to accurately capture and identify signs in varying lighting conditions and different sign variations. Ensuring real-time performance was a struggle with the amount of data processed during sign detection and translation.
Accomplishments we're proud of
Successfully integrating real-time ASL translation to captions using machine learning. Making the website responsive, clean, and accessible for different devices. Ensuring the translation happens almost immediately with a user-friendly interface.
What we learned
Gained hands-on experience with TensorFlow.js and machine learning models for real-time applications. Learned more about gesture recognition and its limitations in real-world usage. Improved problem-solving skills related to real-time video processing and webcam integration.
What's next
Expanding the range of ASL signs supported, including more complex gestures and phrases. Integrating voice recognition to allow for a two-way communication flow between the user and the application. Adding a feedback system to improve the accuracy of sign recognition based on user input and corrections.
This is a Next.js project bootstrapped with create-next-app.
Getting Started
First, run the development server:
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
Open http://localhost:3000 with your browser to see the result.
You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.
This project uses next/font to automatically optimize and load Geist, a new font family for Vercel.
Learn More
To learn more about Next.js, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
You can check out the Next.js GitHub repository - your feedback and contributions are welcome!
Deploy on Vercel
The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.
Check out our Next.js deployment documentation for more details.
Analysis
View
Metric
- 2
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- Next.jsIn code
- ReactIn code
- Tailwind CSSIn code
- TypeScriptIn code
- JavaScriptClaimed
5 of 6 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
14 KB
Source files
15
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
nicoleleehy1/asl-live-captions
27 files · 270 KB · @ d7f6ec0
Structure
Interface
12 files · 44%Screens, components and styles rendered to the user.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- TypeScript87%
- Markdown10%
- CSS2%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
package.json
npm · 18- @tensorflow-models/handpose
- @tensorflow/tfjs
- @tensorflow/tfjs-backend-webgl
- fingerpose
- lucide-react
- next
- react
- react-dom
- react-webcam
- +9 more
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.