# Project export: ASL Live Translator

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2025
- Tagline: Increases the independence for those who are hard of hearing by translating ASL to Live Captions during in-person conversations.
- Devpost: https://devpost.com/software/asl-live-translator-rqekhp
- GitHub: https://github.com/nicoleleehy1/asl-live-captions
- Video: https://www.youtube.com/embed/XZBG1w1wRsU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Nicole (2 commits)

## Devpost submission (written by the team)

### Inspiration

The inspiration behind the ASL Live Translator came from a desire to bridge communication gaps between the hearing and deaf communities. Many people struggle with language barriers, especially when it comes to sign language. This website aims to make communication more accessible and inclusive.

### What it does

The ASL Live Translator uses a webcam to capture American Sign Language (ASL) signs, which are then converted into live captions. This allows users to communicate seamlessly with individuals who are deaf or hard of hearing by translating ASL into text.

### How we built it

We built the website using Next.js, incorporating: TensorFlow.js and the Handpose model for recognizing ASL signs. Fingerpose for gesture estimation. Webcam feed integration using the react-webcam library. Text-to-Speech integration for real-time speech conversion. The project involves custom components like WebcamFeed, LiveCaptions, and SpeechSynthesis.

### Challenges we ran into

Webcam integration was a bit tricky, especially with handling different camera states and permissions. ASL recognition accuracy was challenging, as the model had to accurately capture and identify signs in varying lighting conditions and different sign variations. Ensuring real-time performance was a struggle with the amount of data processed during sign detection and translation.

### Accomplishments we're proud of

Successfully integrating real-time ASL translation to captions using machine learning. Making the website responsive, clean, and accessible for different devices. Ensuring the translation happens almost immediately with a user-friendly interface.

### What we learned

Gained hands-on experience with TensorFlow.js and machine learning models for real-time applications. Learned more about gesture recognition and its limitations in real-world usage. Improved problem-solving skills related to real-time video processing and webcam integration.

### What's next

Expanding the range of ASL signs supported, including more complex gestures and phrases. Integrating voice recognition to allow for a two-way communication flow between the user and the application. Adding a feedback system to improve the accuracy of sign recognition based on user input and corrections.

## README (from the GitHub repository)

This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.


## Detected evidence (automated analysis)

Indexed codebase: 15 recognized source files, 14 KB.
- CSS (language) — detected in the code
- Next.js (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- JavaScript (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (20 of 20)

```
.gitignore
app/components/ASLInterpreter.tsx
app/components/LiveCaptions.tsx
app/components/SpeechSynthesis.tsx
app/components/ui/Button.tsx
app/components/ui/Card.tsx
app/components/WebcamFeed.tsx
app/globals.css
app/layout.tsx
app/page.tsx
app/utils/cn.ts
app/utils/drawHand.ts
app/utils/fingerpose.d.ts
eslint.config.mjs
next.config.ts
package.json
postcss.config.mjs
README.md
tailwind.config.ts
tsconfig.json
```

### Dependencies

- package.json: @eslint/eslintrc@^3, @tensorflow-models/handpose@^0.1.0, @tensorflow/tfjs@^4.22.0, @tensorflow/tfjs-backend-webgl@^4.22.0, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@15.1.7, fingerpose@^0.1.0, lucide-react@^0.475.0, next@15.1.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, react-webcam@^7.2.0, tailwindcss@^3.4.1, typescript@^5

### Recent commits (newest first)

- ASL Live Captions
- Initial commit from Create Next App

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "nicole-project",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@tensorflow-models/handpose": "^0.1.0",
    "@tensorflow/tfjs": "^4.22.0",
    "@tensorflow/tfjs-backend-webgl": "^4.22.0",
    "fingerpose": "^0.1.0",
    "lucide-react": "^0.475.0",
    "next": "15.1.7",
    "react": "^19.0.0",
    "react-dom": "^19.0.0",
    "react-webcam": "^7.2.0"
  },
  "devDependencies": {
    "@eslint/eslintrc": "^3",
    "@types/node": "^20",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "eslint": "^9",
    "eslint-config-next": "15.1.7",
    "postcss": "^8",
    "tailwindcss": "^3.4.1",
    "typescript": "^5"
  }
}

```

### app/layout.tsx

```typescript
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import "./globals.css";

const geistSans = Geist({
  variable: "--font-geist-sans",
  subsets: ["latin"],
});

const geistMono = Geist_Mono({
  variable: "--font-geist-mono",
  subsets: ["latin"],
});

export const metadata: Metadata = {
  title: "Create Next App",
  description: "Generated by create next app",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body
        className={`${geistSans.variable} ${geistMono.variable} antialiased`}
      >
        {children}
      </body>
    </html>
  );
}

```

### app/page.tsx

```typescript
"use client";
import React, { useState } from "react";
import WebcamFeed from "./components/WebcamFeed";
import LiveCaptions from "./components/LiveCaptions";
import SpeechSynthesis from "./components/SpeechSynthesis";
import { Card, CardContent } from "./components/ui/Card"; // Import the Card and CardContent components

export default function Home() {
  const [detectedSign, setDetectedSign] = useState("");

  return (
    <div style={{ textAlign: "center", padding: "20px" }}>
      <main className="flex min-h-screen flex-col items-center justify-center p-24">
        <h1 className="text-4xl font-bold mb-8">ASL Live Captions</h1>
        
        {/* Wrap webcam feed and detected sign inside a Card */}
        <Card className="mb-8">
          <CardContent>
            <WebcamFeed onSignDetected={setDetectedSign} />
            <h2>Detected Sign: {detectedSign}</h2>
          </CardContent>
        </Card>

        {/* Wrap live captions and speech synthesis in another Card */}
        <Card className="mb-8">
          <CardContent>
            <LiveCaptions text={detectedSign} />
            <SpeechSynthesis text={detectedSign} />
          </CardContent>
        </Card>
      </main>
    </div>
  );
}



```

### next.config.ts

```typescript
/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  experimental: {
    appDir: true, // Ensure this is enabled
  },
};

module.exports = nextConfig;


```

### tailwind.config.ts

```typescript
import type { Config } from "tailwindcss";

export default {
  content: [
    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
    "./components/**/*.{js,ts,jsx,tsx,mdx}",
    "./app/**/*.{js,ts,jsx,tsx,mdx}",
  ],
  theme: {
    extend: {
      colors: {
        background: "var(--background)",
        foreground: "var(--foreground)",
      },
    },
  },
  plugins: [],
} satisfies Config;

```

### app/globals.css

```css
@tailwind base;
@tailwind components;
@tailwind utilities;

:root {
  --background: #ffffff;
  --foreground: #171717;
}

@media (prefers-color-scheme: dark) {
  :root {
    --background: #0a0a0a;
    --foreground: #ededed;
  }
}

body {
  color: var(--foreground);
  background: var(--background);
  font-family: Arial, Helvetica, sans-serif;
}

```

### app/utils/cn.ts

```typescript
export function cn(...classes: (string | undefined | false)[]): string {
    return classes.filter(Boolean).join(" ");
  }
```

### app/components/LiveCaptions.tsx

```typescript
import { Card, CardContent } from "@/components/ui/Card"

interface LiveCaptionsProps {
  text: string
}

export default function LiveCaptions({ text }: LiveCaptionsProps) {
  return (
    <Card className="mb-4">
      <CardContent className="p-4">
        <h2 className="text-lg font-semibold mb-2">Live Captions</h2>
        <p className="text-xl">{text || "Waiting for ASL interpretation..."}</p>
      </CardContent>
    </Card>
  )
}
```

### app/utils/fingerpose.d.ts

```typescript
declare module "fingerpose" {
    export class GestureEstimator {
      constructor(gestures: Array<{ name: string; confidence: number }>);
      estimate(handPose: number[][], confidence: number): {
        gestures: Array<{ name: string; confidence: number }>;
      };
    }
  
    export const Gestures: {
      ThumbsUpGesture: { name: string; confidence: number };
      VictoryGesture: { name: string; confidence: number };
    };
  }
  
```

### app/utils/drawHand.ts

```typescript
export const drawHand = (predictions: { landmarks: [number, number, number][] }[], ctx: CanvasRenderingContext2D | null) => {
  if (!predictions || !ctx) return;

  predictions.forEach((prediction) => {
    const { landmarks } = prediction;

    for (let i = 0; i < landmarks.length; i++) {
      const [x, y] = landmarks[i]; // Extract only x and y

      ctx.beginPath();
      ctx.arc(x, y, 5, 0, 3 * Math.PI);
      ctx.fillStyle = "red";
      ctx.fill();
    }
  });
};

```

[5 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]