# Project export: Klaw

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2025
- Tagline: Klaw is an accessibility tool that enables seamless laptop interaction through hand gestures, facial expressions, and voice recognition, empowering individuals with mobility and speech challenges.
- Devpost: https://devpost.com/software/klaw
- GitHub: https://github.com/NikhilSura04/KlawTreeHacks2025
- Video: https://www.youtube.com/embed/pZWX64logH0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Nikhil Surapaneni (11 commits)

## Devpost submission (written by the team)

### Inspiration

Klaw was inspired by the need for more intuitive, hands-free ways to interact with technology, especially for individuals with mobility or speech impairments. As technology becomes increasingly integrated into daily life, traditional input methods like keyboards and mice are sometimes insufficient for the nuanced interactions that modern applications demand. We wanted to create an accessibility tool that bridges the gap between users and their devices, allowing the former to take a more relaxed, agile approach towards making the most of their laptops.

### What it does

Klaw enables seamless laptop control through hand gestures, facial expressions, and voice recognition, allowing users to navigate without a keyboard or mouse. It specifically supports features like gesture-based clicking, head-tilt volume control, and real-time speech-to-text captions, making technology more accessible and intuitive for users with mobility and speech impairments. Hand Gesture Control: Move your cursor by pointing with your index finger, perform clicks by pinching your fingers together, and navigate with swipe gestures. This mechanism creates a touch-free alternative to traditional input methods and ensures smooth and responsive tracking, allowing users to interact naturally without additional hardware. Hand Gesture Control: Move your cursor by pointing with your index finger, perform clicks by pinching your fingers together, and navigate with swipe gestures. This mechanism creates a touch-free alternative to traditional input methods and ensures smooth and responsive tracking, allowing users to interact naturally without additional hardware. Facial Expression Detection: Tilt your head left or right to adjust volume, raise your eyebrows to trigger actions, or smile to confirm selections. Facial Expression Detection: Tilt your head left or right to adjust volume, raise your eyebrows to trigger actions, or smile to confirm selections. Real-Time Speech-to-Text: Converts spoken words into live subtitles, aiding users with speech impairments or those in environments where typing is difficult. This feature opens the door for more effective communication and accessibility in various settings, from workspaces to online education platforms. Real-Time Speech-to-Text: Converts spoken words into live subtitles, aiding users with speech impairments or those in environments where typing is difficult. This feature opens the door for more effective communication and accessibility in various settings, from workspaces to online education platforms. Accessible Interaction: Klaw eliminates the need for a physical mouse or keyboard, making laptops more accessible for users with limited mobility. Accessible Interaction: Klaw eliminates the need for a physical mouse or keyboard, making laptops more accessible for users with limited mobility.

### How we built it

We used Google MediaPipe for hand and face tracking, OpenCV for image processing, SpeechRecognition for real-time subtitles, Numpy for the joint angle calculations, Pyautogui for the scrolling behavior, and Pygame to render the test interface. The project was written in Python and with the assistance of related libraries. The core of gesture tracking relies on MediaPipe Hands, which detects hand landmarks and extracts the index finger and thumb positions to enable cursor movement and gesture-based clicking. The program continuously reads frames from the webcam using OpenCV, processes them, and updates the cursor position via PyAutoGUI. Pinch detection, which is used for clicking, is implemented by calculating the Euclidean distance between the index fingertip and thumb tip (mostly by way of Numpy functions). For facial expression-based controls, MediaPipe Face Mesh extracts key facial landmarks to detect movements such as head tilting. The head tilt feature actually compares the vertical positions of the left and right ear landmarks to determine the direction of tilt, which then triggers volume up or down commands. Speech recognition runs in a separate thread to prevent lag, utilizing the SpeechRecognition library to capture microphone input and convert spoken words into real-time text subtitles. This text is then rendered using Pygame, ensuring that subtitles update dynamically on-screen while other interactions remain smooth. Also, to prevent unintended rapid gestures or speech inputs, we implemented threshold-based debouncing for clicking and volume control, as well as speech recognition cooldown timers to avoid excessive updates and oversensitive cursor activity.

### Challenges we ran into

Fine-tuning gesture detection to avoid false positives. Fine-tuning gesture detection to avoid false positives. Handling speech recognition delays and improving accuracy in noisy environments. Handling speech recognition delays and improving accuracy in noisy environments. Preventing over-sensitive cursor clicking when user pinches. Preventing over-sensitive cursor clicking when user pinches. Integrating the real-time angle calculations into the PyGame simulation Integrating the real-time angle calculations into the PyGame simulation

### Accomplishments we're proud of

Successfully integrating multi-modal input (gesture, face, and voice) into a single tool. Successfully integrating multi-modal input (gesture, face, and voice) into a single tool. Improving gesture accuracy for seamless, hands-free control. Improving gesture accuracy for seamless, hands-free control. Making an accessible and already functional prototype that can genuinely help users right away. Making an accessible and already functional prototype that can genuinely help users right away.

### What we learned

How to apply CV libraries in order to track body movements How to apply CV libraries in order to track body movements How to fine-tune thresholds for gesture recognition to balance sensitivity with accuracy How to fine-tune thresholds for gesture recognition to balance sensitivity with accuracy Challenges in live speech-to-text processing and how to improve clarity Challenges in live speech-to-text processing and how to improve clarity

### What's next

Some type of customizable gestures for more personalized controls Some type of customizable gestures for more personalized controls Machine learning enhancements to improve speech-to-text accuracy Machine learning enhancements to improve speech-to-text accuracy Less sensitive mouse controls Less sensitive mouse controls

## README (from the GitHub repository)

# KlawTreeHacks2025
Klaw is an accessibility tool that enables seamless laptop interaction through hand gestures, facial expressions, and voice recognition, empowering individuals with mobility and speech challenges. 

## Inspiration
Klaw was inspired by the need for more intuitive, hands-free ways to interact with technology, especially for individuals with mobility or speech impairments.  As technology becomes increasingly integrated into daily life, traditional input methods like keyboards and mice are sometimes insufficient for the nuanced interactions that modern applications demand. We wanted to create an accessibility tool that bridges the gap between users and their devices, allowing the former to take a more relaxed, agile approach towards making the most of their laptops.

## What it does
Klaw enables seamless laptop control through hand gestures, facial expressions, and voice recognition, allowing users to navigate without a keyboard or mouse. It specifically supports features like gesture-based clicking, head-tilt volume control, and real-time speech-to-text captions.

Hand Gesture Control: Move your cursor by pointing with your index finger, perform clicks by pinching your fingers together, and navigate with swipe gestures.

Facial Expression Detection: Tilt your head left or right to adjust volume, raise your eyebrows to trigger actions, or smile to confirm selections.

Real-Time Speech-to-Text: Converts spoken words into live subtitles, aiding users with speech impairments or those in environments where typing is difficult.

Accessible Interaction: Klaw eliminates the need for a physical mouse or keyboard, making laptops more accessible for users with limited mobility.

## How we built it
We used Google MediaPipe for hand and face tracking, OpenCV for image processing, SpeechRecognition for real-time subtitles, Numpy for the joint angle calculations, and Pygame to render the test interface. The project was entirely done in Python. 

## Challenges we ran into
Fine-tuning gesture detection to avoid false positives.

Handling speech recognition delays and improving accuracy in noisy environments.

Preventing over-sensitive cursor clicking when user pinches.

Integrating the real-time angle calculations into the PyGame simulation

## Accomplishments that we're proud of
Successfully integrating multi-modal input (gesture, face, and voice) into a single tool.

Improving gesture accuracy for seamless, hands-free control.

Making an accessible and already functional prototype that can genuinely help users right away.

## What we learned
How to apply CV libraries in order to track body movements
  
How to fine-tune thresholds for gesture recognition to balance sensitivity with accuracy
  
Challenges in live speech-to-text processing and how to improve clarity

## What's next for Klaw
Some type of customizable gestures for more personalized controls

Machine learning enhancements to improve speech-to-text accuracy

Less sensitive mouse controls


## Detected evidence (automated analysis)

Indexed codebase: 2 recognized source files, 13 KB.
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (2 of 2)

```
Klaw.py
README.md
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Update README.md
- Update README.md
- Update README.md
- Fixed hand scrolling behavior
- Updated volume control with face tilt
- Parser removal
- Cursor motion with index finger
- Face Detection Contours
- Added speech recognition capabilities
- Added first trial run of hand tracking
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### Klaw.py

```python
# Nikhil Surapaneni
# Rishi Bengani
# TreeHacks 2025


import pygame
import cv2
import mediapipe as mp
import time
import numpy as np
import pickle
import argparse
import serial
import pyautogui
import time
import speech_recognition as sr
import threading

# User defined constants
WIDTH, HEIGHT = 640, 480
RECORD_TIME = 10

data = []

# This is the function to calculate the angle between three points
def calculate_angle(point1, point2, point3):
    """
    Calculate the angle between three points
    """
    point1 = np.array(point1)
    point2 = np.array(point2)  # This is the joint point
    point3 = np.array(point3)
    
    vector1 = point1 - point2
    vector2 = point3 - point2
    
    cosine_angle = np.dot(vector1, vector2) / (np.linalg.norm(vector1) * np.linalg.norm(vector2))
    cosine_angle = np.clip(cosine_angle, -1.0, 1.0) 
    
    angle = np.arccos(cosine_angle)
    angle = np.degrees(angle)  # Convert to degrees
    
    return angle

parser = argparse.ArgumentParser()
parser.add_argument("--gesture", help="Gesture to be recorded", type=str, default=None)
parser.add_argument("--data_dir", help="Directory to save data to", type=str, default=None)
args = parser.parse_args()

# Establish serial connection if a port is specified
arduino_serial = None
if args.port:
    arduino_serial = serial.Serial(args.port, 230400)
    print("The port %s is available" % arduino_serial)

if args.knn:
    knn_file = open(args.knn, 'rb')     
    knn = pickle.load(knn_file)

# Mediapipe and Pygame initializations
mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
mp_hands = mp.solutions.hands
mp_face_mesh = mp.solutions.face_mesh  # Add face mesh module
hands = mp_hands.Hands(model_complexity=0, max_num_hands=1)
face_mesh = mp_face_mesh.FaceMesh(max_num_faces=1)  # Initialize face mesh

pygame.init()
screen = pygame.display.set_mode((WIDTH, HEIGHT))
clock = pygame.time.Clock()
font = pygame.font.SysFont(None, 36)  # Font for displaying text

start = time.time()
gesture = ""
face_info = ""

if __name__ == '__main__':
    cap = cv2.VideoCapture(0)
    recognizer = sr.Recognizer()
    subtitle_text = ""  # Store recognized text

    def recognize_speech():
        """Continuously listens for speech and updates subtitle_text."""
        global subtitle_text
        with sr.Microphone() as source:
            recognizer.adjust_for_ambient_noise(source)  # Reduce background noise
            while True:
                try:
                    # adjust phrase time limit depending on our duration of speech
                    audio = recognizer.listen(source, phrase_time_limit=8)  # Listen for speech
                    subtitle_text = recognizer.recognize_google(audio)  # Convert speech to text
                except sr.UnknownValueError:
                    subtitle_text = "..."
                except sr.RequestError:
                    subtitle_text = "Speech recognition unavailable"

    # Run speech recognition in a separate thread to avoid blocking Pygame
    speech_thread = threading.Thread(target=recognize_speech, daemon=True)
    speech_thread.start()
    while cap.isOpened():
        clock.tick(60)
        success, image = cap.read()
        if not success:
            print("Ignoring empty camera frame.")
            continue
        
        # Process the image and detect hands
        results_hands = hands.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
        results_face = face_mesh.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
        
        if results_hands.multi_hand_landmarks:
            for hand_landmarks in results_hands.multi_hand_landmarks:
                # Calculate and display angle for a specific joint as an example
                try:
                    wrist = [hand_landmarks.landmark[mp_hands.HandLandmark.WRIST].x,
                             hand_landmarks.landmark[mp_hands.HandLandmark.WRIST].y,
                             hand_landmarks.landmark[mp_hands.HandLandmark.WRIST].z]

                    index_mcp = [hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_MCP].x,
                                 hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_MCP].y,
                                 hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_MCP].z]

                    index_pip = [hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_PIP].x,
                                 hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_PIP].y,
                                 hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_PIP].z]

                    # Calculate the angle
                    angle = calculate_angle(wrist, index_mcp, index_pip)
                    gesture = f"Hand Angle: {angle:.2f}"  # Display the calculated angle
                except Exception as e:
                    print(f"Error calculating angle: {e}")
                
                # Draw hand landmarks on the original image
                mp_drawing.draw_landmarks(
                    image,
                    hand_landmarks,
                    mp_hands.HAND_CONNECTIONS,
                    mp_drawing_styles.get_default_hand_landmarks_style(),
                    mp_drawing_styles.get_default_hand_connections_style())

        SCREEN_WIDTH, SCREEN_HEIGHT = pyautogui.size()  # Get screen resolution

        CLICK_DELAY = 0.5  # Time in seconds to prevent multiple clicks (adjust as needed)
        last_click_time = 0  # Initialize last click timestamp
        if results_hands.multi_hand_landmarks:
            for hand_landmarks in results_hands.multi_hand_landmarks:
                # Get index finger tip coordinates
                index_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
                thumb_tip = hand_landmarks.landmark[mp_hands.HandLandmark.THUMB_TIP]

                # Flip X-coordinates for natural movement
                screen_x = SCREEN_WIDTH - (index_tip.x * SC
[truncated — 4434 more characters]
```