# Project export: Eyes On AI

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: The future of Accessibility, a smarter way to control your computer.
- Devpost: https://devpost.com/software/eyes-on-ai
- GitHub: https://github.com/Balpreetkaur291/EyesOnAI
- Video: https://www.youtube.com/embed/MN9tr4tc5vs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Balpreet Kaur (5 commits)

## Devpost submission (written by the team)

### Inspiration

A lot of people in our lives have visual impairment issues, our family members, friends, and some of our favorite teachers. We noticed that a lot of the time the accessibility tools that they use are time consuming and tedious to use and our plan was to leverage AI to fix these issues.

### What it does

We created an AI Assistant that runs locally on a users machine and is able to control the operating system and different applications with voice commands. Our project also implements the functionality of other AI accessibility tools like OCR and TTS.

### How we built it

We actually didn't vibe code it! Our app runs locally on the computer while also running a local web server with Flask so its a little too unique for tools like Vercel. We used Python as our main language, Flask for our UI web server, LMNT for TTS, Google's Imagen 4 for OCR, Claude Anthropic 4 for our main LLM, VAPI for AI phone calls, and then we gave the LLMS access to system control scripts and some pretty overlays.

### Challenges we ran into

We spent a lot of time setting up the various AI tools and connecting them together took some debugging. We were also creating software that has an unusual tech stack so we didn't have a template to go off of. We also maxed out API credits a couple times, and at one point we sent our private api keys to a public Github repo and broke our whole app!

### Accomplishments we're proud of

Honestly most of us are just proud we finished, when we initially came up with the idea it was like nothing we had ever built before and we weren't sure if we would be able to make a prototype in time. We are also proud of being about to create a unique AI tool that solves a problem that a lot of us have seen the people we love struggle with. I'm excited to share this project with my political science professor who is blind and also my other teammates' visually impaired uncle. We hope it can make their lives a little easier.

### What we learned

We learned a lot about the new AI tools out there, we didn't know about groq and lmnts fast outputs. We learned about traditional AI models for edge detection for the restricted OCR. We learned about when to use general purpose AI vs when to use more specialized models. We met a lot of smart people and learned more about the industry. We learned how to create operating system control tools. Using selenium and other power control commands.

### What's next

We want to give our tool more access to the computer. Right now we have a list of things we are able to control with the assistant but we want to enable them to control everything. An example would be using the OCR to find the exact pixel value of intractable things on the screen and be able to have better control of the system.

## README (from the GitHub repository)

# EyesOnAi

<p align="center">
	<img width="500" src="logo.png">
</p>

A voice controlled Agentic Accessibility tool that lets visually impaired users operate a laptop hands free. The backend runs on Python and Flask, Whisper handles speech to text, Claude as a ReAct Agent and LMNT reads responses back out loud.

## Motivation

Many of us have loved ones, family members, friends, and even teachers who live with visual impairments. We noticed that while accessibility tools exist, they are often slow, unintuitive, or difficult to use. With the rapid advancements in AI, we saw an opportunity to reimagine accessibility that is faster, smarter, and more user centric.

## How it works

Press the global hotkey to start talking. Speech is transcribed, then Claude reads the request and decides which tools to call to carry it out, using pyautogui for mouse and keyboard actions and subprocess calls to open and control applications. Tool results feed back into the conversation so Claude can chain multiple steps (for example "open TextEdit and write a grocery list" means opening the app, typing the text, and saving the file) before replying, and that reply is read back out loud.

A vision feature lets Gemini look at the screen (or a region the user drags out with an interactive tkinter overlay) and describe what it sees. A summarization step also condenses older parts of the conversation so long sessions do not run out of context.

Destructive actions (deleting files/folders, shutdown, restart) require an explicit spoken "yes confirm" before they run, and file operations are restricted to `~/Documents`, `~/Desktop`, `~/Downloads`, `~/Pictures`, `~/Music`, and `~/Movies`.

## Architecture

| File | Role |
|---|---|
| [app.py](app.py) | Entry point. Runs the Flask server, a PyQt5 always-on-top overlay, and a global hotkey listener. Hosts `AccessibilityChatbot`, the ReAct loop that calls Claude with tool definitions and executes whatever it returns. |
| [computer_commands.py](computer_commands.py) | `AccessibilityCommands` - the actual tool implementations: window/app control, file CRUD, browser navigation, mouse/keyboard automation, screenshots, and vision screen description. |
| [voice_commands.py](voice_commands.py) | `VoiceRecognition` - records microphone audio and transcribes it via Groq's hosted Whisper. |
| [lmnt_utils.py](lmnt_utils.py) | Text-to-speech playback via LMNT. |
| [region_selector.py](region_selector.py) | Standalone tkinter overlay for dragging out a screen region to describe with vision. |
| [templates/index.html](templates/index.html) | Browser UI - chat/status view driven by a server-sent-events stream. |


## Setup

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

Copy `.env.example` to `.env` and fill in your keys:

```
ANTHROPIC_API_KEY=
GROQ_API_KEY=
LMNT_API_KEY=
```

Run it:

```bash
python app.py
```



## Detected evidence (automated analysis)

Indexed codebase: 8 recognized source files, 100 KB.
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- Python (language) — detected in the code
- CSS (language) — claimed on Devpost, not found in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (9 of 9)

```
app.py
chat.py
computer_commands.py
README.md
requirements.txt
sfx.py
templates/index.html
vapiassist.py
voice_commands.py
```

### Dependencies

- requirements.txt: flask, google-generativeai, keyboard, lmnt, psutil, pyautogui, pygame, PyQt5, replicate, requests, selenium, sounddevice, vapi_server_sdk, wavio

### Recent commits (newest first)

- Revise
- Update README.md
- Update README.md
- added logo
- Update README.md
- added logo
- Update app.py
- Added code files
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
replicate
keyboard
PyQt5
psutil
pyautogui
flask
requests
selenium
sounddevice
wavio
pygame
lmnt
google-generativeai
vapi_server_sdk

```

### app.py

```python
from flask import Flask, render_template, request, jsonify
import json
import replicate
import re
import os
from computer_commands import AccessibilityCommands
from voice_commands import VoiceRecognition
import keyboard
import threading
import requests
import sys
import time
from PyQt5 import QtWidgets, QtCore, QtGui
import signal
from sfx import speak, speak_and_save, play_wake_sound
import google as genai


app = Flask(__name__)
app.config['SECRET_KEY'] = 'your-secret-key-here'

class AccessibilityChatbot:
    
    def __init__(self):
        """Initialize the chatbot with Replicate API token"""
        self.commands = AccessibilityCommands()
        self.voice_recognition = VoiceRecognition()
        
        # Initialize replicate client
        replicate.api_token = os.environ["REPLICATE_API_TOKEN"]
        
        # System prompt for the LLM
        self.system_prompt = """You are an accessibility assistant that helps users control their computer through voice commands and text. 

You can execute the following types of commands:

WINDOW MANAGEMENT:
- open_application(app_name) - Open an application
- close_application(app_name) - Close an application  
- minimize_window() - Minimize active window
- maximize_window() - Maximize active window
- close_window() - Close active window
- switch_window() - Switch between windows
- get_active_windows() - List active applications

FILE OPERATIONS:
- create_file(filepath, content) - Create a new file
- create_folder(folderpath) - Create a new folder
- delete_file(filepath) - Delete a file
- delete_folder(folderpath) - Delete a folder
- copy_file(source, destination) - Copy a file
- move_file(source, destination) - Move a file
- rename_file(old_path, new_path) - Rename a file/folder
- search_files(directory, pattern) - Search for files
- list_directory(directory) - List directory contents

SYSTEM SETTINGS:
- set_brightness(level) - Set brightness 0-100 (Windows only)
- shutdown_system() - Shutdown computer
- restart_system() - Restart computer

BROWSER OPERATIONS:
- open_url(url) - Open a URL
- browser_back() - Go back
- browser_forward() - Go forward  
- refresh_page() - Refresh page
- new_tab() - Open new tab
- close_tab() - Close current tab
- switch_tab(direction) - Switch tabs ("next" or "previous")

ACCESSIBILITY FEATURES:
- click_at(x, y) - Click at coordinates
- type_text(text) - Type text
- press_key(key) - Press key (use + for combinations like "ctrl+c")
- scroll(direction, clicks) - Scroll ("up" or "down")
- read_screen(x, y) - Take screenshot or check pixel color
- ocr_screen() - Perform OCR on the screen
- regional_ocr_interactive() - Perform Regional OCR on the screen
- start_vapi_call() - Start VAPI call
When a user asks you to do something, determine which function(s) to call and format your response as:
EXECUTE: function_name(parameters)

For example:
- "What's on my screen?" → EXECUTE: ocr_screen()
- "Open Regional Ocr" → EXECUTE: regional_ocr_interactive()
- "Open Chrome" → EXECUTE: open_application("chrome")
- "Create a file called test.txt" → EXECUTE: create_file("test.txt", "")
- "Press Ctrl+C" → EXECUTE: press_key("ctrl+c")

Always be helpful and explain what you're doing. If you need clarification about a command, ask the user and keep the responses brief and friendly assume the os is windows."""

    def parse_and_execute_command(self, llm_response):
        """Parse LLM response and execute any commands"""
        results = []
        
        # Look for EXECUTE: commands in the response
        execute_pattern = r'EXECUTE:\s*(\w+)\((.*?)\)'
        matches = re.findall(execute_pattern, llm_response)
        
        for function_name, params_str in matches:
            try:
                # Parse parameters
                if params_str.strip():
                    # Handle string parameters with quotes
                    params = []
                    current_param = ""
                    in_quotes = False
                    quote_char = None
                    
                    i = 0
                    while i < len(params_str):
                        char = params_str[i]
                        
                        if char in ['"', "'"] and not in_quotes:
                            in_quotes = True
                            quote_char = char
                        elif char == quote_char and in_quotes:
                            in_quotes = False
                            quote_char = None
                            params.append(current_param)
                            current_param = ""
                        elif char == ',' and not in_quotes:
                            if current_param.strip():
                                # Try to convert to int if it's a number
                                param = current_param.strip()
                                try:
                                    param = int(param)
                                except ValueError:
                                    pass
                                params.append(param)
                            current_param = ""
                        elif in_quotes or char != ' ':
                            current_param += char
                        
                        i += 1
                    
                    # Add the last parameter
                    if current_param.strip():
                        param = current_param.strip()
                        try:
                            param = int(param)
                        except ValueError:
                            pass
                        params.append(param)
                else:
                    params = []
                
                # Execute the command
                if hasattr(self.commands, function_name):
                    function = getattr(self.commands, 
[truncated — 18253 more characters]
```

### vapiassist.py

```python
from vapi import Vapi

client = Vapi(token="vapi_api_key")  # Replace with your actual API key

def make_outbound_call(assistant_id: str, phone_number: str):
    try:
        call = client.calls.create(
            assistant_id=assistant_id,
            phone_number_id="VAPI_PHONE_ID", 
            customer={
                "number": phone_number, 
            },
        )
        
        print(f"Outbound call initiated: {call.id}")
        return call
    except Exception as error:
        print(f"Error making outbound call: {error}")
        raise error


```

### sfx.py

```python
import os
import asyncio
import pygame
import io
from lmnt.api import Speech

def play_wake_sound(sound_file="wake.mp3"):
    """
    Play a wake sound effect from an MP3 file
    
    Args:
        sound_file (str): Path to the MP3 file to play (default: 'wake.mp3')
    """
    try:
        # Initialize pygame mixer if not already initialized
        if not pygame.mixer.get_init():
            pygame.mixer.init()
        
        # Load and play the sound file
        pygame.mixer.music.load(sound_file)
        pygame.mixer.music.play()
            
    except pygame.error as e:
        print(f"Error playing wake sound: {e}")
    except FileNotFoundError:
        print(f"Wake sound file not found: {sound_file}")


def speak(text, voice='leah'):
    """
    Convert text to speech and play it immediately
    
    Args:
        text (str): Text to convert to speech
        voice (str): Voice to use (default: 'leah')
    """
    asyncio.run(_speak_async(text, voice))

async def _speak_async(text, voice):
    """Internal async function for speech synthesis"""
    async with Speech() as speech:
        synthesis = await speech.synthesize(text, voice)
    
    # Initialize pygame mixer if not already initialized
    if not pygame.mixer.get_init():
        pygame.mixer.init()
    
    # Play audio directly from memory
    audio_data = io.BytesIO(synthesis['audio'])
    pygame.mixer.music.load(audio_data)
    pygame.mixer.music.play()
    
    # Wait for playback to finish
    while pygame.mixer.music.get_busy():
        pygame.time.wait(100)

def speak_and_save(text, filename, voice='leah'):
    """
    Convert text to speech, save to file, and play it
    
    Args:
        text (str): Text to convert to speech
        filename (str): Path to save the audio file
        voice (str): Voice to use (default: 'leah')
    """
    asyncio.run(_speak_and_save_async(text, filename, voice))

async def _speak_and_save_async(text, filename, voice):
    """Internal async function for speech synthesis with file saving"""
    async with Speech() as speech:
        synthesis = await speech.synthesize(text, voice)
    
    # Save to file
    with open(filename, 'wb') as f:
        f.write(synthesis['audio'])
    
    # Initialize pygame mixer if not already initialized
    if not pygame.mixer.get_init():
        pygame.mixer.init()
    
    # Play audio directly from memory
    audio_data = io.BytesIO(synthesis['audio'])
    pygame.mixer.music.load(audio_data)
    pygame.mixer.music.play()
    
    # Wait for playback to finish
    while pygame.mixer.music.get_busy():
        pygame.time.wait(100)

```

### voice_commands.py

```python
import sounddevice as sd
import wavio
import replicate
import os
import tempfile
from collections import deque
import numpy as np
class VoiceRecognition:
    def __init__(self, samplerate=16000, channels=1):
        self.samplerate = samplerate
        self.channels = channels
        self.audio_buffer = deque(maxlen=samplerate*10)  # 10-sec buffer
        self.stream = None
        self.is_active = False
        
        # Set up Replicate API token if not already set
        if not os.environ.get("REPLICATE_API_TOKEN"):
            print("Warning: REPLICATE_API_TOKEN not set. Please set it with:")


    def start_voice(self):
        """Start non-blocking recording"""
        if self.stream and self.is_active:
            return "Already recording!"
        
        try:
            # Clear the buffer for new recording
            self.audio_buffer.clear()
            self.is_active = True
            
            self.stream = sd.InputStream(
                samplerate=self.samplerate,
                channels=self.channels,
                callback=self._audio_callback,
                dtype=np.float32  # Specify data type
            )
            self.stream.start()
            return "Recording started"
            
        except Exception as e:
            self.is_active = False
            return f"Error starting recording: {str(e)}"
        
    def _audio_callback(self, indata, frames, time, status):
        """Callback function for audio stream"""
        if status:
            print(f"Audio callback status: {status}")
        
        # Flatten the audio data and extend buffer
        if self.is_active:
            audio_data = indata.flatten() if self.channels == 1 else indata
            self.audio_buffer.extend(audio_data)

    def stop_voice(self):
        """Stop recording and transcribe audio"""
        try:
            if not self.is_active: 
                return "Not currently recording"
            
            self.is_active = False
            
            # Stop and close the audio stream
            if self.stream:
                self.stream.stop()
                self.stream.close()
                self.stream = None
            
            # Check if we have audio data
            if len(self.audio_buffer) == 0:
                return "No audio recorded"
            
            # Convert buffer to numpy array
            audio_data = np.array(list(self.audio_buffer), dtype=np.float32)
            print(f"Audio data shape: {audio_data.shape}")
            print(f"Audio data length: {len(audio_data)/self.samplerate:.2f} seconds")
            
            # Ensure audio data is in the right format for wavio
            if len(audio_data.shape) == 1:
                audio_data = audio_data.reshape(-1, 1)
            
            # Create temporary file and save audio
            with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
                tmp_path = tmp.name
                
            try:
                # Write audio to temporary file
                # Convert float32 to int16 for better compatibility
                audio_int16 = (audio_data * 32767).astype(np.int16)
                wavio.write(tmp_path, audio_int16, self.samplerate, sampwidth=2)
                
                # Transcribe using Replicate
                with open(tmp_path, "rb") as f:
                    output = replicate.run(
                        "vaibhavs10/incredibly-fast-whisper:3ab86df6c8f54c11309d4d1f930ac292bad43ace52d10c80d87eb258b3c9f79c",
                        input={
                            "task": "transcribe",
                            "audio": f, 
                            "language": "None",
                            "timestamp": "chunk",
                            "batch_size": 64,
                            "diarise_audio": False
                        }
                    )
                
                # Clean up temporary file
                os.unlink(tmp_path)
                
                # Return transcription
                if isinstance(output, dict):
                    return output.get("text", output.get("transcription", "No transcription returned"))
                elif isinstance(output, str):
                    return output
                else:
                    return str(output)
                    
            except Exception as e:
                # Clean up temp file in case of error
                if os.path.exists(tmp_path):
                    os.unlink(tmp_path)
                return f"Error during transcription: {str(e)}"
                
        except Exception as e:
            return f"Error stopping recording: {str(e)}"
    
    def get_status(self):
        """Get current recording status"""
        return {
            "is_active": self.is_active,
            "buffer_length": len(self.audio_buffer),
            "buffer_duration_seconds": len(self.audio_buffer) / self.samplerate if self.audio_buffer else 0,
            "samplerate": self.samplerate,
            "channels": self.channels
        }
    
    def cleanup(self):
        """Clean up resources"""
        if self.stream:
            try:
                self.stream.stop()
                self.stream.close()
            except:
                pass
            self.stream = None
        self.is_active = False
        self.audio_buffer.clear()

# Example usage:
if __name__ == "__main__":
    # Make sure to set your Replicate API token
    # os.environ["REPLICATE_API_TOKEN"] = "your_token_here"
    
    voice_rec = VoiceRecognition()
    
    print("Voice Recognition Test")
    print("Commands: 'start' to begin recording, 'stop' to end and transcribe, 'status' for info, 'quit' to exit")
    
    try:
        while True:
            command = input("\nEnter command: ")
[truncated — 858 more characters]
```

### chat.py

```python
import replicate
import json
import re
import os
from computer_commands import AccessibilityCommands

class AccessibilityChatbot:
    def __init__(self):
        """Initialize the chatbot with Replicate API token"""
        self.commands = AccessibilityCommands()
        
        # Initialize replicate client
        replicate.api_token = os.environ["REPLICATE_API_TOKEN"]
        
        # System prompt for the LLM
        self.system_prompt = """You are an accessibility assistant that helps users control their computer through voice commands and text. 

You can execute the following types of commands:

WINDOW MANAGEMENT:
- open_application(app_name) - Open an application
- close_application(app_name) - Close an application  
- minimize_window() - Minimize active window
- maximize_window() - Maximize active window
- close_window() - Close active window
- switch_window() - Switch between windows
- get_active_windows() - List active applications

FILE OPERATIONS:
- create_file(filepath, content) - Create a new file
- create_folder(folderpath) - Create a new folder
- delete_file(filepath) - Delete a file
- delete_folder(folderpath) - Delete a folder
- copy_file(source, destination) - Copy a file
- move_file(source, destination) - Move a file
- rename_file(old_path, new_path) - Rename a file/folder
- search_files(directory, pattern) - Search for files
- list_directory(directory) - List directory contents

SYSTEM SETTINGS:
- set_brightness(level) - Set brightness 0-100 (Windows only)
- shutdown_system() - Shutdown computer
- restart_system() - Restart computer

BROWSER OPERATIONS:
- open_url(url) - Open a URL
- browser_back() - Go back
- browser_forward() - Go forward  
- refresh_page() - Refresh page
- new_tab() - Open new tab
- close_tab() - Close current tab
- switch_tab(direction) - Switch tabs ("next" or "previous")

ACCESSIBILITY FEATURES:
- click_at(x, y) - Click at coordinates
- type_text(text) - Type text
- press_key(key) - Press key (use + for combinations like "ctrl+c")
- scroll(direction, clicks) - Scroll ("up" or "down")
- read_screen(x, y) - Take screenshot or check pixel color

When a user asks you to do something, determine which function(s) to call and format your response as:
EXECUTE: function_name(parameters)

For example:
- "Open Chrome" → EXECUTE: open_application("chrome")
- "Create a file called test.txt" → EXECUTE: create_file("test.txt", "")
- "Press Ctrl+C" → EXECUTE: press_key("ctrl+c")

Always be helpful and explain what you're doing. If you need clarification about a command, ask the user."""

    def parse_and_execute_command(self, llm_response):
        """Parse LLM response and execute any commands"""
        results = []
        
        # Look for EXECUTE: commands in the response
        execute_pattern = r'EXECUTE:\s*(\w+)\((.*?)\)'
        matches = re.findall(execute_pattern, llm_response)
        
        for function_name, params_str in matches:
            try:
                # Parse parameters
                if params_str.strip():
                    # Handle string parameters with quotes
                    params = []
                    current_param = ""
                    in_quotes = False
                    quote_char = None
                    
                    i = 0
                    while i < len(params_str):
                        char = params_str[i]
                        
                        if char in ['"', "'"] and not in_quotes:
                            in_quotes = True
                            quote_char = char
                        elif char == quote_char and in_quotes:
                            in_quotes = False
                            quote_char = None
                            params.append(current_param)
                            current_param = ""
                        elif char == ',' and not in_quotes:
                            if current_param.strip():
                                # Try to convert to int if it's a number
                                param = current_param.strip()
                                try:
                                    param = int(param)
                                except ValueError:
                                    pass
                                params.append(param)
                            current_param = ""
                        elif in_quotes or char != ' ':
                            current_param += char
                        
                        i += 1
                    
                    # Add the last parameter
                    if current_param.strip():
                        param = current_param.strip()
                        try:
                            param = int(param)
                        except ValueError:
                            pass
                        params.append(param)
                else:
                    params = []
                
                # Execute the command
                if hasattr(self.commands, function_name):
                    function = getattr(self.commands, function_name)
                    result = function(*params)
                    results.append(f"✓ {function_name}: {result}")
                else:
                    results.append(f"✗ Unknown command: {function_name}")
                    
            except Exception as e:
                results.append(f"✗ Error executing {function_name}: {str(e)}")
        
        return results

    def get_llm_response(self, user_input, conversation_history=""):
        """Get response from Replicate LLM"""
        try:
            # Prepare the full prompt
            full_prompt = f"{conversation_history}\nUser: {user_input}\nAssistant:"
            
            input_data = {
                "prompt": full_prompt,
                "system_prompt": self.syste
[truncated — 3185 more characters]
```

### computer_commands.py

```python
import os
import sys
import subprocess
import psutil
import pyautogui
import time
import shutil
import glob
from pathlib import Path
import platform
import json
import webbrowser
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
import google.generativeai as genai
from PIL import Image
from sfx import speak
import vapiassist


# Configure pyautogui safety
pyautogui.FAILSAFE = True
pyautogui.PAUSE = 0.5

class AccessibilityCommands:
    def __init__(self):
        self.system = platform.system().lower()
        self.browser_driver = None
    
    def start_vapi_call(self):
        print("Starting VAPI call...")
        vapiassist.make_outbound_call("vapi_id", "target_num")

    def regional_ocr_interactive(self):
        """Interactive regional OCR with immediate processing on mouse release"""
        import tkinter as tk
        from PIL import Image, ImageTk
        
        try:
            # Take a full screenshot first
            screenshot = pyautogui.screenshot()
            
            # Create overlay window
            root = tk.Tk()
            root.title("Select Region for OCR")
            root.attributes('-topmost', True)
            root.configure(bg='black')
            
            # Get screen dimensions
            screen_width = root.winfo_screenwidth()
            screen_height = root.winfo_screenheight()
            
            # Make window fullscreen
            root.geometry(f"{screen_width}x{screen_height}+0+0")
            root.overrideredirect(True)  # Remove window decorations
            
            # Convert PIL image to PhotoImage for tkinter
            display_image = screenshot.resize((screen_width, screen_height), Image.Resampling.LANCZOS)
            photo = ImageTk.PhotoImage(display_image)
            
            # Create canvas
            canvas = tk.Canvas(root, width=screen_width, height=screen_height, 
                            highlightthickness=0, bg='black')
            canvas.pack()
            
            # Display screenshot
            canvas.create_image(0, 0, anchor=tk.NW, image=photo)
            
            # Add semi-transparent overlay
            overlay_id = canvas.create_rectangle(0, 0, screen_width, screen_height, 
                                            fill='black', stipple='gray50')
            
            # Selection variables
            selection_rect = None
            start_x = start_y = current_x = current_y = 0
            ocr_result = None
            
            def start_selection(event):
                nonlocal start_x, start_y, selection_rect
                start_x, start_y = event.x, event.y
                if selection_rect:
                    canvas.delete(selection_rect)
                selection_rect = canvas.create_rectangle(start_x, start_y, start_x, start_y, 
                                                    outline='red', width=10)
            
            def update_selection(event):
                nonlocal current_x, current_y, selection_rect
                current_x, current_y = event.x, event.y
                if selection_rect:
                    canvas.coords(selection_rect, start_x, start_y, current_x, current_y)
            
            def end_selection(event):
                nonlocal ocr_result
                if abs(current_x - start_x) > 10 and abs(current_y - start_y) > 10:
                    # Show processing message
                    canvas.delete("instruction")
                    root.update()  # Force update display
                    
                    try:
                        # Calculate actual coordinates on original screenshot
                        scale_x = screenshot.width / screen_width
                        scale_y = screenshot.height / screen_height
                        
                        x1 = int(min(start_x, current_x) * scale_x)
                        y1 = int(min(start_y, current_y) * scale_y)
                        x2 = int(max(start_x, current_x) * scale_x)
                        y2 = int(max(start_y, current_y) * scale_y)
                        
                        # Configure Gemini API
                        genai.configure(api_key=self.GEMINI_API_KEY)
                        model = genai.GenerativeModel('gemini-1.5-flash')
                        
                        # Extract region from original screenshot
                        region = screenshot.crop((x1, y1, x2, y2))
                        
                        # Create prompt for OCR
                        prompt = """Please tell me what this image is in a concise simple way"""
                        
                        # Generate response
                        response = model.generate_content([prompt, region])
                        ocr_result = response.text.strip()
                        
                    except Exception as e:
                        ocr_result = f"OCR Error: {str(e)}"
                    
                    # Close the overlay immediately after processing
                    root.quit()
                else:
                    # Selection too small, show error briefly then continue
                    canvas.delete("instruction")
                    canvas.create_text(screen_width//2, screen_height//2, 
                                    text="Selection too small! Try again.", 
                                    fill='red', font=('Arial', 16, 'bold'), tags="error")
                    root.after(1500, lambda: canvas.delete("error"))  # Remove error after 1.5 seconds
                    root.after(1500, lambda: canvas.create_text(50, 50, text=instruction_text, 
                                  
[truncated — 18692 more characters]
```

### templates/index.html

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Accessibility Assistant</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }

        html, body {
            height: 100%;
            overflow: hidden;
        }

        body {
            font-family: 'SF Pro Display', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
            background: linear-gradient(135deg, #1a0b2e 0%, #2d1b69 30%, #4c1d95 60%, #6b21a8 100%);
            position: relative;
            padding: 16px;
            display: flex;
            align-items: center;
            justify-content: center;
        }
        
        body::before {
            content: '';
            position: fixed;
            top: 0;
            left: 0;
            right: 0;
            bottom: 0;
            background: 
                radial-gradient(circle at 20% 20%, rgba(147, 51, 234, 0.4) 0%, transparent 50%),
                radial-gradient(circle at 80% 80%, rgba(168, 85, 247, 0.3) 0%, transparent 50%),
                radial-gradient(circle at 40% 60%, rgba(196, 181, 253, 0.2) 0%, transparent 50%);
            animation: backgroundPulse 15s ease-in-out infinite;
            z-index: -2;
        }
        
        body::after {
            content: '';
            position: fixed;
            top: 0;
            left: 0;
            right: 0;
            bottom: 0;
            background: 
                radial-gradient(circle at 60% 20%, rgba(139, 92, 246, 0.1) 0%, transparent 60%),
                radial-gradient(circle at 20% 80%, rgba(124, 58, 237, 0.15) 0%, transparent 60%);
            animation: backgroundFloat 20s ease-in-out infinite reverse;
            z-index: -1;
        }
        
        @keyframes backgroundPulse {
            0%, 100% { 
                transform: scale(1) rotate(0deg);
                opacity: 1;
            }
            50% { 
                transform: scale(1.1) rotate(5deg);
                opacity: 0.8;
            }
        }
        
        @keyframes backgroundFloat {
            0%, 100% { transform: translateY(0px) translateX(0px); }
            33% { transform: translateY(-20px) translateX(10px); }
            66% { transform: translateY(10px) translateX(-15px); }
        }
        
        .container {
            width: 100%;
            max-width: 900px;
            height: calc(100vh - 32px);
            backdrop-filter: blur(25px);
            background: rgba(255, 255, 255, 0.08);
            border-radius: 24px;
            border: 2px solid rgba(255, 255, 255, 0.2);
            box-shadow: 
                0 20px 60px rgba(0, 0, 0, 0.4),
                0 8px 32px rgba(139, 92, 246, 0.15),
                inset 0 1px 0 rgba(255, 255, 255, 0.2),
                inset 0 -1px 0 rgba(255, 255, 255, 0.1);
            overflow: hidden;
            animation: containerFloat 1s ease-out;
            position: relative;
            display: flex;
            flex-direction: column;
        }
        
        .container::before {
            content: '';
            position: absolute;
            top: 0;
            left: 0;
            right: 0;
            bottom: 0;
            background: linear-gradient(135deg, rgba(255, 255, 255, 0.12) 0%, transparent 50%, rgba(139, 92, 246, 0.08) 100%);
            z-index: -1;
        }
        
        @keyframes containerFloat {
            from {
                opacity: 0;
                transform: translateY(40px) scale(0.9);
            }
            to {
                opacity: 1;
                transform: translateY(0) scale(1);
            }
        }
        
        .header {
            padding: 20px 24px;
            text-align: center;
            background: rgba(255, 255, 255, 0.08);
            backdrop-filter: blur(20px);
            border-bottom: 2px solid rgba(255, 255, 255, 0.15);
            position: relative;
            overflow: hidden;
            flex-shrink: 0;
        }
        
        .header::before {
            content: '';
            position: absolute;
            top: -2px;
            left: -2px;
            right: -2px;
            bottom: -2px;
            background: linear-gradient(45deg, 
                transparent 0%, 
                rgba(196, 181, 253, 0.15) 25%, 
                rgba(139, 92, 246, 0.08) 50%, 
                rgba(196, 181, 253, 0.15) 75%, 
                transparent 100%);
            z-index: -1;
            animation: borderShimmer 4s linear infinite;
        }
        
        @keyframes borderShimmer {
            0% { transform: translateX(-100%) rotate(45deg); }
            100% { transform: translateX(200%) rotate(45deg); }
        }
        
        .header h1 {
            font-size: 24px;
            font-weight: 700;
            background: linear-gradient(135deg, #f8fafc 0%, #e2e8f0 30%, #c084fc 100%);
            -webkit-background-clip: text;
            -webkit-text-fill-color: transparent;
            background-clip: text;
            margin-bottom: 6px;
            position: relative;
            z-index: 1;
            text-shadow: 0 0 30px rgba(196, 181, 253, 0.3);
        }
        
        .header p {
            color: rgba(255, 255, 255, 0.85);
            font-size: 14px;
            font-weight: 400;
            position: relative;
            z-index: 1;
        }
        
        .status {
            padding: 12px 24px;
            text-align: center;
            font-weight: 600;
            font-size: 14px;
            background: rgba(255, 255, 255, 0.08);
            backdrop-filter: blur(15px);
            border-bottom: 2px solid rgba(255, 255, 255, 0.12);
            color: 
[truncated — 25563 more characters]
```