# Project export: Turing This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Automate mundane, repetitive desktop tasks that don't need you. - Devpost: https://devpost.com/software/turing - GitHub: https://github.com/ekagra1602/turing - Team: 3 GitHub contributor(s) — Aryan Keluskar (27 commits), ekagra1602 (10 commits), Soham Daga (4 commits) ## Devpost submission (written by the team) ### Inspiration Millions of hours are collectively wasted on computers every single day. Instead of going out to lunch with your friends, you're resolving repetitive support tickets and answering questions that have been asked 50 other times. Steve Jobs said in the 1980s that computers should be natural extensions of humans, like bicycles for the mind. Moreover, he said that computers would anticipate exactly what the user wants to do next by themselves. Over 40 years later, computers are still fully reliant on users directly interacting through the mouse and keyboard interface. Turing is built to modernize this with context of how you use your computer and with a faster speed and degree of accuracy than the blind desktop agents that exist today are just by themselves. ### What it does Turing watches how you do tasks, how you reply to messages, how you work, and personalizes workflows to follow your patterns and automate your desktop use while you go out and grab a meal with your friends. ### How we built it We combine context retained from recorded desktop workflows processed and encoded by Vision Language Models (Gemini 2.5 Flash), which analyze screen recordings to extract semantic actions. For instance, it learns "click Submit button" and not just "click at co-ordinates (543, 210)". These workflows are decoded and executed by personalized desktop agents powered by Gemini Computer Use, which provides vision-based screen understanding and adaptive execution. Workflow storage and retrieval uses Snowflake Cloud, storing semantic actions, parameters, and metadata. Workflow matching uses Snowflake Vector Search for direct semantic similarity analysis, comparing user requests to stored workflow intentions. We enable speech-to-text via Groq Whisper and TTS human-like responses with ElevenLabs (using Eleven Turbo v2.5) through LiveKit for real-time voice pipelines. The voice agent uses Groq Llama 3.3 70B for conversational intelligence. This enables a truly autonomous agent that learns to automate your workflows with no hands required. ### Challenges we ran into One of the major challenges from the start was to get the desktop agent to convert raw text instructions into actionable steps from the desktop assistant. Another huge challenge was to encoding the actions using the screen recordings. We ran into a few issues with screen recording alongside tkinter on python, since the process would get locked and Mac OS would throw a SIGTRAP error. So, we switched to using the AVFoundation codex with ffmpeg to record videos on a subprocess and that resolved the screen recording bug with Mac OS. It took us several iterations of trial and error to find the right amount of detail that the VLM would output based on screenshots and action timestamps, such that the executor agent can both generalize and reproduce workflows that the user would have recorded earlier. ### What's next One of the immediate first things would be to make it come with TONS of pre-recorded workflows for diverse software and use-cases, so that it can be smarter on new workflows but also not need a recording for existing ones. We would also love to make it easier to chain workflows so you can mix-and-match steps or build on top of other workflows, kinda like Lego. Another idea we would love to explore is to enable all of this via the cloud. This could let you, theoretically, chain and run complicated workflows on your computer using your phone, even when you are nowhere physically close to it. ## README (from the GitHub repository) # Turing - Learn by Observation **Turing** is an AI agent that learns by watching you work. Like an intern that shadows you, learns your workflows, and then executes them autonomously. ## 🎯 Vision Imagine telling your computer: > "Open my DataVis class on Canvas and clone the notebook" And it just... does it. Because it watched you do it once for your Machine Learning class. That's Turing. ## ✨ Features ### 🔴 Record Mode - Click "record", perform your workflow naturally - System captures: - Every click, scroll, and keystroke - Screenshots before/after each action - Visual context (what you clicked on) - OCR of text elements ### 🧠 Visual Learning - AI analyzes your recording to understand: - What steps you took - What the workflow accomplishes - Which values are parameters (e.g., class names) - Visual signatures of UI elements ### 🔄 Smart Replay - Tell it what you want in natural language - System: - Finds matching workflow - Extracts new parameters from your request - Executes workflow with visual guidance - Uses OCR to locate elements dynamically ### 📚 Workflow Library - Store unlimited workflows - Search by name, description, tags - Export/import workflow packages - Track usage statistics ## 🏗️ Architecture image ``` ┌─────────────────────────────────────────────┐ │ User Interface │ │ "Open my DataVis class on Canvas" │ └─────────────────┬───────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────┐ │ Workflow Matching Engine │ │ • Find similar learned workflows │ │ • Extract parameters from user request │ │ • Calculate confidence score │ └─────────────────┬───────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────┐ │ Visual Memory │ │ workflows/ │ │ ├── {uuid}/ │ │ │ ├── metadata.json │ │ │ ├── steps/ │ │ │ │ ├── step_001.json │ │ │ │ ├── step_001_before.png │ │ │ │ └── step_001_after.png │ └─────────────────┬───────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────┐ │ Visual-Guided Execution │ │ 1. Take screenshot │ │ 2. Use OCR to find target element │ │ 3. Use Vision LLM to understand UI │ │ 4. Calculate click coordinates │ │ 5. Execute action │ │ 6. Verify state change │ └─────────────────────────────────────────────┘ ``` ## 🚀 Quick Start ### Installation ```bash # Navigate to backend directory cd Turing/backend # Activate virtual environment source venv/bin/activate # Install dependencies pip install -r requirements.txt ``` ### Set API Key ```bash export GOOGLE_API_KEY='your_gemini_api_key_here' ``` ### Run Enhanced Agent ```bash python agent_enhanced.py ``` ## 📖 Usage Guide ### Recording a Workflow 1. Start the enhanced agent: ```bash python agent_enhanced.py ``` 2. Enter `record` command 3. Provide workflow details: ``` Workflow name: Open Canvas Class Description: Navigate to Canvas and open a specific class Tags: canvas, education ``` 4. **Perform your workflow naturally** - the system is watching! - Open browser - Navigate to canvas.asu.edu - Click on your class - Do whatever you need to do 5. When done, enter `stop` command 6. System analyzes and identifies parameters: ``` 📊 Identified Parameters: - class_name: Name of the class to open Example: Machine Learning ✅ Workflow saved! ``` ### Using a Learned Workflow Just describe what you want: ``` 💬 Open my DataVis class on Canvas ✨ Found matching workflow: Open Canvas Class Confidence: 90% Execute this workflow? [Y/n]: y 🎬 Executing learned workflow... ✅ Done! ``` ### List All Workflows ``` 💬 list 📚 Learned Workflows: ================================================================= Open Canvas Class └─ Navigate to Canvas and open a specific class Steps: 3 | Uses: 5 Parameters: class_name Tags: canvas, education Download Bank Statement └─ Log into bank and download statement PDF Steps: 8 | Uses: 2 Parameters: month, year Tags: finance, banking ``` ## 🛠️ Components ### 1. **visual_memory.py** Stores workflows with complete visual context. ```python from visual_memory import VisualWorkflowMemory memory = VisualWorkflowMemory() # Create workflow wf_id = memory.create_workflow( name="My Workflow", description="What it does", tags=["tag1", "tag2"] ) # Add steps memory.add_step( workflow_id=wf_id, action_type='click', action_data={'x': 500, 'y': 300}, screenshot_before=screenshot, screenshot_after=screenshot, visual_context={'clicked_text': 'Submit'} ) # Finalize memory.finalize_workflow(wf_id, parameters=[...]) ``` ### 2. **recorder.py** Monitors user actions and captures visual context. ```python from recorder import WorkflowRecorder recorder = WorkflowRecorder() # Start recording wf_id = recorder.start_recording("My Workflow") # User performs actions... # System automatically captures everything # Stop recording recorder.stop_recording() ``` ### 3. **visual_analyzer.py** Extracts meaning from screenshots using OCR and computer vision. ```python from visual_analyzer import VisualAnalyzer analyzer = VisualAnalyzer() # Analyze what was clicked context = analyzer.analyze_click_context( screenshot, click_x=500, click_y=300 ) print(context['clicked_text']) # "Submit Button" # Find text in screenshot matches = analyzer.find_text_in_screenshot( screenshot, target_text="Machine Learning" ) for match in matches: print(f"Found at: {match['center']}") ``` ### 4. **agent_enhanced.py** Main interface with recording and learned execution. ## 🔬 Advanced Topics ### Parameter Identification The system uses Google's Gemini LLM to analyze workflows and identify parameters: ``` Workflow: Open Canvas Class Steps: 1. Navigate to https://canvas.asu.edu 2. Click on "Machine Learning" 3. Click on "Assignments" AI identifies: - "Machine Learning" is a parameter (varies per class) - "Assignments" is NOT a parameter (always same) ``` ### Visual Element Matching When executing with new parameters, system uses multiple strategies: 1. **OCR Text Matching**: Find text "DataVis" on screen 2. **Visual Similarity**: Compare to recorded element appearance 3. **Position Heuristics**: Similar elements often in same region 4. **Vision LLM**: Ask AI "where is the DataVis class link?" ### Confidence Scoring ```python if confidence > 0.9: # Execute automatically elif confidence > 0.7: # Ask for confirmation else: # Ask user to demonstrate ``` ## 📊 Storage Format Workflows are stored as structured directories: ``` workflows/ ├── 550e8400-e29b-41d4-a716-446655440000/ │ ├── metadata.json │ ├── steps/ │ │ ├── step_001.json │ │ ├── step_001_before.png │ │ ├── step_001_after.png │ │ ├── step_002.json │ │ ├── step_002_before.png │ │ └── step_002_after.png ``` **metadata.json**: ```json { "workflow_id": "550e8400-...", "name": "Open Canvas Class", "description": "Navigate to Canvas and open class", "tags": ["canvas", "education"], "created": "2025-10-25T10:30:00", "status": "ready", "steps_ [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 251 recognized source files, 2213 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository - AI coding agent: Cursor — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 277) ``` .cursor/rules/markdown-files.mdc .gitignore backend/.gitignore backend/ADD_WORKDAY_WORKFLOW_GUIDE.txt backend/check_canvas.py backend/DEMO_READY.txt backend/END_TO_END_TUTORIAL.md backend/execute_workflow.py backend/extract_structure.py backend/gemini_computer_use.py backend/gemini_workflow_executor.py backend/gemini_workflow_executor.py.backup backend/intelligent_workflow_system.py backend/parameterize_workday.py backend/QUICK_START.md backend/QUICKSTART.md backend/README.md backend/READY_FOR_DEMO.md backend/RECORD_WORKDAY_WORKFLOW.txt backend/RECORD_WORKFLOW_GUIDE.txt backend/recorder_ui.py backend/recorder.py backend/requirements_fast.txt backend/semantic_action_analyzer.py backend/semantic_workflow_matcher.py backend/simple_player.py backend/simple_recorder.py backend/snowflake_workflow_memory.py backend/snowflake_workflow_memory.py.backup backend/START_HERE.txt backend/START_RECORDER.sh backend/test_avfoundation.py backend/test_click_accuracy.py backend/test_executor_workflows.py backend/test_video_encoding.py backend/test_workflow_templates.py backend/test_workflows_system_prompt.py backend/video_analyzer.py backend/VIDEO_RECORDER_README.md backend/video_recorder.py backend/visual_memory.py backend/voice/.env.example backend/voice/.gitignore backend/voice/agent.py backend/voice/frontend/.env.example backend/voice/frontend/.gitignore backend/voice/frontend/app/api/token/route.ts backend/voice/frontend/app/globals.css backend/voice/frontend/app/layout.tsx backend/voice/frontend/app/layout.tsx.backup backend/voice/frontend/app/page.tsx backend/voice/frontend/components/VoiceOrb.tsx backend/voice/frontend/ELECTRON.md backend/voice/frontend/electron/main.js backend/voice/frontend/electron/preload.js backend/voice/frontend/launch-overlay.sh backend/voice/frontend/lib/audioAnalyser.ts backend/voice/frontend/next.config.js backend/voice/frontend/package.json backend/voice/frontend/postcss.config.js backend/voice/frontend/README.md backend/voice/frontend/start.sh backend/voice/frontend/tailwind.config.ts backend/voice/IMPLEMENTATION_SUMMARY.md backend/voice/OVERVIEW.md backend/voice/QUICKSTART.md backend/voice/README.md backend/voice/requirements.txt backend/voice/run_all.sh backend/voice/run_overlay.sh backend/voice/run_voice_integration.sh backend/voice/setup.sh backend/voice/test_apis.py backend/voice/test_voice_integration.py backend/voice/test_voice.py backend/voice/VOICE_INTEGRATION.md backend/voice/voice_workflow_bridge.py backend/WORKDAY_TEMPLATE_SNIPPET.py backend/workflow_cli.py backend/workflow_gui.py backend/WORKFLOW_TEMPLATES_QUICK_START.md backend/workflow_templates.py backend/WORKFLOWS_SYSTEM_PROMPT.md calibration.json CLAUDE.md database/__init__.py database/client.py database/examples.py database/helpers.py database/operations.py database/README.md DEMO_GUIDE.md docs/ARCHITECTURE.md docs/CALHACKS_DEMO.md docs/CLICK_ACCURACY_FIX.md docs/DEBUG_CLICKS.md docs/ENHANCED_MODE.md docs/FAST_COMPUTER_USE_ARCHITECTURE.md docs/FIXES_APPLIED.md docs/IMPLEMENTATION_STATUS.md docs/INTELLIGENT_WORKFLOWS.md docs/LATEST_FIX.md docs/OVERLAY_GUIDE.md docs/PERMISSIONS_FIX.md docs/PROJECT_STATUS.md docs/PROJECT_SUMMARY.md docs/QUICKREF.md docs/QUICKSTART.md docs/README.md docs/RESEARCH.md docs/SEMANTIC_WORKFLOW_SYSTEM.md docs/SNOWFLAKE_INTEGRATION.md docs/VIDEO_SYSTEM.md docs/WHATS_NEW.md docs/WORKFLOW_TEMPLATES.md README_PRO.md requirements.txt src/action_player.py src/action_recorder.py src/enhanced_overlay.py [157 more files omitted for size] ``` ### Dependencies - backend/voice/frontend/package.json: @livekit/components-react@^2.6.3, @types/node@^20, @types/react@^18, @types/react-dom@^18, autoprefixer@^10.4.20, concurrently@^8.2.2, electron@^28.1.0, electron-builder@^24.9.1, eslint@^8, eslint-config-next@14.2.15, framer-motion@^11.11.11, livekit-client@^2.5.7, livekit-server-sdk@^2.6.1, next@^14.2.33, postcss@^8.4.49, react@^18.3.1, react-dom@^18.3.1, tailwindcss@^3.4.15, typescript@^5, wait-on@^7.2.0 - backend/voice/requirements.txt: livekit-agents[silero,turn-detector]@~=1.2, livekit-plugins-elevenlabs@~=1.2, livekit-plugins-groq@~=1.2, livekit-plugins-noise-cancellation@~=0.2, python-dotenv@~=1.0 - requirements.txt: customtkinter@>=5.0.0, pillow@>=9.0.0, pyautogui@>=0.9.50, pynput@>=1.7.0 - WALT/pyproject.toml: absl-py@>=1.0.0, aiofiles@>=23.0.0, aiolimiter@>=1.0.0, beartype@>=0.22.2, beautifulsoup4@>=4.9.0, black@>=23.0.0, boto3@>=1.28.0, cachetools@>=5.0.0, evaluate@>=0.4.0, faiss-cpu@>=1.7.4, fastapi@>=0.104.0, ffmpy@==0.6.3, langchain@>=0.1.0, langchain_community@>=0.3.31, langchain-anthropic@>=0.1.0, langchain-aws@>=0.2.35, langchain-google-genai@>=2.1.12, langchain-openai@>=0.0.5, lxml@>=4.9.0, markdownify@>=0.11.0, mypy@>=1.0.0, nltk@>=3.8.0, numpy@>=1.24.0, openai@>=1.0.0, patchright@>=1.0.0, pillow@>=10.0.0, playwright@>=1.40.0, posthog@>=3.0.0, pydantic@>=2.0.0, pydantic-settings@>=2.0.0, pytest@>=7.0.0, pytest-asyncio@>=0.21.0, python-dotenv@>=1.0.0, pyyaml@>=6.0.0, requests@>=2.31.0, rich@>=13.0.0, rouge-score@>=0.1.2, ruff@>=0.1.0, scikit-image@>=0.25.2, sentence-transformers@>=2.2.0, torch@>=2.0.0, tqdm@>=4.65.0, transformers@>=4.30.0, typer@>=0.9.0, uvicorn@>=0.24.0, walt[dev,recorder] ### Recent commits (newest first) - push - Merge remote-tracking branch 'origin/main' - workday is work - added package json - Updated name to turing - Merge remote-tracking branch 'origin/main' - workday - Changed electron to run both together - more fixes - Update README.md with architecture image - Added electron app - Update model references from "gemini-2.0-flash-exp" to "gemini-2.0-flash" across multiple files, enhancing consistency in model usage. Introduce task completion verification and adaptive action planning in GeminiWorkflowExecutor for improved execution accuracy. - Implement step-by-step intelligent execution in GeminiWorkflowExecutor, enhancing context-awareness and adaptive planning based on current screen state. - Enhance GeminiWorkflowExecutor with robust action execution and adaptive strategies - soham - 🐻 - pog - cal-winner - kms - revert ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Project Overview AgentFlow is a dual-purpose macOS automation tool: 1. **Basic/Pro Mode** (root `/src`): Simple GUI overlay for recording and replaying user interactions (clicks, keyboard, movements) 2. **Backend AI Agent** (`/backend`): Advanced AI-powered system that learns workflows by observation using Google Gemini for visual analysis and natural language understanding ## Core Architecture ### Two Independent Systems **GUI Frontend** (`/src`): - `minimal_overlay.py` - Basic mode: click-only recording with instant replay - `enhanced_overlay.py` - Pro mode: full interaction recording (movements, keyboard, scrolling, drag-drop) - `action_recorder.py` / `action_player.py` - Basic recording/playback engine - `enhanced_recorder.py` / `enhanced_player.py` - Pro recording/playback with timing - `window_manager.py` - Window state management utilities **AI Backend** (`/backend`): - `agent_enhanced.py` - Main conversational interface with learn-by-observation capabilities - `visual_memory.py` - Stores workflows with screenshots and visual context in structured directories - `recorder.py` - Monitors user actions and captures visual context during demonstrations - `visual_analyzer.py` - OCR and computer vision for extracting UI element information - `agent_interface.py` - Base agent with Gemini integration for computer control - `computer_use_simple.py` - Core computer control implementation using Gemini ### Visual Workflow Storage Format Workflows are stored as structured directories under `workflows/`: ``` workflows/{workflow_id}/ ├── metadata.json # Name, description, tags, parameters ├── steps/ │ ├── step_001.json # Action data + visual context │ ├── step_001_before.png │ └── step_001_after.png ``` The `metadata.json` contains workflow metadata including identified parameters (values that can vary between executions, e.g., class names). The system uses Gemini LLM to automatically identify parameters by analyzing recorded workflows. ### Parameter Identification System The backend uses AI to analyze workflows and identify which values are parameters (variables) vs constants: - Recorded workflow steps include visual context (clicked text, OCR results, element types) - After recording, Gemini analyzes the workflow description to identify parameters - Parameters are stored with name, type, example value, step number, and description - During execution, the system extracts parameter values from natural language requests ## Development Commands ### GUI Frontend (Basic/Pro Mode) ```bash # Launch basic mode (clicks only, fast) ./start_basic.sh # Launch pro mode (full interactions: movements, keyboard, drags) ./start_pro.sh # Both scripts auto-create venv_gui and install dependencies on first run # Dependencies: pyautogui, pynput, pillow # Test coordinate accuracy ./venv_gui/bin/python tools/test [truncated — 4142 more characters] ``` ### DEMO_GUIDE.md ```markdown # AgentFlow Demo Guide ## Intelligent Workflow Automation System --- ## 🎯 Demo Overview **What you'll demonstrate:** 1. **Record** a Canvas workflow once (download assignment for ML class) 2. **Execute** it automatically for other classes (Data Mining, Data Visualization) 3. Show the agent **learns intent**, not just clicks **Time**: 5-10 minutes **Impact**: Shows programming by demonstration → generalizable automation --- ## 🛠️ Setup (Before Demo) ### 1. Set Environment Variables ```bash # Required export GOOGLE_API_KEY='your_gemini_api_key_here' export SNOWFLAKE_ACCOUNT='your_account' export SNOWFLAKE_WAREHOUSE='COMPUTE_WH' export SNOWFLAKE_DATABASE='AGENTFLOW_DB' export SNOWFLAKE_SCHEMA='PUBLIC' ``` ### 2. Install Dependencies ```bash cd /Users/aryank/Developer/CalHacks2025/agentflow/backend pip3 install -r requirements_fast.txt ``` ### 3. Test System ```bash cd /Users/aryank/Developer/CalHacks2025/agentflow/backend python3 -c "from intelligent_workflow_system import IntelligentWorkflowSystem; system = IntelligentWorkflowSystem(); print('✅ System ready!')" ``` --- ## 📋 Demo Script: Canvas Assignment Download ### **Phase 1: Recording the Workflow** (3 minutes) **Setup:** - Open Terminal - Have Canvas website ready in browser - Have 2-3 classes visible on Canvas **Script:** ```bash cd /Users/aryank/Developer/CalHacks2025/agentflow/backend python3 workflow_cli.py ``` **In the CLI:** ``` 💬 Your request: record Download Canvas Assignment for ML # System starts recording... # YOU SAY: "Now I'll demonstrate downloading an assignment from Canvas" ``` **Perform the workflow naturally:** 1. **Open Browser** (if not open): - Press `Cmd+Space` - Type "chrome" - Press Enter 2. **Navigate to Canvas**: - Click address bar - Type "canvas.asu.edu" (or your Canvas URL) - Press Enter 3. **Select ML Course**: - Click on "Machine Learning" course card/link 4. **Go to Assignments**: - Click "Assignments" in left sidebar 5. **Download Assignment**: - Click on latest assignment (e.g., "Homework 3") - Click "Download" or "Submit" button **Stop Recording:** ``` 💬 Your request: stop # System analyzes workflow... # 🧠 Analyzing workflow to understand intent... # ✅ Workflow understood! # 5 semantic actions # 1 parameter identified: course_name ``` **What just happened:** - ✅ System recorded raw clicks/keys - 🧠 Gemini analyzed and understood: "User wants to download assignment for a course" - 🎯 Identified parameter: `course_name = "Machine Learning"` - ☁️ Stored in Snowflake (or local) --- ### **Phase 2: Executing with Different Parameters** (2 minutes) **YOU SAY:** "Now watch - I can use natural language to do this for ANY class" **Execute for Data Mining:** ``` 💬 Your request: Download Canvas assignment for Data Mining # System output: # 🔍 Finding similar workflows... # ✓ Found 1 similar workflow(s): # 1. Download Canvas Assignment for ML (similarity: 92%) # # 🎯 Extracting parameters fro [truncated — 6116 more characters] ``` ### requirements.txt ``` pyautogui>=0.9.50 pynput>=1.7.0 customtkinter>=5.0.0 pillow>=9.0.0 # Required by pyautogui ``` ### WALT/pyproject.toml ``` [build-system] requires = ["setuptools>=65", "wheel"] build-backend = "setuptools.build_meta" [project] name = "sfr-walt" version = "0.1.0" description = "Web Agents that Learn Tools - Automatic tool discovery from websites" readme = "README.md" requires-python = ">=3.11" license = {text = "Apache-2.0"} authors = [ {name = "Viraj Prabhu", email = "viraj.prabhu@salesforce.com"} ] keywords = ["web-agents", "llm", "automation", "tool-discovery", "browser-use"] classifiers = [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.11", "Topic :: Scientific/Engineering :: Artificial Intelligence", ] dependencies = [ "playwright>=1.40.0", "patchright>=1.0.0", "langchain>=0.1.0", "langchain-openai>=0.0.5", "langchain-anthropic>=0.1.0", "pydantic>=2.0.0", "pydantic-settings>=2.0.0", "typer>=0.9.0", "rich>=13.0.0", "aiofiles>=23.0.0", "python-dotenv>=1.0.0", "markdownify>=0.11.0", "requests>=2.31.0", "numpy>=1.24.0", "sentence-transformers>=2.2.0", "pyyaml>=6.0.0", "faiss-cpu>=1.7.4", "posthog>=3.0.0", "beautifulsoup4>=4.9.0", "lxml>=4.9.0", "boto3>=1.28.0", "langchain-google-genai>=2.1.12", "langchain-aws>=0.2.35", "ffmpy==0.6.3", "langchain_community>=0.3.31", ] [project.optional-dependencies] dev = [ # Testing "pytest>=7.0.0", "pytest-asyncio>=0.21.0", # Code quality "black>=23.0.0", "ruff>=0.1.0", "mypy>=1.0.0", # Benchmarking "torch>=2.0.0", "transformers>=4.30.0", "pillow>=10.0.0", "nltk>=3.8.0", "evaluate>=0.4.0", "beartype>=0.22.2", "aiolimiter>=1.0.0", "cachetools>=5.0.0", "tqdm>=4.65.0", "openai>=1.0.0", "scikit-image>=0.25.2", "absl-py>=1.0.0", "rouge-score>=0.1.2", ] recorder = [ "uvicorn>=0.24.0", "fastapi>=0.104.0", ] all = [ "walt[dev,recorder]", ] [project.scripts] walt = "walt.cli:main" [project.urls] Homepage = "https://github.com/salesforceairesearch/walt" Documentation = "https://github.com/salesforceairesearch/walt#readme" Repository = "https://github.com/salesforceairesearch/walt" Issues = "https://github.com/salesforceairesearch/walt/issues" [tool.setuptools] package-dir = {"" = "src"} [tool.setuptools.packages.find] where = ["src"] include = ["walt*"] [tool.black] line-length = 100 target-version = ['py311'] [tool.ruff] line-length = 100 target-version = "py311" [tool.mypy] python_version = "3.11" warn_return_any = true warn_unused_configs = true disallow_untyped_defs = false ``` ### backend/voice/requirements.txt ``` # LiveKit Agents Framework livekit-agents[silero,turn-detector]~=1.2 livekit-plugins-noise-cancellation~=0.2 # AI Plugins livekit-plugins-groq~=1.2 # Groq (STT + LLM) livekit-plugins-elevenlabs~=1.2 # ElevenLabs (TTS) # Environment management python-dotenv~=1.0 ``` ### backend/voice/frontend/package.json ``` { "name": "agentflow-voice-overlay", "version": "0.1.0", "private": true, "main": "electron/main.js", "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint", "electron": "NODE_ENV=development electron .", "electron:dev": "concurrently \"npm run dev\" \"wait-on http://localhost:3000 && npm run electron\"", "electron:build": "next build && electron-builder" }, "dependencies": { "@livekit/components-react": "^2.6.3", "framer-motion": "^11.11.11", "livekit-client": "^2.5.7", "livekit-server-sdk": "^2.6.1", "next": "^14.2.33", "react": "^18.3.1", "react-dom": "^18.3.1" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.4.20", "concurrently": "^8.2.2", "electron": "^28.1.0", "electron-builder": "^24.9.1", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8.4.49", "tailwindcss": "^3.4.15", "typescript": "^5", "wait-on": "^7.2.0" }, "build": { "appId": "com.agentflow.voice", "productName": "AgentFlow Voice", "directories": { "output": "dist" }, "files": [ "electron/**/*", "out/**/*", "node_modules/**/*", "package.json" ], "mac": { "category": "public.app-category.productivity", "target": ["dmg", "zip"] } } } ``` ### backend/voice/frontend/app/layout.tsx ```typescript import type { Metadata } from "next"; import "./globals.css"; export const metadata: Metadata = { title: "AgentFlow Voice", description: "AI-powered voice assistant with visual feedback", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( <> {children} ); } ``` ### backend/voice/frontend/electron/main.js ```javascript const { app, BrowserWindow } = require('electron'); const path = require('path'); let mainWindow; function createWindow() { mainWindow = new BrowserWindow({ width: 380, height: 480, transparent: true, frame: false, alwaysOnTop: true, resizable: false, skipTaskbar: false, hasShadow: false, backgroundColor: '#00000000', useContentSize: true, show: false, // we'll position first, then show webPreferences: { nodeIntegration: false, contextIsolation: true, enableRemoteModule: false, backgroundThrottling: false, preload: path.join(__dirname, 'preload.js'), }, }); // Position window in top-right corner (flush with edges, below menu bar) const { screen } = require('electron'); const primaryDisplay = screen.getPrimaryDisplay(); const { x: workX, y: workY, width: workW, height: workH } = primaryDisplay.workArea; // safer with notch/menu bar // Flush to the right edge, with a small adjustable margin from top const TOP_OFFSET = parseInt(process.env.ELECTRON_Y_OFFSET || '8', 10); // minimal default const CONTENT_W = 380; const CONTENT_H = 480; const applyPosition = () => { const x = Math.round(workX + workW - CONTENT_W); const y = Math.round(workY + TOP_OFFSET); console.log('[Electron] setContentBounds', { x, y, width: CONTENT_W, height: CONTENT_H, workArea: { workX, workY, workW, workH } }); mainWindow.setContentBounds({ x, y, width: CONTENT_W, height: CONTENT_H }); }; // Ensure content size is exact before showing mainWindow.setContentSize(CONTENT_W, CONTENT_H); applyPosition(); // Load Next.js dev server or production build const isDev = process.env.NODE_ENV !== 'production'; if (isDev) { mainWindow.loadURL('http://localhost:3000'); // Open DevTools in development // mainWindow.webContents.openDevTools(); } else { mainWindow.loadFile(path.join(__dirname, '../out/index.html')); } // Make window draggable mainWindow.setIgnoreMouseEvents(false); // Ensure background stays transparent once content loads mainWindow.webContents.on('did-finish-load', () => { mainWindow.setBackgroundColor('#00000000'); }); // After content is ready to show, re-apply position then show to avoid flicker mainWindow.once('ready-to-show', () => { applyPosition(); mainWindow.show(); }); mainWindow.on('closed', () => { mainWindow = null; }); } // macOS: Keep app running when all windows are closed app.on('window-all-closed', () => { if (process.platform !== 'darwin') { app.quit(); } }); app.on('activate', () => { if (mainWindow === null) { createWindow(); } }); // Wait for app to be ready app.whenReady().then(() => { createWindow(); }); // Enable click-through for transparent areas (optional) // Uncomment if you want clicks to pass through transparent parts // app.commandLine.appendSwitch('enable-transparent-visuals'); ``` ### WALT/src/walt/cli.py ```python """ WALT CLI Main command-line interface for WALT (Web Agents that Learn Tools). """ import asyncio import sys from pathlib import Path from typing import Optional import typer from rich.console import Console from rich.table import Table app = typer.Typer( name="walt", help="🪄 WALT: Web Agents that Learn Tools - Automatic tool discovery from websites", no_args_is_help=True, add_completion=False, ) console = Console() @app.command() def init(): """Initialize WALT configuration with .env file.""" console.print("[bold cyan]🚀 Initializing WALT configuration[/bold cyan]") env_content = """# WALT Configuration # ============================================================================== # LLM API Keys (OpenAI by default,configure according to your LLM provider) # ============================================================================== OPENAI_API_KEY=your-openai-key-here # ANTHROPIC_API_KEY=your-anthropic-key-here # GOOGLE_API_KEY=your-google-key-here # ============================================================================== # Benchmark URLs (For research reproduction only) # ============================================================================== # Uncomment and configure based on your benchmark setup: # VisualWebArena URLs # DATASET=visualwebarena # CLASSIFIEDS=http://localhost:9980 # CLASSIFIEDS_RESET_TOKEN=4b61655535e7ed388f0d40a93600254c # SHOPPING=http://localhost:7770 # REDDIT=http://localhost:9999 # WIKIPEDIA=http://localhost:8888 # HOMEPAGE=http://localhost:4399 # WebArena URLs # GITLAB=http://localhost:8023 # MAP=http://localhost:3000 # SHOPPING_ADMIN=http://localhost:7780/admin # ============================================================================== # Logging & Telemetry # ============================================================================== ANONYMIZED_TELEMETRY=false BROWSER_USE_LOGGING_LEVEL=info # ============================================================================== # Advanced Settings # ============================================================================== # Browser settings # HEADLESS=true # Performance # MAX_STEPS=30 # MAX_PROCESSES=16 """ if Path(".env").exists(): console.print("[yellow]⚠️ .env file already exists[/yellow]") overwrite = typer.confirm("Overwrite existing .env file?") if not overwrite: console.print("[dim]Cancelled[/dim]") return with open(".env", "w") as f: f.write(env_content) console.print("[green]✅ Created .env file[/green]") console.print("[dim]Please edit .env and add your OPENAI_API_KEY[/dim]") @app.command() def version(): """Show WALT version.""" from walt import __version__ console.print(f"[bold cyan]WALT[/bold cyan] version {__version__}") @app.command() def discover( url: str = typer.Option( ..., "--url", help="Base URL to discover tools from (e.g., https://example.com)" ), output_dir: Optional[str] = typer.Option( None, "--output", "-o", help="Output directory for discovered tools" ), llm: str = typer.Option("gpt-5-mini", "--llm", help="LLM model to use"), planner_llm: Optional[str] = typer.Option( None, "--planner-llm", help="Planner LLM model (defaults to same as --llm)" ), auth_file: Optional[str] = typer.Option( None, "--auth-file", help="Playwright storage_state JSON file for authentication" ), max_processes: int = typer.Option(16, "--max-processes", "-p", help="Max concurrent processes"), force_regenerate: bool = typer.Option( False, "--force-regenerate", help="Force regeneration of existing tools" ), skip_test: bool = typer.Option(False, "--skip-test", help="Skip testing generated tools"), optimize: bool = typer.Option(False, "--optimize", help="Generate optimized versions of tools"), ): """ Discover and generate tools from any website. Examples: walt discover --url https://example.com walt discover --url http://localhost:9980 --output walt-tools/mysite walt discover --url https://example.com --auth-file .auth/state.json walt discover --url https://example.com --llm gpt-4o --max-processes 8 The command automatically: 1. Explores the website to discover possible tools 2. Generates tool definitions with parameters 3. Tests each tool to verify it works 4. Saves tools to the output directory """ console.print(f"[bold cyan]🔍 Discovering tools from:[/bold cyan] {url}") # Build args for generic discovery system from types import SimpleNamespace args = SimpleNamespace( url=url, base_url=url, llm=llm, planner_llm=planner_llm or llm, auth_file=auth_file, max_processes=max_processes, force_regenerate=force_regenerate, test=not skip_test, optimize=optimize, discover=True, generate=True ) # Derive output directory from URL if not specified if not output_dir: domain = url.replace("https://", "").replace("http://", "").split("/")[0] args.output_dir = f"walt-tools/{domain}" else: args.output_dir = output_dir if auth_file: console.print(f"[dim]🔑 Using authentication: {auth_file}[/dim]") # Run discovery try: asyncio.run(discovery_main_async(args)) console.print(f"\n[bold green]✅ Discovery complete![/bold green]") console.print(f"[dim]Tools saved to: {args.output_dir}[/dim]") except KeyboardInterrupt: console.print("\n[yellow]⚠️ Discovery interrupted by user[/yellow]") raise typer.Exit(130) except Exception as e: console.print(f"\n[bold red]❌ Error:[/bold red] {e}") import traceback console.print(f"[dim]{traceback.format_exc()}[/dim]") raise typer.Exit(1) async def discovery_main_async(args): """Run the generic discovery pipeline.""" from [truncated — 20708 more characters] ``` ### backend/voice/frontend/app/page.tsx ```typescript "use client"; import { useEffect, useState, useRef } from "react"; import { Room, RoomEvent, Track, TrackEvent } from "livekit-client"; import { AudioAnalyser } from "@/lib/audioAnalyser"; import VoiceOrb from "@/components/VoiceOrb"; import { motion } from "framer-motion"; export default function Home() { const [room, setRoom] = useState(null); const [isConnected, setIsConnected] = useState(false); const [audioLevel, setAudioLevel] = useState(0); const [isListening, setIsListening] = useState(false); const [isSpeaking, setIsSpeaking] = useState(false); const [transcript, setTranscript] = useState([]); const [error, setError] = useState(""); const analyserRef = useRef(null); const animationFrameRef = useRef(); // Track if agent is active (speaking or listening) const isActive = isSpeaking || isListening; // Get LiveKit connection token from backend const getToken = async () => { try { const response = await fetch("/api/token", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ identity: "user-" + Date.now() }), }); if (!response.ok) { throw new Error("Failed to get token"); } const data = await response.json(); return data.token; } catch (err) { console.error("Token fetch error:", err); throw err; } }; // Connect to LiveKit room const connectToRoom = async () => { try { setError(""); const token = await getToken(); const newRoom = new Room({ adaptiveStream: true, dynacast: true, }); // Set up event listeners newRoom.on(RoomEvent.Connected, () => { console.log("Connected to room"); setIsConnected(true); }); newRoom.on(RoomEvent.Disconnected, () => { console.log("Disconnected from room"); setIsConnected(false); }); newRoom.on(RoomEvent.TrackSubscribed, (track, publication, participant) => { console.log("Track subscribed:", track.kind); if (track.kind === Track.Kind.Audio) { // Agent is speaking setIsSpeaking(true); // Attach audio element for playback (hidden so it doesn't affect layout) const audioElement = track.attach(); audioElement.style.display = 'none'; audioElement.style.position = 'fixed'; audioElement.style.pointerEvents = 'none'; document.body.appendChild(audioElement); // Set up audio analysis if (track.mediaStreamTrack) { const audioContext = new AudioContext(); const source = audioContext.createMediaStreamSource( new MediaStream([track.mediaStreamTrack]) ); analyserRef.current = new AudioAnalyser(source, audioContext); startAudioLevelMonitoring(); } track.on(TrackEvent.Ended, () => { setIsSpeaking(false); audioElement.remove(); stopAudioLevelMonitoring(); }); } }); newRoom.on(RoomEvent.TrackUnsubscribed, (track) => { if (track.kind === Track.Kind.Audio) { setIsSpeaking(false); } }); // Connect to LiveKit Cloud const livekitUrl = process.env.NEXT_PUBLIC_LIVEKIT_URL || "ws://localhost:7880"; await newRoom.connect(livekitUrl, token); setRoom(newRoom); // Enable microphone await newRoom.localParticipant.setMicrophoneEnabled(true); setIsListening(true); } catch (err: any) { console.error("Connection error:", err); setError(err.message || "Failed to connect to voice agent"); } }; // Monitor audio levels for visualization const startAudioLevelMonitoring = () => { const updateLevel = () => { if (analyserRef.current) { const level = analyserRef.current.getAudioLevel(); setAudioLevel(level); } animationFrameRef.current = requestAnimationFrame(updateLevel); }; updateLevel(); }; const stopAudioLevelMonitoring = () => { if (animationFrameRef.current) { cancelAnimationFrame(animationFrameRef.current); } setAudioLevel(0); }; // Disconnect from room const disconnect = async () => { if (room) { await room.disconnect(); setRoom(null); setIsConnected(false); setIsListening(false); setIsSpeaking(false); stopAudioLevelMonitoring(); } }; // Cleanup on unmount useEffect(() => { return () => { disconnect(); }; }, []); // Check if running in Electron const isElectron = typeof window !== 'undefined' && ((window as any).electron?.isElectron === true || window.navigator.userAgent.includes('Electron') || (window as any).process?.type === 'renderer'); return (
{/* Floating Voice Assistant Window */}
{/* Header */} [truncated — 2433 more characters] ``` [221 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]