# Project export: Cortex This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Your desktop is now one ping away. Message it to schedule tasks, monitor runs, save results, and stop costly cloud instances, all without remote desktop, works even on extremely low signal ! - Devpost: https://devpost.com/software/glyph-7hjrkp - GitHub: https://github.com/stripathy1999/Cortex-AI.git - Video: https://www.youtube.com/embed/dz8SoNqyDzA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Cursor (3 commits), Aditya (2 commits), Sakshi Tripathy (1 commits) ## Devpost submission (written by the team) ### Inspiration Cloud compute is expensive, and humans are unreliable. Teams forget to stop GPU instances. Students leave Colab runs unattended. Long training jobs fail silently. Remote desktop tools exist, but they assume stable bandwidth, logins, and setup on both devices, exactly what breaks when you’re on deadlines and on emergency situations outside, or on patchy networks. We wanted a simpler mental model: If you can message a friend, to do things for you, you should be able to message your computer too. In this era of agentic AI, where every field has been taken over by a swarm of AI agents orchestrating every workflow, why not use our desktops and build it an agent and give it life. So we built a messaging-first “virtual presence” that lives on your Mac/Windows machine and executes tasks the moment you text it—so work keeps moving even when you’re away. ### What it does Our agent sits on your desktop and is reachable through everyday chat apps: • Mac + iPhone: iMessage control, with optional FaceTime for live voice + screenshare. • Windows: Discord/WhatsApp entrypoints, with optional Zoom / Google Meet link creation so you can join from your phone. You can send instructions like: • “Check my Colab training. If it’s done, download outputs, summarize logs into a doc, and shut down the GPU.” • “Stop the AWS/GCP instance if utilization is low.” • “Search this topic and save the key points to Notes.” • “Start a call, share screen, and walk me through what’s happening.” The key promise is outcome-based control: you don’t remote into a UI, you message an intent, and the agent plans, executes and confirms. ### How we built it Cortex is a messaging-first control plane for your own computer. • Message ingestion (multi-channel): We listen for incoming commands from iMessage (macOS) and Discord/WhatsApp (Windows), normalize them into a single internal Command schema, and attach metadata (sender, platform, timestamps, permissions). • Orchestration: Claude is the main orchestrator. It classifies intent (monitor vs execute vs research), drafts a step-by-step plan, and selects the right execution path. • Desktop execution: We use AgentS (Simular.ai) to perform OS-native actions (open apps, click/type, manage windows, change settings, save notes). • Browser execution: For web workflows (cloud consoles, Colab tabs, downloads, link creation), we use Stagehand (Browserbase) for robust browser automation. • Fast research + summarization: For “fetch info while I’m away,” we use Bright Data scraping to retrieve targeted sources quickly, then generate structured summaries and save them as artifacts. • Live presence mode: On macOS we can pivot into FaceTime for “talk + screenshare.” On Windows we can generate Meet/Zoom links, join from mobile, and continue with voice-driven instructions. • Guardrails: We implemented allowlisting for sensitive actions, confirmation steps for destructive operations, and continuous progress updates (“Step 2/5…”) so the user always knows what’s happening. ### Challenges we ran into • Agents in the real world are messy: Pop-ups, permission dialogs, notifications, and inconsistent UI states made automation unpredictable. We added retries, state checks, and “recover + re-plan” fallbacks. • Latency and long tasks: Some workflows (downloads, training checks, cloud console navigation) can take minutes. We had to design for asynchronous progress with frequent chat updates and final confirmations. • Cross-OS parity: The same instruction means different APIs/UX paths on macOS vs Windows. Building an OS adapter layer with consistent behavior took a lot of debugging. • Messaging reliability: Each platform has different delivery semantics and formats. Normalizing command parsing was a key engineering focus. • Safety vs autonomy: Giving an agent desktop power is risky. We had to build guardrails so it stays useful without becoming reckless. ### Accomplishments we're proud of • You can message Cortex from your phone and it completes real desktop with browser workflows and confirms back with artifacts/results. • Messaging-first control stays usable on weak networks and doesn’t require remembering hostnames/IPs or setting up a full remote session. • The same high-level instruction routes to macOS or Windows execution paths without rewriting the core logic. • We can monitor a run and trigger shutdown actions so expensive compute doesn’t keep running unattended. • While you’re away, Cortex can scrape targeted sources and leave behind a clean summary doc ready to read. ### What we learned • We learnt working with new technologies and integrating them creatively. After investing a lot of time in brainstorming ideas, we came up with something ambitious enough to pull it off in such a short time! We had lots of fun! ### What's next • We plan to integrate Cortex and build a whole multi agentic ecosystem that is useful not only to corporate people, but also as a human aid for elderly that struggle to navigate through evolving handheld devices. We plan to make this project integrate with wearable devices and make it a personal aid for human. ## README (from the GitHub repository)

Cortex-AI Logo

Cortex-AI

Control your Mac or PC with AI — from your phone.

iMessage · Discord · Slack · WhatsApp · FaceTime voice

View on GitHub

Platform Python License

--- ## ✨ What it does Send a message from your phone. Your computer does the rest. | You send | Your Mac/PC does | |----------|------------------| | **"FaceTime me"** | Starts FaceTime, shares screen — then use **voice** to control the computer | | **"Open Chrome and search X on Google"** | Opens Chrome, runs the search, opens the first result | | **"Open Notepad"** | Opens Notepad and types a test message | | **"Open Calculator"** | Opens the Calculator app | | **"Go to YouTube Music and play some music"** | Opens Chrome, navigates to YouTube Music | Only **your** messages are processed; everyone else is ignored. --- ## 🖼️ How it looks

Cortex-AI in action — message your PC from phone, agent status and Bright Data scraping

Message from your phone → task runs on your computer. Architecture image generated by AI and made with love by Aditya Punjani and Sakshi Tripathy.

--- ## 🎬 Watch the demo

Watch Cortex-AI on YouTube

▶ Watch on YouTube

--- ## 🏗️ How it works ```mermaid flowchart LR subgraph Phone A[iMessage / Discord / Slack / WhatsApp] end subgraph Your Computer B[Message bridge] C[Backend] D[FaceTime voice] E[Stagehand · Browser] F[Agent-S · Desktop] end A --> B --> C C --> D C --> E C --> F D --> G[Your Mac/PC] E --> G F --> G ``` | Layer | Role | |-------|------| | **Message bridge** | Receives your messages (iMessage, Discord, Slack, WhatsApp). | | **Backend** | Routes each message to the right service. | | **FaceTime** | Text *"FaceTime me"* → call starts, screen shares; then use **voice** to control the Mac. | | **Stagehand** | Browser tasks (Chrome, Google, YouTube, Gmail, etc.). | | **Agent-S** | Desktop tasks (Notepad, Calculator, files, apps). | --- ## 🚀 Quick start ### Mac (iMessage + FaceTime) ```bash git clone https://github.com/AdityaYC/Cortex-Treehacks.git cd Cortex-Treehacks cp .env.example .env # Add API keys, FACETIME_APPLE_ID, etc. ./run_order.sh ``` Then text your Mac from iMessage. Say **"FaceTime me"** for a call + screen share and voice control, or send any other task (browser or desktop). ### Windows ```powershell git clone https://github.com/AdityaYC/Cortex-Treehacks.git cd Cortex-Treehacks copy .env.example .env .\run_windows.ps1 ``` This starts the message bridge, backend, Agent-S, Stagehand, and frontend. --- ## 📱 Message your Mac/PC | Platform | ngrok? | Setup | |----------|--------|-------| | **iMessage** (Mac) | No | Run with `run_order.sh`; text from your iPhone to trigger tasks and *"FaceTime me"* for voice | | **Discord** | No | [Discord & Slack setup →](docs/DISCORD_SLACK_SETUP.md) | | **Slack** | No | [Discord & Slack setup →](docs/DISCORD_SLACK_SETUP.md) | | **WhatsApp** | Yes | [WhatsApp setup →](docs/WHATSAPP_SETUP.md) | --- ## 🔑 Required API keys Add to `.env`: ```env # Agent-S (required) ANTHROPIC_API_KEY=sk-ant-your-key # Stagehand (browser tasks) OPENAI_API_KEY=sk-your-openai-key # Discord / Slack / WhatsApp — see setup docs ``` --- ## 📁 Project structure | Folder | Purpose | |--------|---------| | `Agent-S` | Desktop automation (vision + pyautogui) | | `backend` | Flask server, message bridges, Stagehand service | | `frontend` | Electron UI | | `docs` | Setup guides (Discord, Slack, WhatsApp, testing) | | `scripts` | Utilities and run scripts | --- ## 📚 Documentation | Doc | Description | |-----|-------------| | [Video calls & FaceTime](docs/VIDEO_CALL_SETUP.md) | *"FaceTime me"*, Zoom, Meet, WhatsApp | | [Windows setup](docs/WINDOWS_ONLY_SETUP.md) | Full Windows-only setup | | [Discord & Slack](docs/DISCORD_SLACK_SETUP.md) | Bot tokens and allowed user IDs | | [WhatsApp (Twilio)](docs/WHATSAPP_SETUP.md) | Twilio and ngrok | | [Testing](docs/TESTING.md) | How to test messages and flows | --- ## 🙏 Credits - **Cortex-AI** — Logo and architecture imagery generated by AI and made with love by **Aditya Punjani** and **Sakshi Tripathy**. - **[Agent-S](https://github.com/simular-ai/Agent-S)** — computer-use agent (simular-ai) - **[Stagehand](https://github.com/browserbase/stagehand)** — browser automation (browserbase) ---

Last updated: February 2026

## Detected evidence (automated analysis) Indexed codebase: 149 recognized source files, 858 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AWS (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 179) ``` .DS_Store .env.example .gitignore Agent-S/.gitignore Agent-S/.python-version Agent-S/LICENSE Agent-S/models.md Agent-S/pyproject.toml Agent-S/README.md Agent-S/run_claude.sh Agent-S/run_demo_best.sh Agent-S/run_demo_fast.sh Agent-S/run_grok.sh Agent-S/run_groq.sh Agent-S/run_openai.sh Agent-S/src/__init__.py Agent-S/src/s3/__init__.py Agent-S/src/s3/action_analysis_agent.py Agent-S/src/s3/agents/__init__.py Agent-S/src/s3/agents/agent_s.py Agent-S/src/s3/agents/code_agent.py Agent-S/src/s3/agents/grounding.py Agent-S/src/s3/agents/worker.py Agent-S/src/s3/app.py Agent-S/src/s3/bbon/__init__.py Agent-S/src/s3/bbon/behavior_narrator.py Agent-S/src/s3/bbon/comparative_judge.py Agent-S/src/s3/cli_app.py Agent-S/src/s3/core/__init__.py Agent-S/src/s3/core/engine.py Agent-S/src/s3/core/mllm.py Agent-S/src/s3/core/module.py Agent-S/src/s3/memory/__init__.py Agent-S/src/s3/memory/procedural_memory.py Agent-S/src/s3/utils/__init__.py Agent-S/src/s3/utils/common_utils.py Agent-S/src/s3/utils/formatters.py Agent-S/src/s3/utils/local_env.py Agent-S/src/utils.py Agent-S/test.py Agent-S/tests/listen_current_action.py Agent-S/tests/test_action_analysis_agent.py Agent-S/tests/test_asi_one.py Agent-S/tests/test_client.py Agent-S/uv.lock Agent-S/WAA_setup.md backend/.gitignore backend/.python-version backend/AUDIO_DEBUG_README.md backend/audio_level_monitor.html backend/AUDIO_STREAMING_ARCHITECTURE.md backend/audio.py backend/browserbase_playwright.py backend/browserbase_session.py backend/call.py backend/cursor_service.py backend/DEBUG_AUDIO_FLOW.md backend/fast_browser.py backend/imessage_bridge.py backend/main.py backend/out_of_memory_error.py backend/pyproject.toml backend/research_service.py backend/run_output_audio.py backend/stagehand_service.py backend/test_agent_s.py backend/test_audio_levels.py backend/test_audio_output_states.py backend/test_call.py backend/test_vad_integration.py backend/test_video_call.py backend/test_vision_cursor.py backend/TROUBLESHOOTING_AUDIO_LEVELS.md backend/uv.lock backend/VAD_ARCHITECTURE.md backend/video_call_service.py backend/vision_cursor_bridge.py backend/vision_service.py backend/windows_message_bridge.py BRIGHT_DATA_API_SETUP.md docs/BRIGHT_DATA_SETUP.md docs/BRIGHT_DATA_USE_CASES_AND_PROMPTS.md docs/BROWSERBASE_CLOUD_SETUP.md docs/DISCORD_FIX.md docs/DISCORD_SLACK_SETUP.md docs/FACETIME_SETUP.md docs/HOW_VISION_AND_CURSOR_WORK.md docs/IMOVIE_WORKFLOW.md docs/TASK_DIVISION_WINDOWS_MAC.md docs/TESTING.md docs/VIDEO_CALL_SETUP.md docs/VISION_AND_CLAUDE_FOR_COMPLEX_TASKS.md docs/VISION_MIRROR_INTEGRATION.md docs/VOICE_VISION_SETUP.md docs/WHATSAPP_SETUP.md docs/WINDOWS_ONLY_SETUP.md docs/WINDOWS_SETUP.md fix_cookie_sync.py frontend/.gitattributes frontend/.gitignore frontend/.npmrc frontend/API-INTEGRATION.md frontend/doc.md frontend/electron/ApiServerHelper.ts frontend/electron/ipcHandlers.ts frontend/electron/main.ts frontend/electron/preload.ts frontend/electron/ProcessingHelper.ts frontend/electron/ScreenshotHelper.ts frontend/electron/shortcuts.ts frontend/electron/tsconfig.json frontend/electron/WindowHelper.ts frontend/index.html frontend/LICENSE frontend/package.json frontend/postcss.config.js frontend/README.md frontend/renderer/.gitignore frontend/renderer/package.json frontend/renderer/public/index.html [59 more files omitted for size] ``` ### Dependencies - Agent-S/pyproject.toml: anthropic, backoff, black, dotenv@>=0.9.9, fastapi, flask@>=3.1.2, google-genai, numpy, openai, paddleocr, paddlepaddle, pandas, pyautogui, pyobjc, pytesseract, pytest@>=8.4.2, pywin32, pywinauto, scikit-learn, selenium, tiktoken, together, toml, typing-extensions@>=4.15.0, uagents@>=0.20.1, uvicorn, websockets - backend/pyproject.toml: discord-py@>=2.6.4, discord.py@>=2.0.0, fish-audio-sdk@>=1.0.0, flask@>=3.1.2, flask-cors@>=4.0.0, flask-socketio@>=5.3.0, mediapipe@>=0.10.0, mss@>=9.0.0, numpy@>=1.26.0, openai@>=1.0.0, opencv-python-headless@>=4.8.0, playwright@>=1.40.0, pyautogui@>=0.9.54, pycryptodome@>=3.19.0, pydub@>=0.25.1, pyperclip@>=1.8.0, python-dotenv@>=1.0.1, python-socketio[client]@>=5.11.0, requests@>=2.32.5, slack-bolt@>=1.27.0, slack-bolt@>=1.18.0, sounddevice@>=0.4.6, soundfile@>=0.12.1, stagehand@>=3.0.0, twilio@>=9.0.0, twilio@>=9.10.1 - frontend/package.json: @radix-ui/react-dialog@^1.1.2, @radix-ui/react-toast@^1.2.2, @types/color@^4.2.0, @types/diff@^6.0.0, @types/electron@^1.4.38, @types/node@^22.9.0, @types/react@^18.3.12, @types/react-dom@^18.3.1, @types/react-syntax-highlighter@^15.5.13, @types/screenshot-desktop@^1.12.3, @types/uuid@^9.0.8, @typescript-eslint/eslint-plugin@^8.14.0, @typescript-eslint/parser@^8.14.0, @vitejs/plugin-react@^4.3.3, autoprefixer@^10.4.20, axios@^1.7.7, class-variance-authority@^0.7.0, clsx@^2.1.1, concurrently@^9.1.0, cross-env@^7.0.3, diff@^7.0.0, electron@^33.4.11, electron-builder@^25.1.8, electron-is-dev@^3.0.1, form-data@^4.0.1, lucide-react@^0.460.0, postcss@^8.4.49, react@^18.3.1, react-code-blocks@^0.1.6, react-dom@^18.3.1, react-icons@^5.3.0, react-query@^3.39.3, react-syntax-highlighter@^15.6.1, rimraf@^6.0.1, screenshot-desktop@^1.15.0, sharp@^0.33.5, tailwind-merge@^2.5.4, tailwindcss@^3.4.15, tesseract.js@^5.0.5, typescript@^5.6.3, uuid@^11.0.3, vite@^5.4.11, vite-plugin-electron@^0.28.8, vite-plugin-electron-renderer@^0.14.6, wait-on@^8.0.1 - frontend/renderer/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @types/jest@^27.5.2, @types/node@^16.18.119, @types/react@^18.3.12, @types/react-dom@^18.3.1, react@^18.3.1, react-dom@^18.3.1, react-scripts@5.0.1, typescript@^4.9.5, web-vitals@^2.1.4 ### Recent commits (newest first) - README: add logo, architecture images, YouTube demo, and credits - iMessage agent flow, FaceTime vision+AppleScript, completetask routing, docs - Stagehand: smarter + faster (Haiku planner, Sonnet agent, Gemini option, Browserbase scripts) - Video call, share screen, voice filtering, and testing - Update README for Cortex-AI project - Cortex-AI: Shadow-OS with Discord, Slack, WhatsApp, Stagehand, Agent-S - Merge remote-tracking branch 'origin/main' - Initial commit: FaceTimeOS with Claude integration, updated READMEs, fixed notification system, and configured for Feb 14 2026 - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### SYSTEM_STATUS.md ```markdown # Cortex AI System Status ## Architecture **Agent-S (Local Desktop)** + **Browserbase (Cloud Browser)** + **Electron UI** ``` ┌─────────────────────────────────────────────────────────────┐ │ Cortex AI System │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Discord/iMessage → Backend → Router │ │ │ │ │ ┌─────────────┴─────────────┐ │ │ │ │ │ │ Desktop Task? Browser Task? │ │ │ │ │ │ ▼ ▼ │ │ Agent-S (Local) Browserbase/Stagehand │ │ - Notepad - Chrome automation │ │ - Calculator - LinkedIn, Gmail │ │ - File Explorer - Web searches │ │ - Desktop apps - Cloud browser │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ## Running Services ### ✅ Core Services - **Backend API:** http://127.0.0.1:8003 - **Agent-S:** Local desktop automation - **Stagehand (Browserbase):** http://127.0.0.1:5005 - **Research Service (Bright Data):** http://127.0.0.1:5006 - **Electron UI:** http://localhost:5173 ### ✅ Messaging - **Discord Bot:** Running - **iMessage Bridge:** http://127.0.0.1:8100 ## Configuration ### Browserbase ```bash STAGEHAND_USE_BROWSERBASE=true BROWSERBASE_API_KEY=bb_live_Ege52HyNvC7tR6xdYptdA71_QVw BROWSERBASE_PROJECT_ID=70656bf4-7817-4800-9b38-5c258b9e8790 ``` ### Bright Data ```bash BRIGHT_DATA_API_KEY=f2244c0c-a41c-4468-b976-4a5847e65f37 BRIGHT_DATA_ZONE=web_unlocker2 ``` ## Task Routing ### Desktop Tasks → Agent-S (Fast, Local) - "Open Notepad" - "Open Calculator" - "Open File Explorer" - Desktop applications ### Browser Tasks → Browserbase (Cloud) - "Go to google.com" - "Search for AI news" - "Open LinkedIn" - "Check Gmail" - Any website/browser task ### Research Tasks → Bright Data (Bot Bypass) - Bot-protected sites - Massive data extraction - Competitor research ## Usage ### Via Discord Send message in Discord channel ### Via iMessage Send message to configured number ### Via Electron UI Open http://localhost:5173 ### Via API ```bash curl -X POST http://127.0.0.1:8003/api/new_imessage \ -H "Content-Type: application/json" \ -d '{"text": "your command", "phone_number": "+16504440949"}' ``` ## Performance - **Desktop tasks (Agent-S):** Instant (~1-2 seconds) - **Browser tasks (Browserbase):** ~60-90 seconds (AI-powered) - **Research tasks (Bright Data):** ~5-10 seconds ## Start All Services ```bash ./start_all_services.sh ``` Or manually: ```bash pyt [truncated — 410 more characters] ``` ### BRIGHT_DATA_API_SETUP.md ```markdown # Bright Data Web Unlocker API - Quick Setup ## ✅ What's Ready Bright Data integration is implemented and ready to use. Just add your API key! ## 🚀 Quick Setup (2 minutes) ### 1. Get Your API Key 1. Go to: https://brightdata.com/cp/api_tokens 2. Click "Create API Token" 3. Copy your API key (starts with `brd-...`) ### 2. Add to .env Edit `.env` and add: ```bash BRIGHT_DATA_API_KEY=your_api_key_here BRIGHT_DATA_ZONE=web_unlocker1 ``` ### 3. Restart Research Service Run from the **backend** directory so dependencies (e.g. `requests`, `flask`) are available: ```bash pkill -f research_service cd backend && uv run research_service.py & ``` ### 4. Test It ```bash ./test_brightdata_api.sh ``` ## 📝 Example API Call ```bash curl https://api.brightdata.com/request \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "zone": "web_unlocker1", "url": "https://example.com", "format": "raw" }' ``` ## 🎯 How It Works ### Automatic Smart Routing ``` Your Request ↓ Standard Fetch (Free) ↓ Success? → YES → Return content ↓ NO Bot detected (403/CAPTCHA)? ↓ YES Bright Data API (Paid) ↓ Bypass protection → Return content ``` ### When Bright Data is Used **Automatically triggered:** - ✅ Bot detection (403, CAPTCHA, Cloudflare) - ✅ Rate limiting (429 errors) - ✅ Massive extraction keywords ("scrape 50 pages", "bulk extract") - ✅ Standard fetch fails **Always free first:** - Standard fetch tries first - Only uses Bright Data when needed - Saves money automatically ## 💰 Pricing **Standard Fetch:** FREE **Bright Data:** Pay-as-you-go - Single page: ~$0.01 - 10 pages: ~$0.10 - 100 pages: ~$1.00 ## 📱 Usage Examples ### Via API ```bash curl -X POST http://127.0.0.1:5006/api/research \ -H "Content-Type: application/json" \ -d '{ "query": "Research competitor pricing", "urls": ["https://competitor.com/products"] }' ``` ### Via Poke Send to +1 (650) 444-0949: ``` "research latest AI news from techcrunch" "scrape pricing from competitor.com" "extract product data from 50 pages" ``` ### Via Agent-S Agent-S automatically uses it when: - Standard web tools fail - User asks to scrape protected sites - Competitor analysis needed ## 🧪 Testing Run the test suite: ```bash ./test_brightdata_api.sh ``` **Tests:** 1. ✅ Standard fetch (free) 2. ✅ Bot detection fallback 3. ✅ Massive extraction detection ## 🔧 Configuration ### Default Settings ```bash BRIGHT_DATA_API_KEY=your_key_here BRIGHT_DATA_ZONE=web_unlocker1 ``` ### Custom Zone If you have a different zone: ```bash BRIGHT_DATA_ZONE=your_custom_zone ``` ## 📊 Monitoring ### Check Usage View dashboard: https://brightdata.com/cp/zones ### View Logs ```bash # Research service logs tail -f logs/research_service.log # Check Bright Data usage grep "bright_data_api" logs/research_service.log ``` ## ✅ Status Check ```bash # Test standard fetch curl -X POST http://127.0.0.1:5006/api/research \ -H "Conte [truncated — 1051 more characters] ``` ### backend/pyproject.toml ``` [project] name = "backend" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.12" dependencies = [ "playwright>=1.40.0", "pycryptodome>=3.19.0", "stagehand>=3.0.0", "flask>=3.1.2", "flask-socketio>=5.3.0", "flask-cors>=4.0.0", "requests>=2.32.5", "openai>=1.0.0", "python-dotenv>=1.0.1", "fish-audio-sdk>=1.0.0", "numpy>=1.26.0", "sounddevice>=0.4.6", "soundfile>=0.12.1", "pydub>=0.25.1", "python-socketio[client]>=5.11.0", "pyautogui>=0.9.54", "pyperclip>=1.8.0", "mss>=9.0.0", "twilio>=9.10.1", "discord-py>=2.6.4", "slack-bolt>=1.27.0", ] [project.optional-dependencies] vision = [ "mediapipe>=0.10.0", "opencv-python-headless>=4.8.0", ] # For Windows message bridge discord = [ "discord.py>=2.0.0", ] slack = [ "slack-bolt>=1.18.0", ] whatsapp = [ "twilio>=9.0.0", ] ``` ### Agent-S/pyproject.toml ``` [project] name = "gui-agents" version = "0.3.1" description = "A library for creating general purpose GUI agents using multimodal LLMs." readme = "README.md" authors = [ { name = "Simular AI", email = "eric@simular.ai" }, { name = "Dylan Lu", email = "dylanelu@gmail.com" }, ] license = { text = "Apache-2.0" } requires-python = ">=3.9, <=3.13" keywords = ["ai", "llm", "gui", "agent", "multimodal"] dependencies = [ "numpy", "backoff", "pandas", "openai", "anthropic", "fastapi", "uvicorn", "paddleocr", "paddlepaddle", "together", "scikit-learn", "websockets", "tiktoken", "selenium", "pyautogui", "toml", "pytesseract", "google-genai", 'pyobjc; platform_system == "Darwin"', 'pywinauto; platform_system == "Windows"', 'pywin32; platform_system == "Windows"', "uagents>=0.20.1", "pytest>=8.4.2", "flask>=3.1.2", "typing-extensions>=4.15.0", "dotenv>=0.9.9", ] [project.optional-dependencies] dev = ["black"] [project.scripts] agent_s = "src.s3.cli_app:main" [tool.setuptools.packages.find] include = ["src*"] exclude = ["tests*", "images*", "logs*"] ``` ### frontend/package.json ``` { "name": "interview-coder", "version": "1.0.0", "main": "./dist-electron/main.js", "scripts": { "clean": "rimraf dist dist-electron", "dev": "vite", "build": "npm run clean && tsc && vite build", "preview": "vite preview", "postinstall": "cross-env SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm rebuild sharp", "electron:dev": "tsc -p electron/tsconfig.json && cross-env NODE_ENV=development electron .", "electron:build": "tsc -p electron/tsconfig.json && cross-env NODE_ENV=production electron .", "app:dev": "concurrently \"npm run dev -- --port 5180\" \"wait-on http://localhost:5180 && npm run electron:dev\"", "app:build": "npm run build && electron-builder", "watch": "tsc -p electron/tsconfig.json --watch", "start": "npm run app:dev", "dist": "npm run app:build" }, "build": { "appId": "com.electron.meeting-notes", "productName": "Meeting Notes Coder", "files": [ "dist/**/*", "dist-electron/**/*", "package.json", "node_modules/**/*" ], "directories": { "output": "release", "buildResources": "assets" }, "extraResources": [ { "from": "assets/", "to": "assets/" } ], "mac": { "category": "public.app-category.productivity", "target": [ { "target": "dmg", "arch": [ "x64", "arm64" ] } ], "icon": "assets/icons/mac/icon.icns", "hardenedRuntime": true, "entitlements": "assets/entitlements.mac.plist" }, "win": { "target": [ { "target": "nsis", "arch": [ "x64", "ia32" ] }, { "target": "portable", "arch": [ "x64" ] } ], "icon": "assets/icons/win/icon.ico", "requestedExecutionLevel": "asInvoker" }, "linux": { "target": [ { "target": "AppImage", "arch": [ "x64" ] }, { "target": "deb", "arch": [ "x64" ] } ], "icon": "assets/icons/png/", "category": "Office" }, "nsis": { "oneClick": false, "perMachine": false, "allowToChangeInstallationDirectory": true, "deleteAppDataOnUninstall": false }, "publish": [ { "provider": "github", "owner": "ibttf", "repo": "interview-coder-frontend" } ] }, "keywords": [], "author": "", "license": "ISC", "description": "", "devDependencies": { "@types/color": "^4.2.0", "@types/diff": "^6.0.0", "@types/electron": "^1.4.38", "@types/node": "^22.9.0", "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "@types/react-syntax-highlighter": "^15.5.13", "@types/screenshot-desktop": "^1.12.3", "@types/uuid": "^9.0.8", "@typescript-eslint/eslint-plugin": "^8.14.0", "@typescript-eslint/parser": "^8.14.0", "@vitejs/plugin-react": "^4.3.3", "autoprefixer": "^10.4.20", "concurrently": "^9.1.0", "cross-env": "^7.0.3", "electron": "^33.4.11", "electron-builder": "^25.1.8", "electron-is-dev": "^3.0.1", "postcss": "^8.4.49", "rimraf": "^6.0.1", "tailwindcss": "^3.4.15", "typescript": "^5.6.3", "vite": "^5.4.11", "vite-plugin-electron": "^0.28.8", "vite-plugin-electron-renderer": "^0.14.6", "wait-on": "^8.0.1" }, "dependencies": { "@radix-ui/react-dialog": "^1.1.2", "@radix-ui/react-toast": "^1.2.2", "axios": "^1.7.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "diff": "^7.0.0", "form-data": "^4.0.1", "lucide-react": "^0.460.0", "react": "^18.3.1", "react-code-blocks": "^0.1.6", "react-dom": "^18.3.1", "react-icons": "^5.3.0", "react-query": "^3.39.3", "react-syntax-highlighter": "^15.6.1", "screenshot-desktop": "^1.15.0", "sharp": "^0.33.5", "tailwind-merge": "^2.5.4", "tesseract.js": "^5.0.5", "uuid": "^11.0.3" } } ``` ### frontend/renderer/package.json ``` { "name": "renderer", "version": "0.1.0", "private": true, "dependencies": { "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "@types/jest": "^27.5.2", "@types/node": "^16.18.119", "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "react": "^18.3.1", "react-dom": "^18.3.1", "react-scripts": "5.0.1", "typescript": "^4.9.5", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### frontend/src/main.tsx ```typescript import React from "react" import ReactDOM from "react-dom/client" import App from "./App" import "./index.css" ReactDOM.createRoot(document.getElementById("root")!).render( ) ``` ### frontend/src/App.tsx ```typescript import { ToastProvider } from "./components/ui/toast" import Queue from "./_pages/Queue" import { ToastViewport } from "@radix-ui/react-toast" import { useEffect, useRef, useState } from "react" import Solutions from "./_pages/Solutions" import { QueryClient, QueryClientProvider } from "react-query" declare global { interface Window { electronAPI: { //RANDOM GETTER/SETTERS updateContentDimensions: (dimensions: { width: number height: number }) => Promise getScreenshots: () => Promise> setWindowClickThrough: (clickThrough: boolean) => Promise //GLOBAL EVENTS //TODO: CHECK THAT PROCESSING NO SCREENSHOTS AND TAKE SCREENSHOTS ARE BOTH CONDITIONAL onUnauthorized: (callback: () => void) => () => void onScreenshotTaken: ( callback: (data: { path: string; preview: string }) => void ) => () => void onProcessingNoScreenshots: (callback: () => void) => () => void onResetView: (callback: () => void) => () => void takeScreenshot: () => Promise //INITIAL SOLUTION EVENTS deleteScreenshot: ( path: string ) => Promise<{ success: boolean; error?: string }> onSolutionStart: (callback: () => void) => () => void onSolutionError: (callback: (error: string) => void) => () => void onSolutionSuccess: (callback: (data: any) => void) => () => void onProblemExtracted: (callback: (data: any) => void) => () => void onDebugSuccess: (callback: (data: any) => void) => () => void onDebugStart: (callback: () => void) => () => void onDebugError: (callback: (error: string) => void) => () => void moveWindowLeft: () => Promise moveWindowRight: () => Promise moveWindowUp: () => Promise moveWindowDown: () => Promise quitApp: () => Promise sendChatPrompt: (prompt: string) => Promise sendShadowTrigger: (prompt: string) => Promise sendCursorMove: (x: number, y: number) => Promise sendVisionFrame: (frameBase64: string) => Promise pauseAgent: () => Promise resumeAgent: () => Promise stopAgent: () => Promise // Current Action Updates onCurrentActionUpdate: ( callback: ( action: { original: string text_summary: string voice_summary: string mode?: string } ) => void ) => () => void getContactLinks: () => Promise<{ contact_links: Array<{ platform: string; label: string; instruction: string; value: string }>; message: string }> invoke: (channel: string, ...args: any[]) => Promise } } } const queryClient = new QueryClient({ defaultOptions: { queries: { staleTime: Infinity, cacheTime: Infinity } } }) const App: React.FC = () => { const [view, setView] = useState<"queue" | "solutions" | "debug">("queue") const containerRef = useRef(null) // Effect for height monitoring useEffect(() => { const cleanup = window.electronAPI.onResetView(() => { console.log("Received 'reset-view' message from main process.") queryClient.invalidateQueries(["screenshots"]) queryClient.invalidateQueries(["problem_statement"]) queryClient.invalidateQueries(["solution"]) queryClient.invalidateQueries(["new_solution"]) setView("queue") }) return () => { cleanup() } }, []) useEffect(() => { if (!containerRef.current) return const updateHeight = () => { if (!containerRef.current) return const height = containerRef.current.scrollHeight const width = containerRef.current.scrollWidth window.electronAPI?.updateContentDimensions({ width, height }) } const resizeObserver = new ResizeObserver(() => { updateHeight() }) // Initial height update updateHeight() // Observe for changes resizeObserver.observe(containerRef.current) // Also update height when view changes const mutationObserver = new MutationObserver(() => { updateHeight() }) mutationObserver.observe(containerRef.current, { childList: true, subtree: true, attributes: true, characterData: true }) return () => { resizeObserver.disconnect() mutationObserver.disconnect() } }, [view]) // Re-run when view changes useEffect(() => { const cleanupFunctions = [ window.electronAPI.onSolutionStart(() => { setView("solutions") console.log("starting processing") }), window.electronAPI.onUnauthorized(() => { queryClient.removeQueries(["screenshots"]) queryClient.removeQueries(["solution"]) queryClient.removeQueries(["problem_statement"]) setView("queue") console.log("Unauthorized") }), // Update this reset handler window.electronAPI.onResetView(() => { console.log("Received 'reset-view' message from main process") queryClient.removeQueries(["screenshots"]) queryClient.removeQueries(["solution"]) queryClient.removeQueries(["problem_statement"]) setView("queue") console.log("View reset to 'queue' via Command+R shortcut") }), window.electronAPI.onProblemExtracted((data: any) => { if (view === "queue") { console.log("Problem extracted successfully") queryClient.invalidateQueries(["problem_statement"]) queryClient.setQueryData(["problem_statement"], data) } }) ] return () => cleanupFunctions.forEach((cleanup) => cleanup()) }, []) return (
{view === "queue" ? ( ) : view === "solutions" ? [truncated — 212 more characters] ``` ### frontend/electron/main.ts ```typescript import { app, BrowserWindow, Tray, Menu, nativeImage } from "electron" import { initializeIpcHandlers } from "./ipcHandlers" import { WindowHelper } from "./WindowHelper" import { ScreenshotHelper } from "./ScreenshotHelper" import { ShortcutsHelper } from "./shortcuts" import { ProcessingHelper } from "./ProcessingHelper" import { ApiServerHelper } from "./ApiServerHelper" export class AppState { private static instance: AppState | null = null private windowHelper: WindowHelper private screenshotHelper: ScreenshotHelper public shortcutsHelper: ShortcutsHelper public processingHelper: ProcessingHelper public apiServerHelper: ApiServerHelper private tray: Tray | null = null // View management private view: "queue" | "solutions" = "queue" private problemInfo: { problem_statement: string input_format: Record output_format: Record constraints: Array> test_cases: Array> } | null = null // Allow null private hasDebugged: boolean = false // Processing events public readonly PROCESSING_EVENTS = { //global states UNAUTHORIZED: "procesing-unauthorized", NO_SCREENSHOTS: "processing-no-screenshots", //states for generating the initial solution INITIAL_START: "initial-start", PROBLEM_EXTRACTED: "problem-extracted", SOLUTION_SUCCESS: "solution-success", INITIAL_SOLUTION_ERROR: "solution-error", //states for processing the debugging DEBUG_START: "debug-start", DEBUG_SUCCESS: "debug-success", DEBUG_ERROR: "debug-error" } as const constructor() { // Initialize WindowHelper with this this.windowHelper = new WindowHelper(this) // Initialize ScreenshotHelper this.screenshotHelper = new ScreenshotHelper(this.view) // Initialize ProcessingHelper this.processingHelper = new ProcessingHelper(this) // Initialize ShortcutsHelper this.shortcutsHelper = new ShortcutsHelper(this) // Initialize ApiServerHelper this.apiServerHelper = new ApiServerHelper(this) } public static getInstance(): AppState { if (!AppState.instance) { AppState.instance = new AppState() } return AppState.instance } // Getters and Setters public getMainWindow(): BrowserWindow | null { return this.windowHelper.getMainWindow() } public getView(): "queue" | "solutions" { return this.view } public setView(view: "queue" | "solutions"): void { this.view = view this.screenshotHelper.setView(view) } public isVisible(): boolean { return this.windowHelper.isVisible() } public getScreenshotHelper(): ScreenshotHelper { return this.screenshotHelper } public getProblemInfo(): any { return this.problemInfo } public setProblemInfo(problemInfo: any): void { this.problemInfo = problemInfo } public getScreenshotQueue(): string[] { return this.screenshotHelper.getScreenshotQueue() } public getExtraScreenshotQueue(): string[] { return this.screenshotHelper.getExtraScreenshotQueue() } // Window management methods public createWindow(): void { this.windowHelper.createWindow() } public hideMainWindow(): void { this.windowHelper.hideMainWindow() } public showMainWindow(): void { this.windowHelper.showMainWindow() } public toggleMainWindow(): void { console.log( "Screenshots: ", this.screenshotHelper.getScreenshotQueue().length, "Extra screenshots: ", this.screenshotHelper.getExtraScreenshotQueue().length ) this.windowHelper.toggleMainWindow() } public setWindowDimensions(width: number, height: number): void { this.windowHelper.setWindowDimensions(width, height) } public clearQueues(): void { this.screenshotHelper.clearQueues() // Clear problem info this.problemInfo = null // Reset view to initial state this.setView("queue") } // Screenshot management methods public async takeScreenshot(): Promise { if (!this.getMainWindow()) throw new Error("No main window available") const screenshotPath = await this.screenshotHelper.takeScreenshot( () => this.hideMainWindow(), () => this.showMainWindow() ) return screenshotPath } public async getImagePreview(filepath: string): Promise { return this.screenshotHelper.getImagePreview(filepath) } public async deleteScreenshot( path: string ): Promise<{ success: boolean; error?: string }> { return this.screenshotHelper.deleteScreenshot(path) } // New methods to move the window public moveWindowLeft(): void { this.windowHelper.moveWindowLeft() } public moveWindowRight(): void { this.windowHelper.moveWindowRight() } public moveWindowDown(): void { this.windowHelper.moveWindowDown() } public moveWindowUp(): void { this.windowHelper.moveWindowUp() } public centerAndShowWindow(): void { this.windowHelper.centerAndShowWindow() } public createTray(): void { // Create a simple tray icon const image = nativeImage.createEmpty() // Try to use a system template image for better integration let trayImage = image try { // Create a minimal icon - just use an empty image and set the title trayImage = nativeImage.createFromBuffer(Buffer.alloc(0)) } catch (error) { console.log("Using empty tray image") trayImage = nativeImage.createEmpty() } this.tray = new Tray(trayImage) const contextMenu = Menu.buildFromTemplate([ { label: 'Show Cortex', click: () => { this.centerAndShowWindow() } }, { label: 'Toggle Window', click: () => { this.toggleMainWindow() } }, { type: 'separator' }, { label: 'Take Screenshot (Cmd+H)', click: async () => { try { const screenshotPath = await this.takeSc [truncated — 2229 more characters] ``` ### frontend/src/types/index.tsx ```typescript export interface Screenshot { id: string path: string timestamp: number thumbnail: string // Base64 thumbnail } export interface Solution { initial_thoughts: string[] thought_steps: string[] description: string code: string } ``` [130 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]