# Project export: The Cookbook
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: Master any software with a real-time autopilot. Cookbook captures your live screen, breaks down your goal, and overlays exactly where to click. Turn any complex software into a step-by-step recipe.
- Devpost: https://devpost.com/software/the-cookbook-9rqgb2
- GitHub: https://github.com/shamit05/the_cookbook
- Video: https://www.youtube.com/embed/0WoY9THRDJk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — Shamit Surana (17 commits), Aarav Arora (8 commits), Cursor (5 commits)
## Devpost submission (written by the team)
### Inspiration
We’ve all been there–you’re trying to learn a new tool, and you can’t find the right video tutorial to help you get off the ground. Your computer already has 100 windows open, and you’re switching between screens, watching a video that's not even relevant to your task. Even worse, the software version used in the video is incompatible. Our goal is to simplify the process for users trying to learn new softwares, such as Photoshop, Cursor, Canva, and more.
### What it does
Meet The Cookbook! It overlays real-time visual instructions on your screen to guide you through complex software tasks step-by-step. Just open the app, describe what you need help with, and receive on-screen highlights and text instructions with a detailed plan for your task. The instructions are catered to your application version, your style of learning, and your screen exactly as it looks. The main features are: Goal-to-steps generation from live screenshots: Captures the current screen and sends it to a multi-agent pipeline using OpenAI and Gemini APIs to generate precise coordinates for the next user click and a detailed instructional plan. Two-stage spatial grounding pipeline: Generates a 40-marker coordinate grid to help the agent identify a specific "zoom zone," then runs OmniParser on that high-resolution crop to ensure pixel-perfect bounding box accuracy without the clutter of full-screen detection. Interactive overlay guidance: Renders highlighted targets and instruction text as an on-screen overlay so users can follow each step without switching contexts. Click-driven progression with live replanning: After each user click, captures a fresh screenshot and calls backend to determine next step: whether to continue, retry, or complete task, enabling dynamic adaptation. Voice input: Supports voice-based task capture/transcription flow to feed as input to agent. Additional features System-wide hotkey and input monitoring: For quick launch from anywhere. Screen recording and accessibility-aware UX: For macOS permission-sensitive flows. Schema-validated AI responses: Enforces structured output consistency between the backend and client.
### How we built it
Our stack uses a native + AI pipeline architecture: Frontend/Desktop client: Swift, SwiftUI, AppKit (NSPanel overlays), Carbon/CGEvent-based global input handling, and macOS capture/accessibility integrations. Backend/API layer: Python, FastAPI, Uvicorn, Pydantic, multipart form endpoints, and request-level logging/error handling. AI processing pipeline: Multimodal planning and refinement pipeline utilizing GPT-4o, Gemini 3 Flash, and Claude 3.5 Sonnet; we benchmarked these models to select the optimal engine for structured JSON output and reasoning. Computer vision and image handling: OmniParser, Pillow, Ultralytics, and refinement pipeline variants. Voice pipeline: Modal-backed transcription integration with backend endpoint orchestration.
### Challenges we ran into
Post-click timing correctness: Ensuring screenshots are captured after UI state changes, not prematurely on click-down events. Cross-system coordinate alignment: Keeping overlay target mapping accurate across normalized coordinates, screen bounds, and different display setups. Permission complexity on macOS: Handling Screen Recording and Accessibility permission states without breaking the UX.
### Accomplishments we're proud of
Built a functioning end-to-end AI guidance overlay for native macOS workflows. Implemented dynamic step progression (continue / retry / done) with fresh screenshot context each click. Established a clean architecture split across overlay/UI, capture/input, agent pipeline, and state machine ownership. Successfully integrated a voice input path while preserving the core UX.
### What we learned
Iterative visual refinement using a "Set-of-Mark" grid for localization followed by targeted OmniParser crops provides significantly higher fidelity than attempting to parse a complex screen in a single shot. Designing reliable human-in-the-loop automation requires careful event timing and state transitions. Shared schema contracts dramatically reduce integration drift between the frontend and backend. macOS-level integrations (capture/input/permissions) need architecture decisions as much as coding effort.
### What's next
1. Lightning-Fast Plan Generation (Latency Optimization) Hybrid Model Orchestration: Using highly efficient Small Language Models (SLMs) for fast, repetitive routing tasks while reserving larger models exclusively for complex reasoning to drastically cut down the initial "Time to First Token". Token Optimization & Caching: Reducing overall wait times by leveraging Key-Value (KV) caching for static prompt components, and enforcing strict output constraints since generating output tokens is the most computationally expensive phase of the response. Parallel Execution: Running independent agent processes (like guardrail checks, visual parsing, and plan generation) simultaneously rather than sequentially to optimize system throughput. 2. Multi-App Workflows & Community "Recipes" Cross-Application Automation: Expanding the agent's capability to guide users through complex tasks that span multiple software programs simultaneously (e.g., extracting data from Excel, formatting it in Word, and sending it via Slack). Community Recipe Sharing: Creating a platform where power users can record, refine, and share their own custom "Cookbook" workflows, allowing the community to crowdsource interactive tutorials for niche software. Predictive Next-Steps: Anticipating the user's overall goal based on their first few actions and seamlessly queuing up the next logical "Recipe" steps before they even have to ask. Our Vision: To make The Cookbook the default “AI copilot layer” for desktop productivity—turning any complex UI workflow into clear, guided, real-time steps.
## README (from the GitHub repository)
# The Cookbook
A macOS system-wide AI guidance overlay. Press a hotkey, describe what you want to do, and get step-by-step visual instructions overlaid on top of any app.
## Quick Start
### Agent Server (Python)
```bash
cd agent-server
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # fill in your OPENAI_API_KEY
uvicorn app.main:app --reload
```
Mock mode (no API key needed):
```bash
MOCK_MODE=true uvicorn app.main:app --reload
```
Server runs at `http://localhost:8000`. Health check: `GET /health`.
### Mac Client (Swift)
**Recommended — run as .app (shows in Accessibility):**
```bash
cd mac-client
chmod +x run.sh
./run.sh
```
On first run, add **The Cookbook** to System Settings > Privacy & Security > Accessibility (click + and select the TheCookbook.app that appears). Then press **Cmd+Option+O** to toggle the overlay.
**Or build and run directly:**
```bash
cd mac-client
swift run TheCookbook
```
(Requires adding Terminal to Accessibility, or the binary path when prompted.)
> Requires macOS 13+, Screen Recording permission, and Accessibility permission for the hotkey.
## Architecture
```
User presses hotkey
→ Overlay UI appears
→ User types goal
→ Screenshot captured
→ POST /plan to agent-server
→ Agent returns StepPlan JSON
→ Overlay renders step highlights
→ User clicks target → step advances
→ Completion screen
```
## Repo Structure
```
mac-client/ # Swift macOS app
OverlayGuide/
App/ # Entry point, AppDelegate
Overlay/ # Overlay windows + highlight rendering
Capture/ # Screenshot capture + coordinate mapping
Input/ # Global hotkey + mouse click detection
State/ # State machine + session state
Networking/ # HTTP client to agent-server
Models/ # Shared data models (StepPlan, etc.)
UI/ # SwiftUI views (goal input, completion, onboarding)
agent-server/ # Python FastAPI backend
app/
main.py # App entry + CORS
routers/plan.py # POST /plan endpoint
schemas/step_plan.py # Pydantic models
prompts/ # Agent prompt templates
services/agent.py # AI model integration
services/mock.py # Mock mode for demos
shared/ # Cross-platform artifacts
step_plan_schema.json # Source of truth JSON schema
example_step_plan.json # Example plan for testing
docs/
spec.md # Full project spec
```
## Engineer Ownership
Each engineer has a detailed runbook in `docs/` — **read yours before starting**.
| Engineer | Area | Directories | Runbook |
|----------|------|-------------|---------|
| **Eng 1** | Overlay + UI | `mac-client/.../Overlay/`, `mac-client/.../UI/` | [`docs/eng1-overlay-ui.md`](docs/eng1-overlay-ui.md) |
| **Eng 2** | Capture + Input | `mac-client/.../Capture/`, `mac-client/.../Input/` | [`docs/eng2-capture-input.md`](docs/eng2-capture-input.md) |
| **Eng 3** | Agent Pipeline | `agent-server/`, `mac-client/.../Networking/`, `shared/` | [`docs/eng3-agent.md`](docs/eng3-agent.md) |
| **Eng 4** | State Machine | `mac-client/.../State/`, `mac-client/.../Models/`, `mac-client/.../App/` | [`docs/eng4-state.md`](docs/eng4-state.md) |
## Key Conventions
- **Coordinates**: Normalized `[0,1]`, top-left origin `(0,0)`
- **Schema**: All AI outputs validate against `shared/step_plan_schema.json`
- **Request IDs**: UUID in `X-Request-ID` header, logged on both client and server
- **Commit prefixes**: `[overlay]`, `[capture]`, `[agent]`, `[state]`
See `.cursor/rules/project.mdc` for the full coding rules.
## Testing Overlay Output Updates
- Use backend mock mode to exercise both `POST /plan` and `POST /next`.
- The mac state machine now supports applying raw API payloads directly via `applyPlanJSON(_:asNextPlan:)` for UI-only testing.
- Start with `shared/example_step_plan.json` as the initial payload, then paste a `/next`-shaped `StepPlan` JSON payload and call `applyPlanJSON(..., asNextPlan: true)` to verify the overlay refreshes in place.
### Quick UI Tester (CLI)
Run the mac client in synthetic UI test mode and pass parameters:
```bash
cd mac-client
swift run TheCookbook --ui-test --goal "Create calendar event" --steps 4 --x 0.22 --y 0.24 --w 0.18 --h 0.05 --next-after 3
```
- Prints the generated `StepPlan` JSON to terminal (`initial`, then `next` if `--next-after` is provided)
- Renders the overlay directly with those synthetic steps
- Skips hotkey/click monitor setup in this mode to make testing deterministic
See all test flags:
```bash
cd mac-client
swift run TheCookbook --ui-test-help
```
## Detected evidence (automated analysis)
Indexed codebase: 48 recognized source files, 466 KB.
- FastAPI (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- Swift (language) — detected in the code
- AI coding agent: Cursor — evidence: config files committed to the repository; commit authorship or trailers
## Codebase structure (from repository index)
### Files (69 of 69)
```
.cursor/rules/project.mdc
.cursor/worktrees.json
.gitignore
agent-server/.env.example
agent-server/app/__init__.py
agent-server/app/main.py
agent-server/app/prompts/gemini_identify_prompt.txt
agent-server/app/prompts/gemini_locate_next_prompt.txt
agent-server/app/prompts/gemini_locate_prompt.txt
agent-server/app/prompts/gemini_next_prompt.txt
agent-server/app/prompts/gemini_plan_prompt.txt
agent-server/app/prompts/next_step_prompt.txt
agent-server/app/prompts/omniparser_plan_prompt.txt
agent-server/app/prompts/omniparser_refine_prompt.txt
agent-server/app/prompts/plan_prompt.txt
agent-server/app/prompts/refine_prompt.txt
agent-server/app/prompts/replan_prompt.txt
agent-server/app/prompts/som_plan_prompt.txt
agent-server/app/prompts/som_refine_prompt.txt
agent-server/app/prompts/verify_element_prompt.txt
agent-server/app/routers/__init__.py
agent-server/app/routers/next_step.py
agent-server/app/routers/plan.py
agent-server/app/routers/refine.py
agent-server/app/routers/replan.py
agent-server/app/schemas/__init__.py
agent-server/app/schemas/step_plan.py
agent-server/app/scripts/bbox.py
agent-server/app/scripts/yolo_visualize.py
agent-server/app/services/__init__.py
agent-server/app/services/agent.py
agent-server/app/services/debug.py
agent-server/app/services/mock.py
agent-server/app/services/omniparser.py
agent-server/app/services/search.py
agent-server/requirements.txt
agent-server/test_accuracy.py
agent-server/test_live.py
agent-server/test_search.py
docs/eng1-overlay-ui.md
docs/eng2-capture-input.md
docs/eng3-agent.md
docs/eng4-state.md
docs/spec.md
mac-client/OverlayGuide/App/AppDelegate.swift
mac-client/OverlayGuide/App/OverlayGuideApp.swift
mac-client/OverlayGuide/Capture/CoordinateMapper.swift
mac-client/OverlayGuide/Capture/ScreenCaptureService.swift
mac-client/OverlayGuide/Input/GlobalInputMonitor.swift
mac-client/OverlayGuide/Input/VoiceInputService.swift
mac-client/OverlayGuide/Models/SessionState.swift
mac-client/OverlayGuide/Models/StepPlan.swift
mac-client/OverlayGuide/Models/UserPreferences.swift
mac-client/OverlayGuide/Networking/AgentNetworkClient.swift
mac-client/OverlayGuide/Overlay/OnboardingWindowController.swift
mac-client/OverlayGuide/Overlay/OverlayContentView.swift
mac-client/OverlayGuide/Overlay/OverlayWindowController.swift
mac-client/OverlayGuide/Overlay/TargetHighlightOverlayView.swift
mac-client/OverlayGuide/State/GuidanceStateMachine.swift
mac-client/OverlayGuide/Testing/OverlayUITester.swift
mac-client/OverlayGuide/UI/CompletionView.swift
mac-client/OverlayGuide/UI/GoalInputView.swift
mac-client/OverlayGuide/UI/OnboardingView.swift
mac-client/Package.swift
mac-client/run.sh
README.md
shared/example_step_plan.json
shared/step_plan_schema.json
som_zoom_stitch_integration.md
```
### Dependencies
- agent-server/requirements.txt: fastapi, google-generativeai, gradio_client, httpx, huggingface_hub, openai, pillow, pydantic@>=2.0, python-dotenv, python-multipart, ultralytics, uvicorn[standard]
### Recent commits (newest first)
- last version
- some version
- stream fix and trying to make more accurate with two pass
- iterative claude object detection
- working version, has off by one error occasionally
- SECOND WORKING VERSION
- THIS VERSION IS THE GOATTTT
- somewhat working
- current version
- current v
- Improve SoM pipeline: model-adaptive params, blue box overlay, stable signing
- WORKING VERSION 1.0
- added SOM improvements
- eliminate permissions requirement each time rebuilding
- Enhance screen capture and overlay functionality
- highlight now shows up
- sends request to agent but highlight not working
- Merge remote-tracking branch 'origin/shamit/agent-pipeline' into aarav
- Merge branch 'origin/shamit/overlay-next-ui-integration' into aarav
- changed design of input box
## Key source files (fetched from GitHub, selected and truncated for size)
### som_zoom_stitch_integration.md
```markdown
# SoM + Zoom + Stitch Integration Guide (Cursor Implementation File)
This file describes how to integrate a **Set-of-Mark (SoM) + Zoom Refinement + Stitch-back** pipeline into the existing overlay agent architecture.
Goal:
- Improve target accuracy for small / ambiguous UI elements
- Keep latency low (1–2 model calls per step)
- Work across *any* app (including custom-rendered UIs like Blender)
---
## 1) High-level Approach
Instead of asking the model to output a full-screen bounding box directly:
1. **Coarse grounding (SoM):**
- Render the screenshot with numbered markers (or regions) overlaid.
- Ask the model: *which marker corresponds to the clickable UI element?*
- Model outputs `marker_id` (integer).
2. **Refinement (optional zoom):**
- Take a crop around the chosen marker.
- Ask the model to output a tight bbox **within the crop** (normalized to the crop).
- Model outputs `{x,y,w,h}` in crop-normalized coordinates.
3. **Stitch back to full-screen normalized coords:**
- Convert crop bbox → full-image normalized bbox.
- Render overlay using the stitched bbox.
This avoids fragile “raw coordinate guessing” and gives precision without heavy ML infra.
---
## 2) Data Structures
### 2.1 Marker Definition (client-side)
Markers are generated on the client for each screenshot.
Recommended MVP:
- uniform grid markers (fast)
- later: proposal-based markers (OCR boxes, edge boxes, etc.)
Represent each marker as:
```json
{
"id": 37,
"cx": 0.412,
"cy": 0.281,
"radius": 0.012,
"screen_id": "main"
}
```
Where `cx,cy` are normalized point coords in the full screenshot coordinate space (origin top-left, normalized [0,1]).
### 2.2 Crop Definition
When refining, define a crop rectangle around the marker center:
```json
{
"cx": 0.412,
"cy": 0.281,
"cw": 0.20,
"ch": 0.20
}
```
Where:
- `(cx,cy)` is top-left of the crop in full-image normalized coords
- `(cw,ch)` is width/height of the crop in full-image normalized coords
**Crop sizing rules (MVP):**
- Use a fixed normalized crop size like `cw=ch=0.18` (tune)
- Clamp crop to image bounds [0,1]
---
## 3) Stitching Math (Core)
Given:
- crop rect in full-image normalized coords: `(cx, cy, cw, ch)`
- model returns bbox in crop-normalized coords: `(x, y, w, h)`
Stitched bbox in full-image normalized coords:
```text
x' = cx + x * cw
y' = cy + y * ch
w' = w * cw
h' = h * ch
```
All values remain normalized in [0,1] (clamp to bounds).
---
## 4) When to Refine (Latency Control)
Only do zoom refinement if any are true:
- target likely small (heuristic):
- crop bbox predicted from marker radius would be < ~0.002 of screen area
- model confidence is low
- user miss-clicked twice on this step
- the selected marker is in a dense region (many nearby markers / OCR text clusters)
Otherwise:
- Use a default bbox around the marker (fast path)
### Fast-path bbox (no refine)
Create a bbox around marker center:
```text
w = h = 0.06 (tune)
x = cx_marker - w/2
[truncated — 4227 more characters]
```
### docs/eng1-overlay-ui.md
```markdown
# Eng 1 — Overlay + UI
## What You Own
You are responsible for **everything the user sees**. The overlay windows, the highlight rectangles, the instruction bubbles, the goal input screen, the completion screen, animations, and visual polish. Your code makes or breaks the demo.
### Your Directories
```
mac-client/OverlayGuide/
Overlay/ # NSPanel window management + overlay content
OverlayWindowController.swift # Creates/manages one NSPanel per display
OverlayContentView.swift # SwiftUI view: highlights + instruction bubble
UI/ # Standalone SwiftUI views
GoalInputView.swift # Text input for the user's goal
CompletionView.swift # "All done!" success screen
```
### Your Commit Prefix
`[overlay]` — e.g. `[overlay] add pulsing animation to highlight rects`
---
## What You Need to Build (MVP)
### 1. Overlay Window System (`Overlay/OverlayWindowController.swift`)
Create one `NSPanel` per connected display that:
- Is borderless, transparent, always-on-top
- Appears on all Spaces and over fullscreen apps (`.canJoinAllSpaces`, `.fullScreenAuxiliary`)
- Does NOT steal focus (`.nonactivatingPanel`)
- Can toggle `ignoresMouseEvents` — pass-through when guiding, capture when showing input
Key API:
- `showOverlay()` — create panels for all `NSScreen.screens`
- `hideAll()` — close and remove all panels
- The content of each panel is a SwiftUI `NSHostingView`
### 2. Highlight Rendering (`Overlay/OverlayContentView.swift`)
For the current step, draw:
- **Semi-transparent dark backdrop** (Color.black.opacity(0.4)) covering the full screen
- **Cut-out highlight rectangles** around each target (bright border, slightly lighter fill)
- **Instruction bubble** near the target area with the step's `instruction` text
- **Step progress indicator** — "Step 2 of 5"
Targets come as normalized `[0,1]` coordinates. Convert to screen pixels:
```
pixelX = target.x * screenBounds.width
pixelY = target.y * screenBounds.height
pixelW = target.w * screenBounds.width
pixelH = target.h * screenBounds.height
```
### 3. Goal Input Screen (`UI/GoalInputView.swift`)
When `stateMachine.phase == .inputGoal`:
- Show a centered text field with placeholder like "What do you need help with?"
- Submit on Enter or click the send button
- Call `stateMachine.submitGoal(text)` on submit
### 4. Completion Screen (`UI/CompletionView.swift`)
When `stateMachine.phase == .completed`:
- Show a success message with the original goal
- "Dismiss" button that calls `stateMachine.reset()`
### 5. Loading State
When `stateMachine.phase == .loading`:
- Show a spinner / "Thinking..." indicator so the user knows the agent is working
### 6. Error State
When `stateMachine.phase == .error(message)`:
- Show the error message with a "Try Again" button
---
## What You Read From (Don't Modify)
You observe `GuidanceStateMachine` (owned by Eng 4). It's an `ObservableObject` with:
| Property | Type | What it tell
[truncated — 2106 more characters]
```
### agent-server/requirements.txt
```
fastapi
uvicorn[standard]
pydantic>=2.0
openai
google-generativeai
python-multipart
pillow
python-dotenv
gradio_client
ultralytics
huggingface_hub
httpx
```
### agent-server/app/main.py
```python
# app/main.py
# Owner: Eng 3 (Agent Pipeline)
#
# FastAPI application entry point.
# Registers routers, loads env, configures CORS, and sets up error handling.
import os
import time
import uuid
from contextlib import asynccontextmanager
from dotenv import load_dotenv
from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
load_dotenv() # load .env before anything else
from app.routers import next_step, plan, refine, replan # noqa: E402
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Startup / shutdown events."""
mock_mode = os.getenv("MOCK_MODE", "false").lower() == "true"
model = os.getenv("OPENAI_MODEL", "gpt-4o")
gemini_key = bool(os.getenv("GEMINI_API_KEY"))
openai_key = bool(os.getenv("OPENAI_API_KEY"))
openrouter_key = bool(os.getenv("OPENROUTER_API_KEY"))
provider = "Gemini" if gemini_key else ("OpenAI" if openai_key else ("OpenRouter" if openrouter_key else "NONE"))
has_key = gemini_key or openai_key or openrouter_key
print(f"[server] The Cookbook Agent Server starting")
print(f"[server] MOCK_MODE={mock_mode}")
print(f"[server] PROVIDER={provider}")
print(f"[server] MODEL={model}")
if not mock_mode and not has_key:
print("[server] WARNING: No API key set and mock mode is off. /plan will fail.")
yield
print("[server] Shutting down.")
app = FastAPI(
title="The Cookbook Agent Server",
description="AI agent that generates step-by-step UI guidance plans from screenshots",
version="0.1.0",
lifespan=lifespan,
)
# CORS — allow the mac client to connect from localhost
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# ---------------------------------------------------------------------------
# Request ID + timing middleware
# ---------------------------------------------------------------------------
@app.middleware("http")
async def request_id_middleware(request: Request, call_next):
request_id = request.headers.get("x-request-id") or str(uuid.uuid4())
request.state.request_id = request_id
start = time.time()
response = await call_next(request)
elapsed_ms = round((time.time() - start) * 1000)
response.headers["X-Request-ID"] = request_id
print(f"[server] {request.method} {request.url.path} -> {response.status_code} ({elapsed_ms}ms) rid={request_id}")
return response
# ---------------------------------------------------------------------------
# Global exception handler — return JSON, never HTML stacktraces
# ---------------------------------------------------------------------------
@app.exception_handler(Exception)
async def global_exception_handler(request: Request, exc: Exception):
request_id = getattr(request.state, "request_id", "unknown")
print(f"[server] Unhandled error rid={request_id}: {type(exc).__name__}: {exc}")
return JSONResponse(
status_code=500,
content={
"error": "internal_server_error",
"message": str(exc),
"request_id": request_id,
},
)
# Register routers
app.include_router(plan.router)
app.include_router(refine.router)
app.include_router(next_step.router)
app.include_router(replan.router)
app.include_router(refine.router)
@app.get("/health")
async def health():
"""Health check endpoint. Returns mock status if MOCK_MODE is enabled."""
mock_mode = os.getenv("MOCK_MODE", "false").lower() == "true"
return {
"status": "ok",
"mock_mode": mock_mode,
"model": os.getenv("OPENAI_MODEL", "gpt-4o"),
}
```
### mac-client/Package.swift
```swift
// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift Package Manager required to build this package.
import PackageDescription
let package = Package(
name: "TheCookbook",
platforms: [
.macOS(.v13)
],
targets: [
.executableTarget(
name: "TheCookbook",
path: "OverlayGuide"
),
]
)
```
### mac-client/run.sh
```shell
#!/bin/bash
# Launch The Cookbook from a stable app bundle path.
# By default this ONLY launches to preserve macOS permissions.
# Use --rebuild when you intentionally want to rebuild/update the app binary.
# Usage:
# ./run.sh
# ./run.sh --rebuild
set -e
cd "$(dirname "$0")"
BINARY=".build/debug/TheCookbook"
APP_NAME="TheCookbook.app"
INSTALL_DIR="$HOME/Applications"
APP_PATH="$INSTALL_DIR/$APP_NAME"
CONTENTS="$APP_PATH/Contents"
MACOS="$CONTENTS/MacOS"
RESOURCES="$CONTENTS/Resources"
DO_REBUILD=false
SIGNING_IDENTITY="${DEVELOPER_IDENTITY:-}"
if [ "${1:-}" = "--rebuild" ]; then
DO_REBUILD=true
fi
if [ ! -f "$APP_PATH/Contents/MacOS/TheCookbook" ]; then
DO_REBUILD=true
fi
if [ "$DO_REBUILD" = true ]; then
echo "Building The Cookbook..."
swift build
fi
# Keep a stable app bundle path so macOS permissions persist.
mkdir -p "$MACOS"
mkdir -p "$RESOURCES"
# Copy latest built binary into stable app bundle only on rebuild/install.
if [ "$DO_REBUILD" = true ]; then
cp "$BINARY" "$MACOS/TheCookbook"
chmod +x "$MACOS/TheCookbook"
fi
# Create Info.plist only once (keep bundle identity stable).
if [ ! -f "$CONTENTS/Info.plist" ]; then
cat > "$CONTENTS/Info.plist" << 'EOF'
CFBundleExecutable
TheCookbook
CFBundleIdentifier
com.thecookbook.app
CFBundleName
The Cookbook
CFBundlePackageType
APPL
LSMinimumSystemVersion
13.0
NSMicrophoneUsageDescription
The Cookbook uses the microphone for voice input so you can speak your goal instead of typing.
NSSpeechRecognitionUsageDescription
The Cookbook uses speech recognition to convert your voice into text for goal input.
EOF
fi
choose_signing_identity() {
if [ -n "$SIGNING_IDENTITY" ]; then
echo "$SIGNING_IDENTITY"
return
fi
security find-identity -v -p codesigning 2>/dev/null | /usr/bin/awk -F\" '/Apple Development/ {print $2; exit}'
}
sign_app_if_needed() {
local identity
identity="$(choose_signing_identity)"
if [ -n "$identity" ]; then
echo "Signing app with identity: $identity"
codesign --force --deep --sign "$identity" "$APP_PATH"
echo "Code signature verification:"
codesign --verify --deep --strict "$APP_PATH"
else
echo "No Apple Development signing identity found."
echo "Falling back to ad-hoc signing (works for local dev)."
codesign --force --deep --sign - "$APP_PATH"
echo "Code signature verification:"
codesign --verify --deep --strict "$APP_PATH"
fi
}
# Only sign once — on first install. After that, skip signing so
# the CDHash stays the same and macOS permissions persist across rebuilds.
# The binary changes but the signature stays, so macOS may complain once
# about a "damaged" app — just right-click > Open to bypass that one time.
SIGNED_MARKER="$CONTENTS/.signed_once"
if [ "$DO_REBUILD" = true ] && [ ! -f "$SIGNED_MARKER" ]; then
sign_app_if_needed
touch "$SIGNED_MARKER"
elif [ "$DO_REBUILD" = true ]; then
echo "Skipping re-sign to preserve Screen Recording & Accessibility permissions."
fi
echo "Installed to $APP_PATH"
echo "Bundle identifier: com.thecookbook.app"
if [ "$DO_REBUILD" = true ]; then
echo "Mode: rebuild + launch"
else
echo "Mode: launch only (no rebuild)"
fi
echo ""
echo "Launching The Cookbook..."
echo ""
echo "If hotkey (Cmd+Option+O) doesn't work:"
echo " 1. System Settings > Privacy & Security > Accessibility"
echo " 2. Remove TheCookbook if listed, then click +"
echo " 3. Press Cmd+Shift+G, paste: $INSTALL_DIR"
echo " 4. Select TheCookbook.app, click Open"
echo " 5. Quit The Cookbook (Cmd+Q) and run ./run.sh again"
echo ""
echo "NOTE: Use ./run.sh for launch-only to preserve Screen Recording permission."
echo " Use ./run.sh --rebuild only when you intentionally want new code."
echo ""
# Ensure stale instance is closed before launching the fresh build
pkill -x TheCookbook >/dev/null 2>&1 || true
open -n "$APP_PATH"
```
### agent-server/test_accuracy.py
```python
#!/usr/bin/env python3
"""
Accuracy test for the SoM two-pass pipeline.
Takes a screenshot, runs the full pipeline offline (no server needed),
saves debug images at every step, and draws the final bbox on the
original screenshot so you can visually check accuracy.
Usage:
cd agent-server
source .venv/bin/activate
python test_accuracy.py
python test_accuracy.py "Click the File menu"
Then open /tmp/som_test/ in Finder to inspect all debug images.
"""
import asyncio
import io
import json
import os
import subprocess
import sys
import tempfile
import time
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()
from PIL import Image, ImageDraw, ImageFont
# Import pipeline functions
from app.routers.plan import (
_generate_markers_and_image,
_som_plan_to_step_plan,
_crop_and_draw_sub_markers,
_REFINED_BBOX_PAD,
SOM_COLUMNS,
SOM_ROWS,
REFINE_SUB_COLS,
REFINE_SUB_ROWS,
)
from app.schemas.step_plan import ImageSize, TargetRect
from app.services.agent import generate_som_plan, generate_som_refine
OUT_DIR = Path("/tmp/som_test")
DEFAULT_GOAL = "Click the File menu to save the file"
def take_screenshot() -> tuple[bytes, int, int]:
"""Capture screenshot, return (png_bytes, actual_w, actual_h)."""
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as f:
tmp = f.name
subprocess.run(["screencapture", "-x", "-C", "-m", tmp], capture_output=True, timeout=10)
png = Path(tmp).read_bytes()
Path(tmp).unlink()
img = Image.open(io.BytesIO(png))
return png, img.width, img.height
def draw_bbox_on_image(img_bytes: bytes, targets: list[dict], label: str) -> bytes:
"""Draw colored bboxes on an image. targets = [{x,y,w,h,label,color}]."""
img = Image.open(io.BytesIO(img_bytes)).convert("RGB")
draw = ImageDraw.Draw(img)
try:
font = ImageFont.truetype("/System/Library/Fonts/Helvetica.ttc", 20)
except Exception:
font = ImageFont.load_default()
w, h = img.size
for t in targets:
x1 = int(t["x"] * w)
y1 = int(t["y"] * h)
x2 = int((t["x"] + t["w"]) * w)
y2 = int((t["y"] + t["h"]) * h)
color = t.get("color", (0, 255, 0))
draw.rectangle([x1, y1, x2, y2], outline=color, width=3)
lbl = t.get("label", "")
if lbl:
draw.text((x1, max(0, y1 - 22)), lbl, fill=color, font=font)
buf = io.BytesIO()
img.save(buf, format="PNG")
return buf.getvalue()
async def run_test(goal: str):
OUT_DIR.mkdir(exist_ok=True)
model = os.getenv("OPENAI_MODEL", "gpt-4o")
print(f"\n{'='*70}")
print(f" SoM Accuracy Test")
print(f" Model: {model}")
print(f" Goal: {goal}")
print(f" Grid: {SOM_COLUMNS}x{SOM_ROWS} coarse, {REFINE_SUB_COLS}x{REFINE_SUB_ROWS} fine")
print(f" Output: {OUT_DIR}/")
print(f"{'='*70}\n")
# 1. Take screenshot
print("[1/5] Taking screenshot...")
t0 = time.time()
png_bytes, actual_w, actual_h = take_screenshot()
print(f" {actual_w}x{actual_h}, {len(png_bytes):,} bytes ({time.time()-t0:.1f}s)")
(OUT_DIR / "01_raw_screenshot.png").write_bytes(png_bytes)
# Get logical size (for image_size param)
# On Retina, logical = actual / 2
logical_w = actual_w // 2 if actual_w > 2000 else actual_w
logical_h = actual_h // 2 if actual_h > 2000 else actual_h
image_size = ImageSize(w=logical_w, h=logical_h)
print(f" Logical: {logical_w}x{logical_h}, Actual: {actual_w}x{actual_h}")
# 2. Generate coarse markers
print("\n[2/5] Generating coarse markers...")
t0 = time.time()
markers, marked_bytes = _generate_markers_and_image(png_bytes)
print(f" {len(markers)} markers, {len(marked_bytes):,} bytes ({time.time()-t0:.1f}s)")
(OUT_DIR / "02_coarse_markers.png").write_bytes(marked_bytes)
# 3. Pass 1: model picks coarse markers
print(f"\n[3/5] Pass 1: asking {model} to pick coarse markers...")
t0 = time.time()
som_plan = await generate_som_plan(
goal=goal,
image_size=image_size,
screenshot_bytes=marked_bytes,
request_id="test-accuracy",
)
elapsed1 = time.time() - t0
print(f" {len(som_plan.steps)} steps ({elapsed1:.1f}s)")
marker_map = {m.id: m for m in markers}
for step in som_plan.steps:
ids = [st.marker_id for st in step.som_targets]
positions = [(f"{marker_map[i].cx:.3f},{marker_map[i].cy:.3f}" if i in marker_map else "?") for i in ids]
print(f" Step {step.id}: markers={ids} pos=[{', '.join(positions)}] label={step.som_targets[0].label!r}")
# Convert to coarse plan
coarse_plan = _som_plan_to_step_plan(som_plan, markers)
# Draw coarse targets on screenshot
coarse_targets = []
for step in coarse_plan.steps:
for t in step.targets:
coarse_targets.append({"x": t.x, "y": t.y, "w": t.w, "h": t.h,
"label": f"{step.id}: {t.label}", "color": (255, 0, 0)})
coarse_vis = draw_bbox_on_image(png_bytes, coarse_targets, "coarse")
(OUT_DIR / "03_coarse_targets.png").write_bytes(coarse_vis)
# 4. Pass 2: refine each target
print(f"\n[4/5] Pass 2: refining with {REFINE_SUB_COLS}x{REFINE_SUB_ROWS} sub-grid...")
refined_targets_all = []
for si, step in enumerate(coarse_plan.steps):
for ti, target in enumerate(step.targets):
t0 = time.time()
crop_rect, marked_crop, sub_markers = _crop_and_draw_sub_markers(png_bytes, target)
crop_name = f"04_crop_s{si}_t{ti}.png"
(OUT_DIR / crop_name).write_bytes(marked_crop)
img_crop = Image.open(io.BytesIO(marked_crop))
cell_w = img_crop.width / REFINE_SUB_COLS
cell_h = img_crop.height / REFINE_SUB_ROWS
min_cell = min(cell_w, cell_h)
marker_r = max(8, int(min_cell * 0.25))
print(f" Step {step.id}:
[truncated — 3370 more characters]
```
### agent-server/test_live.py
```python
#!/usr/bin/env python3
"""
Quick live test for the agent server.
Takes a screenshot of your screen, sends it to POST /plan with a goal,
and prints the returned step plan. No frontend needed.
Usage:
# 1. Start the server (in another terminal):
# cd agent-server && source .venv/bin/activate
# uvicorn app.main:app --reload
#
# 2. Run this script (SoM pipeline — default):
# python test_live.py
# python test_live.py "Open System Settings and go to Wi-Fi"
#
# 3. Run with legacy (raw coord) pipeline:
# python test_live.py --legacy "Open System Settings and go to Wi-Fi"
"""
import io
import json
import subprocess
import sys
import tempfile
import time
import json
from pathlib import Path
import httpx
SERVER_URL = "http://localhost:8000"
DEFAULT_GOAL = "Find and open the Downloads folder in Finder"
# SoM grid configuration (match MarkerGenerator.swift defaults)
SOM_COLUMNS = 16
SOM_ROWS = 10
MARKER_PIXEL_RADIUS = 14
MARKER_FONT_SIZE = 11
def build_grid_markers(columns: int = 24, rows: int = 14) -> list[dict]:
markers: list[dict] = []
marker_id = 1
for row in range(rows):
for col in range(columns):
markers.append(
{
"id": marker_id,
"cx": (col + 0.5) / columns,
"cy": (row + 0.5) / rows,
"radius": 0.012,
"screen_id": "main",
}
)
marker_id += 1
return markers
def take_screenshot() -> tuple[bytes, int, int]:
"""Capture the main display using macOS screencapture. Returns (png_bytes, width, height)."""
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as f:
tmp_path = f.name
# -x = no sound, -C = capture cursor, -m = main display only
result = subprocess.run(
["screencapture", "-x", "-C", "-m", tmp_path],
capture_output=True,
timeout=10,
)
if result.returncode != 0:
raise RuntimeError(f"screencapture failed: {result.stderr.decode()}")
png_bytes = Path(tmp_path).read_bytes()
Path(tmp_path).unlink()
# Get screen dimensions via system_profiler
try:
sp = subprocess.run(
["system_profiler", "SPDisplaysDataType"],
capture_output=True,
text=True,
timeout=5,
)
for line in sp.stdout.splitlines():
if "Resolution" in line:
parts = line.split(":")[1].strip().split()
w, h = int(parts[0]), int(parts[2])
if "Retina" in line:
w, h = w // 2, h // 2
return png_bytes, w, h
except Exception:
pass
return png_bytes, 1920, 1080
def generate_markers(img_w: int, img_h: int) -> tuple[list[dict], bytes]:
"""
Generate SoM markers on a grid and draw them onto the screenshot.
Returns (markers_list, marked_png_bytes).
Requires Pillow for drawing.
"""
from PIL import Image, ImageDraw, ImageFont
markers = []
marker_id = 0
normalized_radius = MARKER_PIXEL_RADIUS / max(img_w, img_h)
for row in range(SOM_ROWS):
for col in range(SOM_COLUMNS):
cx = (col + 0.5) / SOM_COLUMNS
cy = (row + 0.5) / SOM_ROWS
markers.append({
"id": marker_id,
"cx": round(cx, 6),
"cy": round(cy, 6),
"radius": round(normalized_radius, 6),
})
marker_id += 1
return markers
def draw_markers_on_image(png_bytes: bytes, markers: list[dict], img_w: int, img_h: int) -> bytes:
"""Draw numbered marker circles onto the screenshot. Returns marked PNG bytes."""
from PIL import Image, ImageDraw, ImageFont
img = Image.open(io.BytesIO(png_bytes))
# Resize to match logical dimensions if needed (Retina screenshots are 2x)
if img.width != img_w or img.height != img_h:
# The PNG may be at Retina resolution; draw at the actual pixel size
pass # Draw at actual size, coordinates will be scaled
draw = ImageDraw.Draw(img)
# Try to load a basic font
try:
font = ImageFont.truetype("/System/Library/Fonts/Helvetica.ttc", MARKER_FONT_SIZE)
except Exception:
font = ImageFont.load_default()
actual_w, actual_h = img.size
for m in markers:
# Convert normalized to pixel coords at actual image size
px = m["cx"] * actual_w
py = m["cy"] * actual_h
r = MARKER_PIXEL_RADIUS
# Draw white filled circle with red border
circle_bbox = [px - r, py - r, px + r, py + r]
draw.ellipse(circle_bbox, fill=(255, 255, 255, 230), outline=(255, 50, 50, 220), width=2)
# Draw marker ID text
text = str(m["id"])
text_bbox = draw.textbbox((0, 0), text, font=font)
tw = text_bbox[2] - text_bbox[0]
th = text_bbox[3] - text_bbox[1]
draw.text((px - tw / 2, py - th / 2), text, fill=(0, 0, 0), font=font)
buf = io.BytesIO()
img.save(buf, format="PNG")
return buf.getvalue()
def test_plan(goal: str, use_som: bool = True) -> dict | None:
"""Send a screenshot + goal to POST /plan and print the result."""
mode = "SoM" if use_som else "Legacy"
print(f"\n{'='*60}")
print(f" GOAL: {goal}")
print(f" MODE: {mode}")
print(f"{'='*60}\n")
# Health check
print("[1/4] Checking server health...")
try:
resp = httpx.get(f"{SERVER_URL}/health", timeout=5)
health = resp.json()
print(f" Server: {health['status']}, mock_mode={health['mock_mode']}, model={health['model']}")
except httpx.ConnectError:
print(" ERROR: Cannot connect to server. Is it running?")
print(" Start it with:")
print(" cd agent-server && source .venv/bin/activate")
print(" uvicorn app.main:app --reload")
sys.exit(1)
# Screenshot
print("[2/4] Taking screenshot.
[truncated — 7621 more characters]
```
### agent-server/test_search.py
```python
#!/usr/bin/env python3
"""
Modular tests for the Bright Data web search integration.
Tests each layer independently so you can pinpoint failures fast:
1. Bright Data SERP API connectivity
2. HTML → text extraction
3. LLM search-query generation (uses project's configured provider)
4. Full search_for_goal pipeline
5. In-memory store persistence (simulates /plan → /next flow)
6. End-to-end /plan with search via the running server
Usage:
# Make sure .env is loaded (script loads it automatically).
cd agent-server && source .venv/bin/activate
# Run ALL tests:
python test_search.py
# Run a SINGLE test by name:
python test_search.py brightdata
python test_search.py html
python test_search.py queries
python test_search.py pipeline
python test_search.py store
python test_search.py server
"""
import asyncio
import json
import os
import subprocess
import sys
import tempfile
import time
from pathlib import Path
from dotenv import load_dotenv
load_dotenv() # load .env so API keys are available
# ---------------------------------------------------------------------------
# Utilities
# ---------------------------------------------------------------------------
PASS = "\033[92m✓ PASS\033[0m"
FAIL = "\033[91m✗ FAIL\033[0m"
SKIP = "\033[93m⊘ SKIP\033[0m"
HEADER = "\033[1;36m"
RESET = "\033[0m"
results: list[tuple[str, str, str]] = [] # (name, status, detail)
def record(name: str, passed: bool, detail: str = ""):
status = PASS if passed else FAIL
results.append((name, status, detail))
print(f" {status} {name}")
if detail:
for line in detail.split("\n"):
print(f" {line}")
def record_skip(name: str, reason: str):
results.append((name, SKIP, reason))
print(f" {SKIP} {name} ({reason})")
def section(title: str):
print(f"\n{HEADER}{'─'*60}")
print(f" {title}")
print(f"{'─'*60}{RESET}\n")
def take_screenshot() -> tuple[bytes, int, int]:
"""Capture screen. Returns (png_bytes, w, h)."""
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as f:
tmp_path = f.name
subprocess.run(
["screencapture", "-x", "-C", "-m", tmp_path],
capture_output=True, timeout=10,
)
png_bytes = Path(tmp_path).read_bytes()
Path(tmp_path).unlink()
return png_bytes, 1920, 1080
# ===================================================================
# TEST 1: Bright Data SERP API — raw connectivity
# ===================================================================
async def test_brightdata_api():
"""Call Bright Data SERP API directly with a simple query."""
section("Test 1: Bright Data SERP API connectivity")
api_key = os.getenv("BRIGHTDATA_API_KEY")
if not api_key:
record_skip("brightdata_api_key_present", "BRIGHTDATA_API_KEY not set in .env")
return
record("brightdata_api_key_present", True, f"Key: {api_key[:8]}...{api_key[-4:]}")
import httpx
import urllib.parse
query = "how to open System Settings on macOS"
url = f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
data = {
"zone": "serp_api1",
"url": url,
"format": "raw",
}
start = time.time()
try:
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.post(
"https://api.brightdata.com/request",
json=data,
headers=headers,
)
elapsed = round((time.time() - start) * 1000)
record(
"brightdata_http_status",
resp.status_code == 200,
f"Status: {resp.status_code} ({elapsed}ms)",
)
if resp.status_code == 200:
raw = resp.text
record(
"brightdata_response_has_content",
len(raw) > 100,
f"Response body: {len(raw):,} chars",
)
# SERP API returns JSON even with format=raw; verify structure
try:
body = resp.json()
is_json = isinstance(body, dict) and "organic" in body
except Exception:
is_json = False
record(
"brightdata_response_is_serp_json",
is_json,
f"Parseable as SERP JSON with 'organic' key: {is_json}",
)
else:
record("brightdata_response_has_content", False, f"Error: {resp.text[:300]}")
except Exception as e:
elapsed = round((time.time() - start) * 1000)
record("brightdata_http_request", False, f"{type(e).__name__}: {e} ({elapsed}ms)")
# ===================================================================
# TEST 2: HTML → plain text extraction
# ===================================================================
def test_html_extraction():
"""Verify the HTML stripping helper works correctly."""
section("Test 2: HTML → text extraction")
from app.services.search import _html_to_text
# Simple tags
result = _html_to_text("Hello world
")
record(
"html_strips_basic_tags",
"Hello" in result and "world" in result and "<" not in result,
f"Result: {result!r}",
)
# HTML entities
result = _html_to_text("& < > "")
record(
"html_decodes_entities",
"&" in result and "<" in result and ">" in result,
f"Result: {result!r}",
)
# Whitespace collapsing
result = _html_to_text(" lots of \n\n space
")
record(
"html_collapses_whitespace",
" " not in result and "\n" not in result,
f"Result: {result!r}",
)
# Empty input
result = _html_to_text("")
record("html_handles_empty", result == "", f"Result: {result!r}")
# =================================================
[truncated — 10767 more characters]
```
### mac-client/OverlayGuide/App/OverlayGuideApp.swift
```swift
// App/TheCookbookApp.swift
// Owner: Shared (App entry point)
//
// Main entry point for The Cookbook macOS app.
// Wires together the state machine, overlay controller, and input monitor.
import SwiftUI
@main
struct TheCookbookApp: App {
@NSApplicationDelegateAdaptor(AppDelegate.self) var appDelegate
var body: some Scene {
// We use AppDelegate to manage windows directly (NSPanel).
// No SwiftUI WindowGroup needed for the overlay.
// The Settings scene powers the native "Settings..." (Cmd+,) menu item.
Settings {
OnboardingView(
preferences: .shared,
settingsMode: true,
onComplete: {
// Close the settings window when the user saves/dismisses
NSApp.keyWindow?.close()
}
)
}
}
}
```
[32 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]