Project Info

The Cookbook

Inspiration

We’ve all been there–you’re trying to learn a new tool, and you can’t find the right video tutorial to help you get off the ground. Your computer already has 100 windows open, and you’re switching between screens, watching a video that's not even relevant to your task. Even worse, the software version used in the video is incompatible. Our goal is to simplify the process for users trying to learn new softwares, such as Photoshop, Cursor, Canva, and more.

What it does

Meet The Cookbook! It overlays real-time visual instructions on your screen to guide you through complex software tasks step-by-step. Just open the app, describe what you need help with, and receive on-screen highlights and text instructions with a detailed plan for your task. The instructions are catered to your application version, your style of learning, and your screen exactly as it looks. The main features are: Goal-to-steps generation from live screenshots: Captures the current screen and sends it to a multi-agent pipeline using OpenAI and Gemini APIs to generate precise coordinates for the next user click and a detailed instructional plan. Two-stage spatial grounding pipeline: Generates a 40-marker coordinate grid to help the agent identify a specific "zoom zone," then runs OmniParser on that high-resolution crop to ensure pixel-perfect bounding box accuracy without the clutter of full-screen detection. Interactive overlay guidance: Renders highlighted targets and instruction text as an on-screen overlay so users can follow each step without switching contexts. Click-driven progression with live replanning: After each user click, captures a fresh screenshot and calls backend to determine next step: whether to continue, retry, or complete task, enabling dynamic adaptation. Voice input: Supports voice-based task capture/transcription flow to feed as input to agent. Additional features System-wide hotkey and input monitoring: For quick launch from anywhere. Screen recording and accessibility-aware UX: For macOS permission-sensitive flows. Schema-validated AI responses: Enforces structured output consistency between the backend and client.

How we built it

Our stack uses a native + AI pipeline architecture: Frontend/Desktop client: Swift, SwiftUI, AppKit (NSPanel overlays), Carbon/CGEvent-based global input handling, and macOS capture/accessibility integrations. Backend/API layer: Python, FastAPI, Uvicorn, Pydantic, multipart form endpoints, and request-level logging/error handling. AI processing pipeline: Multimodal planning and refinement pipeline utilizing GPT-4o, Gemini 3 Flash, and Claude 3.5 Sonnet; we benchmarked these models to select the optimal engine for structured JSON output and reasoning. Computer vision and image handling: OmniParser, Pillow, Ultralytics, and refinement pipeline variants. Voice pipeline: Modal-backed transcription integration with backend endpoint orchestration.

Challenges we ran into

Post-click timing correctness: Ensuring screenshots are captured after UI state changes, not prematurely on click-down events. Cross-system coordinate alignment: Keeping overlay target mapping accurate across normalized coordinates, screen bounds, and different display setups. Permission complexity on macOS: Handling Screen Recording and Accessibility permission states without breaking the UX.

Accomplishments we're proud of

Built a functioning end-to-end AI guidance overlay for native macOS workflows. Implemented dynamic step progression (continue / retry / done) with fresh screenshot context each click. Established a clean architecture split across overlay/UI, capture/input, agent pipeline, and state machine ownership. Successfully integrated a voice input path while preserving the core UX.

What we learned

Iterative visual refinement using a "Set-of-Mark" grid for localization followed by targeted OmniParser crops provides significantly higher fidelity than attempting to parse a complex screen in a single shot. Designing reliable human-in-the-loop automation requires careful event timing and state transitions. Shared schema contracts dramatically reduce integration drift between the frontend and backend. macOS-level integrations (capture/input/permissions) need architecture decisions as much as coding effort.

What's next

1. Lightning-Fast Plan Generation (Latency Optimization) Hybrid Model Orchestration: Using highly efficient Small Language Models (SLMs) for fast, repetitive routing tasks while reserving larger models exclusively for complex reasoning to drastically cut down the initial "Time to First Token". Token Optimization & Caching: Reducing overall wait times by leveraging Key-Value (KV) caching for static prompt components, and enforcing strict output constraints since generating output tokens is the most computationally expensive phase of the response. Parallel Execution: Running independent agent processes (like guardrail checks, visual parsing, and plan generation) simultaneously rather than sequentially to optimize system throughput. 2. Multi-App Workflows & Community "Recipes" Cross-Application Automation: Expanding the agent's capability to guide users through complex tasks that span multiple software programs simultaneously (e.g., extracting data from Excel, formatting it in Word, and sending it via Slack). Community Recipe Sharing: Creating a platform where power users can record, refine, and share their own custom "Cookbook" workflows, allowing the community to crowdsource interactive tutorials for niche software. Predictive Next-Steps: Anticipating the user's overall goal based on their first few actions and seamlessly queuing up the next logical "Recipe" steps before they even have to ask. Our Vision: To make The Cookbook the default “AI copilot layer” for desktop productivity—turning any complex UI workflow into clear, guided, real-time steps.

The Cookbook

A macOS system-wide AI guidance overlay. Press a hotkey, describe what you want to do, and get step-by-step visual instructions overlaid on top of any app.

Quick Start

Agent Server (Python)

cd agent-server
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # fill in your OPENAI_API_KEY
uvicorn app.main:app --reload

Mock mode (no API key needed):

MOCK_MODE=true uvicorn app.main:app --reload

Server runs at http://localhost:8000. Health check: GET /health.

Mac Client (Swift)

Recommended — run as .app (shows in Accessibility):

cd mac-client
chmod +x run.sh
./run.sh

On first run, add The Cookbook to System Settings > Privacy & Security > Accessibility (click + and select the TheCookbook.app that appears). Then press Cmd+Option+O to toggle the overlay.

Or build and run directly:

cd mac-client
swift run TheCookbook

(Requires adding Terminal to Accessibility, or the binary path when prompted.)

Requires macOS 13+, Screen Recording permission, and Accessibility permission for the hotkey.

Architecture

User presses hotkey
  → Overlay UI appears
  → User types goal
  → Screenshot captured
  → POST /plan to agent-server
  → Agent returns StepPlan JSON
  → Overlay renders step highlights
  → User clicks target → step advances
  → Completion screen

Repo Structure

mac-client/                 # Swift macOS app
  OverlayGuide/
    App/                    # Entry point, AppDelegate
    Overlay/                # Overlay windows + highlight rendering
    Capture/                # Screenshot capture + coordinate mapping
    Input/                  # Global hotkey + mouse click detection
    State/                  # State machine + session state
    Networking/             # HTTP client to agent-server
    Models/                 # Shared data models (StepPlan, etc.)
    UI/                     # SwiftUI views (goal input, completion, onboarding)
agent-server/               # Python FastAPI backend
  app/
    main.py                 # App entry + CORS
    routers/plan.py         # POST /plan endpoint
    schemas/step_plan.py    # Pydantic models
    prompts/                # Agent prompt templates
    services/agent.py       # AI model integration
    services/mock.py        # Mock mode for demos
shared/                     # Cross-platform artifacts
  step_plan_schema.json     # Source of truth JSON schema
  example_step_plan.json    # Example plan for testing
docs/
  spec.md                   # Full project spec

Engineer Ownership

Each engineer has a detailed runbook in docs/ — read yours before starting.

Engineer	Area	Directories	Runbook
Eng 1	Overlay + UI	`mac-client/.../Overlay/`, `mac-client/.../UI/`	`docs/eng1-overlay-ui.md`
Eng 2	Capture + Input	`mac-client/.../Capture/`, `mac-client/.../Input/`	`docs/eng2-capture-input.md`
Eng 3	Agent Pipeline	`agent-server/`, `mac-client/.../Networking/`, `shared/`	`docs/eng3-agent.md`
Eng 4	State Machine	`mac-client/.../State/`, `mac-client/.../Models/`, `mac-client/.../App/`	`docs/eng4-state.md`

Key Conventions

Coordinates: Normalized [0,1], top-left origin (0,0)
Schema: All AI outputs validate against shared/step_plan_schema.json
Request IDs: UUID in X-Request-ID header, logged on both client and server
Commit prefixes: [overlay], [capture], [agent], [state]

See .cursor/rules/project.mdc for the full coding rules.

Testing Overlay Output Updates

Use backend mock mode to exercise both POST /plan and POST /next.
The mac state machine now supports applying raw API payloads directly via applyPlanJSON(_:asNextPlan:) for UI-only testing.
Start with shared/example_step_plan.json as the initial payload, then paste a /next-shaped StepPlan JSON payload and call applyPlanJSON(..., asNextPlan: true) to verify the overlay refreshes in place.

Quick UI Tester (CLI)

Run the mac client in synthetic UI test mode and pass parameters:

cd mac-client
swift run TheCookbook --ui-test --goal "Create calendar event" --steps 4 --x 0.22 --y 0.24 --w 0.18 --h 0.05 --next-after 3

Prints the generated StepPlan JSON to terminal (initial, then next if --next-after is provided)
Renders the overlay directly with those synthetic steps
Skips hotkey/click monitor setup in this mode to make testing deterministic

See all test flags:

cd mac-client
swift run TheCookbook --ui-test-help

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

FastAPIIn code
OpenAIIn code
PythonIn code
SwiftIn code

4 of 4 appear in the indexed code.

AI coding agents

CursorConfig · Commits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

466 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Project Info

The Cookbook

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments we're proud of

What we learned

What's next

The Cookbook

Quick Start

Agent Server (Python)

Mac Client (Swift)

Architecture

Repo Structure

Engineer Ownership

Key Conventions

Testing Overlay Output Updates

Quick UI Tester (CLI)

Analysis

Technology

AI coding agents

Codebase size

Structure

Interface

Application logic

Data & schema

Languages

Dependencies