# Project export: Iris This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Iris is an always-on agentic AI that makes working across your Apple devices effortless. - Devpost: https://devpost.com/software/iris-dgucei - GitHub: https://github.com/ThePickleGawd/iris - Video: https://www.youtube.com/embed/HxawNsre4xo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Dylan Lu (29 commits), Ishan Katpally (16 commits), PrajwalVandana (10 commits), Rohan Shankar (6 commits) ## Devpost submission (written by the team) ### Inspiration Putting pen on paper (or Apple Pencil on iPad) is one of the most natural ways of getting early thoughts out and giving shape to ideas. It's often much easier to describe a complicated software system, or a tricky geometry problem, with a simple diagram. However, agentic AI tools today tend to heavily favor textual input. Sure, you could take a quick snap of a drawing and upload it—but what if you wanted to make a tiny edit? What if agentic tools on your Mac could read your iPad screen? Write some code on your Mac, have Iris draw a diagram on your iPad, edit the diagram, and have Iris edit the code—this is just one of the possibilities with Iris, the agentic AI that allows you to work cross-device. ### What it does Iris is an always-on AI that spans your Apple devices. It watches your screens, listens to your voice, and acts—drawing diagrams on your iPad, overlaying information on your Mac, and coordinating across both simultaneously. Sees: watches your Mac and iPad screens, understands what you're working on Hears: streams speech-to-text input from any device (including iPhones!) for natural voice interaction Draws: renders diagrams, widgets, and visual plans directly onto your iPad canvas with Apple Pencil interaction Thinks across devices: your Mac and iPad are one unified workspace, not two isolated screens ### How we built it We built Iris as a local-first, cross-device system centered on a Mac-hosted backend. Backend (Python + Flask): We implemented a unified API for sessions, transcript ingestion, screenshot upload, device commands, and chat sync. Data is stored on the filesystem (backend/data/...) so the whole system is easy for agents to inspect and act on. iPad app (SwiftUI + PencilKit): The iPad app's infinite canvas was built using a custom coordinate system. It also exposes a local canvas API for agent-driven actions. iPhone app (SwiftUI): We built a lightweight mobile companion for session control, push-to-talk voice capture (with on-device transcription), camera/photo uploads, and live chat/status viewing. Mac app (Electron + React): The Mac acts as the orchestration/control center, with session management and agent interaction UI. Agent routing: We support multiple model/provider paths (including Codex/Claude Code-linked sessions) so the same Iris session can map to external coding-agent conversations. Proactive behavior: Iris periodically analyzes incoming screen context and can propose or place useful UI help (widgets/suggestions) instead of only reacting to direct prompts. ### Challenges we ran into Getting proactive visual understanding to work reliably was our hardest problem. Foundation models are good at raw image understanding, but not so much at visual interactivity. We solved this by building a dual visual pipeline: (1) live screenshot capture from iPad + Mac at inference time so each agent turn sees current state, and (2) screenshot ingestion + structured image parsing for proactive monitoring. We also passed normalized canvas/viewport coordinate snapshots so visual reasoning could map back to actionable placement. In practice, this shifted us from “the model can describe the screen” to “the agent can interact visually.” ### Accomplishments we're proud of Implemented proactive behavior: Iris can analyze incoming visual context and suggest useful next actions instead of waiting for explicit prompts. Built a true cross-device AI workflow: Iris treats Mac, iPad, and iPhone as one shared workspace, not separate apps. Shipped an iPad infinite-canvas app with agent control: The agent can place widgets, draw/trace SVGs, and interact with the canvas using coordinate-aware APIs. Integrated external coding agents cleanly: Codex and Claude Code can be linked as first-class session backends, with synchronized chat history across Mac/iPhone/iPad views, taking advantage of these tools' impressive capabilities without reinventing the wheel—and allowing users to stick with what they already know. ### What we learned Proactive AI needs to be carefully controlled—without gating and confidence thresholds, “helpful” suggestions flood the user. Multimodal agents need structure: screenshots alone weren’t enough—coordinate snapshots and consistent structural schemas made visual reasoning usable. ### What's next Move from periodic sync to low-latency shared state so all devices update more smoothly. Better context management, with long-horizon context and cross-provider continuity. Shared workspaces and role-based workflows with stronger auth/permissions scoping, allowing Iris to be used by teams. ## README (from the GitHub repository) # Iris An AI assistant that sees, hears, and draws across your Apple devices. Project Demo (YouTube) > Click the image to watch the full demo on YouTube. ## The Problem Siri can't look at your screen. It can't draw you a diagram. It can't watch what you're doing on your Mac and help you on your iPad at the same time. Current AI assistants are stuck in one device, one modality, one conversation. ## What Iris Does Iris is an always-on AI that spans your Apple devices. It watches your screens, listens to your voice, and acts — drawing diagrams on your iPad, overlaying information on your Mac, and coordinating across both simultaneously. - **Sees** — watches your Mac and iPad screens, understands what you're working on - **Hears** — streams audio from any device for natural voice interaction - **Draws** — renders diagrams, widgets, and visual plans directly onto your iPad canvas with Apple Pencil interaction - **Thinks across devices** — your Mac and iPad are one unified workspace, not two isolated screens 46649 ## Use Cases **Planning** — ask Iris to turn a plan into a diagram you can interact with on your iPad using Apple Pencil. **Coding** — Iris watches your screen, understands your code, and can execute changes while you sketch architecture on the iPad. **Math** — write a problem by hand, get hints and solutions overlaid right on your work. **Visual** — sketch a rough diagram, get back a polished image or video. Iris is **all-to-all** — every device in your Apple ecosystem can see, talk to, and control every other device. The AI doesn't live on one screen. It lives across all of them. ## Prerequisites - **uv** (Python package/runtime manager) - **Node.js + npm** (for the Mac Electron app) - **Claude Code CLI** (`claude`) for live Claude integration - **D2** — diagram renderer for flowcharts, architecture, and sequence diagrams: ``` brew install d2 ``` - **TinyTeX** — lightweight LaTeX distribution needed by Manim for math rendering: ``` curl -sL "https://tinytex.yihui.org/install-bin-unix.sh" | sh tlmgr install standalone preview doublestroke relsize fundus-calligra wasysym physics dvisvgm rsfs wasy setspace babel-english ``` - **Backend Python dependencies** (Flask, Manim, etc.) — managed with [uv](https://docs.astral.sh/uv/): ``` cd backend && uv sync ``` ## Setup 1. Create your local env file: ```bash cp .env.example .env ``` 2. Install backend dependencies: ```bash cd backend uv sync cd .. ``` 3. Install Mac app dependencies: ```bash cd mac npm install cd .. ``` 4. Install Iris helper CLIs globally: ```bash tools/claudei install ``` This installs: - `claudei` → live Claude Code bridge - `iris` → iPad tools CLI (`draw`, `push-widget`, `read-screenshot`, `read-widget`) ## Run - Start backend + Mac app together: ```bash ./run.sh ``` - Or run individually: - backend: `bash backend/run.sh` - mac app: `bash mac/run.sh` ## Claude Code + iPad Workflow 1. Link this Mac to the iPad: ```bash claudei link ``` 2. Start a live Claude session from any project folder: ```bash claudei ``` Optional: - `claudei --cwd /path/to/project` - `claudei --resume ` 3. Send prompts from Iris on iPad. On the first message of a linked Claude session, Iris now auto-injects a CLI bootstrap so Claude knows to use: - `iris tools list` - `iris tools describe ` - `iris draw` - `iris push-widget` - `iris read-screenshot` - `iris read-widget` ## Iris CLI (Global) After install, `iris` works from any directory: ```bash iris tools list iris tools describe draw iris draw --svg-file /tmp/diagram.svg --scale 1.5 iris push-widget --html-file /tmp/widget.html --width 360 --height 260 iris read-screenshot --image-out /tmp/ipad.jpg iris read-widget --name calculator ``` Default iPad base URL is `http://dylans-ipad.local:8935` and can be overridden with `IRIS_IPAD_URL`. ## Browser Automation Service `browser/` contains a standalone browser-use integration that can execute browser actions from text + image context using Claude. Quick start: ``` cd browser ./run.sh ``` See `browser/README.md` for API payload format and Playwright setup. ## More Docs - `backend/README.md` — backend API details and endpoints - `iPad/README.md` — iPad canvas API (`/api/v1/*`) - `mac/README.md` — Electron app setup and packaging - `CLAUDE.md` — Claude-focused workflow and tooling notes ## Detected evidence (automated analysis) Indexed codebase: 134 recognized source files, 1298 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 175) ``` .claude/agents/iris-draw.md .claude/agents/iris-widget.md .claude/settings.json .env.example .gitignore .python-version backend/.gitignore backend/agent.py backend/app.py backend/claude_commander.py backend/grounding/__init__.py backend/grounding/coordinates.py backend/grounding/providers/__init__.py backend/grounding/providers/claude_grounding.py backend/grounding/providers/noop.py backend/grounding/service.py backend/grounding/store.py backend/grounding/types.py backend/orchestration/__init__.py backend/orchestration/arrow_policy.py backend/orchestration/coordinates.py backend/orchestration/intent.py backend/orchestration/widgets.py backend/prompts/browser-use-SKILL.md backend/pyproject.toml backend/README.md backend/run.sh backend/uv.lock CLAUDE.md iPad/iris-app.xcodeproj/project.pbxproj iPad/iris-app.xcodeproj/project.xcworkspace/contents.xcworkspacedata iPad/iris-app.xcodeproj/project.xcworkspace/xcuserdata/ishan.xcuserdatad/UserInterfaceState.xcuserstate iPad/iris-app/Configs/App.local.xcconfig.example iPad/iris-app/Configs/App.xcconfig iPad/iris-app/ContentView.swift iPad/iris-app/Extensions/UIColor+Compare.swift iPad/iris-app/Info.plist iPad/iris-app/IrisApp.swift iPad/iris-app/Models/CanvasObject.swift iPad/iris-app/Models/CanvasState.swift iPad/iris-app/Models/Document.swift iPad/iris-app/Resources/iris-design-system.css iPad/iris-app/Services/AgentClient.swift iPad/iris-app/Services/AgentCursorController.swift iPad/iris-app/Services/AgentHTTPServer.swift iPad/iris-app/Services/AudioCaptureService.swift iPad/iris-app/Services/CanvasObjectManager.swift iPad/iris-app/Services/SpeechTranscriber.swift iPad/iris-app/Views/AgentCursorView.swift iPad/iris-app/Views/CanvasObjectWebView.swift iPad/iris-app/Views/CanvasView.swift iPad/iris-app/Views/HomeView.swift iPad/iris-app/Views/SiriGlowView.swift iPad/iris-app/Views/ToolbarView.swift iPad/LLM_HELPER.md iPad/project.yml iPad/README.md iPhone/iris-app.xcodeproj/project.pbxproj iPhone/iris-app.xcodeproj/project.xcworkspace/contents.xcworkspacedata iPhone/iris-app/ContentView.swift iPhone/iris-app/Extensions/UIColor+Compare.swift iPhone/iris-app/Info.plist iPhone/iris-app/IrisApp.swift iPhone/iris-app/Models/CanvasState.swift iPhone/iris-app/Models/Document.swift iPhone/iris-app/Services/AudioCaptureService.swift iPhone/iris-app/Views/CanvasView.swift iPhone/iris-app/Views/HomeView.swift iPhone/iris-app/Views/SiriGlowView.swift iPhone/iris-app/Views/ToolbarView.swift iPhone/project.yml iPhone/README.md mac/.npmrc mac/docs/agent-protocol.md mac/electron/backendUploader.ts mac/electron/DeviceDiscovery.ts mac/electron/env.ts mac/electron/ipcHandlers.ts mac/electron/LLMHelper.ts mac/electron/main.ts mac/electron/preload.ts mac/electron/ProcessingHelper.ts mac/electron/ScreenMonitor.ts mac/electron/ScreenshotHelper.ts mac/electron/shortcuts.ts mac/electron/tsconfig.json mac/electron/WidgetWindowManager.ts mac/electron/WindowHelper.ts mac/index.html mac/package.json mac/postcss.config.js mac/README.md mac/renderer/package.json mac/renderer/public/index.html mac/renderer/public/manifest.json mac/renderer/public/robots.txt mac/renderer/README.md mac/renderer/src/App.css mac/renderer/src/App.test.tsx mac/renderer/src/App.tsx mac/renderer/src/index.css mac/renderer/src/index.tsx mac/renderer/src/react-app-env.d.ts mac/renderer/src/reportWebVitals.ts mac/renderer/src/setupTests.ts mac/renderer/tsconfig.json mac/run.sh mac/src/_pages/Queue.tsx mac/src/App.tsx mac/src/components/Queue/QueueCommands.tsx mac/src/components/Queue/ScreenshotItem.tsx mac/src/components/Queue/ScreenshotQueue.tsx mac/src/components/ui/card.tsx mac/src/components/ui/dialog.tsx mac/src/components/ui/toast.tsx mac/src/index.css mac/src/lib/agentProtocol.ts mac/src/lib/agentTransport.ts mac/src/lib/utils.ts mac/src/lib/widgetProtocol.ts [55 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: flask@==3.1.0, httpx, manim, openai - mac/package.json: @radix-ui/react-dialog@^1.1.2, @radix-ui/react-toast@^1.2.2, @types/color@^4.2.0, @types/diff@^6.0.0, @types/electron@^1.4.38, @types/node@^22.9.0, @types/react@^18.3.12, @types/react-dom@^18.3.1, @types/react-syntax-highlighter@^15.5.13, @types/screenshot-desktop@^1.12.3, @types/uuid@^9.0.8, @typescript-eslint/eslint-plugin@^8.14.0, @typescript-eslint/parser@^8.14.0, @vitejs/plugin-react@^4.3.3, autoprefixer@^10.4.20, axios@^1.7.7, bonjour-service@^1.3.0, class-variance-authority@^0.7.0, clsx@^2.1.1, concurrently@^9.1.0, cross-env@^7.0.3, diff@^7.0.0, electron@^33.2.0, electron-builder@^25.1.8, electron-is-dev@^3.0.1, form-data@^4.0.1, katex@^0.16.11, lucide-react@^0.460.0, postcss@^8.4.49, react@^18.3.1, react-code-blocks@^0.1.6, react-dom@^18.3.1, react-icons@^5.3.0, react-markdown@^9.0.1, react-query@^3.39.3, react-syntax-highlighter@^15.6.1, rehype-katex@^7.0.1, remark-gfm@^4.0.1, remark-math@^6.0.0, rimraf@^6.0.1, screenshot-desktop@^1.15.0, sharp@^0.33.5, tailwind-merge@^2.5.4, tailwindcss@^3.4.15, tesseract.js@^5.0.5, typescript@^5.6.3, uuid@^11.0.3, vite@^5.4.11, vite-plugin-electron@^0.28.8, vite-plugin-electron-renderer@^0.14.6, wait-on@^8.0.1 - mac/renderer/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @types/jest@^27.5.2, @types/node@^16.18.119, @types/react@^18.3.12, @types/react-dom@^18.3.1, react@^18.3.1, react-dom@^18.3.1, react-scripts@5.0.1, typescript@^4.9.5, web-vitals@^2.1.4 - trajectory/package.json: @tailwindcss/postcss@^4.0.0, @types/node@^22, @types/react@^19, @types/react-dom@^19, clsx@^2.1.1, lucide-react@^0.460.0, next@^15.1.0, react@^19.0.0, react-dom@^19.0.0, react-resizable-panels@^2.1.7, tailwind-merge@^2.6.0, tailwindcss@^4.0.0, typescript@^5 - widgets/package.json: puppeteer@^24.2.0 ### Recent commits (newest first) - Update flowchart in README - fix quote in readme - Adjust image size in README - Enhance README with image and use case - Update README with project demo link - Enhance README with project demo link and image - Fix pen strokes z-order: make PKCanvasView transparent so ink renders above placed images - browser use and can draw diagrams again - Fix diagram placement: bypass broken on-device SVG renderer - Auto-recover broken browser sessions and bump tool rounds to 14 - Add iris CLI and auto-bootstrap Claude iPad tool context - locked in - Fix Claudei message forwarding across socket protocol variants - Restore Browser Usage changes without generated artifacts - Fix session sorting, iPad message sync, and screenshot timing - Trim browser artifacts from git history - Add claudei link subcommand and fix iPad session detection - Add real-time message sync via SSE between iPad and Mac - Add Claude Code session picker to Mac app, disable Codex on both platforms - Merge branch 'main' of https://github.com/ThePickleGawd/iris ## Key source files (fetched from GitHub, selected and truncated for size) ### skills.md ```markdown # Iris Agent Skills Reference ## Tools The Iris agent has 4 tools: `draw`, `push_widget`, `read_screenshot`, and `read_widget`. --- ### draw Draw a diagram onto the PencilKit canvas with animated cursor tracing. The AI cursor follows each stroke as it appears — as if drawing by hand. **This is the primary tool for all diagrams.** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `svg` | string | **yes** | SVG string content (rendered from D2 or hand-crafted) | | `scale` | number | no | Scale factor for SVG coordinates (default 1.0, use 1.5-2.0 for readable size) | | `speed` | number | no | Animation speed multiplier (default 1.0) | | `color` | string | no | Hex color for strokes (default `#1A1F28`) | | `stroke_width` | number | no | PencilKit stroke width (default 3) | | `x` | number | no | X position offset (default 0) | | `y` | number | no | Y position offset (default 0) | | `coordinate_space` | string | no | `"viewport_offset"` (default). Other spaces are accepted, but drawn images are clamped to stay inside the current viewport. | **Workflow:** Write D2 source -> render with `d2` CLI -> send SVG to draw endpoint. **Supported SVG elements:** path (all commands), rect, line, polyline, polygon, circle, ellipse. Text is skipped. Filled polygons (e.g. arrowheads) are rasterized as PencilKit strokes. ```json { "svg": "...", "scale": 2.0, "speed": 1.5, "color": "#1A1F28", "stroke_width": 3 } ``` Returns 202: ```json { "status": "drawing", "stroke_count": 13, "estimated_duration_seconds": 5.1 } ``` --- ### push_widget Push an interactive HTML widget to a target device. Use for **interactive content only** (calculators, timers, documents with LaTeX) — NOT for diagrams. | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `widget_id` | string | **yes** | Unique identifier (descriptive slug, e.g. `"formula-ref-1"`) | | `target` | string | **yes** | `"mac"` (desktop overlay) or `"ipad"` (canvas widget) | | `type` | string | no | `"html"` (default), `"document"`, `"animation"` | | `source` | string | no | Content for `document` (Markdown+LaTeX) or `animation` (Manim) | | `html` | string | no | Raw HTML for `html` type only (self-contained, inline CSS/JS) | | `width` | number | no | Width in points (default 320) | | `height` | number | no | Height in points (default 220) | | `x` | number | no | X position in coordinate_space (default 0) | | `y` | number | no | Y position in coordinate_space (default 0) | | `coordinate_space` | string | no | `"viewport_offset"` (default), `"canvas_absolute"`, `"document_axis"` | | `anchor` | string | no | `"top_left"` (default) or `"center"` | #### Widget Types **`document`** — Markdown + LaTeX rendering - Set `source` to Markdown. `$...$` for inline math, `$$...$$` for display math. **`animation`** — Manim (ManimCE) scene - Set `source` to a Python class extending `Scene` wi [truncated — 2944 more characters] ``` ### CLAUDE.md ```markdown # Iris — Claude Code Guidelines ## Project Structure - `backend/` — Python Flask API, agent logic (`agent.py`), widget rendering - `iPad/iris-app/` — Swift iPad app (PencilKit canvas, WKWebView widgets, HTTP server on port 8935) - `mac/` — Electron + React desktop app - `widgets/lib/` — Pre-built widget templates (calculator, graph, timer) ## Interacting with iPad/Mac When adding content to the iPad or Mac interface, **act as the Iris agent**. Use the tools below. - **Diagrams** are placed as rasterized images via `POST /api/v1/place`. Write D2, render to SVG, place on canvas. - **Widgets** (interactive HTML, documents) use `POST /api/v1/objects` as WebView overlays. - **Use D2** for diagram source. Render to SVG with `d2` CLI, then send to the place endpoint. ### Device Discovery ```bash # iPad (physical device) curl -s http://dylans-ipad.local:8935/api/v1/health # iPad (simulator) curl -s http://localhost:8935/api/v1/health ``` ### diagram → `POST /api/v1/place` **Place a diagram as a rasterized image.** Write D2 source, render to SVG, place on canvas. The cursor navigates to the target position, clicks, and the image appears. ```bash # 1. Write D2 and render to SVG echo 'a -> b -> c' | d2 - /tmp/diagram.svg --theme=200 --pad=20 # 2. Place SVG image on canvas SVG=$(python3 -c "import sys,json; print(json.dumps(open('/tmp/diagram.svg').read()))") curl -s -X POST http://dylans-ipad.local:8935/api/v1/place \ -H "Content-Type: application/json" \ -d "{\"svg\": $SVG, \"scale\": 1.5}" ``` Parameters: - `svg` (required): SVG string content - `scale`: Scale factor (default 1.0, use 1.5-2.0 for readable size) - `x`, `y`: Position offset (default 0,0) - `coordinate_space`: `viewport_offset` (default). Other spaces are accepted, but placed images are clamped inside the current viewport. - `background`: Hex color for background (default transparent) - Returns 201 with `id`, `width`, `height`, and position info ### push_widget → `POST /api/v1/objects` Place an interactive HTML widget on the canvas. Use for **interactive content only** — not diagrams. ```bash curl -s -X POST http://dylans-ipad.local:8935/api/v1/objects \ -H "Content-Type: application/json" \ -d '{ "html": "", "width": 320, "height": 220, "x": 0, "y": 0, "coordinate_space": "viewport_offset", "animate": true }' ``` ### read_screenshot → `GET /api/v1/canvas` Get canvas state and viewport info for spatial placement decisions. ```bash curl -s http://dylans-ipad.local:8935/api/v1/canvas ``` ### read_widget → Widget Library ```bash cat widgets/lib/manifest.json # List available widgets cat widgets/lib/calculator/widget.html # Get widget HTML ``` Available: `calculator` (320x520), `graph` (420x380), `timer` (320x340). ### Other Endpoints ```bash curl -s http://dylans-ipad.local:8935/api/v1/objects # List widgets curl -s -X DELETE http://dylans-ipad.local:8935/api/v1/objects/{id} [truncated — 2980 more characters] ``` ### widgets/package.json ``` { "name": "iris-widgets", "private": true, "type": "module", "scripts": { "capture": "node capture.mjs", "serve": "npx serve ." }, "dependencies": { "puppeteer": "^24.2.0" } } ``` ### backend/pyproject.toml ``` [build-system] requires = ["setuptools>=69", "wheel"] build-backend = "setuptools.build_meta" [project] name = "iris-backend" version = "0.2.0" description = "Iris server — sessions, agent, widgets" requires-python = ">=3.12" dependencies = [ "flask==3.1.0", "openai", "httpx", "manim", ] [tool.setuptools] py-modules = ["app", "agent"] [tool.setuptools.packages.find] include = ["grounding*", "orchestration*"] ``` ### trajectory/package.json ``` { "name": "iris-trajectory-visualizer", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "next": "^15.1.0", "react": "^19.0.0", "react-dom": "^19.0.0", "react-resizable-panels": "^2.1.7", "lucide-react": "^0.460.0", "clsx": "^2.1.1", "tailwind-merge": "^2.6.0" }, "devDependencies": { "@tailwindcss/postcss": "^4.0.0", "tailwindcss": "^4.0.0", "typescript": "^5", "@types/node": "^22", "@types/react": "^19", "@types/react-dom": "^19" } } ``` ### mac/package.json ``` { "name": "mac-interface", "version": "1.0.0", "main": "./dist-electron/main.js", "scripts": { "clean": "rimraf dist dist-electron", "dev": "vite", "build": "npm run clean && tsc && vite build", "preview": "vite preview", "postinstall": "cross-env SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm rebuild sharp", "electron:dev": "tsc -p electron/tsconfig.json && cross-env NODE_ENV=development electron .", "electron:build": "tsc -p electron/tsconfig.json && cross-env NODE_ENV=production electron .", "app:dev": "concurrently -k \"npm run dev -- --port 5180\" \"wait-on http://localhost:5180 && npm run electron:dev\"", "app:build": "npm run build && electron-builder", "watch": "tsc -p electron/tsconfig.json --watch", "start": "npm run app:dev", "dist": "npm run app:build" }, "build": { "appId": "com.local.mac-interface", "productName": "Mac Interface", "files": [ "dist/**/*", "dist-electron/**/*", "package.json", "node_modules/**/*" ], "directories": { "output": "release", "buildResources": "assets" }, "extraResources": [ { "from": "assets/", "to": "assets/" } ], "mac": { "category": "public.app-category.productivity", "target": [ { "target": "dmg", "arch": [ "x64", "arm64" ] } ], "icon": "assets/icons/mac/icon.icns", "hardenedRuntime": true, "entitlements": "assets/entitlements.mac.plist" }, "win": { "target": [ { "target": "nsis", "arch": [ "x64", "ia32" ] }, { "target": "portable", "arch": [ "x64" ] } ], "icon": "assets/icons/win/icon.ico", "requestedExecutionLevel": "asInvoker" }, "linux": { "target": [ { "target": "AppImage", "arch": [ "x64" ] }, { "target": "deb", "arch": [ "x64" ] } ], "icon": "assets/icons/png/", "category": "Office" }, "nsis": { "oneClick": false, "perMachine": false, "allowToChangeInstallationDirectory": true, "deleteAppDataOnUninstall": false } }, "keywords": [], "author": "", "license": "ISC", "description": "", "devDependencies": { "@types/color": "^4.2.0", "@types/diff": "^6.0.0", "@types/electron": "^1.4.38", "@types/node": "^22.9.0", "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "@types/react-syntax-highlighter": "^15.5.13", "@types/screenshot-desktop": "^1.12.3", "@types/uuid": "^9.0.8", "@typescript-eslint/eslint-plugin": "^8.14.0", "@typescript-eslint/parser": "^8.14.0", "@vitejs/plugin-react": "^4.3.3", "autoprefixer": "^10.4.20", "concurrently": "^9.1.0", "cross-env": "^7.0.3", "electron": "^33.2.0", "electron-builder": "^25.1.8", "electron-is-dev": "^3.0.1", "postcss": "^8.4.49", "rimraf": "^6.0.1", "tailwindcss": "^3.4.15", "typescript": "^5.6.3", "vite": "^5.4.11", "vite-plugin-electron": "^0.28.8", "vite-plugin-electron-renderer": "^0.14.6", "wait-on": "^8.0.1" }, "dependencies": { "@radix-ui/react-dialog": "^1.1.2", "@radix-ui/react-toast": "^1.2.2", "axios": "^1.7.7", "bonjour-service": "^1.3.0", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "diff": "^7.0.0", "form-data": "^4.0.1", "katex": "^0.16.11", "lucide-react": "^0.460.0", "react": "^18.3.1", "react-code-blocks": "^0.1.6", "react-dom": "^18.3.1", "react-icons": "^5.3.0", "react-markdown": "^9.0.1", "react-query": "^3.39.3", "react-syntax-highlighter": "^15.6.1", "rehype-katex": "^7.0.1", "remark-gfm": "^4.0.1", "remark-math": "^6.0.0", "screenshot-desktop": "^1.15.0", "sharp": "^0.33.5", "tailwind-merge": "^2.5.4", "tesseract.js": "^5.0.5", "uuid": "^11.0.3" } } ``` ### mac/renderer/package.json ``` { "name": "renderer", "version": "0.1.0", "private": true, "dependencies": { "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "@types/jest": "^27.5.2", "@types/node": "^16.18.119", "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "react": "^18.3.1", "react-dom": "^18.3.1", "react-scripts": "5.0.1", "typescript": "^4.9.5", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### mac/src/main.tsx ```typescript import React from "react" import ReactDOM from "react-dom/client" import App from "./App" import "./index.css" ReactDOM.createRoot(document.getElementById("root")!).render( ) ``` ### mac/src/App.tsx ```typescript import { ToastProvider } from "./components/ui/toast" import Queue from "./_pages/Queue" import { ToastViewport } from "@radix-ui/react-toast" import { useEffect, useRef } from "react" import { QueryClient, QueryClientProvider } from "react-query" const queryClient = new QueryClient({ defaultOptions: { queries: { staleTime: Infinity, cacheTime: Infinity } } }) const App: React.FC = () => { const containerRef = useRef(null) useEffect(() => { const container = containerRef.current if (!container) return const updateHeight = () => { const height = container.scrollHeight const width = container.scrollWidth window.electronAPI?.updateContentDimensions({ width, height }) } const resizeObserver = new ResizeObserver(() => { updateHeight() }) // Initial height update updateHeight() // Observe for changes resizeObserver.observe(container) // Also update height when view changes const mutationObserver = new MutationObserver(() => { updateHeight() }) mutationObserver.observe(container, { childList: true, subtree: true, attributes: true, characterData: true }) return () => { resizeObserver.disconnect() mutationObserver.disconnect() } }, []) useEffect(() => { const cleanupFunctions = [ window.electronAPI.onUnauthorized(() => { queryClient.removeQueries(["screenshots"]) }), window.electronAPI.onResetView(() => { queryClient.invalidateQueries(["screenshots"]) }) ] return () => cleanupFunctions.forEach((cleanup) => cleanup()) }, []) return (
) } export default App ``` ### mac/electron/main.ts ```typescript import "./env" import { app, BrowserWindow, Tray, Menu, nativeImage } from "electron" import { initializeIpcHandlers } from "./ipcHandlers" import { WindowHelper } from "./WindowHelper" import { ScreenshotHelper } from "./ScreenshotHelper" import { ShortcutsHelper } from "./shortcuts" import { ProcessingHelper } from "./ProcessingHelper" import { ScreenMonitor } from "./ScreenMonitor" import { DeviceDiscovery } from "./DeviceDiscovery" export class AppState { private static instance: AppState | null = null private windowHelper: WindowHelper private screenshotHelper: ScreenshotHelper public shortcutsHelper: ShortcutsHelper public processingHelper: ProcessingHelper private screenMonitor: ScreenMonitor public deviceDiscovery: DeviceDiscovery private tray: Tray | null = null // View management private view: "queue" | "solutions" = "queue" private problemInfo: { problem_statement: string input_format: Record output_format: Record constraints: Array> test_cases: Array> } | null = null // Allow null private hasDebugged: boolean = false // Processing events public readonly PROCESSING_EVENTS = { //global states UNAUTHORIZED: "procesing-unauthorized", NO_SCREENSHOTS: "processing-no-screenshots", //states for generating the initial solution INITIAL_START: "initial-start", PROBLEM_EXTRACTED: "problem-extracted", SOLUTION_SUCCESS: "solution-success", INITIAL_SOLUTION_ERROR: "solution-error", //states for processing the debugging DEBUG_START: "debug-start", DEBUG_SUCCESS: "debug-success", DEBUG_ERROR: "debug-error" } as const constructor() { // Initialize WindowHelper with this this.windowHelper = new WindowHelper(this) // Initialize ScreenshotHelper this.screenshotHelper = new ScreenshotHelper(this.view) // Initialize ProcessingHelper this.processingHelper = new ProcessingHelper(this) // Initialize ShortcutsHelper this.shortcutsHelper = new ShortcutsHelper(this) // Initialize ScreenMonitor this.screenMonitor = new ScreenMonitor(this) // Initialize DeviceDiscovery (Bonjour browser for iPad) this.deviceDiscovery = new DeviceDiscovery() } public static getInstance(): AppState { if (!AppState.instance) { AppState.instance = new AppState() } return AppState.instance } // Getters and Setters public getMainWindow(): BrowserWindow | null { return this.windowHelper.getMainWindow() } public getView(): "queue" | "solutions" { return this.view } public setView(view: "queue" | "solutions"): void { this.view = view this.screenshotHelper.setView(view) } public isVisible(): boolean { return this.windowHelper.isVisible() } public getScreenshotHelper(): ScreenshotHelper { return this.screenshotHelper } public getProblemInfo(): any { return this.problemInfo } public setProblemInfo(problemInfo: any): void { this.problemInfo = problemInfo } public getScreenshotQueue(): string[] { return this.screenshotHelper.getScreenshotQueue() } public getExtraScreenshotQueue(): string[] { return this.screenshotHelper.getExtraScreenshotQueue() } // Window management methods public createWindow(): void { this.windowHelper.createWindow() } public hideMainWindow(): void { this.windowHelper.hideMainWindow() } public showMainWindow(): void { this.windowHelper.showMainWindow() } public toggleMainWindow(): void { console.log( "Screenshots: ", this.screenshotHelper.getScreenshotQueue().length, "Extra screenshots: ", this.screenshotHelper.getExtraScreenshotQueue().length ) this.windowHelper.toggleMainWindow() } public setWindowDimensions(width: number, height: number): void { this.windowHelper.setWindowDimensions(width, height) } public clearQueues(): void { this.screenshotHelper.clearQueues() // Clear problem info this.problemInfo = null // Reset view to initial state this.setView("queue") } // Screenshot management methods public async takeScreenshot(): Promise { if (!this.getMainWindow()) throw new Error("No main window available") const screenshotPath = await this.screenshotHelper.takeScreenshot( () => this.hideMainWindow(), () => this.showMainWindow() ) return screenshotPath } public async getImagePreview(filepath: string): Promise { return this.screenshotHelper.getImagePreview(filepath) } public async deleteScreenshot( path: string ): Promise<{ success: boolean; error?: string }> { return this.screenshotHelper.deleteScreenshot(path) } public async addExistingScreenshotToQueue( sourcePath: string, view: "queue" | "solutions" = "queue" ): Promise { return this.screenshotHelper.addExistingScreenshotToQueue(sourcePath, view) } public startScreenMonitor(): void { this.screenMonitor.start() } public stopScreenMonitor(): void { this.screenMonitor.stop() } // New methods to move the window public moveWindowLeft(): void { this.windowHelper.moveWindowLeft() } public moveWindowRight(): void { this.windowHelper.moveWindowRight() } public moveWindowDown(): void { this.windowHelper.moveWindowDown() } public moveWindowUp(): void { this.windowHelper.moveWindowUp() } public moveWindowBy(dx: number, dy: number): void { this.windowHelper.moveWindowBy(dx, dy) } public centerAndShowWindow(): void { this.windowHelper.centerAndShowWindow() } public createTray(): void { // Create a simple tray icon const image = nativeImage.createEmpty() // Try to use a system template image for better integration let trayImage = image try { // Create a minimal icon - just use an empty image and set the title trayImage = nativeI [truncated — 2991 more characters] ``` [115 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]