# Project export: iMessage is all you need This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: You're entire workspace in iMessage - Devpost: https://devpost.com/software/codex-in-kerbal-space-program - GitHub: https://github.com/Alhwyn/openai-imessage - Demo: https://x.com/alhwynn/status/2079411518307999828?s=20 - Video: https://www.youtube.com/embed/IS_R4WkmwYE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — alhwyn (75 commits), Cursor Agent (4 commits) ## Devpost submission (written by the team) ### Overview This is an opinionated project where I think that you don't need another application, you just need one app. Instead, I put my agent application in iMessage, and the product is the harness. I am a minimalist on my MacBook. I only have three apps in my dock, and I've been looking for an "everything app." The best application was probably ChatGPT. I can query the agent and use a web browser without leaving my application, and I wanted to do that on mobile using iMessage. The reason why I picked iMessage is that it is the communicator I use to talk to my friends and also talk to my agent. I feel like every piece of software can be abstracted into a chat with interactive UI/UX design to keep the magic. There are some core features that the agent can do. It can text and connect to your Gmail and Calendar to do all of your admin for you. Features it has include: Computer use The agent can surf through the browser on its own, and you can watch it in real time. Maps In iMessage, you can share locations with your agent, and it can help you find things around town. MCP Integration For integrations with Mail, Calendar, and Notion, I'm using Composio to integrate all of the software I use on my computer, and you just text the agent the integration and it will do it. Our goal for this project is that we don't want you to leave iMessage. We don't even want you to go to an external link to Safari to do something, and we wanted to keep everything embedded in iMessage. We use the Photon.code framework for this. For all of the integrations, we use Composio. It's incredible that you can ask the agent to connect to Gmail or Google Calendar just by prompting it. For computer use, we give the AI a real Linux desktop in Docker. It looks at screenshots, clicks and types like a person, and you can watch it live from iMessage. We start a task on a shared XFCE desktop. You get a live viewer link, and the status is saved so you can ask how it's going. For maps, we used Google Maps and the Photon.codes SDK, which is essentially a web iframe so you can see it in real time. For the inference provider, we use the ChatGPT 5.6 Terra model on GMI Cloud. The reason why is that I happen to have a lot of credits with them for orchestration. recording and screenshots both grab the X11 display. When they ran at the same time, still captures hung and Docker wedged. Fix: pause the recorder, take one PNG still for the model, then resume. My favourite feature is computer use. Watching it in real time and seeing the agent use its own computer is amazing on my iPhone. I also really like the UI/UX aspect of watching it do stuff. For example, here it is playing Wordle: Agent playing Wordle: https://x.com/alhwynn/status/2079025090336067920?s=20 I also think it would be so cool if you could share your location with your agent and have it help you find tasks that you need. For example, ask the agent to find where you can find peacocks in your city, and it knows: Agent Location: https://x.com/alhwynn/status/2079411518307999828?s=20 Personally, I've been using this agent harness for myself, so I have a specific way I want it to write in iMessage to keep things simple. It should feel like you're talking to a friend, with less punctuation, no em dashes, and no capital letters. I also gave it a personality. Sometimes it'll roast you for fun, and your agent can be opinionated too. Computer use is an OS problem first. Recording and screenshots both grab the X11 display running them together hung captures and wedged Docker. So the solution was Pause the recorder and take hte screenshot then resumes the recording after that After this submission, I'm going to work on making computer use faster and fixing the authentication problem. The biggest problem with computer use is authentication and detecting whether it's a bot as well. That's what my computer use cloud agent has been focused on. ## README (from the GitHub repository) # openai-imessage **Repo:** [https://github.com/Alhwyn/openai-imessage](https://github.com/Alhwyn/openai-imessage) iMessage orchestrator that debounces inbound messages, routes work through an Interaction Agent, and delivers replies back to the conversation. Conversation history and curated memory persist in [Convex](https://convex.dev). Connected accounts (Gmail, Calendar, and other approved apps) go through Composio. Browser and desktop GUI work runs on a local Linux computer-use runtime powered by GPT-5.6. ## Codex & GPT-5.6 ### Codex (how the project was built) [OpenAI Codex](https://openai.com/codex) was the primary coding agent for this repo. It was used to: - Scaffold and iterate the Bun orchestrator, inbound debounce, and iMessage delivery path - Wire the Interaction / Execution agents through the Vercel AI SDK + OpenAI Responses API - Build the local Linux computer-use worker (Docker desktop, screenshot loop, action execution, live viewer, recordings) - Add Convex persistence for history/memory and the durable computer-run state machine - Harden prompts, tool schemas, tests, and plain-text iMessage delivery constraints ### GPT-5.6 (how the product runs) All text and computer-control inference goes through GMI Cloud’s OpenAI-compatible `/v1/responses` endpoint (`GMI_CLOUD_API_KEY`). Model IDs in code: | Model | Constant | Used for | | --- | --- | --- | | `openai/gpt-5.6-luna` | `DEFAULT_MODEL` / `MODEL_ID` in `src/orchestrator/utils/constants/gmi.ts` | Interaction Agent (chat + tool routing) and Execution Agent (background tasks) via `generateText` | | `openai/gpt-5.6-terra` | `COMPUTER_MODEL` in `src/orchestrator/computer/constants.ts` | Computer-use loop: screenshots in → mouse/keyboard `computer` tool actions out (`src/orchestrator/computer/openai.ts`) | **Luna path:** inbound iMessage → Interaction Agent chooses tools (`assign_task`, `assign_computer_task`, Composio Gmail/Calendar, maps, images, etc.) → optional Execution Agent for longer work → plain-text reply. Reasoning effort is forced on (GMI’s prefixed model id) with `effort: none` for low-latency tool turns. **Terra path:** `assign_computer_task` starts a durable Convex run, records the XFCE desktop, and loops GPT-5.6 Terra with the Responses `computer` tool until the goal is verified visually or the step budget ends. Results land in the token-gated viewer and as an iMessage card. Image generation uses Seedream (`seedream-5.0-lite`) on GMI, not GPT-5.6. ## Stack | Layer | Role | | --- | --- | | Bun + Spectrum iMessage | Runtime and messaging transport | | Vercel AI SDK | Agent tool loops | | GMI Cloud | Text, images, and computer-use model calls (`GMI_CLOUD_API_KEY`) | | Convex | Durable messages and memory (`CONVEX_URL`) | | Composio | Per-person OAuth for connected apps | | Docker + KasmVNC/XFCE | Local Linux desktop for computer-use tasks | ## Setup ### Prerequisites - [Bun](https://bun.sh) (runtime + package manager) - [Docker](https://www.docker.com/) (only if you want computer-use) - A [Spectrum](https://spectrum.im) project for iMessage - A [GMI Cloud](https://gmicloud.ai) API key (GPT-5.6 Luna/Terra + images) - A [Convex](https://convex.dev) account Optional: [Composio](https://composio.dev) CLI, [Exa](https://exa.ai), Google Maps, [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/) for tunnels. ### 1. Install ```bash bun install cp .env.example .env ``` Fill `.env` from the tables below. Never commit `.env`. Authoritative names live only in `.env.example`. ### 2. Environment variables #### Required to boot the orchestrator | Variable | What it is | | --- | --- | | `SPECTRUM_PROJECT_ID` | Spectrum project id | | `SPECTRUM_PROJECT_SECRET` | Spectrum project secret | | `GMI_CLOUD_API_KEY` | GMI inference key (Luna chat/workers, Terra computer-use, Seedream images) | | `CONVEX_URL` | Convex deployment URL (written by `bun run convex:dev`, also copy into `.env`) | | `ORCHESTRATOR_BRIDGE_SECRET` | Shared secret between Bun and Convex; must match on both sides | #### Spectrum / phone | Variable | Required? | What it is | | --- | --- | --- | | `PHONE_NUMBER` | Optional | Local phone / identity note for your Spectrum setup | | `SPECTRUM_SIGNING_WEBHOOK` | Optional | Webhook signing secret when Spectrum posts to your tunnel | #### Connected apps (Composio) | Variable | Required? | What it is | | --- | --- | --- | | `COMPOSIO_API_KEY` | For Gmail/Calendar tools | From `composio login` / dashboard | | `COMPOSIO_USER_ID_SALT` | With Composio | Random salt; Spectrum sender ids are hashed with this before becoming Composio user ids | ```bash composio login composio init # then put COMPOSIO_API_KEY + COMPOSIO_USER_ID_SALT in .env ``` #### Computer use (Docker desktop) | Variable | Required? | What it is | | --- | --- | --- | | `COMPUTER_DESKTOP_PASSWORD` | To start the desktop | Passed to Kasm as `VNC_PW` (`bun run computer:up` fails without it) | | `COMPUTER_LIVE_VIEW_URL` | For iMessage live cards | Public HTTPS URL for the desktop stream, e.g. `https://desktop.example.com` (viewer host is derived as `viewer.*`) | | `OPENAI_API_KEY` | Optional / reserved | Listed in `.env.example`; runtime chat + computer-use currently go through `GMI_CLOUD_API_KEY` | #### Research + maps | Variable | Required? | What it is | | --- | --- | --- | | `EXA_API_KEY` | For web search tools | [Exa](https://exa.ai) API key | | `GOOGLE_MAPS_API_KEY` | For maps / geocode | Google Maps Platform key | | `MAPS_PUBLIC_BASE_URL` | For shareable map cards | Public base URL for the maps viewer | | `MAPS_VIEWER_TOKEN_SECRET` | With maps | Secret used to sign token-gated map viewer links | ### 3. Convex ```bash # Terminal A — links a deployment, syncs schema/functions, writes CONVEX_URL bun run convex:dev ``` Put the same bridge secret in local `.env` and on the Convex deployment: ```bash # .env CONVEX_URL=https://….convex.cloud ORCHESTRATOR_BRIDGE_SECRET=some-long-random-string # Convex deployment (same value) bunx convex env set ORCHESTRATOR_BRIDGE_SECRET some-long-random-string ``` Keep `bun run convex:dev` running while you develop so functions stay synced. ### 4. Minimal `.env` checklist ```bash # required SPECTRUM_PROJECT_ID=... SPECTRUM_PROJECT_SECRET=... GMI_CLOUD_API_KEY=... CONVEX_URL=https://….convex.cloud ORCHESTRATOR_BRIDGE_SECRET=... # optional features SPECTRUM_SIGNING_WEBHOOK= COMPOSIO_API_KEY= COMPOSIO_USER_ID_SALT= COMPUTER_DESKTOP_PASSWORD= COMPUTER_LIVE_VIEW_URL= EXA_API_KEY= GOOGLE_MAPS_API_KEY= MAPS_PUBLIC_BASE_URL= MAPS_VIEWER_TOKEN_SECRET= ``` ## Run ```bash # Terminal 1 — Convex bun run convex:dev # Terminal 2 — orchestrator (Bun loads .env automatically) bun run start ``` Text an inbound message to the Spectrum iMessage line. Flow: debounce → Interaction Agent (GPT-5.6 Luna) → tools/workers → reply. ### Computer use Computer-use drives a full XFCE desktop over X11 (screenshots + `xdotool`), not DOM automation. Docker is the local boundary. 1. Set `GMI_CLOUD_API_KEY` and `COMPUTER_DESKTOP_PASSWORD` in `.env`. 2. Start the desktop: ```bash bun run computer:up ``` 3. Open `https://127.0.0.1:6901` and accept the local certificate. Loopback basic auth is disabled; the password is still required by the Kasm image at startup. Display is locked to **1280×800** for stable model coordinates. When `assign_computer_task` runs, the orchestrator: 1. Creates a durable run in Convex 2. Records the session with FFmpeg (still captures pause the recorder so X11 is not dual-grabbed) 3. Sends screenshots to GPT-5.6 Terra and applies returned mouse/keyboard actions 4. Writes `runtime/computer/artifacts//demo.mp4` 5. Serves a token-gated viewer at `http://127.0.0.1:6902` (live timeline + replay) ```bash bun run computer:logs bun run computer:down ``` If the desktop X session dies: ```bash bun run computer:down && bun run computer:up ``` ### Dev tunnel (optional) Expose localhost over HTTPS for Spectrum webhooks an [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 194 recognized source files, 669 KB. - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel AI SDK (technology) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository - AI coding agent: Codex — evidence: config files committed to the repository - AI coding agent: Cursor — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 222) ``` .agents/skills/convex-create-component/agents/openai.yaml .agents/skills/convex-create-component/references/advanced-patterns.md .agents/skills/convex-create-component/references/hybrid-components.md .agents/skills/convex-create-component/references/local-components.md .agents/skills/convex-create-component/references/packaged-components.md .agents/skills/convex-create-component/SKILL.md .agents/skills/convex-migration-helper/agents/openai.yaml .agents/skills/convex-migration-helper/references/migration-patterns.md .agents/skills/convex-migration-helper/references/migrations-component.md .agents/skills/convex-migration-helper/SKILL.md .agents/skills/convex-performance-audit/agents/openai.yaml .agents/skills/convex-performance-audit/references/function-budget.md .agents/skills/convex-performance-audit/references/hot-path-rules.md .agents/skills/convex-performance-audit/references/occ-conflicts.md .agents/skills/convex-performance-audit/references/subscription-cost.md .agents/skills/convex-performance-audit/SKILL.md .agents/skills/convex-quickstart/agents/openai.yaml .agents/skills/convex-quickstart/SKILL.md .agents/skills/convex-setup-auth/agents/openai.yaml .agents/skills/convex-setup-auth/references/auth0.md .agents/skills/convex-setup-auth/references/clerk.md .agents/skills/convex-setup-auth/references/convex-auth.md .agents/skills/convex-setup-auth/references/workos-authkit.md .agents/skills/convex-setup-auth/SKILL.md .agents/skills/convex/SKILL.md .claude/skills/convex-create-component/agents/openai.yaml .claude/skills/convex-create-component/references/advanced-patterns.md .claude/skills/convex-create-component/references/hybrid-components.md .claude/skills/convex-create-component/references/local-components.md .claude/skills/convex-create-component/references/packaged-components.md .claude/skills/convex-create-component/SKILL.md .claude/skills/convex-migration-helper/agents/openai.yaml .claude/skills/convex-migration-helper/references/migration-patterns.md .claude/skills/convex-migration-helper/references/migrations-component.md .claude/skills/convex-migration-helper/SKILL.md .claude/skills/convex-performance-audit/agents/openai.yaml .claude/skills/convex-performance-audit/references/function-budget.md .claude/skills/convex-performance-audit/references/hot-path-rules.md .claude/skills/convex-performance-audit/references/occ-conflicts.md .claude/skills/convex-performance-audit/references/subscription-cost.md .claude/skills/convex-performance-audit/SKILL.md .claude/skills/convex-quickstart/agents/openai.yaml .claude/skills/convex-quickstart/SKILL.md .claude/skills/convex-setup-auth/agents/openai.yaml .claude/skills/convex-setup-auth/references/auth0.md .claude/skills/convex-setup-auth/references/clerk.md .claude/skills/convex-setup-auth/references/convex-auth.md .claude/skills/convex-setup-auth/references/workos-authkit.md .claude/skills/convex-setup-auth/SKILL.md .claude/skills/convex/SKILL.md .cursor/hooks.json .cursor/hooks/ci-check.sh .cursor/hooks/protect-env.sh .cursor/rules/arrow-functions.mdc .cursor/rules/barrel-index-files.mdc .cursor/rules/env-variable-source-of-truth.mdc .cursor/rules/imessage-plain-text.mdc .cursor/rules/no-hardcoded-language-routing.mdc .cursor/rules/one-line-if-statements.mdc .cursor/rules/protect-env-files.mdc .cursor/rules/react-section-separation.mdc .cursor/rules/test-folder-organization.mdc .cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc .env.example .firecrawl/photon.codes-docs-advanced-kits-imessage-error-handling.md .firecrawl/photon.codes-docs-advanced-kits-imessage-getting-started.md .firecrawl/photon.codes-docs-advanced-kits-imessage-locations.md .firecrawl/photon.codes-docs-spectrum-ts-platform-narrowing.md .github/workflows/ci.yml .gitignore AGENTS.md bun.lock CLAUDE.md cloudflared/computer.yml.example cloudflared/config.yml.example convex.json convex/_generated/ai/ai-files.state.json convex/_generated/ai/guidelines.md convex/_generated/api.d.ts convex/_generated/api.js convex/_generated/dataModel.d.ts convex/_generated/server.d.ts convex/_generated/server.js convex/computerRuns.ts convex/lib/bridge.ts convex/lib/computer.ts convex/lib/memoryEdits.ts convex/lib/messageRetention.ts convex/memories.ts convex/messages.ts convex/schema.ts convex/test/memoryEdits.test.ts convex/test/messageRetention.test.ts convex/tsconfig.json eslint.config.mjs index.ts package.json README.md runtime/computer/artifacts/.gitkeep runtime/computer/compose.yaml runtime/computer/Dockerfile runtime/computer/scripts/kasm_post_run_user.sh runtime/computer/scripts/record-start runtime/computer/scripts/record-stop runtime/computer/scripts/screenshot scripts/run-tunnel.sh scripts/setup-tunnel.sh skills-lock.json src/orchestrator/agents/index.ts src/orchestrator/agents/interaction.ts src/orchestrator/agents/interactionTools.ts src/orchestrator/agents/test/outbound.test.ts src/orchestrator/agents/test/turnEffects.test.ts src/orchestrator/agents/test/visibleText.test.ts src/orchestrator/agents/turnEffects.ts src/orchestrator/agents/types.ts src/orchestrator/agents/visibleText.ts src/orchestrator/bounce/failurePolicy.ts src/orchestrator/bounce/inbound.ts src/orchestrator/bounce/index.ts [102 more files omitted for size] ``` ### Dependencies - package.json: @ai-sdk/openai@^4.0.14, @composio/core@^0.14.0, @composio/vercel@^0.11.1, @convex-dev/eslint-plugin@^2.0.0, @photon-ai/advanced-imessage@^1.0.0, @spectrum-ts/core@^11.0.0, @spectrum-ts/imessage@^11.0.0, @types/bun@latest, @types/node@^26.1.1, ai@^7.0.28, convex@^1.42.2, eslint@^9, eslint-plugin-import@^2.32.0, exa-js@^2.16.0, sharp@^0.35.3, typescript@^5, typescript-eslint@^8.64.0, zod@^4.4.3 ### Recent commits (newest first) - fix: stay silent when 15-minute meeting reminder finds none (#16) - fix: stop routing general questions through calendar search (#14) - refactor: update API integration to use GMI_CLOUD_API_KEY, enhance README with detailed setup instructions, and clarify component functionalities - chore: update README for clarity and structure, enhance setup instructions, and improve descriptions of components and functionality - Merge pull request #12 from Alhwyn/chore/skillz-inference - feat: enhance screenshot functionality and improve display error handling in desktop management - feat: update interaction prompts for improved tone and clarify Gmail context usage in connected apps - feat: refactor API integration to use GMI instead of OpenAI, update related error handling, and enhance interaction prompts - Merge pull request #11 from Alhwyn/feat/maps - feat: improve error handling in Find My share request and enhance session store documentation - feat: refactor maps session management and enhance location client integration - feat: update navigation functionality and enhance mini-app card details - feat: update maps viewer styles and improve mini-app card functionality - feat: enhance outbound summarization and improve maps session management - feat: enhance location sharing and session management with Find My integration - feat: add Google Maps integration and enhance location tools - feat: add create_directions_link tool for enhanced navigation support - feat: enhance location discovery tools and update dependencies - Merge pull request #10 from Alhwyn/feat/background - fix: update file permissions for CI hooks and improve error handling in interaction tools ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENTS.md ```markdown This project uses [Convex](https://convex.dev) as its backend. When working on Convex code, **always read `convex/_generated/ai/guidelines.md` first** for important guidelines on how to correctly use Convex APIs and patterns. The file contains rules that override what you may have learned about Convex from training data. Convex agent skills for common tasks can be installed by running `npx convex ai-files install`. ``` ### CLAUDE.md ```markdown --- description: Use Bun instead of Node.js, npm, pnpm, or vite. globs: "*.ts, *.tsx, *.html, *.css, *.js, *.jsx, package.json" alwaysApply: false --- Default to using Bun instead of Node.js. - Use `bun ` instead of `node ` or `ts-node ` - Use `bun test` instead of `jest` or `vitest` - Use `bun build ` instead of `webpack` or `esbuild` - Use `bun install` instead of `npm install` or `yarn install` or `pnpm install` - Use `bun run ``` With the following `frontend.tsx`: ```tsx#frontend.tsx import React from "react"; import { createRoot } from "react-dom/client"; // import .css files directly and it works import './index.css'; const root = createRoot(document.body); export default function Frontend() { return

Hello, world!

; } root.render(); ``` Then, run index.ts ```sh bun --hot ./index.ts ``` For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`. This project uses [Convex](https://convex.dev) as its backend. When working on Convex code, **always read `convex/_generated/ai/guidelines.md` first** for important guidelines on how to correctly use Convex APIs and patterns. The file contains rules that override what you may have learned about [truncated — 148 more characters] ``` ### package.json ``` { "name": "openai-imessage", "module": "index.ts", "type": "module", "private": true, "scripts": { "build": "bun build index.ts --outdir dist --target bun", "check": "bun run lint && bun run typecheck && bun run test && bun run build", "computer:down": "docker compose --env-file .env -f runtime/computer/compose.yaml down", "computer:logs": "docker compose --env-file .env -f runtime/computer/compose.yaml logs -f desktop", "computer:up": "mkdir -p runtime/computer/artifacts runtime/computer/workspace && docker compose --env-file .env -f runtime/computer/compose.yaml up -d --build", "start": "bun --watch index.ts", "convex:dev": "convex dev", "lint": "eslint .", "lint:fix": "eslint . --fix", "test": "bun test", "tunnel": "bash scripts/run-tunnel.sh", "tunnel:setup": "bash scripts/setup-tunnel.sh", "tunnel:quick": "cloudflared tunnel --url http://127.0.0.1:4001", "typecheck": "tsc --noEmit" }, "devDependencies": { "@convex-dev/eslint-plugin": "^2.0.0", "@types/bun": "latest", "@types/node": "^26.1.1", "eslint": "^9", "eslint-plugin-import": "^2.32.0", "typescript-eslint": "^8.64.0" }, "peerDependencies": { "typescript": "^5" }, "dependencies": { "@ai-sdk/openai": "^4.0.14", "@composio/core": "^0.14.0", "@composio/vercel": "^0.11.1", "@photon-ai/advanced-imessage": "^1.0.0", "@spectrum-ts/core": "^11.0.0", "@spectrum-ts/imessage": "^11.0.0", "ai": "^7.0.28", "convex": "^1.42.2", "exa-js": "^2.16.0", "sharp": "^0.35.3", "zod": "^4.4.3" } } ``` ### runtime/computer/Dockerfile ``` FROM kasmweb/desktop:1.18.0@sha256:9a94316372b8858d0238134ba507812e867fcb903d0db599c8cfd9ff8bcd0bfb USER root RUN apt-get update \ && apt-get install -y --no-install-recommends \ ffmpeg \ imagemagick \ jq \ maim \ pulseaudio-utils \ xdotool \ x11-utils \ x11-xserver-utils \ && apt-get purge -y firefox \ && sed -i \ -e '\|ln -sf $HOME/Uploads $HOME/Desktop/Uploads|d' \ -e '\|ln -sf $HOME/Downloads $HOME/Desktop/Downloads|d' \ /dockerstartup/kasm_default_profile.sh \ && rm -f \ /home/kasm-default-profile/Desktop/firefox.desktop \ /home/kasm-user/Desktop/firefox.desktop \ /home/kasm-default-profile/Desktop/Downloads \ /home/kasm-default-profile/Desktop/Uploads \ /home/kasm-user/Desktop/Downloads \ /home/kasm-user/Desktop/Uploads \ /usr/share/applications/firefox.desktop \ && convert -size 1x1 xc:none /usr/share/extra/icons/icon_default.png \ && for panel_config in \ /home/kasm-default-profile/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml \ /home/kasm-user/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml; do \ if [ -f "$panel_config" ]; then \ sed -i '/name="button-icon"/a\ \\n ' "$panel_config"; \ fi; \ done \ && rm -rf /var/lib/apt/lists/* COPY --chmod=755 scripts/ /opt/computer-agent/bin/ COPY --chown=1000:0 assets/harbor-wallpaper.png /usr/share/backgrounds/computer-agent-harbor.png RUN cp /usr/share/backgrounds/computer-agent-harbor.png /usr/share/backgrounds/bg_default.png # Force fixed desktop geometry after session start (Kasm defaults to 1024x768). COPY --chmod=755 scripts/kasm_post_run_user.sh /dockerstartup/kasm_post_run_user.sh USER 1000 ``` ### index.ts ```typescript import { Spectrum, type Message, type Space, type SpectrumInstance } from "@spectrum-ts/core"; import { imessage } from "@spectrum-ts/imessage"; import { COMPUTER_WATCHDOG_INTERVAL_MS, COMPUTER_WORKER_TIMEOUT_MS, } from "./src/orchestrator/computer/constants"; import { startComputerViewer } from "./src/orchestrator/computer/viewer"; import { assertConvexEnv, assertGmiApiKey, createRecentIdTracker, extractInboundImages, extractInboundText, flushPendingOrchestratorTurns, reconcileStaleComputerRuns, scheduleOrchestratorTurn, SEEN_MESSAGE_MAX, SEEN_MESSAGE_TTL_MS, } from "./src/orchestrator/index"; import { registerLocationClients, startMapsViewer } from "./src/orchestrator/maps"; import { resolveIMessageLocationClients } from "./src/orchestrator/resolveIMessageLocationClients"; /** Drop provider redeliveries for a few minutes inside one process. */ const seenInboundMessages = createRecentIdTracker({ ttlMs: SEEN_MESSAGE_TTL_MS, maxSize: SEEN_MESSAGE_MAX, }); const getSpectrumEnv = () => { const projectId = process.env.SPECTRUM_PROJECT_ID?.trim(); const projectSecret = process.env.SPECTRUM_PROJECT_SECRET?.trim(); const webhookSecret = process.env.SPECTRUM_SIGNING_WEBHOOK?.trim(); const missing: string[] = []; if (!projectId) missing.push("SPECTRUM_PROJECT_ID"); if (!projectSecret) missing.push("SPECTRUM_PROJECT_SECRET"); return { projectId, projectSecret, webhookSecret, missing }; }; const senderIdFrom = (message: Message): string | null => { const sender = message.sender; if (sender && typeof sender === "object" && "id" in sender && typeof sender.id === "string") return sender.id; return null; }; const handleInbound = async (space: Space, message: Message): Promise => { if (message.direction === "outbound") return; const sender = message.sender; if (sender && typeof sender === "object" && "kind" in sender && sender.kind === "agent") return; if (!seenInboundMessages.claim(message.id)) { console.log("[app] Skipping duplicate inbound message", { messageId: message.id, spaceId: space.id, }); return; } const inboundText = extractInboundText(message); const inboundImages = await extractInboundImages(message); if (!inboundText && inboundImages.length === 0) { console.log("[app] Ignored unsupported message content", { messageId: message.id, spaceId: space.id, }); return; } const senderId = senderIdFrom(message); console.log("[app] Inbound:", inboundText.slice(0, 80), { messageId: message.id, spaceId: space.id, senderId, images: inboundImages.map((image) => image.filename), }); scheduleOrchestratorTurn({ space, message, text: inboundText, images: inboundImages, senderId, }); }; const main = async () => { assertGmiApiKey(); assertConvexEnv(); const reconciled = await reconcileStaleComputerRuns({ staleBefore: Date.now(), error: "Computer worker stopped when the orchestrator restarted", }); if (reconciled > 0) console.warn(`[computer-agent] Reconciled ${reconciled} orphaned run(s)`); const watchdog = setInterval(() => { void reconcileStaleComputerRuns({ staleBefore: Date.now() - COMPUTER_WORKER_TIMEOUT_MS - COMPUTER_WATCHDOG_INTERVAL_MS, error: "Computer worker stopped reporting progress", }).catch((error: unknown) => { console.error("[computer-agent] Watchdog reconciliation failed", error); }); }, COMPUTER_WATCHDOG_INTERVAL_MS); watchdog.unref(); const computerViewer = startComputerViewer(); const mapsViewer = startMapsViewer(); const { projectId, projectSecret, webhookSecret, missing } = getSpectrumEnv(); if (missing.length > 0) throw new Error(`Missing env: ${missing.join(", ")}`); const app: SpectrumInstance = await Spectrum({ projectId: projectId!, projectSecret: projectSecret!, platforms: [imessage.config()], webhookSecret, }); registerLocationClients(resolveIMessageLocationClients(app)); let stopping = false; const inboundJobs = new Set>(); const stopApp = async (reason: string) => { if (stopping) return; stopping = true; clearInterval(watchdog); console.log(`[app] Stopping Spectrum (${reason})`); try { await Promise.allSettled(inboundJobs); await flushPendingOrchestratorTurns(); await app.stop(); await computerViewer.stop(true); await mapsViewer.stop(true); } catch (error) { console.error("[app] Spectrum stop failed", error); } }; const shutdownAndExit = (reason: string, exitCode = 0) => { void stopApp(reason).finally(() => { process.exit(exitCode); }); }; const onSignal = (signal: NodeJS.Signals) => { shutdownAndExit(signal, 0); }; process.once("SIGINT", onSignal); process.once("SIGTERM", onSignal); process.once("beforeExit", (code) => { shutdownAndExit("beforeExit", code); }); console.log( `[app] Orchestrator listening (Spectrum + GMI). Debounced inbound → assign_task → notify → reply/react`, ); try { for await (const [space, message] of app.messages) { const job = handleInbound(space, message).catch((error: unknown) => { console.error("[app] Failed to handle inbound message", error); }); inboundJobs.add(job); void job.finally(() => { inboundJobs.delete(job); }); } } finally { await stopApp("messages ended"); } }; main().catch((error) => { console.error("[app] Fatal:", error); process.exit(1); }); ``` ### src/orchestrator/index.ts ```typescript import { runInteractionAgent } from "./agents/index"; import { buildDebouncedTurn, flushPendingOrchestratorTurns, scheduleOrchestratorTurn, } from "./bounce/index"; import { assertConvexEnv, reconcileStaleComputerRuns } from "./db/index"; import { assignImageTask, assignTask } from "./handoff/index"; import { assertGmiApiKey, createRecentIdTracker, extractInboundImages, extractInboundText, MODEL, SEEN_MESSAGE_MAX, SEEN_MESSAGE_TTL_MS, } from "./utils/index"; import { runExecutionAgent } from "./workers/execution"; import type { InboundImage, InteractionEvent, InteractionResult, OutboundItem, TapbackKey, } from "./agents/index"; import type { OrchestratorTurn } from "./bounce/index"; import type { RecentIdTracker, RecentIdTrackerOptions } from "./utils/index"; export { assertConvexEnv, assertGmiApiKey, assignImageTask, assignTask, buildDebouncedTurn, createRecentIdTracker, extractInboundImages, extractInboundText, flushPendingOrchestratorTurns, MODEL, reconcileStaleComputerRuns, runExecutionAgent, runInteractionAgent, scheduleOrchestratorTurn, SEEN_MESSAGE_MAX, SEEN_MESSAGE_TTL_MS, }; export type { InboundImage, InteractionEvent, InteractionResult, OrchestratorTurn, OutboundItem, RecentIdTracker, RecentIdTrackerOptions, TapbackKey, }; ``` ### convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define an HTTP action. * * The wrapped function will be used to respond to HTTP requests received * by a Convex deployment if the requests matches the path and method where * this action is routed. Be sure to route your httpAction in `convex/http.js`. * * @param func - The function. It receives an {@link ActionCtx} as its first argument * and a Fetch API `Request` object as its second. * @returns The wrapped function. Import this function from `convex/http.js` and route it to hook it up. */ export const httpAction = httpActionGeneric; ``` ### src/orchestrator/prompts/index.ts ```typescript import { executionSystemPrompt, interactionSystemPrompt, } from "./loader"; export { executionSystemPrompt, interactionSystemPrompt }; ``` ### src/orchestrator/exa/index.ts ```typescript import { searchNearbyPlaces } from "./networkFns"; import type { PlaceEvidence, SearchNearbyPlacesInput, SearchNearbyPlacesResult, } from "./types"; export { searchNearbyPlaces }; export type { PlaceEvidence, SearchNearbyPlacesInput, SearchNearbyPlacesResult, }; ``` ### src/orchestrator/integrations/index.ts ```typescript import { composioUserIdFor, getComposioTools, isComposioAuthUrl, } from "./composio"; import { cleanupImageAlbum, clampImageCount, generateGmiImages, } from "./gmiImages"; export { cleanupImageAlbum, clampImageCount, composioUserIdFor, generateGmiImages, getComposioTools, isComposioAuthUrl, }; ``` [185 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]