Project Info

iMessage is all you need

This is an opinionated project where I think that you don't need another application, you just need one app. Instead, I put my agent application in iMessage, and the product is the harness. I am a minimalist on my MacBook. I only have three apps in my dock, and I've been looking for an "everything app." The best application was probably ChatGPT. I can query the agent and use a web browser without leaving my application, and I wanted to do that on mobile using iMessage. The reason why I picked iMessage is that it is the communicator I use to talk to my friends and also talk to my agent. I feel like every piece of software can be abstracted into a chat with interactive UI/UX design to keep the magic. There are some core features that the agent can do. It can text and connect to your Gmail and Calendar to do all of your admin for you. Features it has include: Computer use The agent can surf through the browser on its own, and you can watch it in real time. Maps In iMessage, you can share locations with your agent, and it can help you find things around town. MCP Integration For integrations with Mail, Calendar, and Notion, I'm using Composio to integrate all of the software I use on my computer, and you just text the agent the integration and it will do it. Our goal for this project is that we don't want you to leave iMessage. We don't even want you to go to an external link to Safari to do something, and we wanted to keep everything embedded in iMessage. We use the Photon.code framework for this. For all of the integrations, we use Composio. It's incredible that you can ask the agent to connect to Gmail or Google Calendar just by prompting it. For computer use, we give the AI a real Linux desktop in Docker. It looks at screenshots, clicks and types like a person, and you can watch it live from iMessage. We start a task on a shared XFCE desktop. You get a live viewer link, and the status is saved so you can ask how it's going. For maps, we used Google Maps and the Photon.codes SDK, which is essentially a web iframe so you can see it in real time. For the inference provider, we use the ChatGPT 5.6 Terra model on GMI Cloud. The reason why is that I happen to have a lot of credits with them for orchestration. recording and screenshots both grab the X11 display. When they ran at the same time, still captures hung and Docker wedged. Fix: pause the recorder, take one PNG still for the model, then resume. My favourite feature is computer use. Watching it in real time and seeing the agent use its own computer is amazing on my iPhone. I also really like the UI/UX aspect of watching it do stuff. For example, here it is playing Wordle: Agent playing Wordle: https://x.com/alhwynn/status/2079025090336067920?s=20 I also think it would be so cool if you could share your location with your agent and have it help you find tasks that you need. For example, ask the agent to find where you can find peacocks in your city, and it knows: Agent Location: https://x.com/alhwynn/status/2079411518307999828?s=20 Personally, I've been using this agent harness for myself, so I have a specific way I want it to write in iMessage to keep things simple. It should feel like you're talking to a friend, with less punctuation, no em dashes, and no capital letters. I also gave it a personality. Sometimes it'll roast you for fun, and your agent can be opinionated too. Computer use is an OS problem first. Recording and screenshots both grab the X11 display running them together hung captures and wedged Docker. So the solution was Pause the recorder and take hte screenshot then resumes the recording after that After this submission, I'm going to work on making computer use faster and fixing the authentication problem. The biggest problem with computer use is authentication and detecting whether it's a bot as well. That's what my computer use cloud agent has been focused on.

openai-imessage

Repo: https://github.com/Alhwyn/openai-imessage

iMessage orchestrator that debounces inbound messages, routes work through an Interaction Agent, and delivers replies back to the conversation.

Conversation history and curated memory persist in Convex. Connected accounts (Gmail, Calendar, and other approved apps) go through Composio. Browser and desktop GUI work runs on a local Linux computer-use runtime powered by GPT-5.6.

Codex & GPT-5.6

Codex (how the project was built)

OpenAI Codex was the primary coding agent for this repo. It was used to:

Scaffold and iterate the Bun orchestrator, inbound debounce, and iMessage delivery path
Wire the Interaction / Execution agents through the Vercel AI SDK + OpenAI Responses API
Build the local Linux computer-use worker (Docker desktop, screenshot loop, action execution, live viewer, recordings)
Add Convex persistence for history/memory and the durable computer-run state machine
Harden prompts, tool schemas, tests, and plain-text iMessage delivery constraints

GPT-5.6 (how the product runs)

All text and computer-control inference goes through GMI Cloud’s OpenAI-compatible /v1/responses endpoint (GMI_CLOUD_API_KEY). Model IDs in code:

Model	Constant	Used for
`openai/gpt-5.6-luna`	`DEFAULT_MODEL` / `MODEL_ID` in `src/orchestrator/utils/constants/gmi.ts`	Interaction Agent (chat + tool routing) and Execution Agent (background tasks) via `generateText`
`openai/gpt-5.6-terra`	`COMPUTER_MODEL` in `src/orchestrator/computer/constants.ts`	Computer-use loop: screenshots in → mouse/keyboard `computer` tool actions out (`src/orchestrator/computer/openai.ts`)

Luna path: inbound iMessage → Interaction Agent chooses tools (assign_task, assign_computer_task, Composio Gmail/Calendar, maps, images, etc.) → optional Execution Agent for longer work → plain-text reply. Reasoning effort is forced on (GMI’s prefixed model id) with effort: none for low-latency tool turns.

Terra path: assign_computer_task starts a durable Convex run, records the XFCE desktop, and loops GPT-5.6 Terra with the Responses computer tool until the goal is verified visually or the step budget ends. Results land in the token-gated viewer and as an iMessage card.

Image generation uses Seedream (seedream-5.0-lite) on GMI, not GPT-5.6.

Stack

Layer	Role
Bun + Spectrum iMessage	Runtime and messaging transport
Vercel AI SDK	Agent tool loops
GMI Cloud	Text, images, and computer-use model calls (`GMI_CLOUD_API_KEY`)
Convex	Durable messages and memory (`CONVEX_URL`)
Composio	Per-person OAuth for connected apps
Docker + KasmVNC/XFCE	Local Linux desktop for computer-use tasks

Setup

Prerequisites

Bun (runtime + package manager)
Docker (only if you want computer-use)
A Spectrum project for iMessage
A GMI Cloud API key (GPT-5.6 Luna/Terra + images)
A Convex account

Optional: Composio CLI, Exa, Google Maps, cloudflared for tunnels.

1. Install

bun install
cp .env.example .env

Fill .env from the tables below. Never commit .env. Authoritative names live only in .env.example.

2. Environment variables

Required to boot the orchestrator

Variable	What it is
`SPECTRUM_PROJECT_ID`	Spectrum project id
`SPECTRUM_PROJECT_SECRET`	Spectrum project secret
`GMI_CLOUD_API_KEY`	GMI inference key (Luna chat/workers, Terra computer-use, Seedream images)
`CONVEX_URL`	Convex deployment URL (written by `bun run convex:dev`, also copy into `.env`)
`ORCHESTRATOR_BRIDGE_SECRET`	Shared secret between Bun and Convex; must match on both sides

Spectrum / phone

Variable	Required?	What it is
`PHONE_NUMBER`	Optional	Local phone / identity note for your Spectrum setup
`SPECTRUM_SIGNING_WEBHOOK`	Optional	Webhook signing secret when Spectrum posts to your tunnel

Connected apps (Composio)

Variable	Required?	What it is
`COMPOSIO_API_KEY`	For Gmail/Calendar tools	From `composio login` / dashboard
`COMPOSIO_USER_ID_SALT`	With Composio	Random salt; Spectrum sender ids are hashed with this before becoming Composio user ids

composio login
composio init
# then put COMPOSIO_API_KEY + COMPOSIO_USER_ID_SALT in .env

Computer use (Docker desktop)

Variable	Required?	What it is
`COMPUTER_DESKTOP_PASSWORD`	To start the desktop	Passed to Kasm as `VNC_PW` (`bun run computer:up` fails without it)
`COMPUTER_LIVE_VIEW_URL`	For iMessage live cards	Public HTTPS URL for the desktop stream, e.g. `https://desktop.example.com` (viewer host is derived as `viewer.*`)
`OPENAI_API_KEY`	Optional / reserved	Listed in `.env.example`; runtime chat + computer-use currently go through `GMI_CLOUD_API_KEY`

Research + maps

Variable	Required?	What it is
`EXA_API_KEY`	For web search tools	Exa API key
`GOOGLE_MAPS_API_KEY`	For maps / geocode	Google Maps Platform key
`MAPS_PUBLIC_BASE_URL`	For shareable map cards	Public base URL for the maps viewer
`MAPS_VIEWER_TOKEN_SECRET`	With maps	Secret used to sign token-gated map viewer links

3. Convex

# Terminal A — links a deployment, syncs schema/functions, writes CONVEX_URL
bun run convex:dev

Put the same bridge secret in local .env and on the Convex deployment:

# .env
CONVEX_URL=https://….convex.cloud
ORCHESTRATOR_BRIDGE_SECRET=some-long-random-string

# Convex deployment (same value)
bunx convex env set ORCHESTRATOR_BRIDGE_SECRET some-long-random-string

Keep bun run convex:dev running while you develop so functions stay synced.

4. Minimal `.env` checklist

# required
SPECTRUM_PROJECT_ID=...
SPECTRUM_PROJECT_SECRET=...
GMI_CLOUD_API_KEY=...
CONVEX_URL=https://….convex.cloud
ORCHESTRATOR_BRIDGE_SECRET=...

# optional features
SPECTRUM_SIGNING_WEBHOOK=
COMPOSIO_API_KEY=
COMPOSIO_USER_ID_SALT=
COMPUTER_DESKTOP_PASSWORD=
COMPUTER_LIVE_VIEW_URL=
EXA_API_KEY=
GOOGLE_MAPS_API_KEY=
MAPS_PUBLIC_BASE_URL=
MAPS_VIEWER_TOKEN_SECRET=

Run

# Terminal 1 — Convex
bun run convex:dev

# Terminal 2 — orchestrator (Bun loads .env automatically)
bun run start

Text an inbound message to the Spectrum iMessage line. Flow: debounce → Interaction Agent (GPT-5.6 Luna) → tools/workers → reply.

Computer use

Computer-use drives a full XFCE desktop over X11 (screenshots + xdotool), not DOM automation. Docker is the local boundary.

Set GMI_CLOUD_API_KEY and COMPUTER_DESKTOP_PASSWORD in .env.
Start the desktop:

bun run computer:up

Open https://127.0.0.1:6901 and accept the local certificate. Loopback basic auth is disabled; the password is still required by the Kasm image at startup. Display is locked to 1280×800 for stable model coordinates.

When assign_computer_task runs, the orchestrator:

Creates a durable run in Convex
Records the session with FFmpeg (still captures pause the recorder so X11 is not dual-grabbed)
Sends screenshots to GPT-5.6 Terra and applies returned mouse/keyboard actions
Writes runtime/computer/artifacts/<taskId>/demo.mp4
Serves a token-gated viewer at http://127.0.0.1:6902 (live timeline + replay)

bun run computer:logs
bun run computer:down

If the desktop X session dies:

bun run computer:down && bun run computer:up

Dev tunnel (optional)

Expose localhost over HTTPS for Spectrum webhooks and computer/map cards:

bun run tunnel:quick          # one-off URL → :4001
bun run tunnel:setup && bun run tunnel   # named tunnel

bun run tunnel typically maps:

Host	Target
`agent.*`	webhook (`127.0.0.1:4001`)
`desktop.*`	Kasm (`127.0.0.1:6901`)
`viewer.*`	computer viewer (`127.0.0.1:6902`)

Point Spectrum’s webhook at the agent.* URL and set SPECTRUM_SIGNING_WEBHOOK if your project uses signed webhooks. Set COMPUTER_LIVE_VIEW_URL to the public desktop.* URL so iMessage cards open the live stream.

Architecture

Code lives under src/orchestrator/:

Inbound text → keyed debounce (bounce/inbound.ts)
Interaction Agent loads memory, recent history, and tools (assign_task, assign_image_task, assign_computer_task, connected-app tools, and similar)
Workers handle generic tasks, image generation, or the Linux desktop
Handoff delivers results to the original conversation
Transcripts and memory updates persist in Convex

db/ is the only Convex client surface; memory/ calls into db.

Connected apps

Composio tools are cached per sender. The Spectrum sender ID is salted and hashed before becoming the Composio user ID. Without a sender ID, connected-app tools stay off. Default toolkits are Gmail and Google Calendar.

For an unconnected service, the agent texts an OAuth URL. The person finishes OAuth in the browser, then retries the request. Credentials are never collected over iMessage.

Smoke checks

“Search for weekend plans and tell me” — ack → worker → reply.
“Call me Al from now on” — memory write; restart the app and confirm the name sticks.
Multi-turn chat survives process restart (history in Convex).
“Open Chrome and go to example.com” — computer card + live viewer; desktop stays healthy through the run.

Analysis

Compare with all teams

View

Metric

alhwyn
75
Cursor Agent
4

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

HTMLIn code
JavaScriptIn code
OpenAIIn code
TypeScriptIn code
Vercel AI SDKIn code
DockerClaimed

5 of 6 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

Claude CodeConfig
CodexConfig
CursorConfig · Commits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

669 KB

Source files

194

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Project Info

iMessage is all you need

openai-imessage

Codex & GPT-5.6

Codex (how the project was built)

GPT-5.6 (how the product runs)

Stack

Setup

Prerequisites

1. Install

2. Environment variables

Required to boot the orchestrator

Spectrum / phone

Connected apps (Composio)

Computer use (Docker desktop)

Research + maps

3. Convex

4. Minimal `.env` checklist

Run

Computer use

Dev tunnel (optional)

Architecture

Connected apps

Smoke checks

Analysis

Technology

AI coding agents

Codebase size

Structure

Application logic

Background jobs

Data & schema

Languages

Dependencies

iMessage is all you need

openai-imessage

Codex & GPT-5.6

Codex (how the project was built)

GPT-5.6 (how the product runs)

Stack

Setup

Prerequisites

1. Install

2. Environment variables

Required to boot the orchestrator

Spectrum / phone

Connected apps (Composio)

Computer use (Docker desktop)

Research + maps

3. Convex

4. Minimal .env checklist

Run

Computer use

Dev tunnel (optional)

Architecture

Connected apps

Smoke checks

Technology

AI coding agents

Codebase size

Structure

Application logic

Background jobs

Data & schema

Languages

Dependencies

4. Minimal `.env` checklist