# Project export: The Orchestration Company of Palo Alto This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: OS of tomorrow, Claude Code of today - Devpost: https://devpost.com/software/the-orchestration-company-of-palo-alto - GitHub: https://github.com/FO214/treehacks - Demo: https://github.com/soooooooot/treehacks-agent-repo - Video: https://www.youtube.com/embed/imelOPlGLgQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([The Interaction Company of California] Build with Poke (Challenge 1: iPhone Air for everyone in the team Challenge 2: iPhone 17 Pro for everyone in the team Challenge 3: iPhone Air for everyone in the team Bonus: up to $100k in cash)) - Team: 5 GitHub contributor(s) — Theodore (31 commits), Ian Korovinsky (14 commits), FO214 (10 commits), Cursor (2 commits), Ruitong Zhang (1 commits) ## Devpost submission (written by the team) ### Inspiration Have you ever had to manage 6 or 7 Claude Code instances at once? Have you ever felt like there's not enough space on your screen for all of the apps you need to code? Fear no more, introducing the Orchestration Company of Palo Alto! What It Does We allow you to manage all of your coding agents through a brand-new AR interface. Instead of having to constantly switch tabs, babysit Claude Code instances, and doomscrolling Twitter until your Cursor prompt finished, we allow you to hyper-accelerate your productivity through this new coding interface in using Apple Vision Pro. How We Built It For a technical diagram, please refer to the last photo. It helps better explain what is going on in our app. Apple Vision Pro is integrated into this simulator on visionOS: the user is placed in an office environment - floor, cubicles, and desks - and a set of agents is represented by spawning one desk per agent. The 3D world grows on demand: one agent at one desk at launch, and requesting another background task spawns one more agent at a new desk. The immersive experience is built with Xcode, SwiftUI, and an app-wide state model (AppModel) that holds agentCount and immersiveSpaceState. ContentView provides the 2D UI (Enter immersive space and Build an AI agent), while ImmersiveView renders the 3D office and dynamically allocates desks. This single source of truth both defines how many agents exist and ensures actions like adding agents are only available when the user is in the immersive space, so the simulation stays in sync with the Vision Pro session state. We have a TreeHacks Fix Agent MCP (Model Context Protocol) server. It’s implemented with FastMCP and exposes tools such as run_fix, run_analysis, and run_fix_default_repo so callers can trigger Modal sandbox runs and Claude Agent SDK fixes over the network. The server listens on localhost (and tunnels to Poke UI) and uses the streamable HTTP transport : clients send HTTP requests to /mcp and the MCP protocol runs over HTTP with Server-Sent Events (SSE) for streaming. So communication is HTTP/SSE, not WebSockets - one HTTP request can open a stream for server-to-client updates (e.g. tool progress or long agent output). The test client uses mcp to get a read/write pair and a ClientSession for initialize(), list_tools(), and call_tool(). The FastAPI backend acts as an MCP client: it connects to an MCP server (e.g. MCP_HTTP_URL, possibly a separate process or the same poke-mcp on another port) via mcp.client.http.http_client, then calls the run_fix tool with the user’s instruction and optional repo URL and returns the tool result as text. visionOS or any client → FastAPI /fix → MCP client session (HTTP) → poke-mcp (streamable HTTP on 8765) → FastMCP tools → Modal sandbox + Claude Agent. The streaming is handled by the MCP streamable HTTP/SSE transport. Challenges We Ran Into Working with VisionOS is very non-linear - we were very new to it, didn't know how to properly implement some of the more complicated aspects of our hack (and had to opt-in for hackier solutions), and didn't properly understand some of the technical limitations we might run into later on. Accomplishments That We're Proud Of This required learning a lot of brand-new tech that we'd never worked with before - working with multi-turn agents and building what (for most of us) was our first-ever AR hack. We also wanted to make the design feel warm and inviting. We designed animated assets to give the hack a whimsical feel, and we hope that it makes you feel at home :) What We Learned We learned that building hacks for the fun of it all is awesome, and we'll definitely doing it again. VisionOS is also really difficult to work with - we should budget more time to iron out technical issues. What's Next for The Orchestration Company of Palo Alto Build, break, ship, and dream :) Also go back to Waterloo and lock in for exams after TreeHacks is over. Please review our technical diagram (last photo) to check out the technical complexity of our hack! OpenAI: Artificial Intelligence Track We used OpenAI models for speech-to-text and text-to-speech which allows for communication between the user and the multi-turn agent. The multi-agent system itself is run by an orchestrator agent, which uses a custom MCP to spin up code sandboxes for remote code execution. Inside the sandboxes, we run a coding agent in a harness to be able to make changes to the codebase and put up a PR. Finally, we use AI to validate that the changes made by the agent were valid - the AI traverses the webpage for frontend fixes and tests expected behaviour. Anthropic: Human Flourishing Track It's evident that humans are reaching a limit where we receive more signals than we can handle. Engineer switch tabs 24/7 to supervise their coding agents. Employees report that they feel "more overstimulated than ever". The need is clear - we need a way to scale the way we process signals. This project is a first attempt on that - we implemented a thesis we had about the future of software engineering (that most engineers will be product engineers, and this medium creates enough space for folks to easily supervise their agents), and created this MVP project to demonstrate it. Modal: Sandbox Challenge As we spin up coding agents to modify new/existing parts of the codebase, we need to be able to apply these changes in an isolated environment. We use sandboxes to create a brand new coding environment, spin up a coding agent inside of it, clone the existing repo, apply the engineering fix, put up a PR, and double check that the expected behaviour is met through browser automations. Sandboxes are a central piece of this - without them, we wouldn't be able to make changes in isolation,. Anthropic: Claude Agent SDK It's evident that humans are reaching a limit where we receive more signals than we can handle. Engineer switch tabs 24/7 to supervise their coding agents. Employees report that they feel "more overstimulated than ever". The need is clear - we need a way to scale the way we process signals, and we started by reimagining what an agent orchestration interface might look like. For this project, we needed to spin up coding agents that make large scale changes to the codebase. To implement these agents, we used the Claude Agent SDK to create an agent, which would implement the requested change, and then put up a PR showcasing it. Human Capital: Fellowship Prize It's evident that humans are reaching a limit where we receive more signals than we can handle. Engineer switch tabs 24/7 to supervise their coding agents. Employees report that they feel "more overstimulated than ever". The need is clear - we need a way to scale the way we process signals. This project is a first attempt on that - we implemented a thesis we had about the future of software engineering (that most engineers will be product engineers, and this medium creates enough space for folks to easily supervise their agents), and created this MVP project to demonstrate it. We're a team of four Waterloo friends with varying backgrounds, and we'd be open to potentially continuing this project through the fellowship! Greylock: Best Multi-Turn Agent In order to make this project work, we needed an orchestrator. The orchestrator is a multi-turn agent with persistent context across terms, consistent personality, and the ability to make educated decisions on what to do. We were able to create a system where a persistent, multi-turn agent, uses MCPs to spin up sandboxes with coding agents inside, which are able to act on instructions and implement fixes to an existing codebase. Interaction: Build With Poke Most Useful: It's evident that humans are reaching a limit where we receive more signals than we can handle. Engineer switch tabs 24/7 to supervise their coding agents. Employees report that they feel "more overstimulated than ever". The need is clear - we need a way to scale the way we process signals, so we took a first-pass at what a coding IDE run by a conversational assistant (Poke) would look like. Most Technically Complex: Please take a look at the technical diagram (last photo). It shows how complex our system really is :) Most Viral: We did everything we could to make the project go viral. PFA the platforms which we posted / interacted on, and the metrics we reached: In-Person: ran into a bunch of people, got them to try our product! Twitter/X: 6,200+ interactions LinkedIn: 13,000+ impressions, 153 likes, 16 comments, 2 reposts Decagon: Best Conversation Assistant Our project is an implementation of what a coding interface would look like if it was a conversational assistant. We implemented a multi-turn agent with persistent context across terms, consistent personality, and the ability to make educated decisions on what to do. It helps you with your tasks, and can refer to previous conversions (either relevantly, or to make fun of you). We were able to create a system where this persistent, multi-turn agent, uses MCPs to spin up sandboxes with coding agents inside, which are able to act on instructions and implement fixes to an existing codebase, and then the multi-turn agent can discuss their results. Browserbase: Best Web Automation with Stagehand The biggest engineering problem at startups is that the pace at which senior engineers review code is much slower than the speed at which junior engineers can generate it using AI. To tackle this problem, we used Browserbase to tackle this issue by implementing fix validation. Using Stagehand, we are able to automatically traverse websites, and validate that the intended result was achieved. We also show the Browserbase recording of the agentic testing of a user's changes once the PR is put up by the agent, and this helps validate that the agent's changes are legit. ## README (from the GitHub repository) # Treehacks FastAPI server that takes a **text-input** (fix instruction), calls an **MCP server**, which scaffolds a **Modal sandbox** with the **Claude Agent SDK**, clones a sample repo, and runs the agent to implement the fix. ## Architecture - **FastAPI** (`server/`): `POST /fix` with `{"text_input": "Fix the bug in auth.py", "repo_url": "https://..."}`. Calls the MCP server over stdio. - **MCP server** (`mcp_server/`): Exposes tool `run_fix(instruction, repo_url)`. When invoked, creates a Modal sandbox, clones the repo, runs Claude Agent SDK with the instruction (Read/Edit/Bash/Glob/Grep), returns output. - **Sample repo**: Default is `https://github.com/modal-labs/modal-examples`. You can add your own and pass `repo_url` in the request. ## Setup 1. **From repo root**, create a venv and install deps for both server and MCP: ```bash python -m venv .venv source .venv/bin/activate # or .venv\Scripts\activate on Windows pip install -r server/requirements.txt pip install -r mcp_server/requirements.txt ``` 2. **Modal**: `pip install modal` and `modal token new` (or set `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET`). 3. **Anthropic**: Create a Modal secret for the sandbox so the agent can call Claude: ```bash modal secret create anthropic-secret ANTHROPIC_API_KEY=sk-ant-... ``` Or set `ANTHROPIC_API_KEY` in your env; the runner will pass it into the sandbox when no Modal secret exists. 4. **Run FastAPI** (from repo root so `python -m mcp_server.main` works): ```bash uvicorn server.main:app --reload ``` 5. **Call the API**: ```bash curl -X POST http://127.0.0.1:8000/fix \ -H "Content-Type: application/json" \ -d '{"text_input": "List all Python files in this repo"}' ``` With your own repo: ```bash curl -X POST http://127.0.0.1:8000/fix \ -H "Content-Type: application/json" \ -d '{"text_input": "Fix the login bug in auth.py", "repo_url": "https://github.com/you/your-sample-repo"}' ``` ## Docker (server + MCP server) This project can run the FastAPI server and the MCP server as separate services. The MCP server is exposed over Streamable HTTP at `/mcp` on port 8001. 1. Create an env file with your secrets (or export these in your shell): ```bash cat > .env <<'EOF' ANTHROPIC_API_KEY=sk-ant-... MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=... EOF ``` 2. Build and run with Docker Compose (starts both `api` and `mcp`): ```bash docker compose up --build ``` 3. Call the API (same as above): ```bash curl -X POST http://127.0.0.1:8000/fix \ -H "Content-Type: application/json" \ -d '{"text_input": "List all Python files in this repo"}' ``` Notes: - The containers need outbound network access for `git clone` and the Modal sandbox. - The MCP HTTP endpoint is `http://127.0.0.1:8001/mcp`. - The FastAPI server uses `MCP_HTTP_URL` (default: `http://127.0.0.1:8001/mcp`) to reach the MCP server. ## Optional: your sample repo Create a small repo with a deliberate bug (e.g. a broken test or a bug in one file). Push it to GitHub and pass its URL as `repo_url`. The agent will clone it in the sandbox and apply the fix from `text_input`. ## Poke MCP Integration To integrate this MCP server with Poke (Streamable HTTP): 1. Go to `https://poke.com/settings/connections/integrations/new`. 2. Use the verified MCP server template for 1-click deploy: `https://github.com/InteractionCo/mcp-server-template`. 3. Configure your MCP connection in Poke after deployment. If you want to run this repo directly instead of the template, point Poke to: `http://:8001/mcp` (this is served by `python -m mcp_server.http_server`). To send messages to Poke programmatically: Bash ```bash API_KEY="your-api-key-here" MESSAGE="Hello from HackMIT!" response=$(curl 'https://poke.com/api/v1/inbound-sms/webhook' \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -X POST \ -d "{\"message\": \"$MESSAGE\"}") echo $response ``` TypeScript ```ts const API_KEY = 'your-api-key-here'; const MESSAGE = 'Hello from HackMIT!'; const response = await fetch('https://poke.com/api/v1/inbound-sms/webhook', { method: 'POST', headers: { 'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ message: MESSAGE }) }); const data = await response.json(); console.log(data); ``` Python ```py import requests API_KEY = 'your-api-key-here' MESSAGE = 'Hello from HackMIT!' response = requests.post( 'https://poke.com/api/v1/inbound-sms/webhook', headers={ 'Authorization': f'Bearer {API_KEY}', 'Content-Type': 'application/json' }, json={'message': MESSAGE} ) print(response.json()) ``` ### Voice service (chat.db poller + STT/TTS endpoints) This repo includes a local voice service at `voice-server.mjs` that does: 1. Initialize a new `Poke` client on startup. 2. Read latest `message.date`/`ROWID` for `POKE_HANDLE_ID` as the startup checkpoint. 3. Poll `chat.db` every few seconds for new inbound messages from that handle. 4. Queue inbound messages in memory. 5. Expose endpoints to run one recording turn, STT, and TTS. Run it from repo root: ```bash npm install npm run voice ``` For auto-restart on `.env` or `voice-server.mjs` changes during development: ```bash npm run voice:dev ``` Required env vars: - `POKE_API_KEY` - `OPENAI_API_KEY` - `POKE_HANDLE_ID` (Messages `handle_id` for Poke contact) Optional env vars: - `POKE_SESSION_BOOT_MESSAGE` (sent once to Poke on startup) - `STT_MODEL` (default `gpt-4o-mini-transcribe`) - `TTS_MODEL` (default `gpt-4o-mini-tts`) - `TTS_VOICE` (default `alloy`) - `TTS_SPEED` (default `1.0`) - `TTS_RESPONSE_FORMAT` (`wav` or `pcm`; default auto: `pcm` with `ffplay`, else `wav`) - `TTS_BATCH_QUEUE` (default `true`; when true, queued inbound messages are concatenated and spoken together) - `TTS_BATCH_SEPARATOR` (default `" "`; separator used between concatenated queued messages) - `SOUND_EFFECTS_ENABLED` (default `true`) - `SOUND_EFFECTS_DIR` (default `./sound-effects`) - `START_RECORDING_SOUND` (default `start-recording.mp3`) - `STOP_RECORDING_SOUND` (default `stop-recording.mp3`) - `NO_RECORDING_SOUND` (default `no-recording.mp3`) - `MIN_AUDIO_BYTES` (default `8000`) - `CHAT_DB_PATH` (default `~/Library/Messages/chat.db`) - `CHAT_POLL_MS` (default `1000`) - `RESPONSE_TIMEOUT_MS` (default `120000`) - `VOICE_HTTP_PORT` (default `8787`) Endpoints: - `POST /record-once` (record -> STT -> send to Poke -> await next inbound chat.db message -> optional TTS talkback) - `POST /stt` with `{ "audioPath": "/abs/path/to/file.wav" }` - `POST /tts` with `{ "text": "hello" }` - `GET /queue` and `POST /queue/speak-next` - `GET /health` Local system tools needed for audio I/O: - `rec` (SoX) for recording with silence detection - `afplay` (macOS) or `ffplay` for playback ## Project layout ``` treehacks/ ├── server/ # FastAPI app │ ├── main.py # POST /fix → MCP client → run_fix │ └── requirements.txt ├── mcp_server/ # MCP server (stdio) │ ├── main.py # FastMCP, run_fix tool │ ├── agent_runner.py # Modal sandbox + Claude Agent SDK │ └── requirements.txt └── README.md ``` ## Detected evidence (automated analysis) Indexed codebase: 46 recognized source files, 231 KB. - Anthropic (technology) — detected in the code - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Swift (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code - Next.js (technology) — claimed on Devpost, not found in the code - TypeScript (language) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (114 of 114) ``` .gitattributes .gitignore .gitmodules bun.lock DEMO-CONNECTION.md docker-compose.yml Dockerfile.api Dockerfile.mcp docs/SYSTEM-DESIGN.md json-schemas/agent-create-thinking.json json-schemas/agent-start-testing.json json-schemas/agent-start-working.json json-schemas/fix-request.json json-schemas/fix-response.json json-schemas/jump-ping.json json-schemas/README.md package.json package/package.json package/README.md poke-0.2.5.tgz poke-mcp/README.md poke-mcp/requirements.txt poke-mcp/server.py poke-mcp/start.sh poke-mcp/test_client.py poke-responses.log README.md server/diagram_mermaid_ink.py server/diagram.py server/event_bus.py server/main.py server/repo_to_png/__init__.py server/repo_to_png/format_message.py server/repo_to_png/local_directory.py server/repo_to_png/mermaid_to_png.py server/repo_to_png/openai_service.py server/repo_to_png/pipeline.py server/repo_to_png/prompts.py server/repo_to_png/README.md server/repo_to_png/test_mermaid_sanitize.py server/requirements.txt server/voice.py setup-tunnel.sh start.sh test-photo.md treehacks26/Packages/RealityKitContent/.swiftpm/xcode/xcuserdata/rachelzhang.xcuserdatad/xcschemes/xcschememanagement.plist treehacks26/Packages/RealityKitContent/.swiftpm/xcode/xcuserdata/theodore.xcuserdatad/xcschemes/xcschememanagement.plist treehacks26/Packages/RealityKitContent/Package.realitycomposerpro/PluginData/AC3EA5A0-F06C-4A0F-99B4-05AEF069D93D/ShaderGraphEditorPluginID/ShaderGraphEditorPluginID treehacks26/Packages/RealityKitContent/Package.realitycomposerpro/ProjectData/main.json treehacks26/Packages/RealityKitContent/Package.realitycomposerpro/WorkspaceData/rachelzhang.rcuserdata treehacks26/Packages/RealityKitContent/Package.realitycomposerpro/WorkspaceData/SceneMetadataList.json treehacks26/Packages/RealityKitContent/Package.realitycomposerpro/WorkspaceData/Settings.rcprojectdata treehacks26/Packages/RealityKitContent/Package.swift treehacks26/Packages/RealityKitContent/README.md treehacks26/Packages/RealityKitContent/Sources/.txt treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/character.usdc treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/computerSpawn_2.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/fixed_whiteboard.usdc treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Ground/DefaultAttenuationMap.exr treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Ground/Ground.usda treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/palmtreelowpolygon.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_backup.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/0/Retro_computer.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/0/Wood_Boss_Office_Table.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/computerSpawn.usda treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/computerSpawn.usdc treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/computerSpawn.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/Susuwatari_modified.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/0/Susuwatari.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/susuwatari.usda treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari_unpacked/susuwatari.usdc treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/Susuwatari.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/textures/color_121212.hdr treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets/thinking.usdz treehacks26/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.swift treehacks26/set-app-icon.sh treehacks26/treehacks26.xcodeproj/project.pbxproj treehacks26/treehacks26.xcodeproj/project.xcworkspace/contents.xcworkspacedata treehacks26/treehacks26.xcodeproj/project.xcworkspace/xcuserdata/rachelzhang.xcuserdatad/UserInterfaceState.xcuserstate treehacks26/treehacks26.xcodeproj/project.xcworkspace/xcuserdata/theodore.xcuserdatad/UserInterfaceState.xcuserstate treehacks26/treehacks26.xcodeproj/xcuserdata/rachelzhang.xcuserdatad/xcschemes/xcschememanagement.plist treehacks26/treehacks26.xcodeproj/xcuserdata/theodore.xcuserdatad/xcschemes/xcschememanagement.plist treehacks26/treehacks26/APIConfig.swift treehacks26/treehacks26/AppModel.swift treehacks26/treehacks26/Assets.xcassets/AccentColor.colorset/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Back.solidimagestacklayer/Content.imageset/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Back.solidimagestacklayer/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Front.solidimagestacklayer/Content.imageset/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Front.solidimagestacklayer/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Middle.solidimagestacklayer/Content.imageset/Contents.json treehacks26/treehacks26/Assets.xcassets/AppIcon.solidimagestack/Middle.solidimagestacklayer/Contents.json treehacks26/treehacks26/Assets.xcassets/Contents.json treehacks26/treehacks26/Assets.xcassets/palo-alto.imageset/Contents.json treehacks26/treehacks26/AVPlayerView.swift treehacks26/treehacks26/AVPlayerViewModel.swift treehacks26/treehacks26/ContentView.swift treehacks26/treehacks26/HandTrackingManager.swift treehacks26/treehacks26/ImmersiveView.swift treehacks26/treehacks26/Info.plist treehacks26/treehacks26/TableAllocator.swift treehacks26/treehacks26/TaskNameMapping.swift treehacks26/treehacks26/ToggleImmersiveSpaceButton.swift treehacks26/treehacks26/treehacks26App.swift treehacks26/treehacks26/TycoonSceneState.swift treehacks26/treehacks26/WebView.swift treehacks26/treehacks26Tests/treehacks26Tests.swift TUNNEL-SETUP.md visionos/OctaviusApp.xcodeproj/project.pbxproj visionos/OctaviusApp.xcodeproj/project.xcworkspace/contents.xcworkspacedata visionos/OctaviusApp.xcodeproj/project.xcworkspace/xcuserdata/iankorovinsky.xcuserdatad/UserInterfaceState.xcuserstate visionos/OctaviusApp.xcodeproj/xcuserdata/iankorovinsky.xcuserdatad/xcschemes/xcschememanagement.plist visionos/RealityKitContent/.swiftpm/xcode/xcuserdata/iankorovinsky.xcuserdatad/xcschemes/xcschememanagement.plist voice-server.mjs ``` ### Dependencies - package.json: dotenv@^17.2.3, express@^5.1.0, groq-sdk@^0.37.0, openai@^6.3.0, poke@^0.2.5 - package/package.json: @anthropic-ai/claude-agent-sdk@^0.1.0, @types/node@^22.14.1, @types/ws@^8.5.10, commander@^12.1.0, esbuild@^0.25.3, open@^10.0.0, qrcode-terminal@^0.12.0, tsx@^4.16.2, typescript@^5.8.2, ws@^8.16.0 - poke-mcp/requirements.txt: anthropic@>=0.40.0, browserbase@>=1.0.0, fastmcp@>=2.0.0, modal@>=0.64.0, playwright@>=1.40.0, python-dotenv@>=1.0.0, stagehand - server/requirements.txt: fastapi@>=0.115.0, groq@>=0.4.0, httpx@>=0.27.0, matplotlib@>=3.8.0, mcp@>=1.24.0, openai@>=1.0.0, pydantic@>=2.0.0, python-dotenv@>=1.0.0, tiktoken@>=0.5.0, uvicorn[standard]@>=0.32.0 ### Recent commits (newest first) - push - potential round robin fit - fix - fix - better - nsadfg - asdfgb - SDfg - valentines - chore: set up Git LFS for binary assets - lfs asdfg - fix all easrfdg - Merge branch 'main' of github.com:FO214/treehacks - stuff - fix - little fixes - erwfgr - stuff - Merge branch 'main' of github.com:FO214/treehacks - ui ## Key source files (fetched from GitHub, selected and truncated for size) ### DEMO-CONNECTION.md ```markdown # Connection Demo: Vision Pro ↔ FastAPI Step-by-step to get the demo block (red/green) working in the Vision Pro app. --- ## Option A: Cloudflare Tunnel (recommended) ### Step 1: Install deps ```bash cd /Users/theodore/treehacks python3 -m venv .venv source .venv/bin/activate pip install -r server/requirements.txt npm install ``` ### Step 2: Start the consolidated server (voice + demo + /fix on port 8000) ```bash ./start.sh ``` This starts the voice server on port 8000 (demo block, gesture-to-sound, /fix proxy). FastAPI runs on 8002 in the background for /fix. ### Step 3: Start the Cloudflare tunnel In a **new terminal**: ```bash cd /Users/theodore/treehacks cloudflared tunnel run treehacks ``` (If not set up yet, run `./setup-tunnel.sh` first.) ### Step 4: Run on Vision Pro 1. Build and run on your Vision Pro (or simulator). 2. Tap **Toggle Immersive Space** to enter the immersive view. 3. A block should appear ~1m in front of you. 4. Every 2 seconds it should alternate **red** (0) and **green** (1). `APIConfig.baseURL` is already set to `https://treehacks.tzhu.dev`. --- ## Option B: Docker + Cloudflare ### Step 1: Start the backend (requires Docker) ```bash cd /Users/theodore/treehacks export MCP_HTTP_URL=http://mcp:8001/mcp docker compose up --build ``` ### Step 2: Start the Cloudflare tunnel In a **new terminal**: ```bash cloudflared tunnel run treehacks ``` ### Step 3: Run on Vision Pro Same as Option A, Step 4. --- ## Option C: Local only (no tunnel) Use when Vision Pro and Mac are on the same Wi‑Fi. ### Step 1: Start the backend (Python venv) ```bash cd /Users/theodore/treehacks source .venv/bin/activate uvicorn server.main:app --host 0.0.0.0 --port 8000 ``` ### Step 2: Get your Mac's IP ```bash ipconfig getifaddr en0 ``` Example: `192.168.1.42` ### Step 3: Update APIConfig ```swift static let baseURL = "http://192.168.1.42:8000" ``` Replace with your Mac's IP. ### Step 4: Run on Vision Pro Same as Option A, Step 4. Vision Pro must be on the same Wi‑Fi as your Mac. --- ## Troubleshooting | Problem | Fix | |--------|-----| | Block stays red | Backend not reachable. Check URL, tunnel, and firewall. | | 502 Bad Gateway | Backend not running. Start `uvicorn server.main:app` or `docker compose up`. | | Block never appears | Enter immersive space with the toggle button. | | Local IP doesn't work | Ensure Vision Pro and Mac are on the same Wi‑Fi. | ### Test the API manually ```bash curl https://treehacks.tzhu.dev/demo/value # Should return: {"value":0} or {"value":1} ``` ``` ### test-photo.md ```markdown # TreeHacks System Design High-level architecture of the voice + fix-agent stack. ## Component overview ```mermaid flowchart TB subgraph client [Client] VP[Vision Pro App] end subgraph server [FastAPI Server :8000] API[REST API] WSPoke["/ws/poke"] WSSpawn["/ws/spawn"] EventBus[event_bus] InternalEvent["POST /internal/event"] Voice[voice module] end subgraph mcp [MCP Server poke-mcp] FastMCP[FastMCP tools] RunFix[run_fix / run_fix_default_repo] end subgraph external [External Services] Modal[Modal] GitHub[GitHub] Poke[Poke API] ChatDB["chat.db"] OpenAI[OpenAI / Groq] Vercel[Vercel] Browserbase[Browserbase] end VP -->|"hand_open / hand_close"| WSPoke VP -->|"connect, receive events"| WSSpawn VP -.->|"optional POST /record-once"| API WSPoke --> Voice Voice -->|"transcript"| Poke Poke --> ChatDB Voice -->|"poll"| ChatDB Voice -->|"STT / TTS"| OpenAI API -->|"call_tool run_fix"| FastMCP RunFix --> Modal RunFix --> GitHub RunFix -->|"optional"| Vercel RunFix -->|"optional"| Browserbase RunFix -->|"POST events"| InternalEvent InternalEvent --> EventBus WSPoke --> EventBus WSSpawn --> EventBus EventBus -->|"broadcast"| VP ``` ## Data flows ### 1. Voice (Vision Pro → server → Poke → talkback) ```mermaid sequenceDiagram participant VP as Vision Pro participant WS as /ws/poke participant Voice as voice participant Poke as Poke API participant DB as chat.db participant TTS as TTS VP->>WS: hand_open WS->>Voice: start_recording() WS->>VP: broadcast listening VP->>WS: hand_close WS->>Voice: stop_and_process() Voice->>Voice: STT (OpenAI/Groq) Voice->>Poke: POST transcript Poke->>DB: (Poke writes reply) Voice->>DB: poll for inbound Voice->>TTS: speak inbound text WS->>VP: broadcast poke_speaking_start/stop ``` ### 2. Fix agent (client → server → MCP → Modal → GitHub → Vision Pro) ```mermaid sequenceDiagram participant Client as MCP Client participant Server as FastAPI participant MCP as poke-mcp participant Modal as Modal Sandbox participant GitHub as GitHub participant Webhook as /internal/event participant VP as Vision Pro Client->>Server: POST /fix (or direct MCP) Server->>MCP: call_tool run_fix MCP->>MCP: run_modal_agent (async/thread) MCP->>Modal: Sandbox.create MCP->>Modal: clone, branch, exec claude-agent-sdk MCP->>Webhook: POST create_agent_thinking MCP->>Webhook: POST agent_start_working Webhook->>VP: broadcast (via event_bus) Modal-->>MCP: agent done MCP->>GitHub: push, create PR opt smoke_test MCP->>Vercel: wait preview MCP->>Browserbase: run smoke test end MCP->>Webhook: POST agent_start_testing (vercel_link, browserbase_link) Webhook->>VP [truncated — 1760 more characters] ``` ### package.json ``` { "type": "module", "scripts": { "voice": "node voice-server.mjs", "voice:dev": "node --watch --watch-path=.env --watch-path=voice-server.mjs voice-server.mjs" }, "dependencies": { "dotenv": "^17.2.3", "express": "^5.1.0", "groq-sdk": "^0.37.0", "openai": "^6.3.0", "poke": "^0.2.5" } } ``` ### docker-compose.yml ```yaml services: api: build: context: . dockerfile: Dockerfile.api ports: - "8000:8000" environment: MCP_HTTP_URL: ${MCP_HTTP_URL} ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY} MODAL_TOKEN_ID: ${MODAL_TOKEN_ID} MODAL_TOKEN_SECRET: ${MODAL_TOKEN_SECRET} volumes: - .:/app mcp: build: context: . dockerfile: Dockerfile.mcp command: ["python", "-m", "mcp_server.main"] ports: - "8001:8001" environment: HOST: 0.0.0.0 PORT: 8001 ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY} MODAL_TOKEN_ID: ${MODAL_TOKEN_ID} MODAL_TOKEN_SECRET: ${MODAL_TOKEN_SECRET} GITHUB_TOKEN: ${GITHUB_TOKEN} MCP_HTTP_URL: ${MCP_HTTP_URL} volumes: - .:/app ``` ### poke-mcp/requirements.txt ``` fastmcp>=2.0.0 python-dotenv>=1.0.0 modal>=0.64.0 browserbase>=1.0.0 playwright>=1.40.0 anthropic>=0.40.0 stagehand ``` ### server/requirements.txt ``` fastapi>=0.115.0 uvicorn[standard]>=0.32.0 mcp>=1.24.0 pydantic>=2.0.0 # repo_to_png pipeline matplotlib>=3.8.0 openai>=1.0.0 tiktoken>=0.5.0 python-dotenv>=1.0.0 # voice server httpx>=0.27.0 groq>=0.4.0 ``` ### package/package.json ``` { "name": "poke", "version": "0.2.5", "type": "module", "description": "Poke.com Developer SDK", "main": "./dist/index.cjs", "module": "./dist/index.mjs", "types": "./dist/index.d.ts", "exports": { ".": { "types": "./dist/index.d.ts", "import": "./dist/index.mjs", "require": "./dist/index.cjs" } }, "bin": { "poke": "./dist/cli.cjs" }, "files": [ "dist" ], "scripts": { "build": "tsc --noEmit && tsc --emitDeclarationOnly --outDir dist && node esbuild.config.js", "dev": "tsx src/cli.ts" }, "dependencies": { "@anthropic-ai/claude-agent-sdk": "^0.1.0" }, "devDependencies": { "@types/node": "^22.14.1", "@types/ws": "^8.5.10", "commander": "^12.1.0", "esbuild": "^0.25.3", "open": "^10.0.0", "qrcode-terminal": "^0.12.0", "tsx": "^4.16.2", "typescript": "^5.8.2", "ws": "^8.16.0" }, "keywords": [ "poke", "cli", "tunnel", "localhost", "mcp" ], "license": "MIT", "engines": { "node": ">=18" } } ``` ### server/main.py ```python """ FastAPI server: /fix, voice endpoints, demo. Single server on port 8000. """ import os from pathlib import Path from dotenv import load_dotenv # Load .env from repo root (parent of server/) load_dotenv(Path(__file__).resolve().parent.parent / ".env") import asyncio import json as _json from contextlib import asynccontextmanager from typing import Any from fastapi import FastAPI, HTTPException, WebSocket, WebSocketDisconnect from fastapi.responses import PlainTextResponse from mcp import ClientSession from mcp.client.streamable_http import streamablehttp_client from pydantic import BaseModel from . import voice from . import event_bus # --------------------------------------------------------------------------- # Agent state store: last event payload per agent_id (1-9). Updated by /internal/event; # status loop re-broadcasts each once per second so clients get a steady stream. # --------------------------------------------------------------------------- _agent_state_store: dict[int, dict[str, Any]] = {} # Connection counts per endpoint (for debugging: is client on /ws/spawn or /ws/poke?) _ws_spawn_connections = 0 _ws_poke_connections = 0 # --------------------------------------------------------------------------- # Atomic agent ID counter (1-9, wrapping). Owned by the main server. # --------------------------------------------------------------------------- import threading _agent_id_counter = 0 _agent_id_lock = threading.Lock() def _next_agent_id() -> int: """Return the next agent ID (1-9, wrapping). First call after startup returns 1.""" global _agent_id_counter with _agent_id_lock: _agent_id_counter = (_agent_id_counter % 9) + 1 return _agent_id_counter class FixRequest(BaseModel): text_input: str repo_url: str | None = None # optional; MCP server has default sample repo class FixResponse(BaseModel): success: bool result: str error: str | None = None async def _agent_status_stream_loop() -> None: """Every 1 second, re-broadcast the last known state (same JSON) for each active agent.""" while True: await asyncio.sleep(1.0) for agent_id, msg in list(_agent_state_store.items()): try: await event_bus.broadcast(msg) except Exception: pass @asynccontextmanager async def lifespan(app: FastAPI): # Register the running event loop so event_bus.broadcast_sync works from threads event_bus.set_loop(asyncio.get_running_loop()) print("[startup] /ws/spawn forwards real agent events from /internal/event only.", flush=True) voice.voice_startup() if voice.TALKBACK_ENABLED and voice.TTS_LOOP_AUTOSTART: loop = asyncio.get_running_loop() voice.start_tts_loop(loop) status_task = asyncio.create_task(_agent_status_stream_loop()) try: yield finally: status_task.cancel() try: await status_task except asyncio.CancelledError: pass voice.stop_tts_loop() @asynccontextmanager async def get_mcp_session(): """Connect to MCP server over HTTP and yield session.""" mcp_url = os.environ.get("MCP_HTTP_URL", "http://127.0.0.1:8001/mcp") async with streamablehttp_client(mcp_url) as (read_stream, write_stream, _): async with ClientSession(read_stream, write_stream) as session: await session.initialize() yield session async def call_run_fix(text_input: str, repo_url: str | None) -> str: """Call MCP tool run_fix and return result text.""" args: dict[str, Any] = {"instruction": text_input} if repo_url: args["repo_url"] = repo_url async with get_mcp_session() as session: result = await session.call_tool("run_fix", arguments=args) if result.isError: first = result.content[0] if result.content else None msg = getattr(first, "text", None) or (first.get("text") if isinstance(first, dict) else None) if first else "Unknown MCP error" raise RuntimeError(msg) # Result is list of ContentBlock (object or dict) out_parts = [] for block in result.content: text = getattr(block, "text", None) or (block.get("text") if isinstance(block, dict) else None) if text: out_parts.append(text) return "\n".join(out_parts) if out_parts else "" app = FastAPI( title="TreeHacks API", description="Fix API + voice server (record, STT, Poke, TTS).", lifespan=lifespan, ) @app.post("/fix", response_class=PlainTextResponse) async def fix(request: FixRequest) -> str: """Run the fix instruction via MCP server (Modal sandbox + Claude Agent SDK).""" try: result = await call_run_fix(request.text_input, request.repo_url) return result except Exception as e: raise HTTPException(status_code=500, detail=str(e)) # --------------------------------------------------------------------------- # WebSocket: /ws/poke — Vision Pro ↔ server voice event bridge # --------------------------------------------------------------------------- @app.websocket("/ws/poke") async def websocket_poke(websocket: WebSocket): """ Vision Pro connects here for voice flow. Receives hand_open / hand_close gestures; broadcasts listening, poke_speaking events back. """ global _ws_poke_connections await websocket.accept() _ws_poke_connections += 1 event_bus.register(websocket) print(f"[ws/poke] client connected (total poke: {_ws_poke_connections})", flush=True) try: while True: raw = await websocket.receive_text() print(f"[ws/poke] ← received: {raw[:120]}{'…' if len(raw) > 120 else ''}") try: msg = _json.loads(raw) except _json.JSONDecodeError: continue msg_type = msg.get("type", "") print(f"[ws/poke] ← parsed type={msg_type!r}") if msg_type == "hand_open": # Start [truncated — 7109 more characters] ``` ### poke-mcp/server.py ```python """ TreeHacks Fix Agent MCP Server Exposes the Modal + Claude Agent SDK functionality via FastMCP tools. After the agent applies a fix, the server automatically creates a branch, commits, pushes, and opens a GitHub pull request. """ import asyncio import json import os import re import subprocess import sys import threading import time import urllib.request # Load keys from poke-mcp/.env (ANTHROPIC_API_KEY, MODAL_*, GITHUB_TOKEN, etc.) try: from dotenv import load_dotenv _env_path = os.path.join(os.path.dirname(__file__), ".env") load_dotenv(_env_path) except ImportError: pass from fastmcp import FastMCP # Get the directory containing this script SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) PROJECT_ROOT = os.path.dirname(SCRIPT_DIR) mcp = FastMCP("TreeHacks Fix Agent") # Default repository for testing DEFAULT_REPO_URL = "https://github.com/soooooooot/treehacks-agent-repo" # If set, run_fix returns immediately and does sandbox/PR in a background thread. # Use this to avoid MCP client timeouts (ClientDisconnect) when the client gives up # before the long-running job finishes. RUN_FIX_IN_BACKGROUND = (os.environ.get("RUN_FIX_IN_BACKGROUND", "").strip().lower() in ("1", "true", "yes")) RUN_FIX_MAX_CONCURRENT = int(os.environ.get("RUN_FIX_MAX_CONCURRENT", "8")) RUN_FIX_SMOKE_TEST = (os.environ.get("RUN_FIX_SMOKE_TEST", "").strip().lower() in ("1", "true", "yes")) _run_fix_semaphore: asyncio.Semaphore | None = None def _get_run_fix_semaphore() -> asyncio.Semaphore: global _run_fix_semaphore if _run_fix_semaphore is None: _run_fix_semaphore = asyncio.Semaphore(RUN_FIX_MAX_CONCURRENT) return _run_fix_semaphore # --------------------------------------------------------------------------- # Event webhook: POST agent progress events to FastAPI /internal/event # --------------------------------------------------------------------------- EVENT_WEBHOOK_URL = os.environ.get("EVENT_WEBHOOK_URL", "http://localhost:8000/internal/event") NEXT_AGENT_ID_URL = os.environ.get("NEXT_AGENT_ID_URL", "http://localhost:8000/internal/next-agent-id") def _next_agent_id() -> int: """Fetch the next agent ID (1-9, wrapping) from the main server.""" try: req = urllib.request.Request(NEXT_AGENT_ID_URL, method="POST") req.add_header("Content-Type", "application/json") with urllib.request.urlopen(req, timeout=5) as resp: data = json.loads(resp.read()) return data.get("agent_id", 1) except Exception as e: print(f"[soot] Failed to get next agent ID from server: {e}", flush=True) return 1 # fallback def _post_event(event: dict) -> None: """Fire-and-forget POST to the FastAPI event webhook.""" payload = json.dumps(event).encode() headers = {"Content-Type": "application/json"} try: req = urllib.request.Request( EVENT_WEBHOOK_URL, data=payload, method="POST", headers=headers, ) with urllib.request.urlopen(req, timeout=5) as resp: resp.read() except Exception as e: print(f"[soot] event webhook failed: {e}", flush=True) # System prompt: frames the task for fix/change flows (make the modification only, # the server handles branching/committing/PR automatically). FIX_SYSTEM_PROMPT = """You are a code assistant working in a cloned repository at /repo. Your task is to: 1. Make the modification or fix requested by the user. 2. Be precise and only change what is needed to fulfill the request. IMPORTANT: - ALWAYS use the Bash tool for ALL file operations (creating, editing, reading files). Use commands like: cat, echo with redirect, tee, sed, etc. Example: To create a file, use bash: cat << 'HEREDOC_EOF' > /repo/filename.ext - Do NOT use the Write or Edit tools — they may not persist to disk in this environment. - All file paths must be absolute, starting with /repo/ - Do NOT create branches, commit, or open PRs yourself — that is handled automatically after you finish.""" # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- def _parse_github_owner_repo(repo_url: str) -> tuple[str, str]: """Extract (owner, repo) from a GitHub URL.""" m = re.match(r"https?://github\.com/([^/]+)/([^/.]+)", repo_url) if not m: raise ValueError(f"Could not parse GitHub owner/repo from: {repo_url}") return m.group(1), m.group(2) def _slugify(text: str, max_len: int = 40) -> str: """Turn a human sentence into a short branch-safe slug.""" slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") return slug[:max_len].rstrip("-") def _exec(sb, *args, **kwargs) -> tuple[str, str, int]: """Run a command in the sandbox; return (stdout, stderr, returncode).""" p = sb.exec(*args, **kwargs) p.wait() stdout = p.stdout.read() if hasattr(p.stdout, "read") else "" stderr = p.stderr.read() if hasattr(p.stderr, "read") else "" return stdout, stderr, p.returncode def _create_pr( owner: str, repo: str, branch: str, base: str, title: str, body: str, token: str, ) -> str: """Create a GitHub pull request via the REST API. Returns the PR URL.""" url = f"https://api.github.com/repos/{owner}/{repo}/pulls" payload = json.dumps({ "title": title, "head": branch, "base": base, "body": body, }).encode() headers = { "Authorization": f"token {token}", "Accept": "application/vnd.github+json", "Content-Type": "application/json", } # Follow 307/308 redirects (urllib doesn't follow them for POST by default) for _ in range(3): req = urllib.request.Request(url, data=payload, method="POST", headers=headers) try: with urllib.request.urlopen(req) as resp: result = json.loads(resp.read()) return result.get("html_url", "") exce [truncated — 34123 more characters] ``` ### start.sh ```shell #!/usr/bin/env bash # Start FastAPI server (fix + voice) on port 8000. # Cloudflare tunnel → 8000 set -e cd "$(dirname "$0")" # Load .env set -a [ -f .env ] && source .env set +a # Need Python >=3.10 for mcp and other deps (try PATH then Homebrew) PYTHON_CMD="" for p in python3.12 python3.11 python3.10 \ /opt/homebrew/opt/python@3.12/bin/python3.12 \ /opt/homebrew/opt/python@3.11/bin/python3.11 \ /usr/local/opt/python@3.12/bin/python3.12 \ /usr/local/opt/python@3.11/bin/python3.11; do if [ -x "$p" ] 2>/dev/null && "$p" -c "import sys; exit(0 if sys.version_info >= (3, 10) else 1)" 2>/dev/null; then PYTHON_CMD="$p" break fi done if [ -z "$PYTHON_CMD" ]; then if command -v python3 &>/dev/null && python3 -c "import sys; exit(0 if sys.version_info >= (3, 10) else 1)" 2>/dev/null; then PYTHON_CMD="python3" fi fi if [ -z "$PYTHON_CMD" ]; then echo "[start] ERROR: Python 3.10+ required for mcp. Install with: brew install python@3.12" echo " Then run ./start.sh again." exit 1 fi # Recreate venv if missing or broken (encodings error) VENV_PYTHON=".venv/bin/python" if [ ! -f "$VENV_PYTHON" ] || ! "$VENV_PYTHON" -c "import encodings" 2>/dev/null; then echo "[start] Creating or fixing virtual environment (using $PYTHON_CMD)..." rm -rf .venv "$PYTHON_CMD" -m venv .venv "$VENV_PYTHON" -m pip install --upgrade pip -q "$VENV_PYTHON" -m pip install -r server/requirements.txt -q fi if ! "$VENV_PYTHON" -c "import uvicorn" 2>/dev/null; then "$VENV_PYTHON" -m pip install --upgrade pip -q "$VENV_PYTHON" -m pip install -r server/requirements.txt -q fi source .venv/bin/activate echo "[start] Starting FastAPI (fix + voice) on :8000..." exec "$VENV_PYTHON" -m uvicorn server.main:app --host 0.0.0.0 --port 8000 ``` [34 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]