# Project export: OpenAssist This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: Talk to OpenAssist. It remembers your life, manages tasks and notes, sends agents across your Mac to find things and finish work, then speaks back the result. - Devpost: https://devpost.com/software/openassist - GitHub: https://github.com/manikv12/OpenAssist - Video: https://www.youtube.com/embed/8LF7_lxSysw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Manik Vahsith (1 commits) ## Devpost submission (written by the team) ### Inspiration Life moves fast. I have work, personal responsibilities, ideas, tasks, and projects all competing for attention. I kept losing track of what needed to be done, forgetting why a decision was made, or coming back to a project with no memory of where I stopped. I did not want another chatbot that starts from zero every time. I wanted a daily organizer I could talk to naturally: one that remembers my notes and projects, manages the things I need to do, and takes real work off my plate. Voice was essential because explaining a busy day out loud is much easier than typing every detail. That need became OpenAssist: a personal agent on my Mac that organizes my life, remembers context, and keeps work moving. I built it alone, and I use it every day. ### What it does OpenAssist brings live voice, a daily planner, projects, notes, long-term knowledge, and real agent work into one macOS app. Talk naturally from anywhere. A global shortcut starts Live Voice without making you stop what you are doing. Manage everyday life. Tasks, backlogs, reminders, daily focus, notes, and recurring work. Keep project context. Conversations, notes, decisions, and progress stay connected to projects, so the assistant can answer where a project stands and what is still needed. Remember useful history. It recalls completed conversations and saved knowledge instead of treating every session as a blank page. Delegate real work. When a request needs more than a quick answer, OpenAssist hands the task to a Codex agent, keeps the voice conversation going while it runs, and brings the finished result back into the same conversation. Use the computer itself. Through Codex Computer Use, the agent can open Mac apps and do the work directly. In the demo video it opens TextEdit and types a launch checklist by itself, recorded live. Analyze the screen. Screen Analysis reads whatever is on screen and helps on the spot: improve a draft, answer a question, insert text at the cursor. It is temporary by design: sessions are not kept in your notes, threads, or memory. Work across the Mac. With approval, it can use a signed-in browser and supported macOS apps such as Finder, Terminal, Calendar, Reminders, Contacts, Notes, Messages, and System Settings. Stay available on the phone. A companion iPhone app gives access to the planner, reminders, notes, projects, and assistant threads connected to the Mac. Keep the user in control. Important actions go through approval steps, and local settings, history, and memory stay on the Mac. A typical flow is conversational: I ask it to add something to my day, recall an older decision, check something on my computer, or delegate a longer task. It tells me when an agent is working, waits for the job to finish, and then speaks the result back. How I built it The engine underneath is the official Codex app server. OpenAssist runs it as the backend brain: chat turns, delegated agent jobs, and Computer Use all run through it, while Live Voice talks to the Realtime API directly. Computer Use comes from Codex's bundled plugin, so the same agent that answers questions can also operate the Mac. The desktop app is Electron, React, TypeScript, Node.js, and Vite. A native Swift helper handles the global keyboard shortcut and macOS-level voice and system integration. Live Voice connects to the OpenAI Realtime API (or Gemini Live) over WebSockets, and a routing layer decides whether a request can be answered directly, needs an OpenAssist tool, or should become a full Codex agent job. For longer tasks, OpenAssist creates the job, keeps the voice session informed while it runs, and returns the result as a natural spoken answer. Memory is deliberately small. Finished text conversations land in the Voice Log, and a short history is restored when a new session begins. Raw audio, partial transcripts, and provider events are never stored. The planner, reminders, notes, projects, approvals, and personal knowledge all live in the same app, and the iPhone companion (React Native and Expo) connects securely to the desktop. Challenges I ran into The hardest part was making voice feel like a dependable assistant instead of a voice skin on top of chat. I had to handle interruptions, microphone echo, repeated responses, reconnects, incomplete audio, and clear states for listening, speaking, delegating, and waiting. Agent handoff was tricky: the voice model must not guess an answer while a real agent is still working. OpenAssist pauses the answer, shows progress, waits for the delegated task, and then speaks the real result. Some bugs were humbling. At one point every Codex turn started silently returning nothing; my installed CLI had quietly fallen behind what the model required on the server side, and the error was swallowed. Another bug in my own cleanup code was killing live Computer Use helpers a few minutes after startup. Finding those taught me more about the app server's process model than any documentation. Computer access also needs trust, so conversation is separated from sensitive actions, and important changes ask for approval first. Accomplishments that I'm proud of A global voice shortcut that starts OpenAssist from any app. A complete voice-to-agent-to-spoken-result loop, shown with real footage in the demo video. Computer Use working end to end: ask, watch the agent operate a Mac app, and get the report back. Same-thread voice continuity across stop, start, and app restart. One connected system for tasks, reminders, notes, projects, knowledge, and agent work. A floating voice interface that shows listening, speaking, delegation, approvals, and errors without opening the main window. A phone companion that keeps the organizer available away from the desk. What I learned A useful personal assistant needs more than a capable model. It needs reliable state, carefully chosen memory, clear progress, safe approvals, and the patience to wait for real work to finish. Voice also changes the product. People speak with less structure than they type, so the system has to understand intent, preserve context, and make its current state obvious without interrupting the conversation. ### What's next Simpler setup and onboarding, better cross-device reliability, more skills and integrations, stronger project summaries and personal recall, and a more proactive daily organizer that still keeps the user in control. ## README (from the GitHub repository) # OpenAssist OpenAssist is a voice-first personal AI assistant for macOS and iPhone. It keeps daily tasks, reminders, notes, projects, and AI agent work in one place. ## Daily Workspace and WebMCP **Public judge demo:** [openassist-daily-workspace.developingadventures.chatgpt.site](https://openassist-daily-workspace.developingadventures.chatgpt.site/) **Public source:** [github.com/manikv12/OpenAssist](https://github.com/manikv12/OpenAssist) The new [`apps/workspace-site`](apps/workspace-site) is a professional browser workspace for Gmail, Tasks, Calendar, notes, memory, accounts, activity, and a synthetic Shopify supply catalog. Its private judge demo gives each judge a separate 24-hour synthetic Cloudflare workspace and cart pointer that they can safely edit through the UI or WebMCP. Owner-only Live mode remains connected to the existing OpenAssist Workspace MCP and Composio; private Google data is never copied into Demo mode. It exposes 29 judge-visible WebMCP tools that ChatGPT's in-app browser and the same-page voice agent can use. Live owner mode registers 14 additional Second Brain tools, for 43 shared definitions in the Site/voice contract. Every user or external write opens an exact, two-minute preview; destructive actions always require an on-screen tap. Private Google content is not copied into the Site database. The Shopify showcase can search products, return their images, inspect policies, and prepare a judge-isolated cart, but it never exposes checkout, payment, or order tools. ### Owner Second Brain The owner-only **Work** view keeps durable projects, research, decisions, future work, and agent results as indexed Markdown in the selected Google Drive. New content is organized under `OpenAssist/Second Brain`, while Cloudflare D1 stores only pointers, hashes, leases, policies, and sync status. Quick Capture can send an idea to the general Inbox or directly to a project; it does not create a Google Task until the work becomes an active personal action. After one approved assignment, an owner-controlled runner can discover queued work, claim it with a short lease, perform one isolated tool-free Codex analysis pass with bounded Second Brain search, and submit a context-based result. It can analyze, draft, and organize, but it cannot edit a repository or browse yet. Agent IDs are routing labels for one trusted owner orchestrator, not separate security identities. The voice gateway lives in [`apps/workspace-voice-gateway`](apps/workspace-voice-gateway). The Site opens on a private access screen; judge credentials are supplied only in the Devpost review instructions. A judge never needs Google access or an API key. When the owner enables it, judges can use a short project-funded synthetic voice session. ChatGPT subscription voice and Live mode remain owner-only. The server API key never enters the browser or a Container. See [the architecture and security boundaries](docs/workspace-architecture.md), [the judge testing guide](docs/webmcp-judge-guide.md), [the complete under-three-minute video script](docs/webmcp-challenge-demo.md), [the Devpost submission draft](docs/webmcp-devpost-submission.md), [the validation report](docs/webmcp-test-report.md), and [the official-requirements readiness map](docs/webmcp-challenge-readiness.md). ### What was added for the WebMCP Challenge OpenAssist existed before the challenge. The Daily Workspace is a new extension built after the August 25, 2026 challenge start. Starting with commit [`aff7044`](https://github.com/manikv12/OpenAssist/commit/aff7044), it added the ChatGPT Site, 29 real `document.modelContext.registerTool` tools, a separate synthetic judge workspace, visible approval previews, the live Workspace bridge, a safe Shopify catalog/cart showcase, and same-page voice-to-WebMCP control. The earlier desktop and Google Workspace features remain separate and unchanged. > **Primary app:** The supported desktop app is now the React + Electron app in [`apps/desktop`](apps/desktop). The old Swift app is archived in [`legacy/swift`](legacy/swift). ## Why I Built It Life moves fast. I had too many things to do, lost track of why decisions were made, and sometimes forgot work that had already happened. I built OpenAssist as a daily organizer that I can talk to. It can remember useful context, keep projects organized, manage tasks and reminders, and hand real computer work to an AI agent when a simple answer is not enough. ## What Makes It Different - **Live Voice coordinator:** Talk naturally through OpenAI Realtime or Gemini Live. - **Real agent delegation:** Browser, CLI, repository, logs, and computer tasks can be handed to Codex, Claude, or another selected worker. - **Conversation during work:** Voice chat stays available while delegated tasks run in the background. - **Reliable results:** OpenAssist tracks each task, reports progress, returns the final answer, and avoids duplicate work. - **Personal Knowledge:** Search notes, planner items, reminders, saved memory, and project history when Knowledge access is enabled. - **Daily organization:** Manage Today, backlog, categories, projects, tasks, due dates, and timed reminders. - **Notes and project continuity:** Keep long-running work connected to the right project and continue from earlier context. - **Approval controls:** Important writes and computer actions still use the existing approval rules. - **Global Live Voice:** A macOS shortcut can start the floating Live Voice HUD from any app. - **Phone companion:** The iPhone app can view and update the same planner, notes, reminders, and voice workflows through the desktop bridge. ## GPT-5.6 Build Week Work The desktop app includes **GPT-5.6 Sol**, **GPT-5.6 Terra**, and **GPT-5.6 Luna** as agent model choices. GPT-5.6 Sol was the daily driver for the real sessions behind this submission, including everything shown in the demo video. Build Week work (July 13 onward), checked against the commit history and my Codex session logs: - Made the React + Electron app the primary product and restructured the repository into `apps/desktop`, archiving the old Swift app under `legacy/swift`. The restructure itself was carried out by a Codex agent. - Updated Live Voice for the new layout: the global shortcut flow, the floating HUD, the Agent Work shelf, and same-thread continuity. - Built the demo from real sessions, recorded live: voice delegation to a Codex agent, and Computer Use opening TextEdit and typing a launch checklist by itself. - Fixed reliability bugs found through real daily use: a cleanup routine that was killing live Computer Use helpers, silent empty turns caused by an outdated Codex CLI, and streamed thinking text rendering with dropped words. - Used it for real work all week on GPT-5.6 Sol: planning notes with diagrams, research comparisons delegated from voice, reminder and memory recall, and filling a browser form through Computer Use. The Live Voice engineering in this build includes one-owner voice turn routing, a shared task registry with progress, cancellation and duplicate protection, same-thread continuity without saving raw audio, Knowledge routing for notes, reminders, planner data and memory, and the Live Voice HUD with task cards and completion notifications. OpenAI Realtime or Gemini Live handles the low-latency speech connection. GPT-5.6 is the work agent that reasons over the request and completes delegated tasks. ## How Codex Powers OpenAssist OpenAssist runs the official **Codex app server** (`codex app-server`) as its agent engine: - Chat turns and delegated agent jobs run through the app server, with GPT-5.6 (Sol, Terra, or Luna) as the reasoning model. - **Computer Use** comes from Codex's bundled plugin. The agent can open Mac apps and do the work directly; the demo video shows it opening TextEdit and typing a launch checklist by itself, recorded live. - Live Voice hands bigger requests to a Codex agent, keeps the conversation going while the job r [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 483 recognized source files, 23222 KB. - C (language) — detected in the code - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - AI coding agent: GitHub Copilot — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 599) ``` .github/workflows/publish-wiki.yml .github/workflows/release.yml .gitignore apps/desktop/.gitignore apps/desktop/.zcode/plans/plan-sess_c6970e3e-6750-4bfa-a9fb-6435f55309ed.md apps/desktop/bin/openassist-knowledge.mjs apps/desktop/COMPLETION_AUDIT.md apps/desktop/docs/computer-use-troubleshooting.md apps/desktop/docs/live-voice-architecture-plan.md apps/desktop/docs/mac-sync-plan.md apps/desktop/docs/perf-audit-idle.md apps/desktop/docs/voice-recall-architecture.md apps/desktop/electron/appleReminderRouting.ts apps/desktop/electron/claudeAgentCore.ts apps/desktop/electron/claudeAgentDriver.ts apps/desktop/electron/connectors.ts apps/desktop/electron/conversationHistoryStore.ts apps/desktop/electron/gitCheckpointService.ts apps/desktop/electron/helpers/app-activation-monitor.swift apps/desktop/electron/helpers/apple-eventkit-helper-entitlements.plist apps/desktop/electron/helpers/apple-eventkit-helper-info.plist apps/desktop/electron/helpers/apple-eventkit-helper.swift apps/desktop/electron/helpers/apple-speech-helper-info.plist apps/desktop/electron/helpers/apple-speech-helper.swift apps/desktop/electron/helpers/short-video-helper.swift apps/desktop/electron/helpers/shortcut-monitor-helper.swift apps/desktop/electron/helpers/text-inserter-helper.swift apps/desktop/electron/helpers/vision-background-helper.swift apps/desktop/electron/helpers/whisper-transcribe-helper.swift apps/desktop/electron/helpers/whisper.framework/Headers apps/desktop/electron/helpers/whisper.framework/Modules apps/desktop/electron/helpers/whisper.framework/Resources apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/ggml-alloc.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/ggml-backend.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/ggml-blas.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/ggml-cpu.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/ggml-metal.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/ggml.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/gguf.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Headers/whisper.h apps/desktop/electron/helpers/whisper.framework/Versions/A/Modules/module.modulemap apps/desktop/electron/helpers/whisper.framework/Versions/A/Resources/Info.plist apps/desktop/electron/helpers/whisper.framework/Versions/A/whisper apps/desktop/electron/helpers/whisper.framework/Versions/Current apps/desktop/electron/helpers/whisper.framework/whisper apps/desktop/electron/imageBackground.ts apps/desktop/electron/kokoroVoiceWorker.ts apps/desktop/electron/liveVoice/capabilityRegistry.ts apps/desktop/electron/liveVoice/codexSubscriptionCoordinator.ts apps/desktop/electron/liveVoice/codexSubscriptionRealtime.ts apps/desktop/electron/liveVoice/contracts.ts apps/desktop/electron/liveVoice/coordinator.ts apps/desktop/electron/liveVoice/informationResolver.ts apps/desktop/electron/liveVoice/narrationArbiter.ts apps/desktop/electron/liveVoice/providerAdapters.ts apps/desktop/electron/liveVoice/resultOutbox.ts apps/desktop/electron/liveVoice/state.ts apps/desktop/electron/liveVoice/trace.ts apps/desktop/electron/liveVoice/workerModelPolicy.ts apps/desktop/electron/liveVoice/workerResultGuard.ts apps/desktop/electron/liveVoice/workerToolPolicy.ts apps/desktop/electron/liveVoiceContinuity.ts apps/desktop/electron/macSyncCore.ts apps/desktop/electron/main.ts apps/desktop/electron/memoryDreamCore.ts apps/desktop/electron/nativeAccess.ts apps/desktop/electron/ollamaModelCatalog.ts apps/desktop/electron/ollamaRuntime.ts apps/desktop/electron/ollamaWebsiteModelCatalog.ts apps/desktop/electron/openAIRealtimeModels.ts apps/desktop/electron/openassistBridge.ts apps/desktop/electron/peerFilesCore.ts apps/desktop/electron/personalRecallCore.ts apps/desktop/electron/plannerStore.ts apps/desktop/electron/preload.ts apps/desktop/electron/realtimeInterruption.ts apps/desktop/electron/realtimeMcpHarness.ts apps/desktop/electron/realtimeProxy.ts apps/desktop/electron/realtimeTaskCoordinator.ts apps/desktop/electron/release-entitlements.plist apps/desktop/electron/release-info.plist apps/desktop/electron/shortVideo.ts apps/desktop/electron/sideChatCore.ts apps/desktop/electron/storageCleanup.ts apps/desktop/electron/voiceRouting.ts apps/desktop/icon.icns apps/desktop/index.html apps/desktop/package.json apps/desktop/PORT_PARITY_AUDIT.md apps/desktop/README.md apps/desktop/scripts/build-mac-release.sh apps/desktop/scripts/build-native-helpers.sh apps/desktop/scripts/capture-native-reference.mjs apps/desktop/scripts/capture-packaged-views.mjs apps/desktop/scripts/capture-running-window.mjs apps/desktop/scripts/measure-idle-electron.mjs apps/desktop/scripts/measure-startup-electron.mjs apps/desktop/scripts/routing-golden-set.mjs apps/desktop/scripts/verify-apple-eventkit.mjs apps/desktop/scripts/verify-apple-reminders-routing.mjs apps/desktop/scripts/verify-chat-engine.mjs apps/desktop/scripts/verify-claude-agent-sdk-packaged.mjs apps/desktop/scripts/verify-claude-agent-sdk.mjs apps/desktop/scripts/verify-claude-turn-recovery.mjs apps/desktop/scripts/verify-codex-image-mcp.mjs apps/desktop/scripts/verify-codex-image-worker.mjs apps/desktop/scripts/verify-codex-realtime-protocol.mjs apps/desktop/scripts/verify-codex-short-video-mcp.mjs apps/desktop/scripts/verify-codex-subscription-coordinator.mjs apps/desktop/scripts/verify-codex-subscription-realtime.mjs apps/desktop/scripts/verify-computer-use-helper-ownership.mjs apps/desktop/scripts/verify-connectors.mjs apps/desktop/scripts/verify-daily-digest.mjs apps/desktop/scripts/verify-dictation-latency.mjs apps/desktop/scripts/verify-external-fast-path.mjs apps/desktop/scripts/verify-git-checkpoint-service.mjs apps/desktop/scripts/verify-idle-cpu-guards.mjs apps/desktop/scripts/verify-image-background-mode.mjs apps/desktop/scripts/verify-insertion-smoke.mjs apps/desktop/scripts/verify-integration-merge.mjs [479 more files omitted for size] ``` ### Dependencies - apps/desktop/package.json: @anthropic-ai/claude-agent-sdk@0.3.215, @electron/packager@^20.0.0, @google/genai@^2.6.0, @huggingface/transformers@^3.8.1, @liquid-dom/react@^0.1.0, @modelcontextprotocol/sdk@^1.29.0, @tiptap/extension-code-block-lowlight@3.23.4, @tiptap/extension-image@^3.23.4, @tiptap/extension-placeholder@^3.23.4, @tiptap/extension-table@^3.23.4, @tiptap/extension-task-item@^3.23.4, @tiptap/extension-task-list@^3.23.4, @tiptap/markdown@^3.23.4, @tiptap/pm@^3.23.4, @tiptap/react@^3.23.4, @tiptap/starter-kit@^3.23.4, @types/node@^25.0.1, @types/qrcode@^1.5.6, @types/react@^19.2.7, @types/react-dom@^19.2.3, @types/react-syntax-highlighter@^15.5.13, @types/ws@^8.18.1, @vitejs/plugin-react@^5.1.2, concurrently@^9.2.1, cross-env@^10.1.0, electron@^41.2.0, framer-motion@^12.23.24, highlight.js@^11.11.1, kokoro-js@^1.2.1, lowlight@^3.3.0, lucide-react@^0.561.0, mermaid@^11.15.0, qrcode@^1.5.4, react@^19.2.3, react-dom@^19.2.3, react-markdown@^10.1.0, react-syntax-highlighter@^15.6.6, remark-gfm@^4.0.1, rollup-plugin-visualizer@^7.0.1, sharp@^0.34.5, smol-toml@^1.7.0, typescript@^5.9.3, vite@^7.3.0, wait-on@^9.0.3, ws@^8.20.1, zod@^4.0.0 - legacy/swift/web/chat/package.json: @panzoom/panzoom@^4.6.1, @tiptap/extension-code-block-lowlight@^3.21.0, @tiptap/extension-placeholder@^3.21.0, @tiptap/extension-table@^3.21.0, @tiptap/extension-task-item@^3.21.0, @tiptap/extension-task-list@^3.21.0, @tiptap/markdown@^3.21.0, @tiptap/pm@^3.21.0, @tiptap/react@^3.21.0, @tiptap/starter-kit@^3.21.0, @types/react@^19.1.0, @types/react-dom@^19.1.0, @types/react-syntax-highlighter@^15.5.13, @vitejs/plugin-react@^4.4.1, lowlight@^3.1.0, lucide-react@^1.7.0, mermaid@^11.13.0, react@^19.1.0, react-collapsed@^4.2.0, react-dom@^19.1.0, react-markdown@^10.1.0, react-syntax-highlighter@^15.6.1, remark-gfm@^4.0.1, typescript@^5.8.3, vite@^6.3.5, vite-plugin-singlefile@^2.0.3 - legacy/swift/web/remote/package.json: @types/react@^19.1.0, @types/react-dom@^19.1.0, @types/react-syntax-highlighter@^15.5.13, @vitejs/plugin-react@^4.4.1, react@^19.1.0, react-dom@^19.1.0, react-markdown@^10.1.0, react-syntax-highlighter@^15.6.1, remark-gfm@^4.0.1, typescript@^5.8.3, vite@^6.3.5, vite-plugin-singlefile@^2.0.3 ### Recent commits (newest first) - Merge pull request #23 from manikv12/agent/openassist-workspace-update - Expand OpenAssist desktop capabilities - Merge pull request #22 from manikv12/agent/openassist-live-voice-update - Prepare OpenAssist Build Week release - Make Electron app primary and update Live Voice - Merge pull request #21 from manikv12/codex/harden-mac-sync - Harden Mac sync - Merge pull request #20 from manikv12/mVashith/live-voice-shortcut-hud - Improve Live Voice shortcut and app integrations - Fix TipTap peer dependency conflict blocking release build - Merge pull request #19 from manikv12/manikv12-patch-1 - Delete .gemini/antigravity/brain/62b05cc5-9aaf-4a0c-bd3a-0a980c258e6e/scratch directory - Merge pull request #18 from manikv12/mVashith/new-features - Add OpenAssist feature updates - Add new feature updates - Merge pull request #17 from manikv12/mVashith/electron-framework-signing - Sign Electron framework internals - Merge pull request #16 from manikv12/mVashith/electron-sign-all-mach-o - Sign all Electron native resources - Merge pull request #15 from manikv12/mVashith/electron-notarization-signing ## Key source files (fetched from GitHub, selected and truncated for size) ### compressed-rolling-bengio.review.md ```markdown ``` Memory Dreaming — background memory for OpenAssist agents ``` ## Context The user wants Codex/ChatGPT-style memory for OpenAssist's own agents. Codex CLI (inspected locally, v0.144.4) runs a two-phase local pipeline: per-thread raw memories distilled after turns, then consolidation into `MEMORY.md` + a synthesized `memory_summary.md` user profile. ChatGPT's "Dreaming" works similarly. OpenAssist must additionally remember **notes and planner/tasks** activity, which Codex doesn't have. Codex's UI shows **memory citations** when a memory slice was injected into a turn (e.g. "1 memory citation — MEMORY.md lines 783-839") — replicate that visibility: since we choose which memory snippets to inject per turn, show them as a small "N memories" activity chip on that turn. Confirmed decisions: dreaming model = **Codex subscription responses endpoint** (same as Daily Digest); cadence = **"however Codex does it" — triggered after each completed turn** (debounced + fingerprint-gated so quiet threads cost nothing), consolidation piggybacking on stage-1 completions (≥3 pending or day rollover). Injection: profile in new-session instructions, fast per-turn relevance block for ALL backends + Live Voice, citation chip in UI. Settings toggle + "Consolidate now". All work in `apps/desktop`. Bridge is ~38k lines — **anchor edits by unique verbatim strings (copy whitespace exactly; some regions are tab-indented), never line numbers**. Electron edits: `tsc -p tsconfig.electron.json` + restart. Renderer: `npm run build` only (renderer tsc pre-broken). Reused machinery (verified): memory store `saveAssistantMemory`/`readAssistantMemoryIndex`/`memoryStoreRoot` (~~:11595/:11647/:11517); session digests `appendSessionDigest` written after every completed turn at anchor `if (!isTransient && userSource !== "realtimeVoice") {` + `appendSessionDigest({` (~~:33916; **verify-memory-and-context.mjs :129 regex-matches this exact sequence — new code goes AFTER the call, inside the if**); LLM pass template `generatePlannerDailyDigest` :11057 / `generateLiveVoiceSessionSummary` :11288 (endpoint `https://chatgpt.com/backend-api/codex/responses`, `resolveCodexTranscriptionAuthContext`, `codexSafeModel`, `readCodexResponsesText`, outermost-{} regex, `fetchWithTimeout(...,90000)`); debounce template `queueLiveVoiceSessionSummary` :11259; circuit breaker `sparkRecallModelUnavailableUntil` :16959; fast search `searchKnowledgeTimeline(query, {sourceTypes, limit})` :16179 (SQLite LIKE, `rank` column; memories AND digests share `sourceType: "assistant_memory"`); per-turn seam anchor `runtimePrompt = promptWithAgentFilesContext(runtimePrompt, session, agentFiles.path);` (~:37015, after the direct-knowledge early return); new-session anchor `...assistantMemoryIndexInstructionSection()` :30007; synthetic-activity template = image-worker activity ~:32966 (unknown activityKind renders fine; `Brain` icon already imported in App.tsx); voice instructions `coordinatorRealtimeInstructions(code [truncated — 12511 more characters] ``` ### apps/desktop/COMPLETION_AUDIT.md ```markdown # Completion Audit Objective: build a React + Electron version of OpenAssist inside the current OpenAssist repo and make it a close one-to-one functional and visual port. Computer Use inspection/testing was originally required, but the user later explicitly allowed skipping Computer Use for now because the local tool transport is broken. ## Status Complete for the Electron port pass, with documented caveats: - Computer Use was skipped per the user's later instruction because the local Computer Use transport returns `Transport closed`. - Local `whisper.cpp` transcription is now wired through the vendored native `whisper.framework` and the shared native model folder. This machine currently has no installed `ggml-*.bin` model, so the verifier can prove the engine is selected/handled and the app shows a real "install a model first" path, but it cannot prove real speech-to-text output from a local model on this machine. The Electron app builds, packages, and passes the packaged verifier. ## Prompt-To-Artifact Checklist | User requirement | Evidence in this folder | Verification | Status | | --- | --- | --- | --- | | Make a React-based Electron app inside the current OpenAssist repo. | `package.json`, `electron/`, `src/`, `vite.config.ts`, `tsconfig.electron.json` | `npm run build`, `npm run verify:packaged` | Covered | | Look at the original app first using Computer Use. | Native code inspection and user screenshots are documented in `PORT_PARITY_AUDIT.md`; `capture:native` can capture native windows through CoreGraphics. | User explicitly allowed skipping Computer Use for now; official tool still fails with `Transport closed`. | Waived for this pass | | Match the original look, transparency, icons, and professional styling. | `src/styles.css`, `src/assets/provider-marks/*.svg`, `icon.icns` | Packaged screenshots from `npm run capture:packaged`; packaged icon hash check in `verify-packaged-electron.mjs` | Covered for current pass | | Copy provider icons. | `src/assets/provider-marks/codex.svg`, `copilot.svg`, `claude.svg`, `ollama.svg` | Asset existence/size checks in `verify-running-electron.mjs` | Covered | | Main provider dropdown should only show native runtime providers. | `runtimeProviderOptions` in `src/App.tsx`; backend guard in `electron/openassistBridge.ts` | Verifier expects only `Codex`, `Copilot`, `Claude`, `Ollama`; rejects OpenAI/Groq/Grok/etc. | Covered | | Other providers should remain in their proper settings areas. | Prompt rewrite and cloud transcription settings in `SettingsView`; Keychain/defaults bridge in `openassistBridge.ts` | Settings and provider-key round trips in verifier | Covered | | Provider change should affect color/status. | Provider CSS variables in `styles.css`; provider status text in `App.tsx` | `providerBrandColors` verifier checks native hues | Covered | | Model selector belongs at the bottom near the model name, not top. | Composer model popover in `src/App.tsx`; top bar open/editor menu | `composerMod [truncated — 4729 more characters] ``` ### package.json ``` { "name": "openassist", "version": "0.1.0", "private": true, "description": "Voice-first personal assistant for macOS and iPhone.", "engines": { "node": "^20.19.0 || >=22.12.0" }, "scripts": { "setup": "npm --prefix apps/desktop install", "dev": "npm --prefix apps/desktop run dev", "build": "npm --prefix apps/desktop run build", "start": "npm --prefix apps/desktop run start", "package:mac": "npm --prefix apps/desktop run package:mac", "release:mac": "npm --prefix apps/desktop run release:mac", "verify:live-voice": "npm --prefix apps/desktop run verify:live-voice-architecture && npm --prefix apps/desktop run verify:live-voice-provider-contracts && npm --prefix apps/desktop run verify:live-voice-worker-models && npm --prefix apps/desktop run verify:native-permissions && npm --prefix apps/desktop run verify:apple-eventkit && npm --prefix apps/desktop run verify:realtime-protocol && npm --prefix apps/desktop run verify:codex-subscription-realtime && npm --prefix apps/desktop run verify:codex-subscription-coordinator && npm --prefix apps/desktop run verify:live-voice-planner-actions && npm --prefix apps/desktop run verify:openai-realtime-models && npm --prefix apps/desktop run verify:realtime-interruption && npm --prefix apps/desktop run verify:live-voice-echo-guard && npm --prefix apps/desktop run verify:live-voice-continuity && npm --prefix apps/desktop run verify:live-voice-orchestration && npm --prefix apps/desktop run verify:live-voice-retrieval && npm --prefix apps/desktop run verify:single-speaker && npm --prefix apps/desktop run verify:live-voice-history && npm --prefix apps/desktop run verify:memory-dream && npm --prefix apps/desktop run verify:memory-pipeline && npm --prefix apps/desktop run verify:memory-scope", "verify": "npm run build && npm run verify:live-voice" } } ``` ### apps/desktop/package.json ``` { "name": "openassist-desktop", "productName": "Open Assist", "version": "0.1.0", "private": true, "description": "The primary OpenAssist desktop app, built with React and Electron.", "main": "dist-electron/main.js", "type": "module", "engines": { "node": "^20.19.0 || >=22.12.0" }, "bin": { "openassist-knowledge": "./bin/openassist-knowledge.mjs" }, "scripts": { "dev": "concurrently -k \"npm:dev:renderer\" \"npm:dev:electron\"", "dev:renderer": "vite --host 127.0.0.1 --port 5187", "dev:electron": "wait-on tcp:5187 && tsc -p tsconfig.electron.json && cross-env VITE_DEV_SERVER_URL=http://127.0.0.1:5187 electron .", "build": "tsc -p tsconfig.electron.json && vite build", "package:mac": "OPENASSIST_SKIP_DMG=1 bash scripts/build-mac-release.sh", "release:mac": "bash scripts/build-mac-release.sh", "verify:running": "node scripts/verify-running-electron.mjs", "verify:packaged": "npm run package:mac && node scripts/verify-packaged-electron.mjs && node scripts/verify-knowledge-bin-bundled.mjs && node scripts/verify-claude-agent-sdk-packaged.mjs && node scripts/verify-insertion-smoke.mjs", "verify:knowledge-bin": "node scripts/verify-knowledge-bin-bundled.mjs", "verify:claude-agent-sdk": "tsc -p tsconfig.electron.json && node scripts/verify-claude-agent-sdk.mjs", "verify:git-checkpoints": "npm run build && node scripts/verify-git-checkpoint-service.mjs", "verify:realtime-protocol": "node scripts/verify-codex-realtime-protocol.mjs", "verify:codex-subscription-realtime": "tsc -p tsconfig.electron.json && node scripts/verify-codex-subscription-realtime.mjs", "verify:codex-subscription-coordinator": "tsc -p tsconfig.electron.json && node scripts/verify-codex-subscription-coordinator.mjs", "verify:openai-realtime-models": "tsc -p tsconfig.electron.json && node scripts/verify-openai-realtime-models.mjs", "verify:openai-realtime-models:live": "tsc -p tsconfig.electron.json && node scripts/verify-openai-realtime-models-live.mjs", "verify:realtime-interruption": "tsc -p tsconfig.electron.json && node scripts/verify-realtime-interruption.mjs", "verify:realtime-local-mcp": "tsc -p tsconfig.electron.json && node scripts/verify-realtime-local-mcp.mjs", "verify:realtime-router": "node scripts/verify-realtime-router.mjs", "verify:realtime-tool-gate": "node scripts/verify-realtime-tool-gate.mjs", "verify:realtime-handoff-narration": "node scripts/verify-realtime-handoff-narration.mjs", "verify:voice-routing": "node scripts/verify-voice-routing.mjs", "verify:live-voice-echo-guard": "node scripts/verify-live-voice-echo-guard.mjs", "verify:live-voice-continuity": "node scripts/verify-live-voice-continuity.mjs", "verify:live-voice-orchestration": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-orchestration.mjs", "verify:live-voice-retrieval": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-retrieval.mjs", "verify:single-speaker": "tsc -p tsconfig.electron.json && node scripts/verify-single-speaker.mjs", "verify:live-voice-history": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-history.mjs", "verify:live-voice-architecture": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-architecture.mjs", "verify:live-voice-provider-contracts": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-provider-contracts.mjs", "verify:live-voice-worker-models": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-worker-models.mjs", "verify:realtime-voice-guards": "node scripts/verify-realtime-voice-guards.mjs", "verify:parallel-delegation": "node scripts/verify-parallel-delegation.mjs", "verify:codex-image-worker": "node scripts/verify-codex-image-worker.mjs", "verify:codex-image-mcp": "node scripts/verify-codex-image-mcp.mjs", "verify:codex-short-video-mcp": "tsc -p tsconfig.electron.json && node scripts/verify-codex-short-video-mcp.mjs", "verify:image-background-mode": "tsc -p tsconfig.electron.json && node scripts/verify-image-background-mode.mjs", "verify:storage-cleanup": "tsc -p tsconfig.electron.json && node scripts/verify-storage-cleanup.mjs", "verify:apple-eventkit": "node scripts/verify-apple-eventkit.mjs", "verify:apple-reminders-routing": "tsc -p tsconfig.electron.json && node scripts/verify-apple-reminders-routing.mjs", "verify:voice-arguments-protocol": "tsc -p tsconfig.electron.json && node scripts/verify-voice-arguments-protocol.mjs", "verify:live-voice-planner-actions": "tsc -p tsconfig.electron.json && node scripts/verify-live-voice-planner-actions.mjs", "verify:live-voice-rotation": "node scripts/verify-live-voice-rotation.mjs", "verify:live-voice-summary": "node scripts/verify-live-voice-summary.mjs", "verify:open-loops": "node scripts/verify-open-loops.mjs", "verify:write-readback": "node scripts/verify-write-readback.mjs", "verify:worker-guardrails": "node scripts/verify-worker-guardrails.mjs", "verify:routing-golden": "tsc -p tsconfig.electron.json && node scripts/verify-routing-golden.mjs", "verify:native-permissions": "tsc -p tsconfig.electron.json && node scripts/verify-native-permissions.mjs", "verify:external-fast-path": "node scripts/verify-external-fast-path.mjs", "verify:idle-cpu-guards": "node scripts/verify-idle-cpu-guards.mjs", "verify:computer-use-ownership": "node scripts/verify-computer-use-helper-ownership.mjs", "verify:chat-engine": "node scripts/verify-chat-engine.mjs", "verify:dictation-latency": "node scripts/verify-dictation-latency.mjs", "verify:memory": "node scripts/verify-memory-and-context.mjs", "verify:memory-dream": "tsc -p tsconfig.electron.json && node scripts/verify-memory-dream.mjs", "verify:memory-pipeline": "tsc -p tsconfig.electron.json && node scripts/verify-memory-pipeline.mjs", "verify:memory-scope": "tsc -p tsconfig.electron. [truncated — 3882 more characters] ``` ### legacy/swift/web/remote/package.json ``` { "name": "openassist-remote", "private": true, "version": "1.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "deploy": "vite build && cp dist/index.html ../../Resources/remote-access.html" }, "dependencies": { "react": "^19.1.0", "react-dom": "^19.1.0", "react-markdown": "^10.1.0", "react-syntax-highlighter": "^15.6.1", "remark-gfm": "^4.0.1" }, "devDependencies": { "@types/react": "^19.1.0", "@types/react-dom": "^19.1.0", "@types/react-syntax-highlighter": "^15.5.13", "@vitejs/plugin-react": "^4.4.1", "typescript": "^5.8.3", "vite": "^6.3.5", "vite-plugin-singlefile": "^2.0.3" } } ``` ### legacy/swift/web/chat/package.json ``` { "name": "openassist-chat", "private": true, "version": "1.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "deploy": "vite build && cp dist/index.html ../../Resources/markdown-chat.html" }, "dependencies": { "@panzoom/panzoom": "^4.6.1", "@tiptap/extension-code-block-lowlight": "^3.21.0", "@tiptap/extension-placeholder": "^3.21.0", "@tiptap/extension-table": "^3.21.0", "@tiptap/extension-task-item": "^3.21.0", "@tiptap/extension-task-list": "^3.21.0", "@tiptap/markdown": "^3.21.0", "@tiptap/pm": "^3.21.0", "@tiptap/react": "^3.21.0", "@tiptap/starter-kit": "^3.21.0", "lowlight": "^3.1.0", "lucide-react": "^1.7.0", "mermaid": "^11.13.0", "react": "^19.1.0", "react-collapsed": "^4.2.0", "react-dom": "^19.1.0", "react-markdown": "^10.1.0", "react-syntax-highlighter": "^15.6.1", "remark-gfm": "^4.0.1" }, "devDependencies": { "@types/react": "^19.1.0", "@types/react-dom": "^19.1.0", "@types/react-syntax-highlighter": "^15.5.13", "@vitejs/plugin-react": "^4.4.1", "typescript": "^5.8.3", "vite": "^6.3.5", "vite-plugin-singlefile": "^2.0.3" } } ``` ### apps/desktop/src/main.tsx ```typescript import React from "react"; import ReactDOM from "react-dom/client"; import { App } from "./App"; import "./styles.css"; ReactDOM.createRoot(document.getElementById("root")!).render( ); ``` ### legacy/swift/web/chat/src/main.tsx ```typescript import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import { App } from "./App"; import "./styles.css"; createRoot(document.getElementById("root")!).render( ); ``` ### legacy/swift/web/remote/src/main.tsx ```typescript import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import { App } from "./App"; import "./styles.css"; createRoot(document.getElementById("root")!).render( ); ``` ### legacy/swift/web/chat/src/App.tsx ```typescript import { startTransition, useCallback, useEffect, useMemo, useRef, useState } from "react"; import { ChatView } from "./components/ChatView"; import { ComposerView } from "./components/ComposerView"; import { FindBar } from "./components/FindBar"; import { SidebarView } from "./components/SidebarView"; import { ThreadNoteDrawer } from "./components/ThreadNoteDrawer"; import { useTextSelection } from "./hooks/useTextSelection"; import type { ActiveWorkState, ActiveTurnState, AssistantComposerActivityState, AssistantComposerControlsState, AssistantComposerState, AssistantSidebarState, ChatMessage, ChatStreamEvent, CodeReviewPanelState, MessageCheckpointInfo, ProviderTone, RewindState, RuntimePanelState, ThreadNoteState, TypingState, } from "./types"; const HISTORY_TRUNCATE_TRANSITION_MS = 110; const SESSION_SWAP_ENTER_TAIL_COUNT = 4; const SESSION_SWAP_ANIMATION_MESSAGE_LIMIT = 24; type AppViewMode = "chat" | "sidebar" | "composer"; function normalizeViewMode(mode?: string | null): AppViewMode { if (mode === "sidebar") return "sidebar"; if (mode === "composer") return "composer"; return "chat"; } function initialViewMode(): AppViewMode { return normalizeViewMode(window.__OPENASSIST_INITIAL_VIEW_MODE); } function clearMessageTransitionState(message: ChatMessage): ChatMessage { if (!message.transitionState) { return message; } const { transitionState, ...stableMessage } = message; return stableMessage; } function stableMessages(messages: ChatMessage[]): ChatMessage[] { return messages .filter((message) => message.transitionState !== "removing") .map(clearMessageTransitionState); } function providerTone(value?: string | null): ProviderTone { const normalized = (value || "").trim().toLowerCase(); if (normalized.includes("copilot")) return "copilot"; if (normalized.includes("codex")) return "codex"; if (normalized.includes("claude") || normalized.includes("anthropic")) return "claude"; return "default"; } function dedupeMessages(messages: ChatMessage[]): ChatMessage[] { const seen = new Set(); const deduped: ChatMessage[] = []; for (let i = messages.length - 1; i >= 0; i--) { if (!seen.has(messages[i].id)) { seen.add(messages[i].id); deduped.unshift(clearMessageTransitionState(messages[i])); } } return deduped; } function upsertMessageInOrder( messages: ChatMessage[], message: ChatMessage, afterMessageID?: string ): ChatMessage[] { const stable = stableMessages(messages); const nextMessage = clearMessageTransitionState(message); const withoutExisting = stable.filter((entry) => entry.id !== nextMessage.id); if (!afterMessageID) { return [...withoutExisting, nextMessage]; } const anchorIndex = withoutExisting.findIndex((entry) => entry.id === afterMessageID); if (anchorIndex < 0) { return [...withoutExisting, nextMessage]; } return [ ...withoutExisting.slice(0, anchorIndex + 1), nextMessage, ...withoutExisting.slice(anchorIndex + 1), ]; } function applyChatStreamEvents( previousMessages: ChatMessage[], previousActiveWork: ActiveWorkState | null, previousTyping: TypingState, previousActiveTurn: ActiveTurnState | null, events: ChatStreamEvent[] ): { messages: ChatMessage[]; activeWork: ActiveWorkState | null; typing: TypingState; activeTurn: ActiveTurnState | null; } { let messages = stableMessages(previousMessages); let activeWork = previousActiveWork; let typing = previousTyping; let activeTurn = previousActiveTurn; for (const event of events) { switch (event.kind) { case "replaceMessages": messages = dedupeMessages(event.messages); break; case "responseTextDelta": messages = messages.map((message) => message.id === event.messageID ? { ...message, text: event.text, isStreaming: event.isStreaming } : message ); break; case "upsertMessage": messages = upsertMessageInOrder(messages, event.message, event.afterMessageID); break; case "removeMessage": messages = messages.filter((message) => message.id !== event.messageID); break; case "setActiveWorkState": activeWork = event.state; break; case "setTypingState": typing = event.state; break; case "setActiveTurnState": activeTurn = event.state; break; } } return { messages, activeWork, typing, activeTurn }; } function buildTruncationTransition( previous: ChatMessage[], next: ChatMessage[] ): ChatMessage[] | null { if (next.length === 0 || next.length >= previous.length) { return null; } const keepsPrefix = next.every((message, index) => previous[index]?.id === message.id); if (!keepsPrefix) { return null; } const removedTail = previous.slice(next.length).map((message) => ({ ...clearMessageTransitionState(message), transitionState: "removing" as const, })); if (removedTail.length === 0) { return null; } return [...next.map(clearMessageTransitionState), ...removedTail]; } function buildExpansionTransition( previous: ChatMessage[], next: ChatMessage[] ): ChatMessage[] | null { if (previous.length === 0 || next.length <= previous.length) { return null; } const keepsPrefix = previous.every((message, index) => next[index]?.id === message.id); if (!keepsPrefix) { return null; } const addedTail = next.slice(previous.length).map((message) => ({ ...clearMessageTransitionState(message), transitionState: "entering" as const, })); if (addedTail.length === 0) { return null; } return [...previous.map(clearMessageTransitionState), ...addedTail]; } function buildSessionSwapTransition( previous: ChatMessage[], next: ChatMessage[] ): ChatMessage[] | null { if (next.length === 0) { return null; } if (previous.length > 0) { const nextIDs = new Set(next.map((mess [truncated — 26537 more characters] ``` [425 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]