# Project export: LyrnOS This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: An AI tutor that reads your handwritten whiteboard, runs live Physics & Chemistry lab simulations, and coaches you through past papers. Grades 7–12, 8 subjects. - Devpost: https://devpost.com/software/lyrnos - GitHub: https://github.com/Tanweer-Ahmed-Chiktay/Lyrn_OS - Demo: https://lyrnos.vercel.app/ - Video: https://www.youtube.com/embed/34ypBbprOPs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Tanweer-Ahmed-Chiktay (1 commits) ## Devpost submission (written by the team) ### Inspiration Grade 12 is a year that defines a lot in South Africa. Your matric results follow you. I remember working through past papers a lot, they were some of the most useful thing I had. But what struck me was that students in the same grade, writing the same exam, were having completely different experiences depending on which class they landed in. A student with a great maths teacher had something the others didn't. A student whose parents could afford a private tutor at R300 an hour had something else entirely. The ones who couldn't? Googling, watching YouTube, hoping to stumble onto the right explanation before the test. Circle geometry clicked for me not from a textbook but because a teacher walked through a diagram with me, pointed at the angle at the centre, the angle at the circumference, showed me why one is twice the other using a drawing right in front of me. That is what tutoring actually is. Not a content library. Someone who watches you work and responds to your specific mistake. Most AI chat apps break the moment you need to show your working. You draw on paper, photograph it, upload it, wait. The tutor feel is gone. And for students whose schools don't run proper chemistry or physics practicals, schools that can't maintain equipment, that don't have the budget, there's a whole category of exam question they go into blind. Lyrn exists to close both gaps. The tutoring gap and the lab gap. The idea that where you were born or what school your parents could afford should determine how well you understand electrostatics or titration curves is something that is genuinely worth trying to fix. ### What it does Most AI tutoring tools respond to what you type. Mira responds to what you draw. At the centre of Lyrn is Mira — a Socratic AI tutor with four modes of working with a student: live whiteboard, chat with inline visual diagrams, voice, and autonomous lab operation. The whiteboard The shared canvas uses Excalidraw. When a student sketches a circle theorem, a force body diagram, or the working for a quadratic, Mira reads it using real computer vision — the canvas is exported to PNG and sent to Claude's vision API, not described in text. Mira sees the actual drawing. She can write back on the board, annotate the student's diagram, add labels, draw new figures. A student who draws the wrong angle relationship sees Mira annotate directly on their figure showing where the reasoning broke. When Mira is teaching on the whiteboard, her board writes are timed to her speech. If she is explaining that the angle at the centre is twice the angle at the circumference, the step she is writing appears on the board at the moment she says it — not before (which gives away the answer) and not after (which breaks the flow). It works the same way a person at a whiteboard actually teaches. If Mira is mid-sentence and the student has figured it out, they can interrupt. She stops speaking. The student records their explanation. Mira listens and responds to what was actually said, not what she was about to say. Chat with inline visual diagrams For theory subjects — Accounting, History, Geography, Computer Science — Mira teaches through the chat interface and embeds live visual diagrams directly inline in the conversation. These are not static images: they are rendered SVG and HTML diagrams, generated by the model and drawn directly inside the chat using fenced diagram blocks. A lesson on double-entry bookkeeping produces an actual T-account diagram. A lesson on the causes of World War One produces an annotated timeline. A lesson on data structures produces a visual tree or linked list rendered in real time as Mira explains it. Students see the concept and the visual at the same moment they read the explanation. The hidden prompt design means students never see the instruction Mira is given to start a lesson — they see only Mira beginning to teach, naturally, as if she chose to explain it that way herself. Live lesson teaching When a student starts a lesson on factorising or the theorem of Pythagoras, Mira does not hand them an explanation. She teaches. She plans a worked example, writes each step on the board while speaking it aloud, then poses a practice problem and waits for the student to attempt it. She reads their working off the board using vision, confirms it or identifies exactly where the reasoning broke down, and either moves forward or reteaches. Voice synthesis makes her sound like a person — not a TTS robot — and the interrupt/resume flow means the student can take over at any point. Physics and chemistry labs Students who attend schools without functioning science labs can describe an experiment to Mira. She will launch a real PhET physics simulation — a browser-based simulation used by universities worldwide — and operate it autonomously on the student's behalf. The agent builds circuits, configures wave interference setups, and adjusts parameters while narrating what it is doing in real time. For chemistry, there is a full titration laboratory: a student adds sodium hydroxide to hydrochloric acid drop by drop, watches a live equivalence curve computed from real stoichiometric equations, and has Mira explain what is happening at each inflection point. The circuit lab runs a physics solver built from scratch: Modified Nodal Analysis with Gaussian elimination. It stamps conductance into a matrix, solves for node voltages and branch currents, handles LDR resistance as a function of light level, blocks current through a reverse-biased LED, detects short circuits, and excludes dangling components from the solve. Parallel circuits carry different currents in different branches because the physics says they should, not because of a lookup table. Adaptive tests and cognitive graph Tests are generated per grade, per subject, across all eight subjects in the CAPS and IEB curricula: Mathematics, Physical Sciences, Life Sciences, English, Accounting, History, Geography, and Computer Science. Results update a cognitive graph — a force-directed node map, with real repulsion and attraction physics in the layout engine — that shows mastery per concept, colour-coded by subject, updated from actual quiz and lesson data. A student can see which nodes are solid and which are gaps before an exam. Everything else A command palette gives keyboard access to every feature. A learning paths view organises lessons by subject and grade. A past papers workspace lets students work through official exam papers with Mira available to help at any step. Grade selection updates the entire experience — lesson content, test difficulty, and curriculum alignment all shift when a student changes their grade from Grade 9 to Grade 12. ### How we built it Frontend The application is a single-page app with no frontend framework — intentional, not a gap. A real-time tutoring experience coordinates live audio, streaming AI text, timed board writes, SSE consumers, and animated graph layouts simultaneously. Explicit control over the event loop makes that coordination cleaner than working against a reconciler. The codebase is 3,600 lines of JavaScript with 249 named functions and a 5,300-line CSS design system covering nine distinct views. The entry bundle is 372KB gzipped. Clerk's 2.8MB authentication SDK is dynamically imported only when authentication is needed. The PhET lab module is a separate chunk loaded on demand. This matters for students on mobile data. Backend and AI gateway The Node.js server exposes a multi-provider AI gateway. Requests fall through from AWS Bedrock (primary) to the direct Anthropic API to Groq as a final fallback — in the same order for both streaming and non-streaming callers. If a lesson is mid-flow and the primary provider has a latency spike, the gateway recovers without the student seeing it. Supabase stores all learning data — events, sessions, assessments, chat history — with row-level security enforced at the database level. Every table policy keys on the authenticated user's ID. Clerk handles authentication. The PhET lab agent The lab agent is 1,600 lines of Python. It uses Browser-Use — an AI agent framework for web automation — to drive a Chromium browser with real keyboard and mouse events. In production it attaches to a Steel-managed cloud browser via Chrome DevTools Protocol (CDP): Steel maintains the browser session and the agent connects to it remotely. In local development it attaches to a locally-running Chromium instance instead, using the same CDP interface — the agent code is identical in both paths. The cloud session runs on ECS Fargate, provisioned on demand when a student opens the lab. There is no idle capacity. The gate requires PHET_ECS_ENABLED=true in the environment and validates that the configured cluster name contains lyrnos before any RunTask call. Sessions are hard-capped at seven minutes with automatic cleanup — Steel sessions are closed, ECS tasks are stopped, and the student's lab instance is marked released. Before touching the simulation canvas, the agent plans the complete build — what components are needed, in what order, where they connect. It identifies terminal positions from a screenshot using vision rather than hard-coded coordinates, because terminal positions shift with zoom level and component orientation. It verifies the completed build from a fresh screenshot before reporting success. Every action is logged so a failure is diagnosable. The trust boundary Board images are validated on the way in: PNG only, 3.75MB ceiling, 120-element snapshot cap, annotation type allowlist. Mira's reading of a board is treated as a candidate interpretation — a separate deterministic solver checks any equations independently. If they disagree, Mira acknowledges uncertainty rather than teaching confidently from a wrong premise. This prevents a class of hallucination that would be genuinely harmful in an educational context: a student writes a wrong answer, the model misreads it as correct, and confidently explains why they got it right. That cannot happen in Lyrn. The trust boundary is covered by 15 unit tests that run on every change. Codex collaboration We used Codex throughout, switching between Luna and Terra based on what each task actually needed. Luna for incremental edits, CSS iteration, and UI fixes where speed mattered more than deep cross-file context. Terra for the multi-file architectural passes — the voice state machine, the ECS safety gate, the trust boundary layer, the streaming gateway — where holding complex invariants across files in context mattered. Working within a credit budget forced the discipline to make that distinction deliberately rather than defaulting to the most capable model for everything. ### Challenges we ran into Timing board writes to speech was the first hard problem. The lesson plan comes back as a sequence of steps with associated board actions. Each written element must appear at the moment Mira finishes saying the corresponding sentence — the implementation tracks audio boundary events from the voice pipeline and sequences DOM writes against them. The whiteboard OCR trust problem is fundamental: you want the model to be responsive to what a student draws, but a model that confidently misreads a student's working and teaches from the wrong premise is worse than no model at all. Treating the vision output as a candidate to be verified — rather than a fact to be acted on — was the right design, but it took careful work to implement without making Mira feel hesitant about clear cases. The PhET agent's terminal identification was genuinely hard. Component terminals in the PhET circuit builder are at positions that depend on zoom, component orientation, and canvas layout. The only reliable solution was vision-based identification from the rendered screenshot — which required building a verification step to catch failures before they are reported as success. Steel CDP session lifecycle across ECS Fargate required careful engineering. An ECS task that starts, fails to attach to Steel, and runs until the 7-minute cap would waste compute and block the student's next session. The agent's attach sequence has explicit failure detection and the hard cap enforces cleanup regardless. ### Accomplishments we're proud of The circuit physics is correct. The MNA solver computes what the laws of physics say — a parallel circuit carries different currents in different branches, a short circuit is detected because the source current exceeds the physical limit, an LDR in a low-light environment has a different resistance than in bright light. No lookup table, no approximation. The lesson teaching experience actually feeling like a tutor. A student working through factorising hears Mira speak and watches each step appear on the board at the moment she says it. That timing is not cosmetic — it is why the experience feels like being taught rather than reading a transcript. The chat diagram rendering putting a T-account or a circuit schematic directly in the conversation, generated on demand, at the exact moment the explanation arrives. No screenshot. No static image. A live visual drawn by the AI as part of its response. The bundle being 372KB for an application that does real-time voice, streaming AI, live board coordination, force-directed graph layout, and circuit physics simulation. Every kilobyte matters when a student is on a limited mobile data plan. Building something genuinely useful to a student whose school has no functioning physics lab, who is writing matric in three months, who cannot afford a tutor — and giving that student the same quality of worked-example, hands-on, voice-guided, diagram-rich experience that a well-resourced student gets from a private tutor. ### What we learned The most useful thing Codex changed was how early architectural problems became visible. Working alone you can tell yourself a design is fine and continue. When you hand the task to a model and read what it produces, you see immediately whether your mental model of the system is actually coherent. The trust boundary layer is the clearest example — having to articulate precisely why board readings cannot be trusted as ground truth produced a cleaner implementation than solo reasoning would have. Luna versus Terra is a real distinction, not just a cost optimisation. The question "do I need iteration speed or cross-file coherence right now" turns out to be a useful frame for any kind of engineering decision, AI-assisted or not. Designing for a learner's cognitive state rather than engineering convenience. The interrupt flow, the step-timed board writes, the practice question before moving on — none of that comes from what is easiest to build. It comes from thinking about what it feels like to be mid-explanation and not quite understand yet. Getting that right required resisting the easier version at every step. ### What's next South Africa first, then other national curricula. The vision tutor, lesson engine, lab agent, and subject database all work independently of which curriculum they serve. CAPS and IEB are the beachhead because that is where the inequity is most immediate and where the product is most directly tested. Exam preparation mode: connecting the cognitive graph to past paper analysis so Mira can look at a student's actual mastery data, identify the concepts with the most remaining gaps given how many weeks are left before the exam, and build a preparation schedule around real weaknesses rather than a generic syllabus order. The PhET simulation library has hundreds of experiments. The agent supports a working set today. Building the guided experiment flows that turn each simulation into a structured lesson — with Mira present throughout, asking questions and testing understanding, is the work that makes the lab a full curriculum track rather than a demonstration. And the classroom version. A teacher who can see which concepts her students are collectively struggling with, in aggregate, in real time, based on what Mira has tried to explain and where students got stuck, changes what is possible in a classroom. That version of Lyrn is worth building. ## README (from the GitHub repository) # LyrnOS An adaptive learning workspace for South African CAPS and IEB learners in Grades 7–12. Mira, an AI tutor, teaches through a live shared whiteboard, voice, and inline visual diagrams — reads what you draw, writes on the board while she speaks, and runs real science labs in a sandboxed browser. Built during the OpenAI Build Week hackathon (Education track). --- ## Built with Codex and GPT-5.6 LyrnOS was built collaboratively with Codex (GPT-5.6) as a pair engineer across more than 40 sessions totalling several hours of active collaboration. Every major system — the whiteboard vision path, the voice interrupt/resume state machine, the Browser-Use circuit lab, the streaming provider gateway, the cognitive graph, the ECS safety gate — was implemented through this loop: describe the learning outcome, let Codex draft end-to-end across front end and backend, review and tighten at the pedagogy and safety layer. GPT-5.6's longer coherent context let whole features (capture → API → model → render → persist) land in one pass. The collaboration model alternated between Luna for fast incremental iteration and Terra (High) for multi-file architectural sessions where cross-file coherence mattered more than speed. See [`codex.md`](./codex.md) for a session-by-session account of what was built, what was hard, and what decisions I made that Codex implemented to spec. --- ## What it does **Mira** is a Socratic tutor with four modes: **Live whiteboard** — the shared canvas uses Excalidraw. Learners sketch circle theorems, force diagrams, or equation working. Mira reads it using real computer vision (canvas exported to PNG → Claude vision API), writes back on the board, annotates the learner's figure. When teaching, she writes each step at the moment she says it — board writes are timed to speech boundary events. Learners can interrupt mid-sentence, speak their question, and Mira responds to what was actually said before resuming. **Chat with inline diagrams** — theory subjects (Accounting, History, Geography, Computer Science) teach through chat. Mira generates live SVG and HTML diagrams rendered directly in the conversation — T-accounts, timelines, data structure trees — at the moment she explains them. A hidden prompt starts the lesson; learners see only Mira beginning to teach. **PhET Physics Lab** — Mira's agent launches and operates a real PhET simulation in a sandboxed Steel cloud browser on ECS Fargate. The agent plans a complete circuit before touching the canvas, identifies terminal positions from a screenshot using vision, places components in a verified batch, then hands wiring to the learner. It verifies the result from a fresh screenshot before reporting success. Sessions are hard-capped at 7 minutes and ECS tasks are stopped on release. **Chemistry Titration Lab** — full stoichiometric titration engine computing pH from concentration and volume in real time. Students add reagent drop by drop and watch the equivalence curve shift. **Circuit Builder** — a physics-correct Modified Nodal Analysis solver with Gaussian elimination computes branch currents and voltages from first principles. Parallel circuits carry different currents in different branches. Short circuits are detected. LDR resistance changes with light level. LED polarity matters. **Adaptive Tests** — Claude-generated quizzes per subject, per grade, per section. Green/red review with correct answer and explanation for every question. Results update a force-directed cognitive graph showing mastery per concept from real quiz and lesson data. **My Work** — drop in a past paper PDF. Claude reads the entire document and returns question count, question text, and type. Learners work on an Excalidraw canvas beside the paper. A live check toggle sends Mira the canvas snapshot + exact question text every 1.8 seconds after a writing pause. **8 subjects, Grades 7–12:** Mathematics, Physical Sciences, Life Sciences, English, Accounting, History, Geography, Computer Science — CAPS and IEB aligned. --- ## Architecture ``` Browser (Vite SPA, 372KB gzipped entry chunk) ├── app.js 3,604 lines, 249 functions — tutor, voice, lessons, tests, graph, chat ├── style.css 5,286 lines — nine-view design system, glassmorphism ├── whiteboard.mjs Excalidraw + PNG export for vision + structured element snapshot ├── lab.mjs PhET Lab UI — simulation selection, embedded viewport, Mira controls └── auth.js Clerk (lazy-loaded, 2.8MB SDK out of initial bundle) Node server (server.mjs, 2,019 lines) ├── Multi-provider AI gateway: Bedrock → Anthropic API → Groq (streaming + non-streaming parity) ├── ECS orchestration: RunTask only when PHET_ECS_ENABLED=true + cluster name contains 'lyrnos' ├── Steel session management: 7-minute hard cap, automatic cleanup └── SSE streaming for chat, lessons, voice, and lab events Serverless API routes (api/) ├── _bedrock.js Provider gateway (211 lines) ├── _utils.js Trust boundary: input sanitisation + output parsing (314 lines, 15 tests) ├── chat.js Multi-turn chat with image/document attachments ├── tutor.js Lesson planning, whiteboard annotation, worked examples ├── synthesize.js Cartesia TTS → 16-bit PCM WAV → Web Audio API ├── quiz.js Claude-generated adaptive tests └── studio.js My Work proactive tutor observer ECS Fargate (on-demand only) └── phet-agent/service.py 1,610 lines — Browser-Use + Steel CDP + Claude Haiku via Bedrock Plans circuit before touching canvas Vision-based terminal identification Screenshot verification before success claim Self-terminates at 420 seconds Deterministic engines (labs/) ├── circuits/solver.mjs MNA + Gaussian elimination (57 lines) ├── chemistry/workbench.mjs Stoichiometric titration engine └── phet-registry.json Allowlisted simulation registry — agent cannot invent URLs Persistence (Supabase, RLS on every table) ├── learning_nodes Mastery per concept, updated from quiz + lesson events ├── assessment_attempts Quiz results, correct/wrong answers ├── lesson_sessions Lesson start, subject, mode, grade └── chat_history Threaded conversations, keyed to Clerk user ID ``` ### AI and safety boundary The PhET agent operates only within the selected simulation from the approved registry — it cannot invent URLs, execute shell commands, or navigate outside the PhET domain. Board readings are treated as candidate interpretations, not ground truth: a separate deterministic solver verifies equations independently. If the two disagree, Mira flags ambiguity instead of teaching from a wrong premise. Every client payload is validated on entry (PNG only, 3.75MB ceiling, 120-element snapshot cap, annotation type allowlist). Every model response is parsed on exit. Provider credentials are server-side only and never reach the browser. --- ## Setup and run locally ### Prerequisites - Node.js 20.19+ (or Node.js 22.12+) - npm 10+ - Git ### Quick start ```bash git clone https://github.com/Tanweer-Ahmed-Chiktay/Lyrn_OS.git cd Lyrn_OS npm install cp .env.example .env npm run dev ``` Open the URL printed by Vite (normally `http://localhost:5173`). The app is usable without credentials: it starts in local demo mode with the local cognitive graph and deterministic circuit and chemistry labs. ### Verify a local build ```bash npm test npm run build npm run preview ``` `npm run preview` serves the production build locally after `npm run build`. ### Environment Copy `.env.example` to `.env`: | Variable | Enables | |----------|---------| | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` | Bedrock-powered Mira (chat, tutor, vision, voice) | | `VITE_CLERK_PUBLISHABLE_KEY` | Real sign-in and user profiles | | `VITE_SUPABASE_URL` / `VITE_SU [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 27 recognized source files, 956 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (53 of 53) ``` .env.example .gitignore api/_bedrock.js api/_utils.js api/chat.js api/quiz.js api/search.js api/studio.js api/synthesize.js api/transcribe.js api/tutor.js app.js audio-capture-worklet.js audio-playback-worklet.js auth.js codex.md design-qa.md index.html lab.mjs labs/catalog.mjs labs/chemistry/reactions.mjs labs/chemistry/workbench.mjs labs/circuit-components.mjs labs/circuit-workbench.mjs labs/circuits/solver.mjs labs/phet-registry.json labs/phet-registry.mjs labs/schema.mjs LICENSE package.json phet-agent/.env.example phet-agent/buildspec.yml phet-agent/Dockerfile phet-agent/ecs-task-definition.example.json phet-agent/iam-bedrock-policy.example.json phet-agent/iam-gateway-policy.example.json phet-agent/README.md phet-agent/requirements.txt phet-agent/service.py pnpm-workspace.yaml README.md server.mjs style.css supabase/migrations/202607150001_learning_workspace.sql supabase/migrations/202607190002_learning_activity_assessments.sql supabase/migrations/202607190003_profile_context.sql test/api-utils.test.mjs test/circuit-solver.test.mjs test/lab-schema.test.mjs test/phet-registry.test.mjs vercel.json vite.config.js whiteboard.mjs ``` ### Dependencies - package.json: @aws-sdk/client-bedrock-runtime@^3.1086.0, @aws-sdk/client-ec2@^3.1087.0, @aws-sdk/client-ecs@^3.1087.0, @clerk/clerk-js@^5.127.1, @excalidraw/excalidraw@^0.18.1, @supabase/supabase-js@^2.110.6, concurrently@^9.2.1, dotenv@^16.6.1, express@^5.1.0, katex@^0.16.47, react@^18.3.1, react-dom@^18.3.1, vite@^7.1.12, ws@^8.21.1 - phet-agent/requirements.txt: boto3@>=1.40.0, browser-use[aws]@==0.13.4, fastapi@==0.116.1, playwright@==1.55.0, python-dotenv@==1.2.2, steel-sdk@>=0.2.0, uvicorn[standard]@==0.35.0 ### Recent commits (newest first) - feat: publish clean LyrnOS codebase ## Key source files (fetched from GitHub, selected and truncated for size) ### design-qa.md ```markdown # Design QA — Workspace routes and whiteboard ## Reference inputs Compared against the supplied 1280×720 references: - `/var/folders/kf/8rhptpxs35x0krq73g2g5q180000gn/T/TemporaryItems/NSIRD_screencaptureui_4PXR0u/Screenshot 2026-07-15 at 17.56.20.png` - `/var/folders/kf/8rhptpxs35x0krq73g2g5q180000gn/T/TemporaryItems/NSIRD_screencaptureui_xJL5vc/Screenshot 2026-07-15 at 17.56.37.png` - `/var/folders/kf/8rhptpxs35x0krq73g2g5q180000gn/T/TemporaryItems/NSIRD_screencaptureui_DrPKQv/Screenshot 2026-07-15 at 17.57.12.png` ## Full-view comparison - `/home`, `/whiteboard`, `/lab`, `/my-work`, `/chat`, and `/learning-paths` are URL-backed views. Navigation uses the History API and keeps the active view synchronized with the address bar. - The My Work hero copy (“Your thinking, in one place.”) is removed. The paper workspace begins directly below the shared top bar. - The Mira launcher is visible only on `/my-work`; it is hidden on `/home`, `/whiteboard`, `/lab`, and `/learning-paths`. - Chat is available as a dedicated `/chat` page and as the My Work floating panel. - The whiteboard route fills the available viewport below the top bar. At 1280×720, the Excalidraw root measured 810×638 and the tutor panel occupied the remaining right column. ## Focused region comparison - The My Work orb is a fixed circular launcher in the lower-right corner and opens a 430×592 floating chat panel with focus moved into the response field. - Excalidraw’s canvas is the pointer target (`elementFromPoint(...) === CANVAS`) and has active pointer events/touch handling. - The freehand tool was selected and a real drag produced a third board element; the live board status changed from “Board linked · 2 elements” to “Board linked · 3 elements”. ## Functional checks - Direct route: `/whiteboard` renders the tutor view after entering the demo workspace. - In-app route navigation: Home → Whiteboard → My Work → Lab updates the URL and active view without leaving the workspace. - Mira open/close: launcher opens the panel, focuses `#chatInput`, and switching routes closes it. - Chat tab: opens `/chat` as a full page, focuses the message field, and does not use the floating-panel class. - Build: `vite build` completed successfully. - Browser console: no application errors; only the expected Clerk development-key warning was present. ## Latest full-bleed iteration Reference comparison used the supplied My Work and Whiteboard captures at the desktop viewport. The My Work shell now reaches the content edges beneath the shared top bar with no outer radius, border, shadow, or page-level gap. The working pane uses one full-width question rail and a constrained grid row so the Excalidraw surface expands instead of pushing controls below the viewport. The whiteboard screenshot captured after the fix shows the canvas occupying the full working stage. The static Excalidraw layer is non-interactive and only the interactive canvas receives pointer input. A freehand drag increased the link [truncated — 984 more characters] ``` ### codex.md ```markdown # Building LyrnOS with Codex (GPT-5.6) This document is written for the OpenAI Build Week judges. It separates the work Codex accelerated from the product, pedagogy, and safety decisions I owned — so the technical depth and the collaboration model are both legible. ## How I worked with Codex I used Codex as a pair engineer, not a code generator. The loop was: describe the outcome I wanted for a learner (not a feature spec), let Codex draft the end-to-end implementation across front end, backend, and Python agent, then review, correct, tighten — especially anywhere the model touched pedagogy or safety. GPT-5.6's longer coherent reasoning context let me hand it whole features (input → API → model → render → persist) in a single pass instead of stitching together snippets. I switched deliberately between **Luna** and **Terra** (High) throughout the build: - **Luna** for incremental edits, CSS iteration, single-file fixes, and anything where iteration speed outweighed cross-file coherence. UI bug fixes, label changes, small layout adjustments. - **Terra** for multi-file architectural passes — the voice state machine, the ECS safety gate, the trust boundary layer, the streaming gateway parity fix, the cognitive graph wiring. Passing a complex constraint across five files at once is where GPT-5.6's window actually earns its cost. Staying within the credit budget forced a discipline that improved decisions. You think harder about whether a multi-file refactor is the right call when it costs more than a targeted edit. ## The sessions — what was built and how hard it actually was ### 1. PhET Simulation Lab (Wednesday 17:40 — 17 minutes) **What I asked for:** build a lab where students can describe an experiment in natural language, the app finds the right PhET simulation from an allowlisted registry, launches it embedded, and an AI agent can guide or operate the simulation on behalf of the learner — but only spin up the browser infrastructure when explicitly needed, not always-on. **What Codex built in one pass:** - Allowlisted PhET registry (`labs/phet-registry.json`) with verified official URLs — model cannot invent simulation URLs - Natural-language selection with suggestions, fast-launch cards, embedded viewport, and Mira guide/operate modes - Bedrock selection route and secure session/agent gateway in `server.mjs` - ECS-ready Browser-Use + Steel + Claude Haiku worker in `phet-agent/service.py` - Docker/ECS/Secrets Manager templates (`Dockerfile`, `ecs-task-definition.example.json`) - On-demand ECS: `RunTask` only on a Mira guide/operate request, `StopTask` on session release The implementation followed Browser-Use's current Bedrock support pattern and Steel's CDP session integration. Codex flagged that the pasted AWS and Steel credentials were compromised and did not reuse or deploy them. ### 2. ECS cost control (Wednesday 18:25 — 9 minutes 51 seconds) **Problem:** always-on ECS Fargate was generating unexpected cost. **What Codex imple [truncated — 15333 more characters] ``` ### package.json ``` { "name": "lyrnos", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview --host 127.0.0.1", "start": "node server.mjs", "test": "node --test" }, "dependencies": { "@aws-sdk/client-bedrock-runtime": "^3.1086.0", "@aws-sdk/client-ec2": "^3.1087.0", "@aws-sdk/client-ecs": "^3.1087.0", "@clerk/clerk-js": "^5.127.1", "@excalidraw/excalidraw": "^0.18.1", "@supabase/supabase-js": "^2.110.6", "dotenv": "^16.6.1", "express": "^5.1.0", "katex": "^0.16.47", "react": "^18.3.1", "react-dom": "^18.3.1", "ws": "^8.21.1" }, "devDependencies": { "concurrently": "^9.2.1", "vite": "^7.1.12" }, "allowScripts": { "esbuild@0.28.1": true } } ``` ### phet-agent/requirements.txt ``` fastapi==0.116.1 uvicorn[standard]==0.35.0 playwright==1.55.0 browser-use[aws]==0.13.4 steel-sdk>=0.2.0 python-dotenv==1.2.2 boto3>=1.40.0 ``` ### phet-agent/Dockerfile ``` FROM public.ecr.aws/docker/library/python:3.11-slim ENV PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 \ PLAYWRIGHT_BROWSERS_PATH=/ms-playwright WORKDIR /app COPY phet-agent/requirements.txt /app/phet-agent/requirements.txt RUN pip install --no-cache-dir -r /app/phet-agent/requirements.txt \ && python -m playwright install --with-deps chromium COPY browser-use /app/browser-use-src RUN pip install --no-cache-dir --no-deps /app/browser-use-src COPY labs/phet-registry.json /app/labs/phet-registry.json COPY phet-agent/service.py /app/phet-agent/service.py EXPOSE 8790 CMD ["uvicorn", "--app-dir", "/app/phet-agent", "service:app", "--host", "0.0.0.0", "--port", "8790"] ``` ### pnpm-workspace.yaml ```yaml allowBuilds: '@clerk/shared': true browser-tabs-lock: true bufferutil: true core-js: true utf-8-validate: true minimumReleaseAgeExclude: - '@supabase/auth-js@2.110.6' - '@supabase/functions-js@2.110.6' - '@supabase/postgrest-js@2.110.6' - '@supabase/realtime-js@2.110.6' - '@supabase/storage-js@2.110.6' - '@supabase/supabase-js@2.110.6' ``` ### audio-capture-worklet.js ```javascript // Nova Sonic requires continuous 16-bit PCM, mono, 16 kHz input. Keep this // processor allocation-light: worklets run on the browser's real-time audio // thread, so array splicing or per-callback allocations cause audible jitter. class NovaPcmCapture extends AudioWorkletProcessor { constructor() { super(); this.phase = 0; this.total = 0; this.samplesInBucket = 0; this.frame = new Int16Array(512); // 32 ms at 16 kHz this.frameOffset = 0; } flushSample(sample) { const clamped = Math.max(-1, Math.min(1, sample)); this.frame[this.frameOffset] = clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff; this.frameOffset += 1; if (this.frameOffset !== this.frame.length) return; this.port.postMessage(this.frame.buffer, [this.frame.buffer]); this.frame = new Int16Array(512); this.frameOffset = 0; } process(inputs) { const source = inputs[0]?.[0]; if (!source) return true; // A small moving-average decimator is enough for speech capture. The phase // accumulator keeps the resulting stream at exactly 16 kHz from both // 44.1 kHz and 48 kHz browser audio contexts. for (let index = 0; index < source.length; index += 1) { this.total += source[index]; this.samplesInBucket += 1; this.phase += 16000; if (this.phase < sampleRate) continue; this.flushSample(this.total / this.samplesInBucket); this.phase -= sampleRate; this.total = 0; this.samplesInBucket = 0; } return true; } } registerProcessor('nova-pcm-capture', NovaPcmCapture); ``` ### audio-playback-worklet.js ```javascript // A single pull-based PCM player is much less vulnerable to WebSocket jitter // than scheduling one AudioBufferSourceNode for every model packet. It also // resamples Nova's 24 kHz output to the listener's hardware sample rate while // preserving continuity between packets. class NovaPcmPlayback extends AudioWorkletProcessor { constructor() { super(); // Complete assistant turns are released together by the server, so retain // enough PCM to play a whole concise reply without dropping its tail. this.capacity = 480_000; // Twenty seconds at Nova Sonic's usual 24 kHz rate. this.maxBufferedSamples = 360_000; // Bound latency at roughly fifteen seconds. this.ring = new Float32Array(this.capacity); this.readIndex = 0; this.writeIndex = 0; this.queuedSamples = 0; this.sourceRate = 24_000; this.step = this.sourceRate / sampleRate; this.phase = 0; this.started = false; this.fadeInSamples = 0; // Keep the first response conversational. The server relays packets as // they arrive, so a short lead is enough to absorb normal WebSocket jitter. this.targetPrebufferSeconds = 0.42; this.maxPrebufferSeconds = 1.1; this.underruns = 0; this.droppedSamples = 0; this.renderedSamplesSinceReport = 0; this.port.onmessage = ({ data }) => { if (data?.type === 'clear') { this.clear(); return; } if (data?.type === 'pcm' && data.pcm instanceof ArrayBuffer) { this.enqueue(data.pcm, data.sampleRate); } }; } clear() { this.readIndex = 0; this.writeIndex = 0; this.queuedSamples = 0; this.phase = 0; this.started = false; this.fadeInSamples = 0; } enqueue(buffer, requestedRate) { const rate = Number(requestedRate) || 24_000; // Nova uses one output rate per session. If a provider ever changes it, // start a clean stream rather than mixing two clock domains. if (this.queuedSamples && rate !== this.sourceRate) this.clear(); this.sourceRate = rate; this.step = this.sourceRate / sampleRate; const pcm = new Int16Array(buffer); for (let index = 0; index < pcm.length; index += 1) { // Do not let a transient upstream burst turn into seconds of delayed // speech. This branch is exceptional; normal playback stays near the // small prebuffer below. if (this.queuedSamples >= this.maxBufferedSamples) { this.readIndex = (this.readIndex + 1) % this.capacity; this.queuedSamples -= 1; this.phase = 0; this.droppedSamples += 1; } this.ring[this.writeIndex] = pcm[index] / 0x8000; this.writeIndex = (this.writeIndex + 1) % this.capacity; this.queuedSamples += 1; } } process(_inputs, outputs) { const output = outputs[0]?.[0]; if (!output) return true; output.fill(0); // A modest adaptive prebuffer absorbs ordinary packet and main-thread // jitter without // making a spoken response feel delayed. if (!this.started) { if (this.queuedSamples < Math.round(this.sourceRate * this.targetPrebufferSeconds)) { this.report(output.length); return true; } this.started = true; this.fadeInSamples = 128; } for (let index = 0; index < output.length; index += 1) { // Linear interpolation needs the next source sample. Waiting for a // fresh prebuffer after an underrun avoids chopped, zipper-like speech. if (this.queuedSamples < 2) { this.underruns += 1; this.targetPrebufferSeconds = Math.min( this.maxPrebufferSeconds, Math.max(0.7, this.targetPrebufferSeconds + 0.2), ); this.clear(); this.report(output.length); return true; } const current = this.ring[this.readIndex]; const next = this.ring[(this.readIndex + 1) % this.capacity]; let sample = current + (next - current) * this.phase; if (this.fadeInSamples > 0) { sample *= (129 - this.fadeInSamples) / 128; this.fadeInSamples -= 1; } output[index] = sample; this.phase += this.step; const consumed = Math.floor(this.phase); if (consumed > 0) { if (consumed >= this.queuedSamples - 1) { this.clear(); this.report(output.length); return true; } this.readIndex = (this.readIndex + consumed) % this.capacity; this.queuedSamples -= consumed; this.phase -= consumed; } } this.report(output.length); return true; } report(renderedSamples) { this.renderedSamplesSinceReport += renderedSamples; if (this.renderedSamplesSinceReport < sampleRate) return; this.renderedSamplesSinceReport = 0; this.port.postMessage({ type: 'stats', bufferedMs: Math.round((this.queuedSamples / this.sourceRate) * 1_000), targetBufferMs: Math.round(this.targetPrebufferSeconds * 1_000), underruns: this.underruns, droppedSamples: this.droppedSamples, started: this.started, }); } } registerProcessor('nova-pcm-playback', NovaPcmPlayback); ``` ### vite.config.js ```javascript import { defineConfig } from 'vite'; import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; import { config as loadEnv } from 'dotenv'; import { spawn } from 'node:child_process'; import http from 'node:http'; loadEnv(); // Track the actual port server.mjs binds (may differ from API_PORT due to EADDRINUSE). let backendPort = null; let backendProcess = null; function ensureBackend() { if (backendProcess) return; backendProcess = spawn(process.execPath, ['server.mjs'], { cwd: process.cwd(), env: process.env, stdio: ['inherit', 'pipe', 'inherit'], }); backendProcess.stdout.on('data', (chunk) => { process.stdout.write(chunk); const m = chunk.toString().match(/listening on http:\/\/127\.0\.0\.1:(\d+)/); if (m) backendPort = Number(m[1]); }); backendProcess.once('exit', () => { backendProcess = null; backendPort = null; }); for (const sig of ['exit', 'SIGINT', 'SIGTERM']) { process.once(sig, () => { backendProcess?.kill(); }); } } /** * Vite plugin that serves /api/* routes from the api/ directory locally, * matching Vercel serverless function behaviour without needing `vercel dev`. * Routes not found in api/ are proxied to server.mjs (handles /api/phet/* etc.). */ function localApiPlugin() { return { name: 'local-api', configureServer(server) { ensureBackend(); // Proxy /api/phet/* to server.mjs at its actual bound port. server.middlewares.use('/api/phet', async (req, res) => { ensureBackend(); // restart if it exited if (!backendPort) { // Wait up to 8 s for server.mjs to report its port before giving up. for (let i = 0; i < 32; i++) { await new Promise((r) => setTimeout(r, 250)); if (backendPort) break; } } if (!backendPort) { res.writeHead(503, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ error: 'Backend starting — retry in a moment' })); return; } const options = { hostname: '127.0.0.1', port: backendPort, path: `/api/phet${req.url}`, method: req.method, headers: { ...req.headers, host: `127.0.0.1:${backendPort}` }, }; const proxy = http.request(options, (backendRes) => { res.writeHead(backendRes.statusCode, backendRes.headers); backendRes.pipe(res, { end: true }); }); proxy.on('error', (err) => { if (!res.headersSent) { res.writeHead(502, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ error: err.message })); } }); req.pipe(proxy, { end: true }); }); server.middlewares.use(async (req, res, next) => { if (!req.url?.startsWith('/api/')) return next(); // Parse URL and query string const [pathname, qs] = req.url.split('?'); const query = Object.fromEntries(new URLSearchParams(qs || '')); // Map /api/chat/stream → api/chat.js?action=stream (vercel rewrite style) const segments = pathname.replace(/^\/api\//, '').split('/'); const handlerName = segments[0]; if (segments[1]) query.action = segments[1]; let handler; try { const filePath = pathToFileURL(resolve(process.cwd(), `api/${handlerName}.js`)).href; const mod = await import(filePath); handler = mod.default; } catch (err) { console.error(`[api] import error for ${handlerName}:`, err.message); return next(); } if (typeof handler !== 'function') return next(); // Collect body const chunks = []; for await (const chunk of req) chunks.push(chunk); const raw = Buffer.concat(chunks).toString(); let body = {}; try { body = raw ? JSON.parse(raw) : {}; } catch {} // Minimal req/res shim const fakeReq = { method: req.method, query, body, headers: req.headers }; const headers = {}; let statusCode = 200; const fakeRes = { status(code) { statusCode = code; return fakeRes; }, setHeader(k, v) { headers[k] = v; }, json(obj) { const payload = JSON.stringify(obj); res.writeHead(statusCode, { 'Content-Type': 'application/json', ...headers }); res.end(payload); }, write(chunk) { if (!res.headersSent) res.writeHead(statusCode, headers); res.write(chunk); }, end(chunk) { if (!res.headersSent) res.writeHead(statusCode, headers); res.end(chunk); }, flushHeaders() { if (!res.headersSent) res.writeHead(statusCode, headers); }, get headersSent() { return res.headersSent; }, }; try { await handler(fakeReq, fakeRes); } catch (err) { if (!res.headersSent) { res.writeHead(500, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ error: err.message })); } } }); }, }; } export default defineConfig({ plugins: [localApiPlugin()], server: { port: 5173, }, }); ``` ### auth.js ```javascript // Clerk (~2.8 MB) and the Supabase client are only ever constructed inside // initAuth(). Importing them dynamically keeps them out of the initial bundle, // so Home/Whiteboard paint from a small critical-path chunk and the // credential-free demo path loads neither provider at all. const clerkKey = import.meta.env.VITE_CLERK_PUBLISHABLE_KEY; const supabaseUrl = import.meta.env.VITE_SUPABASE_URL; const supabaseKey = import.meta.env.VITE_SUPABASE_PUBLISHABLE_KEY || import.meta.env.VITE_SUPABASE_ANON_KEY; let clerk = null; let supabase = null; let demoUser = JSON.parse(localStorage.getItem('lyrnos:demo-user') || 'null'); export const getCurrentUser = () => clerk?.user || demoUser; export const getSupabase = () => supabase; export async function signOut() { try { if (clerk) await clerk.signOut(); demoUser = null; localStorage.removeItem('lyrnos:demo-user'); window.location.assign('/'); } catch (error) { console.error('[LyrnOS] sign out failed', error); } } function userProfile(user) { const meta = user.unsafeMetadata || user.publicMetadata || {}; return { user_id: user.id, display_name: user.firstName || user.username || user.primaryEmailAddress?.emailAddress?.split('@')[0] || 'Learner', grade: meta.grade || 'Grade 10', curriculum: meta.curriculum || 'CAPS', country: meta.country || 'South Africa' }; } async function syncProfile(user) { if (!supabase || !user) return; const profile = userProfile(user); await supabase.from('profiles').upsert(profile, { onConflict: 'user_id' }); } function setUserDetails(user) { if (!user) return; const profile = userProfile(user); document.querySelectorAll('[data-user-name]').forEach((node) => { node.textContent = profile.display_name; }); document.querySelectorAll('[data-user-initials]').forEach((node) => { node.textContent = profile.display_name.slice(0, 2).toUpperCase(); }); document.querySelectorAll('[data-user-grade]').forEach((node) => { node.textContent = profile.grade; }); document.querySelectorAll('[data-user-space]').forEach((node) => { node.textContent = `${profile.display_name}’s space`; }); } export function openProfile() { const user = getCurrentUser(); if (!user) return; const profile = userProfile(user); document.querySelector('#profileName').value = profile.display_name; document.querySelector('#profileGrade').value = profile.grade; document.querySelector('#profileCurriculum').value = profile.curriculum; document.querySelector('#profileCountry').value = profile.country; document.querySelectorAll('[data-profile-initials]').forEach((node) => { node.textContent = profile.display_name.slice(0, 2).toUpperCase(); }); document.querySelector('#profileError').textContent = ''; document.querySelector('#profileModal')?.removeAttribute('hidden'); } async function saveProfile(event) { event.preventDefault(); const user = getCurrentUser(); const name = document.querySelector('#profileName').value.trim() || 'Learner'; const unsafeMetadata = { grade: document.querySelector('#profileGrade').value, curriculum: document.querySelector('#profileCurriculum').value, country: document.querySelector('#profileCountry').value.trim() || 'South Africa' }; try { if (clerk?.user?.update) await clerk.user.update({ firstName: name, unsafeMetadata }); else { demoUser = { ...demoUser, firstName: name, unsafeMetadata }; localStorage.setItem('lyrnos:demo-user', JSON.stringify(demoUser)); } setUserDetails(getCurrentUser()); document.querySelector('#profileModal')?.setAttribute('hidden', ''); } catch (error) { document.querySelector('#profileError').textContent = error?.message || 'Unable to save your profile.'; } } function enterWorkspace(user) { setUserDetails(user); document.body.classList.add('workspace-ready'); document.querySelector('#landingPage')?.setAttribute('aria-hidden', 'true'); document.querySelector('#authModal')?.setAttribute('hidden', ''); window.dispatchEvent(new CustomEvent('lyrnos:auth-ready', { detail: user })); } export async function initAuth() { if (supabaseUrl && supabaseKey) { const { createClient } = await import('@supabase/supabase-js'); supabase = createClient(supabaseUrl, supabaseKey, { accessToken: async () => clerk?.session?.getToken({ template: 'supabase' }) || null }); } if (clerkKey) { const { Clerk } = await import('@clerk/clerk-js'); clerk = new Clerk(clerkKey); await clerk.load(); clerk.addListener(({ user }) => { if (user) { setUserDetails(user); void syncProfile(user); } }); if (clerk.user) { enterWorkspace(clerk.user); void syncProfile(clerk.user); } } else if (demoUser) enterWorkspace(demoUser); return clerk; } export async function openAuth(mode = 'sign-in') { const modal = document.querySelector('#authModal'); const mount = document.querySelector('#clerkAuthMount'); const demo = document.querySelector('#demoAuth'); const title = document.querySelector('#authTitle'); const subtitle = document.querySelector('#authSubtitle'); if (!modal) return; modal.removeAttribute('hidden'); mount.replaceChildren(); demo.hidden = mode !== 'demo'; mount.hidden = mode === 'demo'; const authCopy = { 'sign-up': ['Start learning with Lyrn', 'Create your space and make progress that sticks.'], 'demo': ['Try Lyrn instantly', 'A local demo workspace — no account needed.'], 'sign-in': ['Welcome back to Lyrn', 'Sign in to continue your learning journey.'], }; const [copyTitle, copySubtitle] = authCopy[mode] || authCopy['sign-in']; title.textContent = copyTitle; subtitle.textContent = copySubtitle; if (mode === 'demo') return; if (!clerk) { document.querySelector('#authError').textContent = 'Add VITE_CLERK_PUBLISHABLE_KEY to enable real sign-in, or use the free demo.'; return; } const options = { routing: 'virtual', appearance: { variables: { colorPrimary: '#6856f6', borderRadius: '14px' } } }; mode === 'sign-up' ? clerk.mountSignUp(mount, options) : clerk.mountSignIn(mount [truncated — 1773 more characters] ``` [14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]