# Project export: SimulationOS This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: We automate compliance training, offering instant generation, accordance with corporate or legal requirements, adaptive difficulty, and a conversation based examination system. - Devpost: https://devpost.com/software/simulationos - GitHub: https://github.com/GameStudioAI/GameStudio - Video: https://player.vimeo.com/video/1203256069?byline=0&portrait=0&title=0#t= - Team: 2 GitHub contributor(s) — SurajGajula (6 commits), Cursor (2 commits) ## Devpost submission (written by the team) ### Overview Corporate compliance training is long, boring, and legally required. We can't shorten it — so we made it fun. Enterprise Training Studio turns mandatory training into voice-first roleplay. Learners talk through realistic scenarios with an AI character who pushes back and reacts in real time. Compliance stays complete; the experience stops feeling like punishment. Benefits Employers Audit-ready completion with scored decision points, not passive checkboxes AI competency assessment (mastery score, verdict, per-decision eval) Fast module creation — describe a scenario, publish in minutes Arize traces for quality monitoring and LLM-as-a-Judge evaluators Downloadable session transcripts for HR systems and downstream workflows Employees Practice by speaking, not reading slide decks Immediate coaching feedback at session end Replay and try different approaches Feels like a game; works like training How It Works Studio — Describe a scenario by voice or text Browserbase — Researches the topic on the web during creation (Wikipedia, MDN) Gemini — Generates the roleplay script, decision points, and scene background Deepgram — Powers live voice roleplay (STT, turn detection, Aura TTS) Arize — Receives competency assessment traces via OpenTelemetry Tech Stack React · TypeScript · Vite · Node.js · Express · Deepgram · Gemini · Browserbase · Arize · Sharp Integrations What's Next HR dashboard — Unified participation, completion, and eval data across the org Org-wide leaderboard — Incentivize replay and adaptive difficulty so employees build real competency, not one-and-done completion ## README (from the GitHub repository) # Enterprise Training Studio Voice-first corporate training studio powered by **Deepgram Voice Agent**, **Gemini** (scenario design + assessment), and **Browserbase** (web research during creation). Learners practice through continuous voice roleplay scenarios with a single scene background and competency scoring at the end. ## Run ```bash npm install npm run dev ``` Opens the Vite client at `http://localhost:5173` and the API server at `http://localhost:3001`. ## Environment (`.env`) | Variable | Used for | |----------|----------| | `DEEPGRAM_API_KEY` | Voice Agent WebSocket, speech-to-text, and TTS | | `GEMINI_API_KEY` | Scenario generation, runtime agent brain, image generation, and competency scoring | | `GEMINI_MODEL` | Agent brain (`gemini-3.1-flash-lite`) | | `GEMINI_IMAGE_MODEL` | Scene backgrounds and marketing thumbnails | | `BROWSERBASE_API_KEY` | Researches topics on the web when creating/editing simulations | | `BROWSERBASE_PROJECT_ID` | Optional Browserbase project (inferred from API key if omitted) | | `ARIZE_API_KEY` | **Required** — competency assessment traces to Arize AX (OTLP) | | `ARIZE_SPACE_ID` | **Required** — Arize space ID for trace export | | `ARIZE_PROJECT_NAME` | Optional project name in Arize (default: `gamestudio-learning`) | ## How it works 1. **Studio (voice or type)** — describe an enterprise training scenario 2. **Browserbase (creation)** — opens a cloud browser and gathers factual excerpts from the web (see below) 3. **Gemini (creation)** — designs a voice roleplay scenario with decision points and generates one scene background from the research 4. **Training (play)** — learner has one continuous spoken conversation with a Deepgram-powered voice agent that roleplays the scenario (see below) 5. **Gemini + Arize (assessment)** — at the natural end of the scenario, the full transcript is scored and optionally exported to Arize (see below) ## Deepgram, Browserbase, and Arize ### Deepgram Deepgram powers all real-time voice and spoken audio in the app. The server never streams raw audio to the client for storage — it mints short-lived tokens and builds agent configs; the browser connects directly to Deepgram via the `@deepgram/agents` SDK. **Studio (create / edit flow)** — On the Studio page, the mic opens a Deepgram Voice Agent session (`GET /api/deepgram/token`, `GET /api/deepgram/config`). Deepgram handles speech-to-text and turn detection. **Gemini** is wired in as the agent’s “think” provider: it interprets what you said and decides which studio tool to call (`create_game`, `edit_game`, `publish_game`, etc.). Spoken replies are rendered client-side through `POST /api/deepgram/speak` (Aura TTS), not the agent’s built-in speak pipeline. **Training (roleplay flow)** — On the Training page, a separate lesson agent config is built per session (`POST /api/deepgram/lesson-config`). Here Deepgram listens to the learner’s microphone (with end-of-turn detection) while Gemini drives an in-character roleplay: the agent speaks as the scenario character(s), pushes back on weak answers, and walks through generated **decision points**. When the scenario reaches a natural endpoint, the agent calls a `complete_scenario` tool. Agent lines are spoken via the same REST TTS endpoint so captions stay in sync with audio. **Where it runs** | Surface | Deepgram role | |---------|----------------| | Studio mic | STT + turn taking; Gemini decides studio actions | | Training mic | STT + turn taking; Gemini roleplays the scenario | | All spoken lines | Aura TTS via `POST /api/deepgram/speak` | If `DEEPGRAM_API_KEY` is missing, voice mode is disabled but typed Studio chat and non-voice flows still work. ### Browserbase Browserbase is used **only during creation and editing**, not while a learner is training. When you ask Studio to create or edit a module, the server starts a **Browserbase cloud browser session** and connects to it with Playwright over CDP (`server/browserbase.ts`). The session searches the web for material related to your topic: 1. **Wikipedia** — runs a search, opens the first relevant article, and extracts a text excerpt. 2. **MDN** — same pattern for technical or product-adjacent topics. Those excerpts (URL, title, and body text) are formatted into research notes and passed to **Gemini** when generating: - The scenario outline (setting, persona, opening line, 3–4 decision points with 3–4 choices each) - Edits to an existing module (`edit_game`) Learners never browse the live web during training — Browserbase is a **research step for accurate scenario content**, similar to an instructional designer looking up source material before writing a script. If `BROWSERBASE_API_KEY` is not set, generation falls back to Gemini’s general knowledge without web research. ### Arize Arize receives **observability traces for competency assessments**, not live voice audio or Browserbase sessions. Scoring itself is done by **Gemini** in `POST /api/learning/evaluate`. When a training session ends: 1. The full conversation transcript and scenario decision points are sent to the evaluate endpoint. 2. Gemini maps each decision point to what the learner said and assigns quality (strong / adequate / weak). 3. A weighted mastery score and verdict are computed and returned to the UI. 4. OpenTelemetry spans are emitted to Arize AX via OTLP with OpenInference attributes (`input.value`, `output.value`, `openinference.span.kind`, `session.id`). **Trace hierarchy per session:** | Span name | Kind | Purpose | |-----------|------|---------| | `training.session` | CHAIN | Root trace for one assessment | | `llm.analyze_decisions` | LLM | Gemini decision mapping | | `llm.score_session` | LLM | Gemini mastery scoring | | `learning.decision.eval` | LLM | One span per decision point — **use these for Q&A evaluator** | | `llm.analyze_decisions` | LLM | Internal Gemini decision-mapping call | | `llm.score_session` | LLM | Internal Gemini scoring call | | `training.assessment` | CHAIN | Session summary | **Project filter in Arize:** set `ARIZE_PROJECT_NAME` in `.env` (default `gamestudio-learning`) and select that project in the Arize UI. **LLM-as-a-Judge template:** use **Q&A** with scope **Span** and this filter: ``` name = 'learning.decision.eval' ``` (Arize's default preview filter `openinference.span.kind = LLM` will match these spans.) | Template variable | Map to attribute | |-------------------|------------------| | `{input}` | `attributes.input.value` → JSON field `input` (reference context + choices) | | `{question}` | `attributes.evaluation.question` | | `{output}` | `attributes.output.value` (learner's spoken answer) | Do **not** target `llm.analyze_decisions` or `llm.score_session` — those are internal Gemini prompts, not learner Q&A. Optional: add **User Frustration** on `training.assessment` spans using the transcript in `input.value`. The assessment overlay in the app is the learner-facing result. Arize is **required** for competency assessment — traces are flushed on every evaluation so they appear in Arize AX within ~30 seconds. ## Studio agent flow 1. Tap the mic on **Studio** (or use **Type** mode) → connects to Deepgram Voice Agent or Gemini text chat 2. Describe a scenario — agent transcribes/responds and calls studio functions 3. Tap again to disconnect (voice mode) Example commands: - *"Create a sales discovery call simulation for new enterprise AEs"* - *"Add a branch for handling pricing objections"* - *"Publish this training module"* - *"Make a B2B ad for sales enablement leaders"* ## Training flow 1. Open a module from the **Library** → single background with voice UI overlay 2. The agent opens with the scenario’s in-character **opening line** (spoken via Deepgram TTS) 3. Learner unmutes and responds; the agent roleplays through **decision points** until a natural endpoint 4. After the closing line finishes, the **competency assessment** overlay appears with score, feedback, transcript download, [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 78 recognized source files, 332 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (84 of 84) ``` .gitignore eslint.config.js index.html package.json README.md scripts/arize-smoke-test.ts scripts/browserbase-metadata-test.ts scripts/browserbase-research-test.ts scripts/patch-deepgram-agents.mjs server/arize.ts server/assets.ts server/browserbase.ts server/characterAssets.ts server/chromaKey.ts server/chromaKeyRemoval.ts server/env.ts server/gameEditor.ts server/gamePackage.ts server/gemini.ts server/index.ts server/learningAssessment.ts server/lessonRules.ts server/lessonVoiceAgentConfig.ts server/routes/browserbase.ts server/routes/deepgram.ts server/routes/learning.ts server/routes/studio.ts server/routes/trainings.ts server/runtimeNarrative.ts server/sceneAssets.ts server/speakerResolver.ts server/studioActions.ts server/studioAgentConfig.ts server/textAgent.ts server/visualStyle.ts server/voiceAgentConfig.ts shared/chromaKeyRemoval.ts src/App.tsx src/components/PageNav.tsx src/components/StagePanels.tsx src/components/StudioLoadingModal.tsx src/game/assetLoader.ts src/game/bgm.ts src/game/chromaKey.ts src/game/ConversationRuntime.tsx src/game/dynamicEngine.ts src/game/DynamicVNRuntime.tsx src/game/gameConfig.ts src/game/index.ts src/game/lessonRules.ts src/game/useAssessmentUiReset.ts src/game/VnDialogueFooter.tsx src/game/vnEngine.ts src/game/VNRuntime.tsx src/game/VnSceneDialogueLines.tsx src/hooks/useConversationVoiceAgent.ts src/hooks/useDemoEngine.ts src/hooks/useLessonVoiceAgent.ts src/hooks/useVoiceAgent.ts src/index.css src/lib/assessmentContext.ts src/lib/deepgramClient.ts src/lib/deepgramSpeech.ts src/lib/learningAssessment.ts src/lib/learningThresholds.ts src/lib/runtimeNarrative.ts src/lib/sessionTranscript.ts src/lib/trainingLibrary.ts src/lib/voiceAnswer.ts src/lib/voiceAudio.ts src/lib/voiceCoaching.ts src/main.tsx src/pages/AdsPage.tsx src/pages/GamePage.tsx src/pages/LibraryPage.tsx src/pages/PublishedPage.tsx src/pages/VoicePage.tsx src/types/demo.ts src/types/lessonVoice.ts src/vite-env.d.ts tsconfig.app.json tsconfig.json tsconfig.node.json vite.config.ts ``` ### Dependencies - package.json: @browserbasehq/sdk@^2.14.1, @deepgram/agents@^0.1.1, @eslint/js@^10.0.1, @opentelemetry/api@^1.9.1, @opentelemetry/exporter-trace-otlp-proto@^0.219.0, @opentelemetry/resources@^2.8.0, @opentelemetry/sdk-trace-node@^2.8.0, @types/cors@^2.8.17, @types/express@^5.0.0, @types/node@^24.12.3, @types/react@^19.2.14, @types/react-dom@^19.2.3, @vitejs/plugin-react@^6.0.1, concurrently@^9.1.2, cors@^2.8.5, dotenv@^16.4.7, eslint@^10.3.0, eslint-plugin-react-hooks@^7.1.1, eslint-plugin-react-refresh@^0.5.2, express@^4.21.2, globals@^17.6.0, playwright-core@^1.61.0, react@^19.2.6, react-dom@^19.2.6, sharp@^0.35.2, tsx@^4.19.3, typescript@~6.0.2, typescript-eslint@^8.59.2, vite@^8.0.12 ### Recent commits (newest first) - Fix eval scoring for scenario premise mistakes vs vague protocol answers. - Replace VN training with continuous voice roleplay and session assessment. - Improve voice training UX and harden game generation. - Merge pull request #2 from GameStudioAI/ai-game-editing - Replace platformer runtime with visual novel engine and learning studio. - Merge pull request #1 from GameStudioAI/feature/voice-studio-gemini-sprites - Add voice studio backend, Gemini sprites, Sentry, and player HP - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "gamestudio", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "concurrently -n client,api -c cyan,magenta \"npm run dev:client\" \"npm run dev:server\"", "dev:client": "vite", "dev:server": "tsx watch server/index.ts", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview", "postinstall": "node scripts/patch-deepgram-agents.mjs" }, "dependencies": { "@browserbasehq/sdk": "^2.14.1", "@deepgram/agents": "^0.1.1", "@opentelemetry/api": "^1.9.1", "@opentelemetry/exporter-trace-otlp-proto": "^0.219.0", "@opentelemetry/resources": "^2.8.0", "@opentelemetry/sdk-trace-node": "^2.8.0", "cors": "^2.8.5", "dotenv": "^16.4.7", "express": "^4.21.2", "playwright-core": "^1.61.0", "react": "^19.2.6", "react-dom": "^19.2.6", "sharp": "^0.35.2" }, "devDependencies": { "@eslint/js": "^10.0.1", "@types/cors": "^2.8.17", "@types/express": "^5.0.0", "@types/node": "^24.12.3", "@types/react": "^19.2.14", "@types/react-dom": "^19.2.3", "@vitejs/plugin-react": "^6.0.1", "concurrently": "^9.1.2", "eslint": "^10.3.0", "eslint-plugin-react-hooks": "^7.1.1", "eslint-plugin-react-refresh": "^0.5.2", "globals": "^17.6.0", "tsx": "^4.19.3", "typescript": "~6.0.2", "typescript-eslint": "^8.59.2", "vite": "^8.0.12" } } ``` ### src/main.tsx ```typescript import { StrictMode } from 'react'; import { createRoot } from 'react-dom/client'; import './index.css'; import App from './App.tsx'; const container = document.getElementById('root'); if (!container) { throw new Error('Root element not found'); } const root = createRoot(container); root.render( , ); ``` ### server/index.ts ```typescript import cors from 'cors'; import express from 'express'; import { getAsset } from './assets.js'; import { env, requireKeys } from './env.js'; import { browserbaseRouter } from './routes/browserbase.js'; import { deepgramRouter } from './routes/deepgram.js'; import { studioRouter } from './routes/studio.js'; import { isArizeConfigured, initArizeTracing } from './arize.js'; import { isBrowserbaseConfigured } from './browserbase.js'; import { learningRouter } from './routes/learning.js'; import { trainingsRouter } from './routes/trainings.js'; const app = express(); app.use(cors()); app.use(express.json({ limit: '2mb' })); app.get('/api/health', (_req, res) => { const missing = requireKeys(); res.json({ ok: missing.length === 0, missing, deepgram: Boolean(env.deepgramApiKey), gemini: Boolean(env.geminiApiKey), browserbase: isBrowserbaseConfigured(), arize: isArizeConfigured(), }); }); app.get('/api/assets/:id', (req, res) => { const asset = getAsset(req.params.id); if (!asset) { res.status(404).json({ error: 'Asset not found' }); return; } res.setHeader('Content-Type', asset.mime); res.setHeader('Cache-Control', 'public, max-age=3600'); res.send(asset.buffer); }); app.use('/api/deepgram', deepgramRouter); app.use('/api/studio', studioRouter); app.use('/api/trainings', trainingsRouter); app.use('/api/browserbase', browserbaseRouter); app.use('/api/learning', learningRouter); app.listen(env.port, () => { const missing = requireKeys(); console.log(`API server listening on http://localhost:${env.port}`); if (missing.length) { console.warn(`Missing env: ${missing.join(', ')}`); } if (isArizeConfigured()) { initArizeTracing(); console.log('Arize tracing enabled'); } else { console.warn('Arize not configured — assessments will fail until ARIZE_API_KEY and ARIZE_SPACE_ID are set'); } if (isBrowserbaseConfigured()) { console.log('Browserbase research enabled'); } else { console.warn('Browserbase not configured — training creation will skip web research'); } }); ``` ### src/App.tsx ```typescript import { PageNav } from './components/PageNav'; import { StudioLoadingModal } from './components/StudioLoadingModal'; import { useDemoEngine } from './hooks/useDemoEngine'; import { AdsPage } from './pages/AdsPage'; import { GamePage } from './pages/GamePage'; import { LibraryPage } from './pages/LibraryPage'; import { PublishedPage } from './pages/PublishedPage'; import { VoicePage } from './pages/VoicePage'; function App() { const { state, page, voiceError, voiceAvailable, pendingStudioAction, setPage, triggerMic, handleAssessmentUpdate, libraryRefreshKey, openTraining, editTraining, } = useDemoEngine(); return (
{page === 'voice' && ( )} {page === 'library' && ( )} {page === 'game' && ( )} {page === 'published' && } {page === 'ads' && }
{pendingStudioAction && }
); } export default App; ``` ### src/game/index.ts ```typescript export { VNRuntime } from './VNRuntime'; ``` ### vite.config.ts ```typescript import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' export default defineConfig({ plugins: [react()], server: { proxy: { '/api': { target: 'http://localhost:3001', changeOrigin: true, }, }, }, }) ``` ### index.html ```html Enterprise Training Studio — Demo
``` ### eslint.config.js ```javascript import js from '@eslint/js' import globals from 'globals' import reactHooks from 'eslint-plugin-react-hooks' import reactRefresh from 'eslint-plugin-react-refresh' import tseslint from 'typescript-eslint' import { defineConfig, globalIgnores } from 'eslint/config' export default defineConfig([ globalIgnores(['dist']), { files: ['**/*.{ts,tsx}'], extends: [ js.configs.recommended, tseslint.configs.recommended, reactHooks.configs.flat.recommended, reactRefresh.configs.vite, ], languageOptions: { globals: globals.browser, }, }, ]) ``` ### src/vite-env.d.ts ```typescript /// ``` ### server/voiceAgentConfig.ts ```typescript export { buildVoiceAgentSettings, buildStudioFunctions, buildStudioSystemPrompt, STUDIO_BASE_PROMPT, } from './studioAgentConfig.js'; ``` [68 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]