# Project export: Curious Catalyst This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Curiosity shouldn't be prompted. Let Plato be your curious catalyst. - Devpost: https://devpost.com/software/curiosity-gabstn - GitHub: https://github.com/laasya-konidala/TreeHacks-2026 - Video: https://www.youtube.com/embed/yjS9LLQ7KOE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 5 GitHub contributor(s) — Cursor (22 commits), ishajain985 (19 commits), madihuang8 (7 commits), Arya Gupta (5 commits), Claude Opus 4.6 (2 commits) ## Devpost submission (written by the team) ### Inspiration LLM usage decreases critical thinking, but having so much AI technology at your fingertips should do the exact opposite! In 36 hours, we built Curious Catalyst, starring Plato, a very curious, very alive cat that lives on your screen. ### What it does Plato is a floating cat that lives on your desktop and observes your actions with the computer, the physical content on your computer, and infers your level of understanding/state of mind. When activated, Plato reveals a side panel that presents our learning tools. We have three use cases of learning agents: 1) Conceptual learning which revisits topics and strengthens foundations, 2) Application-based learning which allows for iterations on problems to strengthen problem-solving, 3) Extension-based learning to further your understanding Plato can learn about the way you learn, how you're learning - through Bayesian Knowledge Tracing - and deploy the right agent, dynamically switching to give you the help you need - completely unprompted. Alongside inter-agent communication and content generation, Plato decides whether to generate dynamic, customizable visualizations (to help layer abstractions) or a live phone call integrated with realtime speech-to-speech models that are adjusted to your understanding and pace of learning to walk you through the concept. ### How we built it The live video feed of the model is handled through a two pronged approach - Gemini’s VLM model and Zoom’s Screenshare Render to analyze the stream of content on the screen, user activity. Plato uses Bayesian Knowledge Tracing to attempt to infer how much the user actually knows, and by consequence determine the intervention point of the model. Thus, Plato actively tracks mastery by building a confidence weighted map of your level of understanding and determining which lens of learning you would most benefit from. Plato’s agents interact and are developed through fetch.ai’s agent tooling and are hosted on AgentVerse, and are findable on ASI:One. There are three agents, each for a different lens of learning. They have different decision making capabilities on which tools to call, when to deploy, and how to engage with the user. There is an orchestrate agent, that handles transitions between different learning agents, keeping the transitions seamless while keeping the model personalized. Visualizations are generated dynamically based on your learning habits, live behavior, and the content on your screen. We also use OpenAI's speech to speech API, where the side panel will prompt you with an “incoming” phone call to dive deeper into a concept, either for more foundational explanation or more in-depth innovation. As for Plato, we did our own art, assisted by Sora to bring him to life. ### Challenges we ran into We faced many challenges bringing Curious Catalyst to life. Our core philosophy was that curiosity shouldn't be prompted—but this created our biggest design challenge. How do you build a system that knows when to intervene without the user explicitly asking? We had to architect an entire behavioral learning system that passively observes screen content, tracks micro-interactions (re-reading the same line, pausing on a problem, scrolling back), and infers cognitive state in real-time. The agents needed to collaborate seamlessly while determining intervention timing that felt helpful rather than intrusive. Getting multiple agents to agree on "now is the right moment" without constant user input required building sophisticated inter-agent communication protocols and confidence thresholds. We also spent a lot of time thinking about when should the conceptual agent hand off to the problem-solving agent? Switching too early leaves foundations shaky; switching too late frustrates users who are ready to move forward. To do this, we had to build out state synchronization: each agent needed access to shared context (what the user is working on, what they've tried, their current affect) while maintaining specialized decision-making capabilities. ### Accomplishments we're proud of and what's next for Curious Catalyst As knowledge progresses and the sheer power of AI’s ability to provide endless, customizable knowledge becomes greater, we aim to break the glass ceiling for more concepts. On top of improving the customizability of visuals, we also seek to extend this to dense scientific information and wet-lab experimental workflow. It would also be beneficial to extend Curious Catalyst to thesis and argument defensibility, or information retainment from Additionally, we could use this tool to keep under-resourced schools with a large number of students more focused and on track. Since Plato meets the learner where they are, extending it with some preset curriculum, such as Common Core can make learning and teaching standards more accessible. Furthermore, it can make home-schooling more equitable and robust for those with extenuating circumstances. In a world where literacy rates and mathematical ability correlates to incarceration rates, your ability to learn is the most important thing you have. On the other end of the spectrum, innovation happens every single second. Learning in any field being endlessly customizable, iterative, and active will improve the ability of the world to learn. ### What we learned Multi-agent orchestration is deceptively complex. Coordinating three specialized learning agents isn't just about message passing - it requires sophisticated state management, fault tolerance, and seamless handoffs. Vision models have wildly different strengths. We discovered that no single VLM excels at everything. Screen analysis quality varies dramatically depending on the task—some models better understand mathematical notation, others excel at detecting user frustration signals. Bayesian Knowledge Tracing is elegant and honest. Rather than binary right/wrong tracking, BKT's probabilistic approach to modeling mastery feels more authentic to how humans actually learn. Ambient intelligence requires invisibility. We learned that effective ambient AI must be felt, not seen—present when needed, invisible when not. Plato, the cat avatar, helped make monitoring feel friendly rather than surveillant. ## README (from the GitHub repository) # Ambient Learning Agent System ![tag:innovationlab](https://img.shields.io/badge/innovationlab-3D8BD3) ![tag:hackathon](https://img.shields.io/badge/hackathon-5F43F1) A multi-agent AI tutoring system that **observes what you're studying** and **helps you learn** through contextual questions, visualizations, and guided problem-solving — all without interrupting your flow. Built on [Fetch.ai](https://fetch.ai) uAgents with ASI:One Chat Protocol and Payment Protocol for discoverability and monetization on [Agentverse](https://agentverse.ai). ## Agents | Agent | Name | Description | |-------|------|-------------| | **Orchestrator** | `learning_orchestrator` | Central brain — ASI:One entry point. Detects when to prompt, routes to specialist agents, tracks mastery via BKT. Includes Chat + Payment protocols. | | **Conceptual** | `conceptual_understanding` | Helps students build knowledge via contextual questions and visualizations when watching videos or reading notes. | | **Applied** | `applied_problem_solving` | Scaffolds reasoning for active problem-solving — guides without giving answers. | | **Extension** | `extension_stretch` | Pushes students to make cross-topic connections and tackle stretch challenges. | | **Monitor** | `metrics_monitor` | Sends metrics triggers to orchestrator using ASI-1 ChatProtocol. | ## Architecture ``` ASI:One User │ ChatMessage (chat_protocol_spec) ▼ ┌─────────────────────────────────────────────┐ │ Orchestrator (learning_orchestrator) │ │ ├── Chat Protocol (ASI:One discoverable) │ │ ├── Payment Protocol (FET monetization) │ │ ├── BKT Learner Model │ │ └── Timing Logic (when to prompt) │ └──────────┬──────────┬──────────┬────────────┘ │ │ │ ┌──────▼──┐ ┌─────▼────┐ ┌──▼──────────┐ │Conceptual│ │ Applied │ │ Extension │ │ Agent │ │ Agent │ │ Agent │ └─────────┘ └──────────┘ └─────────────┘ │ │ │ └──────────┴──────────┘ │ Claude LLM (exercise generation) ``` **Data flow:** 1. Electron desktop overlay captures screen → Gemini VLM analysis → FastAPI backend 2. Orchestrator polls backend, updates BKT mastery model, detects natural prompt moments 3. Routes to conceptual/applied/extension agent based on activity mode 4. Agent generates contextual exercise via Claude → response sent to sidebar via WebSocket ## Setup ### Prerequisites - Python 3.10+ - Node.js 18+ (for Electron overlay) - API keys: Gemini, Anthropic (Claude) ### Install ```bash # Python dependencies pip install -r requirements.txt # Electron overlay npm install ``` ### Environment Variables ```bash export GEMINI_API_KEY=your_gemini_key export ANTHROPIC_API_KEY=your_anthropic_key # To enable Agentverse registration: export AGENTVERSE_ENABLED=true # For FET payment (testnet by default): export FET_USE_TESTNET=true ``` ### Run ```bash # Start all agents + API server python run.py # In a separate terminal, start the Electron overlay npm start ``` ### Agent Addresses To see all agent addresses and registration status: ```bash python scripts/register_agentverse.py ``` ## Agentverse Deployment 1. Set `AGENTVERSE_ENABLED=true` and run `python run.py` 2. The orchestrator auto-registers on Agentverse with `mailbox=True` 3. Open the **Local Agent Inspector** URL from terminal output to connect 4. The Chat Protocol manifest is published — ASI:One can discover the agent 5. Go to [ASI:One](https://asi1.ai), enable "Agents" toggle, and search for your agent ## Monetization The Payment Protocol supports FET token payments on the Fetch.ai blockchain: - **Free tier**: 3 interventions/day, behavioral detection only - **Premium tier**: Unlimited interventions + screen analysis + multi-turn dialogue (0.1 FET) - **Per-mastery tier**: Charge on concept mastery milestones ## Tech Stack - **Agents**: Fetch.ai uAgents framework - **LLMs**: Claude (exercise generation), Gemini (screen analysis, chat) - **Frontend**: Electron desktop overlay + Chrome extension - **Backend**: FastAPI + WebSocket - **Learning Model**: Confidence-Weighted Bayesian Knowledge Tracing (BKT) ## Extra Resources - Chrome extension for behavioral signals: `learning-companion-extension/` - Demo scenario: `scripts/demo_scenario.py` - Vision pipeline test: `scripts/test_vision_pipeline.py` ## Detected evidence (automated analysis) Indexed codebase: 44 recognized source files, 332 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - TypeScript (language) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (50 of 50) ``` .gitignore agents/__init__.py agents/agent_applied.py agents/agent_conceptual.py agents/agent_extension.py agents/assessor.py agents/chat_protocol.py agents/config.py agents/deep_diver.py agents/dialogue_session.py agents/learner_model.py agents/models.py agents/monitor.py agents/observation_pipeline.py agents/orchestrator.py agents/payment_protocol.py agents/tools/__init__.py agents/tools/tool_quiz.py agents/tools/tool_review.py agents/tools/tool_visualization.py agents/visualizer.py character.html chrome-extension/background.js chrome-extension/content.js chrome-extension/manifest.json DEMO_SCRIPT.md input_pipeline/__init__.py input_pipeline/screen_analyzer.py input_pipeline/server.py input_pipeline/zoom_client.py learning-companion-extension/background.js learning-companion-extension/manifest.json learning-companion-extension/src/avatar.css learning-companion-extension/src/content.js learning-companion-extension/src/sidebar.html main.js ngrok overlay.html package.json preload.js README.md renderer/overlay.js requirements.txt run.py scripts/demo_scenario.py scripts/register_agentverse.py scripts/setup_agentverse.py scripts/test_vision_pipeline.py tests/__init__.py tests/test_bkt.py ``` ### Dependencies - package.json: @anthropic-ai/sdk@^0.39.0, @google/genai@^1.41.0, dotenv@^16.6.1, electron@^33.4.11, ws@^8.19.0 - requirements.txt: anthropic@>=0.40.0, cosmpy@>=0.11.0, fastapi@>=0.109.0, google-genai@>=1.0.0, httpx@>=0.26.0, manim@>=0.18.0, pydantic@>=2.5.0, pytest@>=7.4.0, python-dotenv@>=1.0.0, uagents@>=0.23.0, uagents-core@>=0.4.0, uvicorn[standard]@>=0.27.0, websockets@>=12.0 ### Recent commits (newest first) - final? - Merge branch 'main' of https://github.com/laasya-konidala/TreeHacks-2026 - Rename to Curious Cat-alyst in sidebar header - UI: replace emojis with color dot, clear mode labels - Fix vertical 'No content' fallback card + harden D3 error handling - Merge branch 'main' of https://github.com/laasya-konidala/TreeHacks-2026 - Fix viz card clipping: re-scroll after render, compact layout - saving voice - Demo timing + fix visualization cutoff in sidebar - Rebalance tool selection: favor visualization, fix detection - Merge branch 'main' of https://github.com/laasya-konidala/TreeHacks-2026 - removed end call - Avatar state machine, agent ring colors, and visualization-first agents - voice call - Harden D3 code generation and fix apostrophe syntax errors - Smart tool selection with forced rotation after 2 consecutive - Fix sidebar UX: stop VLM feed from burying visualizations - phone icon back - Merge remote: resolve conflicts in server.py and requirements.txt - launch zoom meeting with agent ## Key source files (fetched from GitHub, selected and truncated for size) ### DEMO_SCRIPT.md ```markdown # Plato — 1-Minute Demo Script **Duration:** 1 minute **Goal:** Plato knows when you’re stuck, watches with permission, decides when to prompt — no typing “I’m stuck” needed. --- ## Script (~1 min) **"When you’re stuck, you usually have to stop and type ‘I’m stuck.’ We flipped that. This is Plato — an ambient learning companion."** **"You don’t need to prompt when you’re stuck; Plato already knows. You give permission once, and Plato watches every tab and document you’re on. A vision-language model reads your screen in real time, and Plato decides when to prompt you: at natural pauses, when you’re stuck, or when you switch topics. It’s autonomous: no prompting required."** **"Under the hood: we deployed three AI agents on Fetch.ai and two tools — voice call for oral concept checks with Plato, and visualization. We have a Conceptual agent when you’re reading or watching, an Applied agent when you’re solving problems or coding, and an Extension agent when you’re ready to stretch and connect ideas. Right now I’m working on a math problem, so our Applied agent is active. We use LaTeX, D3, Plotly, and Manim so you get active, interactive visuals — equations, diagrams, plots, and 3Blue1Brown-style animations — all driven by what’s on your screen."** **[Demo beat]** Show one intervention (visualization or voice). *"I didn’t type anything. Plato saw the screen and prompted me."* **"That’s Plato — when you’re stuck, it already knows."** --- ## One-liners (if you need to cut or reorder) | Topic | Line | |-------|------| | No prompt | "You don’t need to prompt when you’re stuck — Plato already knows." | | Permission | "When you give permission, Plato watches every tab or document you’re on." | | Who prompts | "Plato decides when to prompt for you." | | VLM | "A vision-language model interprets your screen in real time." | | Stack | "Three agents on Fetch.ai, two tools: voice call and visualization. LaTeX, D3, Plotly, Manim for interactive visuals." | | Oral | "Oral concept checks with Plato — talk through your reasoning." | --- ## Backup - **VLM slow:** Run `python scripts/demo_scenario.py` to simulate stuck context and show agent responses. - **Voice not working:** Say "Plato can also do voice concept checks — we’re showing the visualization here." ``` ### requirements.txt ``` # Ambient Learning Agent System — Python dependencies python-dotenv>=1.0.0 uagents>=0.23.0 uagents-core>=0.4.0 fastapi>=0.109.0 uvicorn[standard]>=0.27.0 httpx>=0.26.0 google-genai>=1.0.0 anthropic>=0.40.0 pydantic>=2.5.0 websockets>=12.0 cosmpy>=0.11.0 manim>=0.18.0 pytest>=7.4.0 ``` ### package.json ``` { "name": "learning-overlay", "version": "1.0.0", "description": "Desktop overlay learning agent with Claude VLM", "main": "main.js", "scripts": { "start": "unset ELECTRON_RUN_AS_NODE && electron ." }, "dependencies": { "@anthropic-ai/sdk": "^0.39.0", "@google/genai": "^1.41.0", "dotenv": "^16.6.1", "ws": "^8.19.0" }, "devDependencies": { "electron": "^33.4.11" } } ``` ### main.js ```javascript const path = require('path'); const http = require('http'); const url = require('url'); require('dotenv').config({ path: path.join(__dirname, '.env') }); const { app, BrowserWindow, screen, desktopCapturer, ipcMain, systemPreferences, shell } = require('electron'); const WebSocket = require('ws'); const Anthropic = require('@anthropic-ai/sdk'); // ─── Config ──────────────────────────────────────────────────────── const ANTHROPIC_API_KEY = process.env.ANTHROPIC_API_KEY; const BACKEND_URL = process.env.BACKEND_URL || 'http://localhost:3000'; const CAPTURE_INTERVAL_MS = 8000; // 8s — balances cost vs responsiveness const CAPTURE_WIDTH = 1280; const CAPTURE_HEIGHT = 720; const VLM_MODEL = 'claude-haiku-4-5'; // cheap + fast for frequent VLM const VLM_MAX_TOKENS = 800; // enough for full JSON with screen_details // ─── State ───────────────────────────────────────────────────────── let sidebarWindow = null; let characterWindow = null; let sidebarVisible = false; let currentAvatar = 'plato'; // 'plato' | 'einstein' let captureInterval = null; let sessionActive = false; let claude = null; // Anthropic client let contextBuffer = []; // rolling buffer of recent observations const MAX_CONTEXT = 10; let agentWs = null; // WebSocket to Python agent backend let zoomMeetingWindow = null; // ─── Create floating character window ─────────────────────────────── function createCharacter() { const { width: screenWidth, height: screenHeight } = screen.getPrimaryDisplay().bounds; characterWindow = new BrowserWindow({ width: 130, height: 130, x: screenWidth - 490, y: screenHeight - 140, frame: false, transparent: true, alwaysOnTop: true, resizable: false, skipTaskbar: true, backgroundColor: '#00000000', hasShadow: false, webPreferences: { preload: path.join(__dirname, 'preload.js'), nodeIntegration: false, contextIsolation: true, }, }); characterWindow.loadFile(path.join(__dirname, 'character.html')); characterWindow.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true }); characterWindow.setIgnoreMouseEvents(false); characterWindow.webContents.once('did-finish-load', () => { characterWindow.webContents.send('avatar', currentAvatar); }); } // ─── Move both windows together (fixed relative position) ────────── ipcMain.on('move-character', (_event, dx, dy) => { if (characterWindow && !characterWindow.isDestroyed()) { const [x, y] = characterWindow.getPosition(); characterWindow.setPosition(x + dx, y + dy); } if (sidebarWindow && !sidebarWindow.isDestroyed()) { const [x, y] = sidebarWindow.getPosition(); sidebarWindow.setPosition(x + dx, y + dy); } }); // ─── Toggle sidebar visibility ───────────────────────────────────── ipcMain.on('toggle-sidebar', () => { if (!sidebarWindow || sidebarWindow.isDestroyed()) return; sidebarVisible = !sidebarVisible; if (sidebarVisible) { sidebarWindow.show(); // Switch avatar back to rest when sidebar opens if (characterWindow && !characterWindow.isDestroyed()) { characterWindow.webContents.send('avatar-state', 'rest'); } } else { sidebarWindow.hide(); } if (characterWindow && !characterWindow.isDestroyed()) { characterWindow.webContents.send('sidebar-visibility', sidebarVisible); } }); // ─── Avatar selection (from sidebar dropdown) ─────────────────────── ipcMain.on('set-avatar', (_event, avatar) => { currentAvatar = avatar; if (characterWindow && !characterWindow.isDestroyed()) { characterWindow.webContents.send('avatar', avatar); } }); // ─── Create the sidebar overlay window ───────────────────────────── function createSidebar() { const { width: screenWidth, height: screenHeight } = screen.getPrimaryDisplay().bounds; const panelWidth = 380; sidebarWindow = new BrowserWindow({ width: panelWidth, height: screenHeight, x: screenWidth - panelWidth, y: 0, frame: false, transparent: true, alwaysOnTop: true, resizable: false, skipTaskbar: true, backgroundColor: '#00000000', hasShadow: false, webPreferences: { preload: path.join(__dirname, 'preload.js'), nodeIntegration: false, contextIsolation: true, }, }); sidebarWindow.loadFile(path.join(__dirname, 'overlay.html')); sidebarWindow.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true }); sidebarWindow.setIgnoreMouseEvents(false); sidebarWindow.once('ready-to-show', () => { sidebarWindow.hide(); sendStatus('ready', 'Click the toggle to start a session.'); }); } // ─── Claude Vision (screen analysis) ─────────────────────────────── const SYSTEM_PROMPT = `You are an intelligent learning assistant observing a student's screen via periodic screenshots. Analyze each screenshot and output a JSON summary: { "activity": "what the student is doing right now", "topic": "subject/topic (e.g. eigenvalues, gradient_descent, photosynthesis)", "subtopic": "more specific sub-topic if identifiable", "mode": "CONCEPTUAL | APPLIED | CONSOLIDATION", "stuck": true/false, "work_status": "correct | incorrect | incomplete | unclear", "content_type": "code | equation | text | diagram | video | mixed", "error_description": null or "specific error if work is incorrect", "natural_pause": true/false, "screen_details": "VERY SPECIFIC description of what is visible on screen — read out exact text, equations, variable names, code snippets, question text, diagram labels, video titles, slide headings. Be as literal as possible so a tutor who cannot see the screen knows exactly what the student is looking at." } Modes (pick one based on what the student is DOING): - CONCEPTUAL: watching a video, reading notes/textbook, learning new theory - APPLIED: solving problems, writing code, doing exercises, practicing - CONSOLIDATION: reviewing notes, summarizing, making flashcards, organizing Timing cues — set natural_pause to true if: - A vi [truncated — 12591 more characters] ``` ### input_pipeline/server.py ```python """ FastAPI server — context ingestion, WebSocket broadcast, reply forwarding. Merges data from two sources: 1. Electron/Gemini VLM (screen analysis: topic, stuck, work_status, confusion) 2. Chrome extension (behavioral: typing speed, deletions, pauses, scroll-back) Runs on BACKEND_PORT from config (default 3000). Also: Zoom OAuth and meeting creation. """ from pathlib import Path from dotenv import load_dotenv load_dotenv(Path(__file__).resolve().parent.parent / ".env") import asyncio import json import logging import os import subprocess import tempfile import time import uuid from pathlib import Path from typing import Optional from fastapi import FastAPI, Query, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import RedirectResponse from fastapi.staticfiles import StaticFiles from pydantic import BaseModel import httpx from input_pipeline.zoom_client import ( exchange_code_for_tokens, get_authorize_url, get_or_create_persistent_meeting, is_connected, reset_persistent_meeting, ) logger = logging.getLogger(__name__) app = FastAPI(title="Ambient Learning Agent Server") app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"], ) # ─── Manim output directory & static serving ─── MANIM_OUTPUT_DIR = Path(__file__).resolve().parent.parent / "manim_output" MANIM_OUTPUT_DIR.mkdir(exist_ok=True) app.mount("/video", StaticFiles(directory=str(MANIM_OUTPUT_DIR)), name="manim_videos") # Track render jobs: job_id → { status, url, error } _manim_jobs: dict[str, dict] = {} # ─── In-memory state ─── # Stores the latest MERGED context (Gemini VLM + behavioral signals) latest_context: dict = {"data": None, "timestamp": 0} # Separate buffers for the two data sources so we can merge them _gemini_buffer: dict = {"data": None, "timestamp": 0} _behavioral_buffer: dict = {"data": None, "timestamp": 0} ws_clients: list[WebSocket] = [] reply_queue: asyncio.Queue = asyncio.Queue() # ─── Models ─── class TouchRequest(BaseModel): message: str = "" user_id: str = "default" class CreateMeetingRequest(BaseModel): topic: str = "Learning Companion Call" # ─── Context Merging ─── def _merge_context() -> dict: """ Merge Gemini VLM analysis with Chrome extension behavioral signals into a single WorkContext dict. Gemini provides the 'eyes' (topic, stuck, work_status), Chrome provides the 'fingers' (typing, deletions, pauses, scrolling). """ gemini = _gemini_buffer["data"] or {} behavioral = _behavioral_buffer["data"] or {} merged = { # Gemini VLM provides these (it actually sees the screen) "screen_content": gemini.get("screen_content") or behavioral.get("screen_content", ""), "screen_content_type": gemini.get("screen_content_type") or behavioral.get("screen_content_type", "text"), "detected_topic": gemini.get("detected_topic") or behavioral.get("detected_topic", ""), "detected_subtopic": gemini.get("detected_subtopic") or behavioral.get("detected_subtopic", ""), # Chrome extension provides these (it monitors keystrokes) "typing_speed_ratio": behavioral.get("typing_speed_ratio", gemini.get("typing_speed_ratio", 1.0)), "deletion_rate": behavioral.get("deletion_rate", gemini.get("deletion_rate", 0.0)), "pause_duration": behavioral.get("pause_duration", gemini.get("pause_duration", 0.0)), "scroll_back_count": behavioral.get("scroll_back_count", gemini.get("scroll_back_count", 0)), # Verbal cues can come from either source "audio_transcript": gemini.get("audio_transcript") or behavioral.get("audio_transcript"), "verbal_confusion_cues": ( gemini.get("verbal_confusion_cues", []) + behavioral.get("verbal_confusion_cues", []) ), # Touch / user message — either source "user_touched_agent": ( behavioral.get("user_touched_agent", False) or gemini.get("user_touched_agent", False) ), "user_message": behavioral.get("user_message") or gemini.get("user_message"), # No screenshot needed (Gemini already analyzed the screen) "screenshot_b64": None, # IDs "user_id": behavioral.get("user_id") or gemini.get("user_id", "default"), "session_id": gemini.get("session_id") or behavioral.get("session_id", ""), "timestamp": gemini.get("timestamp") or behavioral.get("timestamp", ""), # Gemini VLM analysis fields "gemini_stuck": gemini.get("gemini_stuck", False), "gemini_work_status": gemini.get("gemini_work_status", "unclear"), "gemini_confused_about": gemini.get("gemini_confused_about", []), "gemini_understands": gemini.get("gemini_understands", []), "gemini_error": gemini.get("gemini_error"), "gemini_mode": gemini.get("gemini_mode", ""), "gemini_notes": gemini.get("gemini_notes", ""), "gemini_screen_details": gemini.get("gemini_screen_details", ""), "gemini_natural_pause": gemini.get("gemini_natural_pause", False), } return merged # ─── Endpoints ─── @app.post("/context") async def receive_context(ctx: dict): """ Receive context from either Gemini (Electron) or Chrome extension. Automatically detects the source and updates the right buffer, then merges into the unified context that the orchestrator polls. """ source = ctx.get("_source", "") if source == "chrome_extension": # Behavioral signals from Chrome extension _behavioral_buffer["data"] = ctx _behavioral_buffer["timestamp"] = time.time() else: # Gemini VLM analysis from Electron (or direct POST) _gemini_buffer["data"] = ctx _gemini_buffer["timestamp"] = time.time() # Merge both sources into the unified context latest_context["data"] = _merge_context() latest_context["time [truncated — 11477 more characters] ``` ### preload.js ```javascript const { contextBridge, ipcRenderer } = require('electron'); contextBridge.exposeInMainWorld('api', { // Receive Gemini text responses in the sidebar onGeminiResponse: (callback) => { ipcRenderer.on('gemini-response', (_event, data) => callback(data)); }, // Receive status updates onStatusUpdate: (callback) => { ipcRenderer.on('status-update', (_event, data) => callback(data)); }, // Send speech transcript to main process sendSpeechTranscript: (transcript) => { ipcRenderer.send('speech-transcript', transcript); }, // Tell main process to start/stop the session toggleSession: () => { ipcRenderer.send('toggle-session'); }, // Toggle sidebar visibility (from character window) toggleSidebar: () => { ipcRenderer.send('toggle-sidebar'); }, // Move the character window by a delta (for manual drag) moveWindow: (dx, dy) => { ipcRenderer.send('move-character', dx, dy); }, // Mic start/stop signals from main onStartMic: (callback) => { ipcRenderer.on('start-mic', (_event) => callback()); }, onStopMic: (callback) => { ipcRenderer.on('stop-mic', (_event) => callback()); }, // Receive agent responses forwarded from the Python backend via WebSocket onAgentResponse: (callback) => { ipcRenderer.on('agent-response', (_event, data) => callback(data)); }, // Simple agent trigger (Gemini detected user is stuck/confused) onAgentTriggered: (callback) => { ipcRenderer.on('agent-triggered', (_event, data) => callback(data)); }, // Session state for character pulse animation onSessionState: (callback) => { ipcRenderer.on('session-state', (_event, active) => callback(active)); }, // Sidebar open/closed so character can switch plato vs plato2 animation onSidebarVisibility: (callback) => { ipcRenderer.on('sidebar-visibility', (_event, open) => callback(open)); }, // Avatar choice for character window (plato | socrates) onAvatar: (callback) => { ipcRenderer.on('avatar', (_event, avatar) => callback(avatar)); }, // Sidebar: send avatar selection to main setAvatar: (avatar) => { ipcRenderer.send('set-avatar', avatar); }, // Zoom meeting openExternal: (url) => { ipcRenderer.send('open-external', url); }, launchZoomMeeting: (url) => { ipcRenderer.send('launch-zoom-meeting', url); }, closeZoomMeeting: () => { ipcRenderer.send('close-zoom-meeting'); }, onZoomMeetingClosed: (callback) => { ipcRenderer.on('zoom-meeting-closed', () => callback()); }, // Avatar state: 'rest' (plato_at_rest) or 'active' (plato wants attention) onAvatarState: (callback) => { ipcRenderer.on('avatar-state', (_event, state) => callback(state)); }, // Agent mode: 'conceptual' | 'applied' | 'extension' — controls ring color onAgentMode: (callback) => { ipcRenderer.on('agent-mode', (_event, mode) => callback(mode)); }, }); ``` ### run.py ```python """ Main entry point — starts Bureau (agents) + FastAPI server together. Runs: - Orchestrator (routes by VLM context, ASI:One chat + payment protocols) - Conceptual (building knowledge) - Applied (problem solving & scaffolding) - Extension (stretch & connect) - Monitor (metrics trigger via ASI-1 ChatProtocol) """ from pathlib import Path from dotenv import load_dotenv load_dotenv(Path(__file__).parent / ".env") import logging import os import threading from dotenv import load_dotenv load_dotenv() # Load .env before anything reads os.environ import uvicorn from uagents import Bureau from agents.config import ( BACKEND_HOST, BACKEND_PORT, AGENTVERSE_ENABLED, ORCHESTRATOR_PORT, CONCEPTUAL_PORT, APPLIED_PORT, EXTENSION_PORT, MONITOR_PORT, ) # Configure logging logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(name)s] %(levelname)s: %(message)s", datefmt="%H:%M:%S", ) logger = logging.getLogger(__name__) def start_api(): """Start FastAPI server in background thread.""" uvicorn.run( "input_pipeline.server:app", host=BACKEND_HOST, port=BACKEND_PORT, log_level="info", ) if __name__ == "__main__": # Start FastAPI in background thread unless server is already running elsewhere (e.g. terminal 1) skip_api = os.environ.get("SKIP_API", "").lower() in ("1", "true", "yes") if not skip_api: api_thread = threading.Thread(target=start_api, daemon=True) api_thread.start() logger.info(f"API server starting on http://localhost:{BACKEND_PORT}") else: logger.info(f"SKIP_API=1: not starting API (expect server at http://localhost:{BACKEND_PORT})") # Import agents from agents.orchestrator import orchestrator from agents.agent_conceptual import conceptual_agent from agents.agent_applied import applied_agent from agents.agent_extension import extension_agent from agents.monitor import monitor # Start all agents via Bureau bureau = Bureau() bureau.add(orchestrator) bureau.add(conceptual_agent) bureau.add(applied_agent) bureau.add(extension_agent) bureau.add(monitor) print("\n" + "=" * 64) print(" AMBIENT LEARNING AGENT SYSTEM") print("=" * 64) print(f" Orchestrator: {orchestrator.address}") print(f" Port: {ORCHESTRATOR_PORT}") print(f" Protocols: ChatProtocol (ASI:One), PaymentProtocol (FET)") print(f" Conceptual: {conceptual_agent.address}") print(f" Port: {CONCEPTUAL_PORT}") print(f" Applied: {applied_agent.address}") print(f" Port: {APPLIED_PORT}") print(f" Extension: {extension_agent.address}") print(f" Port: {EXTENSION_PORT}") print(f" Monitor: {monitor.address}") print(f" Port: {MONITOR_PORT}") print("=" * 64) print(f" API Server: http://localhost:{BACKEND_PORT}") print(f" WebSocket: ws://localhost:{BACKEND_PORT}/ws") print(f" Health: http://localhost:{BACKEND_PORT}/health") print("=" * 64) print(f" Routing: CONCEPTUAL → conceptual | APPLIED → applied | CONSOLIDATION → extension") print(f" Agentverse: {'ENABLED — agents will register' if AGENTVERSE_ENABLED else 'disabled (local only)'}") if AGENTVERSE_ENABLED: print(f" ASI:One: Orchestrator discoverable via Chat Protocol") print(f" Monetization: Payment Protocol active (FET)") print("=" * 64 + "\n") bureau.run() ``` ### character.html ```html
Plato