# Project export: Memory Mallard This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Memory Mallard uses real-time EEG to detect lapses in attention and instantly re-engage you with our talking duck, Henry. - Devpost: https://devpost.com/software/mindful-mallard - GitHub: http://www.github.com/shawnl57/cal_hacks - Video: https://www.youtube.com/embed/7-mEhr2aCjs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Garyxue213 (11 commits), Claude (9 commits), clarkipeng (6 commits), jaredfellin567 (4 commits) ## Devpost submission (written by the team) ### Inspiration We wanted to make neurofeedback fun and easy to understand. It is usually seen in medical or research settings, filled with charts and numbers. We thought, what if your brain had a mascot instead? A duck that shows up when your mind starts to wander felt like a lighthearted and effective way to help people stay focused. What It Does Memory Mallard is a Chrome extension that uses real-time EEG data from a Muse headband to tell when your focus drops while reading online. When that happens, a small animated duck walks across the screen and fades away. It acts as a gentle reminder to bring your attention back. After you finish reading, the extension gives you short questions based on the moments when you lost focus. These questions help you recall what you might have missed and understand the material better. How We Built It Memory Mallard is built as a full system that connects brain signals to what happens on your screen. The Muse EEG headset records brain activity. A Python backend analyzes the data using signal processing and basic machine learning on brainwave patterns like theta and beta ratios. A Tauri desktop app made with React and Rust shows the data and connects to the browser through WebSocket. The Chrome extension displays the animated duck on websites when focus is lost. The duck also talks using Fish Audio’s text-to-speech API in a Donald Duck-style voice. All parts work together in a loop: Muse → Python → Tauri → Extension → Browser. This lets the system read your attention in real time and give instant feedback. --- ## Challenges We Ran Into It was hard to connect the EEG stream to the browser in a stable way. The Muse headset sends data over OSC, which is not made for the web, so we built our own bridge server. The EEG data was also noisy and different for each user, which made it difficult to find good thresholds for focus detection. Chrome’s messaging system also added some complexity since background scripts and content scripts needed to talk to each other in real time. --- ## Accomplishments We’re Proud Of We managed to bring live brainwave data into a Chrome extension, something that is rarely done. We created a focus tool that is based on real science but still feels playful and friendly. We are proud of how we balanced hardware, software, and design under time pressure. Most of all, we built something that can make people smile while helping them focus. --- ## What We Learned We learned how to connect EEG sensors with web apps and process real-time data streams. We also learned more about how focus works from both a technical and human point of view. We realized that feedback does not have to be complex to be helpful. Sometimes a small, funny reminder like a duck can be enough to bring someone back to the task. Humor and good design can make serious technology feel approachable. --- ## What’s Next for Memory Mallard We want to expand Memory Mallard beyond reading and studying. The same idea can help drivers stay alert on long trips. By using EEG to track attention and giving small sound cues when focus drops, we could help prevent fatigue and improve safety. Our long-term goal is to explore how gentle feedback from brain data can make people more aware, focused, and safe in daily life. ## README (from the GitHub repository) # 🦆 Duck Controller - Complete Application A desktop application that sends random duck messages from Python → Tauri → Browser Extension, displaying them on any webpage. ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Python Backend (Port 5000) │ │ - Random message generator │ │ - Sends messages every 5-10 seconds │ └─────────────────────┬───────────────────────────────────┘ │ HTTP POST ↓ ┌─────────────────────────────────────────────────────────┐ │ Tauri Desktop App (Rust + React) │ │ - HTTP Server (Port 3030) - receives from Python │ │ - WebSocket Server (Port 3030/ws) - sends to extension │ │ - React Dashboard - displays activity │ └─────────────────────┬───────────────────────────────────┘ │ WebSocket ↓ ┌─────────────────────────────────────────────────────────┐ │ Browser Extension (Chrome/Edge) │ │ - Background service worker │ │ - Content script injection │ │ - Displays duck messages on all web pages │ └─────────────────────────────────────────────────────────┘ ``` --- ## 🚀 Quick Start ### Prerequisites - **Rust** (latest stable) - [Install](https://rustup.rs/) - **Node.js 18+** - [Install](https://nodejs.org/) - **Python 3.8+** - [Install](https://www.python.org/) - **Chrome/Edge Browser** - **ffmpeg** - [Install](https://ffmpeg.org/) (for video generation) - **Muse EEG Headset** - [muse-lsl](https://github.com/alexandrebarachant/muse-lsl) for streaming - **API Keys**: - Anthropic Claude API key - Fish Audio API key --- ## 📦 Installation ### 1. Setup Tauri Desktop App ```bash cd calhackproj # Install dependencies npm install # Run in development mode npm run tauri dev ``` This will: - Start the React frontend on `http://localhost:1420` - Start the Rust backend with HTTP server on `http://localhost:3030` - Start WebSocket server on `ws://localhost:3030/ws` ### 2. Setup Python Backend ```bash cd python-backend # Create virtual environment (optional but recommended) python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install -r requirements.txt # Create .env file with API keys echo "ANTHROPIC_API_KEY=your_key_here" > .env echo "FISH_AUDIO_API_KEY=your_key_here" >> .env # Run the backend python main.py ``` The Python backend will: - Start Flask server on port 5000 - Connect to Muse EEG headset for focus detection - Take screenshots every 30 seconds - Generate questions with Claude AI - Create TTS audio with Fish Audio (Donald Duck voice) - Generate lip-sync videos with animated duck - Send video when user regains focus ### 3. Install Browser Extension 1. Open Chrome/Edge 2. Navigate to `chrome://extensions/` (or `edge://extensions/`) 3. Enable **Developer mode** (toggle in top-right) 4. Click **Load unpacked** 5. Select the `browser-extension` folder 6. The extension icon should appear in your toolbar --- ## 🎮 Usage ### Running the Complete System **Terminal 1: Start Tauri App** ```bash cd calhackproj npm run tauri dev ``` **Terminal 2: Start Python Backend** ```bash cd python-backend python main.py ``` **Browser: Enable Extension** - Extension auto-connects to Tauri WebSocket - Visit any website (e.g., google.com, youtube.com) - Duck messages will appear as floating notifications! --- ## 📊 What Each Component Does ### Python Backend (`python-backend/main.py`) - Connects to Muse EEG headset for real-time brain monitoring - Takes screenshots every 30 seconds - Uses Claude AI to generate questions about screen content - Generates TTS audio with Fish Audio (Donald Duck voice) - Creates lip-sync videos with animated duck mouth movements - Detects focus state changes and sends videos when focus is restored - Endpoints: - `GET /health` - Health check - `GET /api/metrics` - Current EEG metrics - `GET /video/` - Serve generated videos - `GET /screenshot/status` - Screenshot generator status - `GET /screenshot/latest` - Latest generated video path ### Tauri Backend (`calhackproj/src-tauri/src/lib.rs`) - **HTTP Server (Port 3030)** - `POST /api/message` - Receives messages from Python - `POST /api/video` - Receives video URLs from Python - `GET /health` - Health check - **WebSocket Server (Port 3030/ws)** - Broadcasts messages to all connected browser extensions - Forwards video URLs to browser for display - **Tauri Commands** - `get_service_status` - Returns status of all services ### React Frontend (`calhackproj/src/App.tsx`) - Beautiful dashboard showing: - Service status (HTTP, WebSocket, Extension) - Message count - Real-time activity log - Setup instructions ### Browser Extension (`AnnoyingDuckExtension/`) - **Background Worker** (`background.js`) - Connects to Tauri WebSocket - Auto-reconnects on disconnect - Forwards messages to content scripts - **Content Script** (`content.ts`) - Spawns animated walking duck GIFs on distractions - Displays lip-sync videos bottom-right with fade in/out - Tracks attention metrics over 2-minute timeline - Saves scroll positions when focus drops - **Popup** (`popup.html`) - Shows EEG connection status - Displays attention timeline chart - Manual quack button for testing - Settings for duck visibility --- ## 🔧 Configuration ### Python Backend Edit `python-backend/main.py`: ```python TAURI_HTTP_URL = "http://localhost:3030/api/message" MESSAGE_INTERVAL_MIN = 5 # seconds MESSAGE_INTERVAL_MAX = 10 # seconds ``` ### Browser Extension Edit `browser-extension/background.js`: ```javascript const WEBSOCKET_URL = 'ws://127.0.0.1:3030/ws'; const RECONNECT_INTERVAL = 3000; // 3 seconds const MAX_RECONNECT_ATTEMPTS = 10; ``` --- ## 🐛 Troubleshooting ### Extension Not Connecting 1. Check Tauri app is running: `http://localhost:3030/health` 2. Check WebSocket server in terminal logs 3. Open extension popup → Click "Reconnect" 4. Check browser console (F12) for errors ### Python Backend Can't Connect 1. Ensure Tauri app is running first 2. Check Tauri logs for HTTP server status 3. Try manual send: `curl -X POST http://localhost:3030/api/message -H "Content-Type: application/json" -d '{"message":"Test","timestamp":"2024-01-01T00:00:00Z","type":"test"}'` ### No Messages Appearing 1. Check Python backend logs - should show "Sent to Tauri" 2. Check Tauri logs - should show "Received from Python" 3. Check extension background worker console: - Right-click extension icon → "Inspect service worker" 4. Check webpage console (F12) - should show "Received duck message" ### Port Already in Use If port 3030 or 5000 is taken: - Change ports in code (see Configuration section) - Kill existing processes: `lsof -ti:3030 | xargs kill -9` --- ## 📝 Development ### Build for Production **Tauri App:** ```bash cd calhackproj npm run tauri build ``` **Python Backend (Executable):** ```bash cd python-backend pip install pyinstaller pyinstaller --onefile main.py ``` **Browser Extension:** - Zip the `browser-extension` folder - Upload to Chrome Web Store --- ## 🎨 Customization ### Add More Duck Messages Edit `python-backend/main.py`: ```python DUCK_MESSAGES = [ "🦆 Your custom message!", "🦆 Another message!", # Add more... ] ``` ### Change Message Display Style Edit `browser-extension/content.js` - modify the `createMessageBox()` function CSS. ### Adjust Message Frequency Edit `python-backend/main.py`: ```python MESSAGE_INTERVAL_MIN = 2 # Faster MESSAGE_INTERVAL_MAX = 5 ``` --- ## 📚 File Structure ``` annoying-duck-extension/ ├── calhackproj/ # Tauri Desktop App │ ├── src/ # React Frontend │ │ ├── App.tsx # Main dashboard │ │ └── App.css [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 263 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Rust (language) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (47 of 47) ``` .env .gitignore AnnoyingDuckExtension/manifest.json AnnoyingDuckExtension/package.json AnnoyingDuckExtension/popup.html AnnoyingDuckExtension/src/content.ts AnnoyingDuckExtension/src/popup.ts AnnoyingDuckExtension/tsconfig.json calhackproj/.gitignore calhackproj/index.html calhackproj/package.json calhackproj/python-backend/assets/example_lipsync.json calhackproj/python-backend/assets/latest_question_lipsync.json calhackproj/python-backend/attention_classifier.py calhackproj/python-backend/lipsync_generator.py calhackproj/python-backend/main.py calhackproj/python-backend/requirements.txt calhackproj/python-backend/screenshot_video_generator.py calhackproj/README.md calhackproj/setup.sh calhackproj/src-tauri/.gitignore calhackproj/src-tauri/build.rs calhackproj/src-tauri/capabilities/default.json calhackproj/src-tauri/Cargo.toml calhackproj/src-tauri/icons/icon.icns calhackproj/src-tauri/src/lib.rs calhackproj/src-tauri/src/main.rs calhackproj/src-tauri/tauri.conf.json calhackproj/src/App.css calhackproj/src/App.tsx calhackproj/src/main.tsx calhackproj/src/TypingTest.css calhackproj/src/TypingTest.tsx calhackproj/src/utils/portDiscovery.ts calhackproj/src/vite-env.d.ts calhackproj/start.sh calhackproj/tsconfig.json calhackproj/tsconfig.node.json calhackproj/vite.config.ts CHANGELOG.md PROJECT_SUMMARY.md QUICKSTART.md README_INTEGRATION.md README.md start-dev.sh start-integrated.sh TESTING.md ``` ### Dependencies - AnnoyingDuckExtension/package.json: @types/chrome@^0.1.24, typescript@^5.0.0 - calhackproj/package.json: @tauri-apps/api@^2, @tauri-apps/cli@^2, @tauri-apps/plugin-opener@^2, @types/react@^19.1.8, @types/react-dom@^19.1.6, @vitejs/plugin-react@^4.6.0, react@^19.1.0, react-dom@^19.1.0, typescript@~5.8.3, vite@^7.0.4 - calhackproj/python-backend/requirements.txt: anthropic, fish-audio-sdk, Flask@==3.0.0, Flask-CORS@==4.0.0, numpy@>=1.26.0, pillow, plotly@==5.17.0, pylsl@==1.16.2, python-dotenv, requests@==2.31.0, scipy@>=1.11.2 - calhackproj/src-tauri/Cargo.toml: axum@0.7, chrono@0.4, futures-util@0.3, reqwest@0.11, serde@1, serde_json@1, tauri@2, tauri-build@2, tauri-plugin-opener@2, tokio@1, tokio-tungstenite@0.21, tower@0.4, tower-http@0.5 ### Recent commits (newest first) - asd - breaking changes - Fix scroll navigation to properly cycle through positions - Add test distraction button for manual position logging - Add clickable navigation buttons to webpage - Disable screenshot analyzer - Add sustained focus detection with scroll tracking - Add single-command startup scripts and streamlined README - Add focus calibration, screenshot analysis, and comprehensive documentation - duck - working - working - ASIUDHSKD - Delete AnnoyinDuckExtension directory - Add files via upload - Delete browser-extension directory - Add files via upload - Add intelligent duck alert system based on 5-second attention window - Add integration documentation and start script - Add Muse 2 EEG attention tracking system ## Key source files (fetched from GitHub, selected and truncated for size) ### README_INTEGRATION.md ```markdown # Muse Attention + Duck Controller Integration ## Quick Start ```bash ./start-integrated.sh ``` ## Manual Start ### 1. Start Python Backend (Muse + API) ```bash cd calhackproj/python-backend pip install -r requirements.txt python integrated_backend.py ``` ### 2. Start Tauri App ```bash cd calhackproj npm install npm run tauri dev ``` ## Features - Real-time EEG monitoring from Muse 2 - Attention/Focus classification - Typing test for focus calibration - Duck controller integration - Browser extension support ## API Endpoints - `GET /health` - Backend health check - `GET /api/metrics` - Current brain metrics - `POST /api/calibrate` - Auto-calibrate from current state - `POST /api/calibrate-with-score` - Calibrate with typing test score - `GET /api/typing-words` - Get random typing test words - `POST /api/clear` - Clear all data buffers ## Architecture ``` ┌─────────────────┐ │ Muse 2 EEG │ └────────┬────────┘ │ LSL ▼ ┌─────────────────┐ │ Python Backend │◄──── REST API ────┐ │ (Flask:5001) │ │ └────────┬────────┘ │ │ │ ▼ │ ┌─────────────────┐ ┌──────────────┐ │ Tauri Desktop │ │ React UI │ │ (Rust HTTP) │◄───────────┤ Components │ └────────┬────────┘ └──────────────┘ │ ▼ ┌─────────────────┐ │ Browser Ext │ │ (WebSocket) │ └─────────────────┘ ``` ## Files - `calhackproj/python-backend/integrated_backend.py` - Main Flask backend - `calhackproj/python-backend/attention_classifier.py` - EEG analysis - `calhackproj/src/TypingTest.tsx` - Focus calibration UI - `calhackproj/src/App.tsx` - Main dashboard ``` ### CHANGELOG.md ```markdown # 🦆 Duck Controller - Changelog ## Version 1.1.0 - 2024-10-25 ### ✨ New Features - **Python Auto-Launch**: Rust backend now automatically launches Python backend as a subprocess - No need to run Python manually in a separate terminal! - Python process is automatically killed when Tauri app closes - Graceful error handling if Python fails to start - **Port Auto-Selection**: Python backend now tries ports 5000-5005 - No more "port already in use" errors! - Automatically finds an available port in the range ### 🔧 Improvements - Simplified startup process - only one command needed - Better error messages if Python launch fails - Clean shutdown of all services when app closes ### 📚 Documentation Updates - Updated QUICKSTART.md with simplified instructions - Added process management details to README - Updated TESTING.md with new subprocess behavior --- ## Version 1.0.0 - 2024-10-25 ### 🎉 Initial Release **Core Features:** - Python backend with random duck message generator - Tauri desktop app with HTTP + WebSocket servers - React dashboard with real-time activity log - Chrome/Edge browser extension (Manifest V3) - Beautiful gradient UI with animations - Auto-reconnection logic - Service status monitoring - Comprehensive documentation **Tech Stack:** - Python 3.8+ (Flask) - Rust (Tauri 2.0, Axum, Tokio) - TypeScript + React 18 - JavaScript (Chrome Extension) **Architecture:** ``` Python → HTTP → Rust/Tauri → WebSocket → Browser Extension ``` --- ## Upgrade Guide ### From 1.0.0 to 1.1.0 **No breaking changes!** The new version is backward compatible. **What changed:** - You no longer need to run Python manually - Python dependencies must be installed before first run: `pip3 install -r python-backend/requirements.txt` - Startup script is now optional (but still works for manual control) **To upgrade:** 1. Pull latest code 2. Install Python dependencies: `cd calhackproj && pip3 install -r ../python-backend/requirements.txt` 3. Run Tauri: `npm run tauri dev` 4. That's it! Python auto-starts now. --- ## Future Roadmap ### Version 1.2.0 (Planned) - [ ] Native messaging support (direct extension ↔ Tauri) - [ ] System tray integration - [ ] Desktop notifications - [ ] Message history persistence ### Version 1.3.0 (Planned) - [ ] Settings panel (UI for configuration) - [ ] Custom duck images/GIFs - [ ] Sound effects - [ ] Multiple message themes ### Version 2.0.0 (Future) - [ ] Python bundled as sidecar (no separate Python install needed) - [ ] Auto-updater - [ ] Plugin system - [ ] Cloud sync ``` ### AnnoyingDuckExtension/package.json ``` { "name": "annoying-duck-extension", "version": "1.0.0", "scripts": { "build": "tsc", "watch": "tsc --watch" }, "devDependencies": { "@types/chrome": "^0.1.24", "typescript": "^5.0.0" } } ``` ### calhackproj/package.json ``` { "name": "calhackproj", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc && vite build", "preview": "vite preview", "tauri": "tauri" }, "dependencies": { "react": "^19.1.0", "react-dom": "^19.1.0", "@tauri-apps/api": "^2", "@tauri-apps/plugin-opener": "^2" }, "devDependencies": { "@types/react": "^19.1.8", "@types/react-dom": "^19.1.6", "@vitejs/plugin-react": "^4.6.0", "typescript": "~5.8.3", "vite": "^7.0.4", "@tauri-apps/cli": "^2" } } ``` ### calhackproj/python-backend/requirements.txt ``` Flask==3.0.0 Flask-CORS==4.0.0 requests==2.31.0 plotly==5.17.0 pylsl==1.16.2 numpy>=1.26.0 scipy>=1.11.2 python-dotenv pillow anthropic fish-audio-sdk ``` ### calhackproj/src-tauri/Cargo.toml ``` [package] name = "calhackproj" version = "0.1.0" description = "A Tauri App" authors = ["you"] edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [lib] # The `_lib` suffix may seem redundant but it is necessary # to make the lib name unique and wouldn't conflict with the bin name. # This seems to be only an issue on Windows, see https://github.com/rust-lang/cargo/issues/8519 name = "calhackproj_lib" crate-type = ["staticlib", "cdylib", "rlib"] [build-dependencies] tauri-build = { version = "2", features = [] } [dependencies] tauri = { version = "2", features = [] } tauri-plugin-opener = "2" serde = { version = "1", features = ["derive"] } serde_json = "1" tokio = { version = "1", features = ["full"] } axum = { version = "0.7", features = ["ws"] } tower = "0.4" tower-http = { version = "0.5", features = ["cors"] } tokio-tungstenite = "0.21" futures-util = "0.3" chrono = "0.4" reqwest = { version = "0.11", features = ["json"] } ``` ### calhackproj/src/main.tsx ```typescript import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render( , ); ``` ### calhackproj/src/App.tsx ```typescript import { useState, useEffect } from "react"; import { invoke } from "@tauri-apps/api/core"; import { listen } from "@tauri-apps/api/event"; import TypingTest from "./TypingTest"; import { fetchMuseMetrics } from "./utils/portDiscovery"; import "./App.css"; interface ServiceStatus { http_server: boolean; websocket_server: boolean; extension_connected: boolean; messages_received: number; muse_connected: boolean; } interface DuckMessage { message: string; timestamp: string; type: string; } interface MuseMetrics { attention: string; focus_score: number; brain_state: string; head_orientation: string; heart_rate: number; movement_intensity: number; theta_beta_ratio: number; } function App() { const [status, setStatus] = useState(null); const [messages, setMessages] = useState([]); const [museMetrics, setMuseMetrics] = useState(null); const [isLoading, setIsLoading] = useState(true); const [showTypingTest, setShowTypingTest] = useState(false); const [museConnected, setMuseConnected] = useState(false); // Load service status async function loadStatus() { try { const result = await invoke("get_service_status"); setStatus(result); setMuseConnected(result.muse_connected); setIsLoading(false); } catch (error) { console.error("Failed to load status:", error); setIsLoading(false); } } // Listen for duck messages from Rust backend useEffect(() => { const unlisten = listen("duck-message", (event) => { console.log("Received duck message:", event.payload); setMessages((prev) => [event.payload, ...prev].slice(0, 50)); // Keep last 50 messages }); return () => { unlisten.then((fn) => fn()); }; }, []); // Load status on mount and every 5 seconds useEffect(() => { loadStatus(); const interval = setInterval(loadStatus, 5000); return () => clearInterval(interval); }, []); // Fetch Muse metrics every 500ms (for displaying metrics, not connection status) useEffect(() => { async function loadMetrics() { try { const data = await fetchMuseMetrics(); setMuseMetrics(data); } catch (error) { setMuseMetrics(null); } } loadMetrics(); const interval = setInterval(loadMetrics, 500); return () => clearInterval(interval); }, []); const getStatusColor = (isActive: boolean) => { return isActive ? "#00ff00" : "#ff3333"; }; const getStatusText = (isActive: boolean) => { return isActive ? "Running" : "Stopped"; }; const getFocusColor = (attention: string) => { if (attention.toLowerCase().includes("high") || attention.toLowerCase().includes("focused")) { return "#00ff00"; } else if (attention.toLowerCase().includes("medium")) { return "#ffaa00"; } else { return "#ff3333"; } }; if (showTypingTest) { return ; } if (isLoading) { return (

🦆 Duck Controller + 🧠 Muse Monitor

Loading...

); } return (

🦆 Duck Focus Monitor

Real-time Brain Activity & Distraction Detection

{/* EEG Connection Status Banner */}
{museConnected ? '🧠' : '⚠️'}

{museConnected ? 'Muse EEG Connected' : 'Muse EEG Disconnected'}

{museConnected ? 'Actively monitoring your brain activity and focus levels' : 'Please connect your Muse headset to start monitoring'}

{/* Muse Brain Metrics */} {museMetrics && museConnected && (

🧠 Live Brain Metrics

🎯 Focus Level
{museMetrics.attention}
{(museMetrics.focus_score * 100).toFixed(0)}% focused
🧘 Brain State
{museMetrics.brain_state}
🔄 Head Position
{museMetrics.head_orientation}