# Project export: Reflections This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Be greater. Be yourself. Reflections is a desktop AI agent that keeps you productive by intervening - as you. - Devpost: https://devpost.com/software/reflection-0v4ijl - GitHub: https://github.com/coach-calhacks/reflections - Video: https://www.youtube.com/embed/pAAL7KySvqE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — caden li (35 commits), “RichardBuild” (28 commits), Mohul Shukla (9 commits), Sunith (8 commits) ## Devpost submission (written by the team) ### Inspiration In a world of distractions, it’s hard to stay focused on your dreams. Most productivity tools only add to the chaos with more dashboards, inboxes, and alerts. We believe the best thing to hold you accountable - is you. So we built Reflections to be different. It uses your own signals - your calendar, your email, your story - to create a blueprint for a more productive you, synthesizing them into meaningful insights every day, and coaching you toward better habits, clearer priorities, and deeper focus long term. No more noise. Just real reflection and real change. ### What it does Your Reflection Understands Your Focus in Real-Time. Reflections securely connects to your work signals. By passively understanding your screen activity and analyzing communication patterns from your Gmail, Google Calendar, Notion, it builds a private, real-time picture of what you're working on, where your time is going, and when you are truly focused. But Reflection goes beyond data. It begins with a guided conversation to understand the intangibles that make you who you are: your story, your dreams, your personality, and what truly drives you. This forms the core of its coaching intelligence. Your Reflection Understands Your Focus in Real-Time. Reflections securely connects to your work signals. By passively understanding your screen activity and analyzing communication patterns from your Gmail, Google Calendar, Notion, it builds a private, real-time picture of what you're working on, where your time is going, and when you are truly focused. But Reflection goes beyond data. It begins with a guided conversation to understand the intangibles that make you who you are: your story, your dreams, your personality, and what truly drives you. This forms the core of its coaching intelligence. Your Reflection Calls You When You’re Unproductive When it senses you've slipped into distraction, it doesn't send a notification you can ignore. A simple, FaceTime-style window appears with your AI twin - a 3D model, modeled after you. Using your own voice, it offers a gentle but firm nudge to get back on track. It’s the voice of your own intentions, helping you stay true to your goals. Your Reflection Calls You When You’re Unproductive When it senses you've slipped into distraction, it doesn't send a notification you can ignore. A simple, FaceTime-style window appears with your AI twin - a 3D model, modeled after you. Using your own voice, it offers a gentle but firm nudge to get back on track. It’s the voice of your own intentions, helping you stay true to your goals. Your Reflection Acts as Your Personal AI Coach. Your coach is always available. You can start a voice or text conversation anytime to talk through challenges, set priorities, or set short or long-term goals. The Reflection remembers every session, getting smarter about your goals and work style over time. Your Reflection Acts as Your Personal AI Coach. Your coach is always available. You can start a voice or text conversation anytime to talk through challenges, set priorities, or set short or long-term goals. The Reflection remembers every session, getting smarter about your goals and work style over time. ### How we built it We built Reflections as a cross‑platform Electron app (Vite + React/TypeScript) with Tailwind and shadcn/ui for a clean, responsive UI and Framer Motion for subtle interactions. Voice and text coaching run through ElevenLabs over WebRTC, and a FaceTime‑style, always‑on‑top popup uses your camera to “call” you when focus slips. The 3D model of the user's face was rendered using Blender and Unreal Engine. In the main process, we sample the screen at user‑set intervals, run Claude function‑calling to classify on‑task vs. distraction and task type, and persist per‑interval stats to Supabase, which power the session/lifetime radar. For signals, we securely link Gmail via Composio, fetch a recent slice of mail, and analyze it with Claude to infer communication patterns, role, and personality - while deep research runs on the Exa API and is summarized and saved. A Claude synthesis step then fuses conversations, email insights, and research into a living system prompt that personalizes coaching over time, with Google sign‑in and a hardened preload bridge keeping secrets isolated and IPC streams reliable. Sponsors: Composio, Claude, Gemini, Eleven Labs, Postman, Bright Data ### Challenges we ran into 3D Modeling - Our vision for Reflections required a deeply personal, visual identity, leading us down the complex path of creating a high-fidelity 3D avatar using Blender and Unreal Engine. The primary challenge was bridging the world of 3D animation with the performance constraints of a real-time Electron app, and the real-time audio streaming. We wrestled with asset optimization and facial animation to ensure the avatar could be integrated seamlessly into the user experience - from the live call popups to the UI - without compromising system performance or app responsiveness. Reflections isn’t just an app; it’s a live system orchestrating multiple, high-latency processes at once. The moment a user drifts, the app has to instantly classify screen content, generate and stream natural-sounding voice, and trigger the interactive call experience. Achieving a smooth, low-latency interaction that felt natural was a massive hurdle. Initial versions suffered from noticeable lag, forcing us to re-architect our entire event-driven system to prioritize the real-time audio and visual cues without crippling the user's machine with background processing and extreme delays. One of the biggest challenges was fusing all the disparate signals that we record into a single, coherent "living system prompt" for your Reflection. Engineering this synthesis step to create a system that genuinely personalizes its coaching over time, rather than just acting on isolated data points, was a major data architecture and AI-prompt-engineering problem. ### Accomplishments we're proud of Accomplishments that we're proud of The "Living System Prompt": A Coach That Genuinely Learns We are incredibly proud of the unified system prompt that serves as the "brain" for the coach. Instead of relying on isolated data points, our system successfully fuses every signal - conversations, email activity, screen activity - into a single, evolving model of the user. This means Reflections isn't just reacting; it's learning. The coaching becomes more insightful, personalized, and effective over time, fulfilling our core promise of a tool that truly grows with you. The "Living System Prompt": A Coach That Genuinely Learns We are incredibly proud of the unified system prompt that serves as the "brain" for the coach. Instead of relying on isolated data points, our system successfully fuses every signal - conversations, email activity, screen activity - into a single, evolving model of the user. This means Reflections isn't just reacting; it's learning. The coaching becomes more insightful, personalized, and effective over time, fulfilling our core promise of a tool that truly grows with you. 3D Assets and Facial Animation We successfully bridged the gap between high-fidelity 3D assets created in Blender and Unreal Engine and a lightweight, performant Electron app. By creating an optimized pipeline for these visuals, we gave Reflections a unique, polished aesthetic that goes far beyond typical productivity tools. Additionally, we’re incredibly proud of the facial animations we were able to produce. This amalgamation of models and facial animations is central to the user experience, making the coach feel more tangible and the interactions more personal and engaging. 3D Assets and Facial Animation We successfully bridged the gap between high-fidelity 3D assets created in Blender and Unreal Engine and a lightweight, performant Electron app. By creating an optimized pipeline for these visuals, we gave Reflections a unique, polished aesthetic that goes far beyond typical productivity tools. Additionally, we’re incredibly proud of the facial animations we were able to produce. This amalgamation of models and facial animations is central to the user experience, making the coach feel more tangible and the interactions more personal and engaging. A Seamless, Real-Time Coaching Experience Our biggest technical achievement is the highly optimized, event-driven pipeline that delivers the core coaching experience. We managed to orchestrate multiple complex, high-latency processes—real-time screen analysis, ElevenLabs voice synthesis, and the interactive call UI - with minimal lag. When your Reflection calls, the interaction feels genuinely responsive and human, not like a series of disjointed API calls. This seamlessness is what makes the coaching feel natural and effective, rather than robotic. A Seamless, Real-Time Coaching Experience Our biggest technical achievement is the highly optimized, event-driven pipeline that delivers the core coaching experience. We managed to orchestrate multiple complex, high-latency processes—real-time screen analysis, ElevenLabs voice synthesis, and the interactive call UI - with minimal lag. When your Reflection calls, the interaction feels genuinely responsive and human, not like a series of disjointed API calls. This seamlessness is what makes the coaching feel natural and effective, rather than robotic. ### What we learned We learned an immense amount about the challenge of integrating disparate, cutting-edge technologies into a single, cohesive experience. On the AI side, we discovered that building a system that truly learns requires a deep focus on data synthesis — fusing structured, unstructured, and conversational data into a single "living prompt." On the visual side, we tackled the steep learning curve of 3D modeling and animation, which gave us a profound respect for the immense technical and artistic challenges that video game developers overcome every day. ### What's next We want to continue to expand Reflections by adding more and more features that makes it a truly transformational tool for people looking to achieve their dreams. We’re looking to add features analyzing your productivity by time of day and day of the week, its coaching will become radically personalized to your unique rhythm. Eventually, the goal is for Reflections to be so attuned to your patterns that it can help you architect your ideal work-life balance, ensuring that your drive toward success is both powerful and sustainable. ## README (from the GitHub repository) Your reflection bringing about change. Reflection is a Desktop Application featuring an AI twin that has all the context on you and allows you to stay true to your goals and self. Reflections securely connects to your work signals. By passively understanding your screen activity and analyzing communication patterns from your Gmail, Google Calendar, Notion, it builds a private, real-time picture of what you're working on, where your time is going, and when you are truly focused. But Reflection goes beyond data. It begins with a guided conversation to understand the intangibles that make you who you are: your story, your dreams, your personality, and what truly drives you. This forms the core of its coaching intelligence. When it senses you've slipped into distraction, it doesn't send a notification you can ignore. A simple, FaceTime-style window appears with your AI twin - a 3D model, modeled after you. Using your own voice, it offers a gentle but firm nudge to get back on track. It’s the voice of your own intentions, helping you stay true to your goals. Your coach is always available. You can start a voice or text conversation anytime to talk through challenges, set priorities, or set short or long-term goals. The Reflection remembers every session, getting smarter about your goals and work style over time. We built Reflections as a cross‑platform Electron app (Vite + React/TypeScript) with Tailwind and shadcn/ui for a clean, responsive UI and Framer Motion for subtle interactions. Voice and text coaching run through ElevenLabs over WebRTC, and a FaceTime‑style, always‑on‑top popup uses your camera to “call” you when focus slips. The 3D model of the user's face was rendered using Blender and Unreal Engine. In the main process, we sample the screen at user‑set intervals, run Claude function‑calling to classify on‑task vs. distraction and task type, and persist per‑interval stats to Supabase, which power the session/lifetime radar. For signals, we securely link Gmail via Composio, fetch a recent slice of mail, and analyze it with Claude to infer communication patterns, role, and personality - while deep research runs on the Exa API and is summarized and saved. A Claude synthesis step then fuses conversations, email insights, and research into a living system prompt that personalizes coaching over time, with Google sign‑in and a hardened preload bridge keeping secrets isolated and IPC streams reliable. ## Detected evidence (automated analysis) Indexed codebase: 58 recognized source files, 325 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (68 of 68) ``` .DS_Store .gitignore coach-electron/.gitignore coach-electron/components.json coach-electron/electron-builder.yml coach-electron/electron.vite.config.ts coach-electron/ELEVENLABS_SETUP.md coach-electron/FIXES_APPLIED.md coach-electron/GEMINI_SETUP.md coach-electron/GOOGLE_OAUTH_SETUP.md coach-electron/migrations/insert_activity_stat_function.sql coach-electron/package.json coach-electron/postcss.config.js coach-electron/README.md coach-electron/src/main/index.ts coach-electron/src/main/lib/composioAnalysis.ts coach-electron/src/main/lib/googleAuth.ts coach-electron/src/main/lib/index.ts coach-electron/src/main/lib/popupWindow.ts coach-electron/src/main/lib/webResearch.ts coach-electron/src/preload/index.d.ts coach-electron/src/preload/index.ts coach-electron/src/renderer/index.html coach-electron/src/renderer/popup.html coach-electron/src/renderer/public/models/mohulpcfull.glb coach-electron/src/renderer/src/__test__/App.test.tsx coach-electron/src/renderer/src/__test__/utils/index.ts coach-electron/src/renderer/src/__test__/utils/svg.mock.ts coach-electron/src/renderer/src/__test__/utils/window.mock.ts coach-electron/src/renderer/src/App.tsx coach-electron/src/renderer/src/assets/index.css coach-electron/src/renderer/src/assets/theme.css coach-electron/src/renderer/src/components/Dashboard.tsx coach-electron/src/renderer/src/components/DashboardChat.tsx coach-electron/src/renderer/src/components/FaceTimeCall.tsx coach-electron/src/renderer/src/components/GLBViewer.tsx coach-electron/src/renderer/src/components/MCPLoadingScreen.tsx coach-electron/src/renderer/src/components/RecordingDashboard.tsx coach-electron/src/renderer/src/components/RecordingSettings.tsx coach-electron/src/renderer/src/components/ResearchDemo.tsx coach-electron/src/renderer/src/components/ui/avatar.tsx coach-electron/src/renderer/src/components/ui/button.tsx coach-electron/src/renderer/src/components/ui/card.tsx coach-electron/src/renderer/src/components/ui/chart.tsx coach-electron/src/renderer/src/components/ui/checkbox.tsx coach-electron/src/renderer/src/components/ui/conversation.tsx coach-electron/src/renderer/src/components/ui/input.tsx coach-electron/src/renderer/src/components/ui/message.tsx coach-electron/src/renderer/src/components/ui/orb.tsx coach-electron/src/renderer/src/components/ui/progress.tsx coach-electron/src/renderer/src/components/ui/response.tsx coach-electron/src/renderer/src/components/ui/scroll-area.tsx coach-electron/src/renderer/src/components/ui/shimmering-text.tsx coach-electron/src/renderer/src/components/ui/tooltip.tsx coach-electron/src/renderer/src/components/Versions.tsx coach-electron/src/renderer/src/components/VoiceChat.tsx coach-electron/src/renderer/src/env.d.ts coach-electron/src/renderer/src/main.tsx coach-electron/src/renderer/src/utils/index.ts coach-electron/src/renderer/src/utils/mcp.py coach-electron/src/shared/types.ts coach-electron/tailwind.config.js coach-electron/tsconfig.json coach-electron/tsconfig.node.json coach-electron/tsconfig.web.json coach-electron/vite.config.mjs FACETIME_PICKLE_SETUP.md README.md ``` ### Dependencies - coach-electron/package.json: @composio/core@^0.2.0, @electron-toolkit/eslint-config-prettier@^2.0.0, @electron-toolkit/eslint-config-ts@^1.0.1, @electron-toolkit/preload@^3.0.1, @electron-toolkit/tsconfig@^1.0.1, @electron-toolkit/utils@^3.0.0, @elevenlabs/react@^0.8.1, @google/genai@^1.27.0, @radix-ui/react-avatar@^1.1.10, @radix-ui/react-checkbox@^1.3.3, @radix-ui/react-progress@^1.1.7, @radix-ui/react-scroll-area@^1.2.10, @radix-ui/react-slot@^1.2.3, @radix-ui/react-tooltip@^1.2.8, @react-three/drei@^10.7.6, @react-three/fiber@^9.4.0, @supabase/supabase-js@^2.76.1, @testing-library/dom@^10.4.0, @testing-library/jest-dom@^6.6.3, @testing-library/react@^16.3.0, @types/jest@^29.5.14, @types/node@^18.19.9, @types/react@19.2.2, @types/react-dom@19.2.2, @types/three@^0.180.0, @vitejs/plugin-react@^4.2.1, @vitest/coverage-v8@^2.1.9, autoprefixer@^10.4.20, class-variance-authority@^0.7.1, clsx@^2.1.1, dotenv@^17.2.3, electron@^32.2.0, electron-builder@^24.9.1, electron-vite@^2.0.0, eslint-plugin-react@^7.33.2, framer-motion@^12.23.24, jest-environment-jsdom@^29.7.0, lucide-react@^0.454.0, motion@^12.23.24, nanoid@^5.1.6, openai@^6.7.0, postcss@^8.4.47, prettier@^3.2.4, react@19.0.0, react-dom@19.0.0, react-markdown@^10.1.0, recharts@^3.3.0, remark-gfm@^4.0.1, streamdown@^1.4.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.14, tailwindcss-animate@^1.0.7, three@^0.180.0, ts-jest@^29.2.5, ts-node@^10.9.2, typescript@^5.3.3, use-stick-to-bottom@^1.1.1, vite@^5.4.14, vitest@^2.1.9 ### Recent commits (newest first) - trying to fix merge conflicts ig - Merge pull request #2 from coach-calhacks/pickle - Merge branch 'main' into pickle - call ending instant now - added 2nd agent id - Merge branch 'rollback' - the bug fix for the button delay was not fixed - Initial README.md for Reflection application - Merge branch 'bug' - fixed phone call ending taking to long bug - miles morales - added call context and phone context to db, and now runs a summery to get system prompt for user - whateves rever - Merge branch 'mail' - mcp for gmail added - the markdown - vibes totally vibes - blue - graph update - graphs ## Key source files (fetched from GitHub, selected and truncated for size) ### FACETIME_PICKLE_SETUP.md ```markdown # FaceTime + Pickle + ElevenLabs Integration Setup Guide ## Overview This implementation enables the following workflow: 1. **User accepts FaceTime call** → Triggers navigation to FaceTime call screen 2. **Pickle app is displayed** → Screen capture shows Pickle application window 3. **ElevenLabs voice connects** → AI voice agent starts speaking with configured prompt 4. **Audio routing** → Voice plays through speakers AND routes to BlackHole for Pickle 5. **Pickle animates** → Pickle app receives audio through BlackHole and animates lip movements 6. **Call ends** → ElevenLabs session closes, audio routing stops ## Features Implemented ### ✅ ElevenLabs Voice Integration - Automatically connects to ElevenLabs when call is accepted - Uses WebRTC for real-time voice communication - Fetches custom prompts from Supabase database - Voice connection status indicator in UI - Automatic disconnection when call ends ### ✅ Audio Routing with BlackHole - Duplicates audio to two destinations: - Default speakers (user hears the voice) - BlackHole virtual audio device (Pickle can listen for animation) - Volume monitoring for debugging - Clean cleanup on call end ### ✅ Supabase Prompt Configuration - Fetches user-specific prompts from `user_prompts` table - Falls back to default prompt if not found - Supports customized warning messages per user ### ✅ Call Lifecycle Management - Proper initialization on call start - Clean resource cleanup on call end - Screen capture pauses during FaceTime calls - Main process aware of call state ## Prerequisites ### 1. Install BlackHole (Virtual Audio Device) ```bash # Install via Homebrew brew install blackhole-2ch # Or download from GitHub # https://github.com/ExistentialAudio/BlackHole ``` ### 2. Configure macOS Audio MIDI Setup 1. Open **Audio MIDI Setup** app (in `/Applications/Utilities/`) 2. Click the **"+"** button in bottom-left corner 3. Select **"Create Multi-Output Device"** 4. In the Multi-Output Device settings: - ✅ Check **"Built-in Output"** (or your default speakers) - ✅ Check **"BlackHole 2ch"** - Set **"Built-in Output"** as the master device 5. (Optional) Rename the Multi-Output Device to "Speakers + BlackHole" ### 3. Set System Audio Output 1. Open **System Preferences** → **Sound** → **Output** 2. Select the **Multi-Output Device** you just created 3. All system audio will now play through speakers AND route to BlackHole **Alternative:** Only set this Multi-Output Device during FaceTime calls by using a tool like [SwitchAudioSource](https://github.com/deweller/switchaudio-osx) ### 4. Configure Pickle App 1. Open Pickle application settings 2. Set **Audio Input Device** to **"BlackHole 2ch"** 3. Pickle will now receive the ElevenLabs voice audio for lip sync ### 5. Supabase Database Setup Create the `user_prompts` table in your Supabase database: ```sql CREATE TABLE user_prompts ( id UUID DEFAULT gen_random_uuid() PRIMARY KEY, email TEXT NOT NULL UNIQUE, prompt TEXT NO [truncated — 7483 more characters] ``` ### coach-electron/GEMINI_SETUP.md ```markdown # Gemini Screenshot Analysis Setup ## Overview Your Electron app now automatically analyzes every captured screenshot using Google's Gemini AI and logs the results to the terminal. ## Setup Instructions ### 1. Get Your Gemini API Key 1. Visit [Google AI Studio](https://aistudio.google.com/app/apikey) 2. Sign in with your Google account 3. Create a new API key 4. Copy the API key ### 2. Configure the API Key 1. Open `/Users/richard/Documents/coach-main/coach-electron/.env` 2. Replace `your_api_key_here` with your actual API key: ``` GEMINI_API_KEY=YOUR_ACTUAL_API_KEY_HERE ``` 3. Save the file ### 3. Run the Application ```bash cd /Users/richard/Documents/coach-main/coach-electron npm run dev ``` ## How It Works 1. When you start screen capture, the app takes screenshots at your specified interval 2. Each screenshot is saved to `~/Pictures/ReflectionsScreenshots/` 3. Immediately after saving, the screenshot is sent to Gemini AI with the prompt: "What is this screenshot about?" 4. Gemini's analysis is logged to the terminal in a formatted output ## Example Terminal Output ``` Screenshot saved: /Users/richard/Pictures/ReflectionsScreenshots/screenshot_2024-10-24_23-45-30.png 🔍 Analyzing screenshot with Gemini AI... ================================================================================ 📸 SCREENSHOT ANALYSIS ================================================================================ File: screenshot_2024-10-24_23-45-30.png Time: 10/24/2024, 11:45:30 PM -------------------------------------------------------------------------------- This screenshot shows a code editor with an Electron application being developed. The code appears to be implementing screen capture functionality with TypeScript... ================================================================================ ``` ## Features - **Automatic Analysis**: Every screenshot is automatically analyzed - **No Interruption**: Analysis happens asynchronously without blocking capture - **Error Handling**: Gracefully handles missing API keys or API failures - **Detailed Logging**: Clear, formatted output for easy reading ## Troubleshooting ### No Analysis Output? - Check that `GEMINI_API_KEY` is set in `.env` - Ensure the API key is valid (not `your_api_key_here`) - Check terminal for any error messages ### API Errors? - Verify your API key is active in Google AI Studio - Check your API quota and usage limits - Ensure you have internet connectivity ## Security Note ⚠️ The `.env` file is already in `.gitignore` to prevent accidentally committing your API key. Never share your API key publicly. ## What Was Installed - `@google/genai` - Official Google Generative AI SDK - `dotenv` - Environment variable management ## Files Modified - `src/main/lib/index.ts` - Added Gemini integration and analysis function - `package.json` - Added new dependencies - `.env` - Created for API key storage (you need to add your key) - `.env.example` - Template for other develo [truncated — 6 more characters] ``` ### coach-electron/package.json ``` { "name": "coach-electron", "version": "0.0.0", "description": "An Electron template with React, TypeScript, and Shadcn UI", "main": "./out/main/index.js", "author": "Author", "homepage": "https://electron-vite.org", "scripts": { "typecheck:node": "tsc --noEmit -p tsconfig.node.json --composite false", "typecheck:web": "tsc --noEmit -p tsconfig.web.json --composite false", "typecheck": "npm run typecheck:node && npm run typecheck:web", "start": "electron-vite preview", "dev": "electron-vite dev", "build": "npm run typecheck && electron-vite build", "postinstall": "electron-builder install-app-deps", "build:unpack": "npm run build && electron-builder --dir", "build:win": "npm run build && electron-builder --win", "build:mac": "electron-vite build && electron-builder --mac", "build:linux": "electron-vite build && electron-builder --linux", "test:unit": "vitest", "test": "npm run test:unit", "coverage": "vitest run --coverage" }, "dependencies": { "@composio/core": "^0.2.0", "@electron-toolkit/preload": "^3.0.1", "@electron-toolkit/utils": "^3.0.0", "@elevenlabs/react": "^0.8.1", "@google/genai": "^1.27.0", "@radix-ui/react-avatar": "^1.1.10", "@radix-ui/react-checkbox": "^1.3.3", "@radix-ui/react-progress": "^1.1.7", "@radix-ui/react-scroll-area": "^1.2.10", "@radix-ui/react-slot": "^1.2.3", "@radix-ui/react-tooltip": "^1.2.8", "@react-three/drei": "^10.7.6", "@react-three/fiber": "^9.4.0", "@supabase/supabase-js": "^2.76.1", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "dotenv": "^17.2.3", "framer-motion": "^12.23.24", "lucide-react": "^0.454.0", "motion": "^12.23.24", "nanoid": "^5.1.6", "openai": "^6.7.0", "react-markdown": "^10.1.0", "recharts": "^3.3.0", "remark-gfm": "^4.0.1", "streamdown": "^1.4.0", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7", "three": "^0.180.0", "use-stick-to-bottom": "^1.1.1" }, "devDependencies": { "@electron-toolkit/eslint-config-prettier": "^2.0.0", "@electron-toolkit/eslint-config-ts": "^1.0.1", "@electron-toolkit/tsconfig": "^1.0.1", "@testing-library/dom": "^10.4.0", "@testing-library/jest-dom": "^6.6.3", "@testing-library/react": "^16.3.0", "@types/jest": "^29.5.14", "@types/node": "^18.19.9", "@types/react": "19.2.2", "@types/react-dom": "19.2.2", "@types/three": "^0.180.0", "@vitejs/plugin-react": "^4.2.1", "@vitest/coverage-v8": "^2.1.9", "autoprefixer": "^10.4.20", "electron": "^32.2.0", "electron-builder": "^24.9.1", "electron-vite": "^2.0.0", "eslint-plugin-react": "^7.33.2", "jest-environment-jsdom": "^29.7.0", "postcss": "^8.4.47", "prettier": "^3.2.4", "react": "19.0.0", "react-dom": "19.0.0", "tailwindcss": "^3.4.14", "ts-jest": "^29.2.5", "ts-node": "^10.9.2", "typescript": "^5.3.3", "vite": "^5.4.14", "vitest": "^2.1.9" } } ``` ### coach-electron/src/preload/index.ts ```typescript import { contextBridge } from "electron"; import { ipcRenderer } from "electron/renderer"; import { GetVersionsFn, StartScreenCaptureFn, StopScreenCaptureFn, GetScreenCaptureStatusFn, SetScreenCaptureIntervalFn, GetScreenCaptureFolderFn, SignInWithGoogleFn, PerformDeepResearchFn, OnResearchEventFn, ResearchEvent, GetTaskStatsFn, StartFaceTimeCallFn, GetDesktopSourcesFn, OnNavigateToCallFn, SetFaceTimeCallActiveFn, GetPromptConfigFn, GetLifetimeTaskStatsFn, AnalyzeUserEmailsFn, OnEmailAnalysisProgressFn, EmailAnalysisProgress, UploadConversationFn, UploadResearchSummaryFn, GenerateSystemPromptFn, } from "@shared/types"; // The preload process plays a middleware role in bridging // the call from the front end, and the function in the main process if (!process.contextIsolated) { throw new Error("Context isolation must be enabled in the Browser window"); } try { // Expose IPC send for popup window contextBridge.exposeInMainWorld("ipcRenderer", { send: (channel: string, data?: any) => ipcRenderer.send(channel, data), }); // Front end can call the function by using window.context. contextBridge.exposeInMainWorld("context", { getVersions: (...args: Parameters) => ipcRenderer.invoke("getVersions", ...args), triggerIPC: () => ipcRenderer.invoke("triggerIPC"), startScreenCapture: (...args: Parameters) => ipcRenderer.invoke("startScreenCapture", ...args), stopScreenCapture: (...args: Parameters) => ipcRenderer.invoke("stopScreenCapture", ...args), getScreenCaptureStatus: (...args: Parameters) => ipcRenderer.invoke("getScreenCaptureStatus", ...args), setScreenCaptureInterval: (...args: Parameters) => ipcRenderer.invoke("setScreenCaptureInterval", ...args), getScreenCaptureFolder: (...args: Parameters) => ipcRenderer.invoke("getScreenCaptureFolder", ...args), signInWithGoogle: (...args: Parameters) => ipcRenderer.invoke("signInWithGoogle", ...args), performDeepResearch: (...args: Parameters) => ipcRenderer.invoke("performDeepResearch", ...args), onResearchEvent: (callback: OnResearchEventFn) => { const subscription = (_event: any, data: ResearchEvent) => callback(data); ipcRenderer.on('research-event', subscription); // Return unsubscribe function return () => { ipcRenderer.removeListener('research-event', subscription); }; }, onStatsUpdated: (callback: () => void) => { const handler = () => callback(); ipcRenderer.on('stats-updated', handler); return () => ipcRenderer.removeListener('stats-updated', handler); }, getTaskStats: (...args: Parameters) => ipcRenderer.invoke("getTaskStats", ...args), startFaceTimeCall: (...args: Parameters) => ipcRenderer.invoke("startFaceTimeCall", ...args), getDesktopSources: (...args: Parameters) => ipcRenderer.invoke("getDesktopSources", ...args), setFaceTimeCallActive: (...args: Parameters) => ipcRenderer.invoke("setFaceTimeCallActive", ...args), getPromptConfig: (...args: Parameters) => ipcRenderer.invoke("getPromptConfig", ...args), onNavigateToCall: (callback: OnNavigateToCallFn) => { const subscription = () => callback(); ipcRenderer.on('navigate-to-call', subscription); // Return unsubscribe function return () => { ipcRenderer.removeListener('navigate-to-call', subscription); }; }, getLifetimeTaskStats: (...args: Parameters) => ipcRenderer.invoke("getLifetimeTaskStats", ...args), analyzeUserEmails: (...args: Parameters) => ipcRenderer.invoke("analyzeUserEmails", ...args), onEmailAnalysisProgress: (callback: OnEmailAnalysisProgressFn) => { const subscription = (_event: any, data: EmailAnalysisProgress) => callback(data); ipcRenderer.on('email-analysis-progress', subscription); // Return unsubscribe function return () => { ipcRenderer.removeListener('email-analysis-progress', subscription); }; }, uploadConversation: (...args: Parameters) => ipcRenderer.invoke("uploadConversation", ...args), uploadResearchSummary: (...args: Parameters) => ipcRenderer.invoke("uploadResearchSummary", ...args), generateSystemPrompt: (...args: Parameters) => ipcRenderer.invoke("generateSystemPrompt", ...args), }); } catch (error) { console.error("Error occured when establishing context bridge: ", error); } ``` ### coach-electron/src/main/index.ts ```typescript import { app, shell, BrowserWindow, ipcMain, desktopCapturer } from "electron"; import { join } from "path"; import { electronApp, optimizer, is } from "@electron-toolkit/utils"; import icon from "../../resources/icon.png?asset"; import { getVersions, triggerIPC, startScreenCapture, stopScreenCapture, getScreenCaptureStatus, setScreenCaptureInterval, getScreenCaptureFolder, initializeScreenCapture, signInWithGoogle, performDeepResearch, getTaskStats, setFaceTimeCallActive, getPromptConfig, getLifetimeTaskStats, analyzeUserEmails, uploadConversation, uploadResearchSummary, generateSystemPrompt, } from "@/lib"; import { GetVersionsFn, StartScreenCaptureFn, StopScreenCaptureFn, GetScreenCaptureStatusFn, SetScreenCaptureIntervalFn, GetScreenCaptureFolderFn, SignInWithGoogleFn, PerformDeepResearchFn, GetTaskStatsFn, DesktopSource, SetFaceTimeCallActiveFn, GetPromptConfigFn, GetLifetimeTaskStatsFn, AnalyzeUserEmailsFn, UploadConversationFn, UploadResearchSummaryFn, GenerateSystemPromptFn, } from "@shared/types"; let mainWindow: BrowserWindow | null = null; function createWindow(): void { // Create the browser window. mainWindow = new BrowserWindow({ width: 900, height: 670, show: false, autoHideMenuBar: true, titleBarStyle: 'hidden', ...(process.platform === "linux" ? { icon } : {}), webPreferences: { preload: join(__dirname, "../preload/index.js"), sandbox: false, // Changed from true - needed for desktopCapturer contextIsolation: true, nodeIntegration: false, }, }); mainWindow.on("ready-to-show", () => { mainWindow?.show(); }); mainWindow.on("closed", () => { mainWindow = null; }); mainWindow.webContents.setWindowOpenHandler((details) => { shell.openExternal(details.url); return { action: "deny" }; }); // HMR for renderer base on electron-vite cli. // Load the remote URL for development or the local html file for production. if (is.dev && process.env["ELECTRON_RENDERER_URL"]) { mainWindow.loadURL(process.env["ELECTRON_RENDERER_URL"]); } else { mainWindow.loadFile(join(__dirname, "../renderer/index.html")); } } // This method will be called when Electron has finished // initialization and is ready to create browser windows. // Some APIs can only be used after this event occurs. app.whenReady().then(() => { // Set app user model id for windows electronApp.setAppUserModelId("com.electron"); // Default open or close DevTools by F12 in development // and ignore CommandOrControl + R in production. // see https://github.com/alex8088/electron-toolkit/tree/master/packages/utils app.on("browser-window-created", (_, window) => { optimizer.watchWindowShortcuts(window); }); createWindow(); app.on("activate", function () { // On macOS it's common to re-create a window in the app when the // dock icon is clicked and there are no other windows open. if (BrowserWindow.getAllWindows().length === 0) createWindow(); }); // IPC events ipcMain.handle( "getVersions", (_, ...args: Parameters) => getVersions(...args) ); ipcMain.handle("triggerIPC", () => triggerIPC()); // Screen capture IPC events ipcMain.handle( "startScreenCapture", (_, ...args: Parameters) => startScreenCapture(...args) ); ipcMain.handle( "stopScreenCapture", (_, ...args: Parameters) => stopScreenCapture(...args) ); ipcMain.handle( "getScreenCaptureStatus", (_, ...args: Parameters) => getScreenCaptureStatus(...args) ); ipcMain.handle( "setScreenCaptureInterval", (_, ...args: Parameters) => setScreenCaptureInterval(...args) ); ipcMain.handle( "getScreenCaptureFolder", (_, ...args: Parameters) => getScreenCaptureFolder(...args) ); // Google OAuth IPC event ipcMain.handle( "signInWithGoogle", (_, ...args: Parameters) => signInWithGoogle(...args) ); // Web Deep Research IPC event ipcMain.handle( "performDeepResearch", async (event, ...args: Parameters) => { // Set up event callback to send events back to renderer const onEvent = (researchEvent: any) => { event.sender.send('research-event', researchEvent); }; return performDeepResearch(args[0], onEvent); } ); // Stats IPC event ipcMain.handle( "getTaskStats", (_, ...args: Parameters) => getTaskStats(...args) ); // FaceTime Call IPC events ipcMain.handle( "startFaceTimeCall", async () => { if (mainWindow && !mainWindow.isDestroyed()) { mainWindow.show(); mainWindow.focus(); mainWindow.webContents.send('navigate-to-call'); } } ); ipcMain.handle( "getDesktopSources", async (): Promise => { try { const sources = await desktopCapturer.getSources({ types: ['window', 'screen'], thumbnailSize: { width: 150, height: 150 } }); return sources.map(source => ({ id: source.id, name: source.name, thumbnail: source.thumbnail.toDataURL() })); } catch (error) { console.error('Error getting desktop sources:', error); return []; } } ); // Set FaceTime call active state ipcMain.handle( "setFaceTimeCallActive", (_, ...args: Parameters) => setFaceTimeCallActive(...args) ); // Get prompt configuration ipcMain.handle( "getPromptConfig", (_, ...args: Parameters) => getPromptConfig(...args) ); // Lifetime Stats IPC event ipcMain.handle( "getLifetimeTaskStats", (_, ...args: Parameters) => getLifetimeTaskStats(...args) ); / [truncated — 1204 more characters] ``` ### coach-electron/src/renderer/src/main.tsx ```typescript import "./assets/index.css"; import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render( , ); ``` ### coach-electron/src/renderer/src/App.tsx ```typescript import { useState, Component, ReactNode, useEffect } from "react"; import { Button } from "@/components/ui/button"; import { Progress } from "@/components/ui/progress"; import googleIcon from "./assets/google_icon.svg"; import VoiceChat from "@/components/VoiceChat"; import { ResearchDemo } from "@/components/ResearchDemo"; import { GLBViewer } from "@/components/GLBViewer"; import { MCPLoadingScreen } from "@/components/MCPLoadingScreen"; import Dashboard from "@/components/Dashboard"; import { FaceTimeCall } from "@/components/FaceTimeCall"; type SetupStep = "mcp" | "research" | "voice" | "complete"; // Error boundary to catch 3D viewer errors class ErrorBoundary extends Component< { children: ReactNode }, { hasError: boolean } > { constructor(props: { children: ReactNode }) { super(props); this.state = { hasError: false }; } static getDerivedStateFromError() { return { hasError: true }; } componentDidCatch(error: Error, errorInfo: any) { console.error("3D Viewer Error:", error, errorInfo); } render() { if (this.state.hasError) { return (

3D viewer unavailable

); } return this.props.children; } } const App = () => { const [isLoading, setIsLoading] = useState(false); const [error, setError] = useState(null); const [userInfo, setUserInfo] = useState(null); const [currentPage, setCurrentPage] = useState<"login" | "dashboard" | "research" | "facetime-call">("login"); const [setupStep, setSetupStep] = useState("mcp"); // Listen for navigation events from main process useEffect(() => { const unsubscribe = window.context.onNavigateToCall(() => { setCurrentPage("facetime-call"); }); return unsubscribe; }, []); // Dev mode: skip to dashboard with 'S' key useEffect(() => { const handleKeyPress = (e: KeyboardEvent) => { if (e.key === 's' || e.key === 'S') { console.log('[Dev Mode] Skipping research, jumping to dashboard...'); setSetupStep("complete"); setCurrentPage("dashboard"); // Set a mock user if not already set if (!userInfo) { setUserInfo({ name: "Dev User", email: "dev@example.com", id: "dev-user-id" }); } } }; window.addEventListener('keydown', handleKeyPress); return () => window.removeEventListener('keydown', handleKeyPress); }, [userInfo]); const handleGoogleSignIn = async () => { setIsLoading(true); setError(null); try { const result = await window.context.signInWithGoogle(); if (result.success && result.userInfo) { console.log("Successfully signed in:", result.userInfo); setUserInfo(result.userInfo); // You can store the tokens in localStorage or state management // localStorage.setItem('accessToken', result.accessToken || ''); // localStorage.setItem('refreshToken', result.refreshToken || ''); } else { setError(result.error || "Failed to sign in with Google"); console.error("Sign in failed:", result.error); } } catch (err) { const errorMessage = err instanceof Error ? err.message : "Unknown error occurred"; setError(errorMessage); console.error("Sign in error:", err); } finally { setIsLoading(false); } }; const getProgressValue = () => { switch (setupStep) { case "mcp": return 25; case "research": return 50; case "voice": return 75; case "complete": return 100; default: return 0; } }; // If on facetime-call page, show the FaceTime call interface if (currentPage === "facetime-call") { return setCurrentPage("dashboard")} />; } // If on dashboard page, show the combined recording dashboard if (currentPage === "dashboard") { return ; } // If on research page, show the research demo if (currentPage === "research") { return (
); } if (userInfo) { // Show setup progress before showing voice chat if (setupStep !== "complete") { return (
{/* Progress bar at top */}

Learning about you, {userInfo.name.split(' ')[0]}

{/* Content */}
{/* Show MCP loading screen during mcp step */} {setupStep === "mcp" && ( { setSetupStep("research"); }} /> )} {/* Show research demo during research step */} {setupStep === "research" && (
{ setSetupStep("voice"); }} />
)} {/* Show continue button for other steps */} {setupStep !== "research" && setupStep !== "mcp" && ( [truncated — 2536 more characters] ``` ### coach-electron/src/renderer/src/utils/index.ts ```typescript import { clsx, type ClassValue } from "clsx"; import { twMerge } from "tailwind-merge"; export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)); } ``` ### coach-electron/src/renderer/src/__test__/utils/index.ts ```typescript export * from "@testing-library/dom"; export { render } from "@testing-library/react"; ``` ### coach-electron/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` [46 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]