# Project export: CappyCoding This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: An embedded device to serve as a developers trusted companion. - Devpost: https://devpost.com/software/cappycoding - GitHub: https://github.com/Sluggish-Solutions/CapyCoding - Video: https://www.youtube.com/embed/__N4VOSAq6Q?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) โ€” Akhil Datla (32 commits), Surendra Jammishetti (23 commits), Kenric (10 commits) ## Devpost submission (written by the team) ### Inspiration We wanted to build upon the tried and true rubber ducky method of debugging, where a developer explains their thought process to a "rubber ducky", forcing themselves to walk through their system, start to finish. We wanted to create a tool that can be used for this as well as some flair! ### What it does We provide a cute screen, ideally held up by our mascot, that shows developers their coding stats such as GitHub Pull Requests, GitHub Actions, commits or more specific to vibe coding like Claude token usage etc. ### How we built it We connected an ESP32 board to an E-Ink display, and used Rust to program the TUI widgets. A Go server then complements the embedded device, which polls for live updates and stats. The device runs standalone due to using the Wi-Fi/BLE stack of the esp32. ### Challenges we ran into Since some of the teammates were new to Rust, some time was spent on learning before building. It was also our first time trying to understand the Wi-Fi and Bluetooth protocols at a deeper level, and then trying to find examples which we could build on for those technologies was very difficult. There was a lot of trail and error. ### Accomplishments we're proud of We are extremely proud of the fact that we built a standalone embedded device that functions on ubiquitous protocols. Just seeing it contact our backend while only connected to power, and then pull that data down and then displaying it; it was incredibly rewarding. ### What we learned We learned so much about interfacing with networking protocols as well as more experience with embedded software development! ### What's next We would love to spend more time on the project and add more features, and even design a nice case so it could be a practical desk-gadget! ## README (from the GitHub repository) # CappyCoding A Tauri-based desktop application with voice-enabled AI assistant powered by LiveKit and Claude. ## Features - ๐ŸŽ™๏ธ **Voice AI Assistant**: Talk to Claude using natural speech - ๐Ÿ”Š **LiveKit Integration**: Professional-grade WebRTC for real-time communication - ๐Ÿค– **Claude Sonnet 4.5**: Advanced AI responses via Anthropic - ๐Ÿ“Š **Usage Metrics**: Track Claude API usage and costs - ๐ŸŽจ **Modern UI**: Built with Svelte 5 and Tauri 2.0 - ๐Ÿ”ง **Easy Configuration**: UI-based setup for API keys and agent management ## Architecture ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ Tauri Desktop App โ”‚ โ”‚ (Svelte 5 Frontend) โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ ๐ŸŽค Voice Input โ†’ LiveKit โ†’ Python Agent โ†’ ๐Ÿ”Š Voice โ”‚ โ”‚ โ†“ โ”‚ โ”‚ LiveKit Inference โ”‚ โ”‚ (Deepgram STT + Cartesia TTS) โ”‚ โ”‚ โ†“ โ”‚ โ”‚ Claude AI (Anthropic) โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` ## Quick Start ### Prerequisites - **Node.js 18+** and **bun** (for frontend) - **Rust** (for Tauri) - **Python 3.13+** (for voice agent) ### 1. Get API Keys You'll need: - **LiveKit Cloud** account: https://cloud.livekit.io/ (free tier available) - **Anthropic API** key: https://console.anthropic.com/ ### 2. Launch the App ```bash cd capycoding-app bun install bun run tauri dev ``` ### 3. Configure Voice Agent (in the app) 1. Find the **"๐Ÿค– Voice Agent Configuration"** panel 2. Enter your LiveKit credentials (URL, API Key, API Secret) 3. Enter your Anthropic API key 4. Click **"๐Ÿ’พ Save Configuration"** 5. Click **"โ–ถ๏ธ Start Agent"** ### 4. Start Talking! 1. Go to **"๐ŸŽ™๏ธ Connect to Voice Session"** 2. Enter a participant identity and room name 3. Click **"Connect to Voice Session"** 4. Start speaking naturally! ## Project Structure ``` CappyCoding/ โ”œโ”€โ”€ agent.py # Python voice agent (LiveKit + Claude) โ”œโ”€โ”€ env/ # Python virtual environment โ”œโ”€โ”€ capycoding-app/ # Tauri desktop application โ”‚ โ”œโ”€โ”€ src/ # Svelte frontend โ”‚ โ””โ”€โ”€ src-tauri/ # Rust backend โ”œโ”€โ”€ capycoding-esp/ # ESP32 firmware (optional) โ”œโ”€โ”€ ble-types/ # Bluetooth type definitions โ”œโ”€โ”€ server/ # Go metrics server โ””โ”€โ”€ docs/ โ”œโ”€โ”€ FRONTEND-CONFIG.md # UI configuration guide โ”œโ”€โ”€ README-AGENT.md # Agent technical details โ””โ”€โ”€ QUICKSTART.md # Quick start guide ``` ## Components ### Voice Agent (`agent.py`) Python agent using LiveKit's agent framework: - **STT**: Deepgram (via LiveKit Inference) - **LLM**: Claude Sonnet 4.5 (via Anthropic plugin) - **TTS**: Cartesia (via LiveKit Inference) - **VAD**: Silero voice activity detection Configuration stored in: - `~/.config/capycoding/agent_config.json` (UI-saved) - `.env` file (manual configuration) - Environment variables (highest priority) ### Tauri App (`capycoding-app/`) Desktop application featuring: - Voice agent configuration and management - LiveKit voice session connection - Claude usage metrics tracking - Real-time audio streaming Built with: - **Frontend**: Svelte 5, Vite, LiveKit Client SDK - **Backend**: Rust, Tauri 2.0, taurpc for IPC ## Documentation - **[QUICKSTART.md](QUICKSTART.md)** - Get started in 1 minute - **[FRONTEND-CONFIG.md](FRONTEND-CONFIG.md)** - UI configuration guide - **[README-AGENT.md](README-AGENT.md)** - Agent technical details ## Costs Using LiveKit Inference simplifies billing to just 2 providers: **LiveKit Cloud** (includes STT + TTS): - Deepgram STT: $0.0043/minute - Cartesia TTS: $0.045/minute - LiveKit transfer: Free tier includes 50 GB/month **Anthropic**: - Claude Sonnet 4.5: ~$3/$15 per million tokens **Typical usage**: ~$0.26/hour of conversation ## Development ### Running the Agent Manually ```bash cd /path/to/CappyCoding source env/bin/activate python agent.py dev ``` ### Building the Tauri App ```bash cd capycoding-app bun run tauri build ``` ### Installing Agent Dependencies Already installed in `/env`, but to reinstall: ```bash cd /path/to/CappyCoding python -m venv env source env/bin/activate pip install livekit "livekit-agents[anthropic,silero,deepgram,cartesia]" ``` ## Environment Variables The agent loads configuration in this priority: 1. **Environment variables** (highest) 2. **UI-saved config** (`~/.config/capycoding/agent_config.json`) 3. **.env file** (lowest) Required variables: ```env LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=APIxxxxxxxxxx LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxx ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxx ``` ## Troubleshooting ### Agent won't start - Verify API keys are correct (no extra spaces) - Check that `/env` virtual environment exists - Try running manually to see error messages ### Can't connect to voice session - Ensure agent is running (check status in UI) - Verify LiveKit credentials in agent configuration - Check that participant identity and room name are filled in ### No audio response - Check LiveKit Inference is enabled (default on new projects) - Verify microphone permissions in your OS - Check browser/Tauri audio settings See [QUICKSTART.md](QUICKSTART.md) for more troubleshooting tips. ## License [Your License Here] ## Contributing [Your Contributing Guidelines Here] ## Detected evidence (automated analysis) Indexed codebase: 412 recognized source files, 584 KB. - CSS (language) โ€” detected in the code - Go (language) โ€” detected in the code - HTML (language) โ€” detected in the code - JavaScript (language) โ€” detected in the code - Python (language) โ€” detected in the code - Rust (language) โ€” detected in the code - Svelte (technology) โ€” detected in the code - Tailwind CSS (technology) โ€” detected in the code - TypeScript (language) โ€” detected in the code ## Codebase structure (from repository index) ### Files (120 of 443) ``` .env.example .gitignore agent.py agent.py.old ble-types/Cargo.lock ble-types/Cargo.toml ble-types/src/lib.rs capycoding-app/.gitignore capycoding-app/.vscode/extensions.json capycoding-app/.vscode/settings.json capycoding-app/biome.jsonc capycoding-app/bun.lock capycoding-app/components.json capycoding-app/dprint.json capycoding-app/justfile capycoding-app/package.json capycoding-app/README.md capycoding-app/src-tauri/.gitignore capycoding-app/src-tauri/build.rs capycoding-app/src-tauri/capabilities/default.json capycoding-app/src-tauri/capabilities/microphone-access.json capycoding-app/src-tauri/Cargo.lock capycoding-app/src-tauri/Cargo.toml capycoding-app/src-tauri/icons/icon.icns capycoding-app/src-tauri/Info.plist capycoding-app/src-tauri/src/ble/mod.rs capycoding-app/src-tauri/src/lib.rs capycoding-app/src-tauri/src/main.rs capycoding-app/src-tauri/src/python/collect_metrics.py capycoding-app/src-tauri/src/types.rs capycoding-app/src-tauri/tauri.conf.json capycoding-app/src/app.css capycoding-app/src/app.html capycoding-app/src/lib/components/LiveKitVoiceConsole.svelte capycoding-app/src/lib/components/RemoteAudioTile.svelte capycoding-app/src/lib/components/ui/accordion/accordion-content.svelte capycoding-app/src/lib/components/ui/accordion/accordion-item.svelte capycoding-app/src/lib/components/ui/accordion/accordion-trigger.svelte capycoding-app/src/lib/components/ui/accordion/accordion.svelte capycoding-app/src/lib/components/ui/accordion/index.ts capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-action.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-cancel.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-content.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-description.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-footer.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-header.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-overlay.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-title.svelte capycoding-app/src/lib/components/ui/alert-dialog/alert-dialog-trigger.svelte capycoding-app/src/lib/components/ui/alert-dialog/index.ts capycoding-app/src/lib/components/ui/alert/alert-description.svelte capycoding-app/src/lib/components/ui/alert/alert-title.svelte capycoding-app/src/lib/components/ui/alert/alert.svelte capycoding-app/src/lib/components/ui/alert/index.ts capycoding-app/src/lib/components/ui/aspect-ratio/aspect-ratio.svelte capycoding-app/src/lib/components/ui/aspect-ratio/index.ts capycoding-app/src/lib/components/ui/avatar/avatar-fallback.svelte capycoding-app/src/lib/components/ui/avatar/avatar-image.svelte capycoding-app/src/lib/components/ui/avatar/avatar.svelte capycoding-app/src/lib/components/ui/avatar/index.ts capycoding-app/src/lib/components/ui/badge/badge.svelte capycoding-app/src/lib/components/ui/badge/index.ts capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb-ellipsis.svelte capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb-item.svelte capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb-link.svelte capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb-list.svelte capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb-page.svelte capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb-separator.svelte capycoding-app/src/lib/components/ui/breadcrumb/breadcrumb.svelte capycoding-app/src/lib/components/ui/breadcrumb/index.ts capycoding-app/src/lib/components/ui/button-group/button-group-separator.svelte capycoding-app/src/lib/components/ui/button-group/button-group-text.svelte capycoding-app/src/lib/components/ui/button-group/button-group.svelte capycoding-app/src/lib/components/ui/button-group/index.ts capycoding-app/src/lib/components/ui/button/button.svelte capycoding-app/src/lib/components/ui/button/index.ts capycoding-app/src/lib/components/ui/calendar/calendar-caption.svelte capycoding-app/src/lib/components/ui/calendar/calendar-cell.svelte capycoding-app/src/lib/components/ui/calendar/calendar-day.svelte capycoding-app/src/lib/components/ui/calendar/calendar-grid-body.svelte capycoding-app/src/lib/components/ui/calendar/calendar-grid-head.svelte capycoding-app/src/lib/components/ui/calendar/calendar-grid-row.svelte capycoding-app/src/lib/components/ui/calendar/calendar-grid.svelte capycoding-app/src/lib/components/ui/calendar/calendar-head-cell.svelte capycoding-app/src/lib/components/ui/calendar/calendar-header.svelte capycoding-app/src/lib/components/ui/calendar/calendar-heading.svelte capycoding-app/src/lib/components/ui/calendar/calendar-month-select.svelte capycoding-app/src/lib/components/ui/calendar/calendar-month.svelte capycoding-app/src/lib/components/ui/calendar/calendar-months.svelte capycoding-app/src/lib/components/ui/calendar/calendar-nav.svelte capycoding-app/src/lib/components/ui/calendar/calendar-next-button.svelte capycoding-app/src/lib/components/ui/calendar/calendar-prev-button.svelte capycoding-app/src/lib/components/ui/calendar/calendar-year-select.svelte capycoding-app/src/lib/components/ui/calendar/calendar.svelte capycoding-app/src/lib/components/ui/calendar/index.ts capycoding-app/src/lib/components/ui/card/card-action.svelte capycoding-app/src/lib/components/ui/card/card-content.svelte capycoding-app/src/lib/components/ui/card/card-description.svelte capycoding-app/src/lib/components/ui/card/card-footer.svelte capycoding-app/src/lib/components/ui/card/card-header.svelte capycoding-app/src/lib/components/ui/card/card-title.svelte capycoding-app/src/lib/components/ui/card/card.svelte capycoding-app/src/lib/components/ui/card/index.ts capycoding-app/src/lib/components/ui/carousel/carousel-content.svelte capycoding-app/src/lib/components/ui/carousel/carousel-item.svelte capycoding-app/src/lib/components/ui/carousel/carousel-next.svelte capycoding-app/src/lib/components/ui/carousel/carousel-previous.svelte capycoding-app/src/lib/components/ui/carousel/carousel.svelte capycoding-app/src/lib/components/ui/carousel/context.ts capycoding-app/src/lib/components/ui/carousel/index.ts capycoding-app/src/lib/components/ui/chart/chart-container.svelte capycoding-app/src/lib/components/ui/chart/chart-style.svelte capycoding-app/src/lib/components/ui/chart/chart-tooltip.svelte capycoding-app/src/lib/components/ui/chart/chart-utils.ts capycoding-app/src/lib/components/ui/chart/index.ts capycoding-app/src/lib/components/ui/checkbox/checkbox.svelte capycoding-app/src/lib/components/ui/checkbox/index.ts capycoding-app/src/lib/components/ui/collapsible/collapsible-content.svelte capycoding-app/src/lib/components/ui/collapsible/collapsible-trigger.svelte capycoding-app/src/lib/components/ui/collapsible/collapsible.svelte [323 more files omitted for size] ``` ### Dependencies - ble-types/Cargo.toml: heapless@0.9.1, postcard@1.1.3, serde@1.0.228, uuid@1.18.1 - capycoding-app/package.json: @biomejs/biome@^2.3.0, @internationalized/date@^3.8.1, @livekit/components-core@^0.12.10, @livekit/components-styles@^1.1.6, @lucide/svelte@^0.544.0, @sveltejs/adapter-static@^3.0.10, @sveltejs/kit@^2.9.0, @sveltejs/vite-plugin-svelte@^5.0.0, @tailwindcss/vite@^4.1.13, @tanstack/table-core@^8.20.5, @tauri-apps/api@^2, @tauri-apps/cli@^2, @tauri-apps/plugin-opener@^2, @thisux/sveltednd@^0.0.20, @types/node@^24.9.1, @xenova/transformers@^2.17.2, bits-ui@^2.11.0, clsx@^2.1.1, embla-carousel-svelte@^8.6.0, formsnap@^2.0.1, layerchart@2.0.0-next.27, livekit-client@^2.5.6, mode-watcher@^1.1.0, neverthrow@^8.2.0, paneforge@^1.0.2, phosphor-svelte@^3.0.1, runed@^0.35.1, svelte@^5.0.0, svelte-check@^4.0.0, svelte-sonner@^1.0.5, sveltekit-superforms@^2.26.1, tailwind-merge@^3.3.1, tailwind-variants@^3.1.1, tailwindcss@^4.1.13, taurpc@^1.8.1, tw-animate-css@^1.4.0, typescript@~5.6.2, typescript-svelte-plugin@^0.3.50, vaul-svelte@^1.0.0-next.7, vite@^6.0.3, vite-plugin-devtools-json@^1.0.0 - capycoding-app/src-tauri/Cargo.toml: anyhow@1.0.100, ble-types, btleplug@0.11.8, chrono@0.4, dirs@5, jsonwebtoken@9, log@0.4.28, reqwest@0.12, serde@1, serde_json@1, specta@=2.0.0-rc.22, tauri@2, tauri-build@2, tauri-plugin-opener@2, taurpc@0.5.2, tokio@1 - capycoding-esp/Cargo.toml: ble-types, bt-hci@0.6.0, critical-section@1.2.0, display-interface-spi@0.5.0, embassy-executor@0.9.0, embassy-futures@0.1.2, embassy-net@0.7.0, embassy-sync@0.7.2, embassy-time@0.5.0, embedded-graphics@0.8.1, embedded-hal-bus@0.2.0, embedded-io@0.7.1, embedded-io-async@0.7.0, embedded-storage@0.3.1, esp-alloc@0.9.0, esp-backtrace@0.18.0, esp-bootloader-esp-idf@0.3.0, esp-hal@=1.0.0-rc.1, esp-println@0.16.0, esp-radio@0.16.0, esp-rtos@0.1.1, esp-storage@0.8.0, heapless@0.9.1, log@0.4.27, mousefood, postcard@1.1.3, profont@0.7.0, ratatui, reqwless@0.13.0, serde@1.0.228, smoltcp@0.12.0, static_cell@2.1.1, trouble-host@0.5.0, weact-studio-epd@0.1.2 - server/go.mod: github.com/cloudflare/circl@v1.3.3, github.com/golang-jwt/jwt@v3.2.2+incompatible, github.com/golang/protobuf@v1.5.3, github.com/google/go-github/v55@v55.0.0, github.com/google/go-querystring@v1.1.0, github.com/labstack/echo/v4@v4.11.4, github.com/labstack/gommon@v0.4.2, github.com/mattn/go-colorable@v0.1.13, github.com/mattn/go-isatty@v0.0.20, github.com/ProtonMail/go-crypto@v0.0.0-20230217124315-7d5c6f04bbb8, github.com/valyala/bytebufferpool@v1.0.0, github.com/valyala/fasttemplate@v1.2.2, golang.org/x/crypto@v0.19.0, golang.org/x/net@v0.21.0, golang.org/x/oauth2@v0.17.0, golang.org/x/sys@v0.17.0, golang.org/x/text@v0.14.0, golang.org/x/time@v0.5.0, google.golang.org/appengine@v1.6.7, google.golang.org/protobuf@v1.31.0 ### Recent commits (newest first) - Merge pull request #13 from Sluggish-Solutions/claude-metrics-wip - Merge branch 'main' into claude-metrics-wip - Final changes - cards - page.svelte - Merge branch 'main' of github.com:Sluggish-Solutions/CappyCoding - FINALLY GOT LIVEKIT TO WORK - erm... - Merge branch 'main' of github.com:Sluggish-Solutions/CappyCoding - fix cargo lock - int fix - add cappy png - add generated types.ts to gitignore - Merge branch 'main' of github.com:Sluggish-Solutions/CappyCoding - connect_device - Merge pull request #12 from Sluggish-Solutions/create-claude-code-api-integration - Merge branch 'main' into create-claude-code-api-integration - Append remote audio elements to DOM - merge - erm... ## Key source files (fetched from GitHub, selected and truncated for size) ### QUICK-START.md ```markdown # ๐Ÿš€ Claude Usage Tracker - Quick Start ## โœ… Status: FULLY OPERATIONAL Everything is installed, configured, and tested. ## ๏ฟฝ๏ฟฝ What You Have **Current Usage (Last 24 Hours):** - Total cost: $0.51 USD - Total tokens: 94,434 - Sessions: 19 - Burn rate: $0.021/hour ## ๐ŸŽฏ 3-Step Quick Start ### 1. Start the Go Server ```bash cd server go run main.go ``` Server starts on `http://localhost:8080` ### 2. Start the Tauri App ```bash cd capycoding-app npm run tauri dev ``` The UI opens automatically. ### 3. Use the UI 1. Navigate to **"Claude Usage Metrics"** section 2. Click **"Collect metrics"** (uses defaults) 3. Click **"Start auto-sync"** for automatic 5-minute updates Done! ๐ŸŽ‰ ## ๐Ÿงช Test Everything ```bash ./demo-usage-tracker.sh ``` Expected output: All checks pass โœ… ## ๐ŸŽค Voice Agent (Optional) ```bash # Start voice agent python agent.py dev # Monitor usage in another terminal tail -f ~/.claude/projects/voice-agent.jsonl ``` ## ๐Ÿ“ API Endpoints - `GET http://localhost:8080/metrics/claude` - Current metrics - `POST http://localhost:8080/metrics/claude` - Push metrics - `GET http://localhost:8080/metrics/claude/history` - History ## ๐Ÿ” Monitor Real-Time ```bash # Watch all Claude usage tail -f ~/.claude/projects/*.jsonl # Pretty print tail -f ~/.claude/projects/*.jsonl | jq ``` ## ๐Ÿ“š More Info - **Full docs:** `USAGE-TRACKER-COMPLETE.md` - **Implementation:** `IMPLEMENTATION-COMPLETE.md` - **Tests:** `test-complete-tracker.sh` - **Demo:** `demo-usage-tracker.sh` --- **Last Updated:** October 26, 2025 **Status:** Production Ready โœ… ``` ### README-AGENT.md ```markdown # LiveKit Voice Agent Setup ## Quick Start The LiveKit voice agent is now consolidated at the project root. ### 1. Environment Setup The agent uses the root `/env` virtual environment with all dependencies installed. ### 2. Configuration Methods You can configure the agent in **3 ways** (in order of precedence): #### Option A: Frontend UI (Recommended) - Open the Tauri app - Navigate to "LiveKit Voice Agent" section - Fill in your API keys - Click "Save Configuration" - Click "Start Agent" Configuration saved to: `~/.config/capycoding/agent_config.json` #### Option B: .env File ```bash cd /Users/akhildatla/GitHub/CappyCoding cp livekit-agent/.env.example .env # Edit .env with your keys ``` #### Option C: Direct Environment Variables ```bash export LIVEKIT_URL="wss://your-project.livekit.cloud" export LIVEKIT_API_KEY="your_key" export LIVEKIT_API_SECRET="your_secret" export ANTHROPIC_API_KEY="your_claude_key" ``` ### 3. Running the Agent **From Frontend** (recommended): - Click "Start Agent" in the Tauri app UI **From Terminal**: ```bash cd /Users/akhildatla/GitHub/CappyCoding source env/bin/activate python agent.py dev ``` ### 4. API Keys Required | Service | Purpose | Get it from | |---------|---------|-------------| | **LiveKit** | Room management + STT/TTS via Inference | https://cloud.livekit.io/ | | **Anthropic** | Claude AI (LLM) | https://console.anthropic.com/ | That's it! Just 2 API keys needed. ## How It Works ``` User โ†’ [Tauri App] โ† LiveKit WebRTC โ†’ [Python Agent] โ†“ LiveKit Inference (Deepgram STT) โ†“ Anthropic Claude (LLM) โ†“ LiveKit Inference (Cartesia TTS) ``` ## Features - โœ… **Automatic speech detection** - No button clicks needed - โœ… **Natural conversation** - Claude Sonnet 4.5 for intelligence - โœ… **Professional quality** - Deepgram STT + Cartesia TTS - โœ… **Simple setup** - Configure from frontend UI - โœ… **One venv** - All dependencies in `/env` ## Costs - **LiveKit Inference STT**: $0.0043/min - **LiveKit Inference TTS**: $0.045/min - **Claude Sonnet 4.5**: ~$3/$15 per million tokens (input/output) **Typical usage**: ~$0.27/hour of conversation ## Troubleshooting **Agent won't start**: - Check API keys are set (via UI or .env) - Verify venv is activated: `source env/bin/activate` - Check dependencies: `pip list | grep livekit` **No transcription**: - Verify LiveKit Inference is enabled (should be automatic) - Check LiveKit dashboard for errors **Can't hear responses**: - Check Tauri app has audio permissions - Verify agent connected (see terminal output) - Check browser/system audio settings ## Files - `/agent.py` - Main agent code - `/env/` - Consolidated Python virtual environment - `/livekit-agent/` - Documentation and examples (can be removed if not needed) - `~/.config/capycoding/agent_config.json` - [truncated โ€” 273 more characters] ``` ### ble-types/Cargo.toml ``` [package] name = "ble-types" version = "0.1.0" edition = "2024" [dependencies] heapless = { version = "0.9.1", features = ["serde"] } postcard = "1.1.3" serde = {version = "1.0.228", default-features = false} uuid = {version = "1.18.1", default-features=false} ``` ### server/go.mod ``` module cappycoding/server go 1.21 require ( github.com/google/go-github/v55 v55.0.0 github.com/labstack/echo/v4 v4.11.4 golang.org/x/oauth2 v0.17.0 ) require ( github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 // indirect github.com/cloudflare/circl v1.3.3 // indirect github.com/golang-jwt/jwt v3.2.2+incompatible // indirect github.com/golang/protobuf v1.5.3 // indirect github.com/google/go-querystring v1.1.0 // indirect github.com/labstack/gommon v0.4.2 // indirect github.com/mattn/go-colorable v0.1.13 // indirect github.com/mattn/go-isatty v0.0.20 // indirect github.com/valyala/bytebufferpool v1.0.0 // indirect github.com/valyala/fasttemplate v1.2.2 // indirect golang.org/x/crypto v0.19.0 // indirect golang.org/x/net v0.21.0 // indirect golang.org/x/sys v0.17.0 // indirect golang.org/x/text v0.14.0 // indirect golang.org/x/time v0.5.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/protobuf v1.31.0 // indirect ) ``` ### capycoding-app/package.json ``` { "name": "capycoding-app", "version": "0.1.0", "description": "", "type": "module", "scripts": { "dev": "vite dev", "build": "vite build", "preview": "vite preview", "check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json", "check:watch": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json --watch", "tauri": "tauri" }, "license": "MIT", "dependencies": { "@livekit/components-core": "^0.12.10", "@livekit/components-styles": "^1.1.6", "@lucide/svelte": "^0.544.0", "@tauri-apps/api": "^2", "@tauri-apps/plugin-opener": "^2", "@thisux/sveltednd": "^0.0.20", "@xenova/transformers": "^2.17.2", "clsx": "^2.1.1", "livekit-client": "^2.5.6", "neverthrow": "^8.2.0", "runed": "^0.35.1", "tailwind-merge": "^3.3.1", "tailwind-variants": "^3.1.1", "taurpc": "^1.8.1", "tw-animate-css": "^1.4.0" }, "devDependencies": { "@biomejs/biome": "^2.3.0", "@internationalized/date": "^3.8.1", "@sveltejs/adapter-static": "^3.0.10", "@sveltejs/kit": "^2.9.0", "@sveltejs/vite-plugin-svelte": "^5.0.0", "@tailwindcss/vite": "^4.1.13", "@tanstack/table-core": "^8.20.5", "@tauri-apps/cli": "^2", "@types/node": "^24.9.1", "bits-ui": "^2.11.0", "embla-carousel-svelte": "^8.6.0", "formsnap": "^2.0.1", "layerchart": "2.0.0-next.27", "mode-watcher": "^1.1.0", "paneforge": "^1.0.2", "phosphor-svelte": "^3.0.1", "svelte": "^5.0.0", "svelte-check": "^4.0.0", "svelte-sonner": "^1.0.5", "sveltekit-superforms": "^2.26.1", "tailwindcss": "^4.1.13", "typescript": "~5.6.2", "typescript-svelte-plugin": "^0.3.50", "vaul-svelte": "^1.0.0-next.7", "vite": "^6.0.3", "vite-plugin-devtools-json": "^1.0.0" } } ``` ### capycoding-esp/Cargo.toml ``` [package] edition = "2024" name = "capycoding-esp" rust-version = "1.88" version = "0.1.0" [[bin]] name = "capycoding-esp" path = "./src/bin/main.rs" [dependencies] esp-hal = { version = "=1.0.0-rc.1", features = [ "esp32c3", "log-04", "unstable", ] } esp-rtos = { version = "0.1.1", features = [ "embassy", "esp-alloc", "esp-radio", "esp32c3", "log-04", ] } esp-bootloader-esp-idf = { version = "0.3.0", features = ["esp32c3"] } log = "0.4.27" embassy-net = { version = "0.7.0", features = [ "dhcpv4", "log", "medium-ethernet", "tcp", "udp", "dns" ] } embedded-io = "0.7.1" embedded-io-async = "0.7.0" esp-alloc = "0.9.0" esp-backtrace = { version = "0.18.0", features = [ "esp32c3", "panic-handler", "println", ] } esp-println = { version = "0.16.0", features = ["esp32c3", "log-04"] } # for more networking protocol support see https://crates.io/crates/edge-net bt-hci = "0.6.0" embassy-executor = { version = "0.9.0", features = ["log"] } embassy-time = { version = "0.5.0", features = ["log"] } esp-radio = { version = "0.16.0", features = [ "ble", "coex", "esp-alloc", "esp32c3", "log-04", "smoltcp", "unstable", "wifi", ] } smoltcp = { version = "0.12.0", default-features = false, features = [ "log", "medium-ethernet", "multicast", "proto-dhcpv4", "proto-dns", "proto-ipv4", "socket-dns", "socket-icmp", "socket-raw", "socket-tcp", "socket-udp", "dns-max-server-count-4", ] } trouble-host = { version = "0.5.0", features = ["gatt"] } critical-section = "1.2.0" static_cell = "2.1.1" display-interface-spi = "0.5.0" embedded-graphics = "0.8.1" embedded-hal-bus = "0.2.0" profont = "0.7.0" weact-studio-epd = { version = "0.1.2", features = ["blocking"] } heapless = { version = "0.9.1", features = ["serde"] } # mousefood = { git = "https://github.com/j-g00da/mousefood", rev = "1028ac24f83920fd95a5585a85ef943901ee63ac", features = ["epd-weact" ]} mousefood = { git = "https://github.com/suri-codes/mousefood.git", branch = "main", features = ["epd-weact", "portable-atomic", "framebuffer"]} ratatui = { git = "https://github.com/suri-codes/ratatui.git", branch = "main", default-features = false} esp-storage = { version = "0.8.0", features = ["esp32c3"] } serde = {version = "1.0.228", default-features = false} postcard = "1.1.3" embedded-storage = "0.3.1" embassy-sync = "0.7.2" reqwless = { version = "0.13.0", default-features = false, features = [ "embedded-tls", ] } embassy-futures = "0.1.2" ble-types = {path = "../ble-types"} [patch.crates-io] ratatui = { git = "https://github.com/suri-codes/ratatui.git", branch = "main" } ratatui-core = { git = "https://github.com/suri-codes/ratatui.git", branch = "main" } [profile.dev] # Rust debug is too slow. # For debug builds always builds with some optimization # opt-level = "s" opt-level = 3 [profile.release] codegen-units = 1 # LLVM can perform better optimizations using a single thread debug = 2 debug-assertions = false incremental = false lto = 'fat' opt-level = 's' overflow-checks = false ``` ### capycoding-app/src-tauri/Cargo.toml ``` [package] name = "capycoding-app" version = "0.1.0" description = "A Tauri App" authors = ["you"] edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [lib] # The `_lib` suffix may seem redundant but it is necessary # to make the lib name unique and wouldn't conflict with the bin name. # This seems to be only an issue on Windows, see https://github.com/rust-lang/cargo/issues/8519 name = "capycoding_app_lib" crate-type = ["staticlib", "cdylib", "rlib"] [build-dependencies] tauri-build = { version = "2", features = [] } [dependencies] tauri = { version = "2", features = [] } tauri-plugin-opener = "2" serde = { version = "1", features = ["derive"] } serde_json = "1" chrono = { version = "0.4", features = ["serde"] } reqwest = { version = "0.12", features = ["json", "rustls-tls"] } jsonwebtoken = "9" dirs = "5" taurpc = "0.5.2" specta = { version = "=2.0.0-rc.22", features = ["derive"] } # specta-typescript = "0.0.9" tokio = { version = "1", features = ["full"] } ble-types= {path="../../ble-types"} anyhow = "1.0.100" btleplug = "0.11.8" log = "0.4.28" ``` ### server/main.go ```go package main import ( "context" "errors" "log" "net/http" "os" "os/signal" "syscall" "time" echov4 "github.com/labstack/echo/v4" "github.com/labstack/echo/v4/middleware" "cappycoding/server/internal/claude" "cappycoding/server/internal/githubclient" httpHandlers "cappycoding/server/internal/http" ) func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() client, err := githubclient.NewClient(ctx) if err != nil { if errors.Is(err, githubclient.ErrMissingToken) { log.Printf("github token not configured: %v", err) log.Printf("requests must include Authorization or X-GitHub-Token headers; continuing without default credentials") client = nil } else { log.Fatalf("failed to create github client: %v", err) } } e := echov4.New() e.HideBanner = true e.Use(middleware.Logger()) e.Use(middleware.Recover()) claudeStore := claude.NewStore(288) httpHandlers.RegisterRoutes(e, client, claudeStore) srv := &http.Server{ Addr: addr(), Handler: e, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, } errCh := make(chan error, 1) go func() { log.Printf("Starting metrics server on %s", srv.Addr) errCh <- srv.ListenAndServe() }() select { case <-ctx.Done(): shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() if err := srv.Shutdown(shutdownCtx); err != nil { log.Printf("graceful shutdown failed: %v", err) } case err := <-errCh: if err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("server error: %v", err) } } } func addr() string { port := os.Getenv("PORT") if port == "" { port = "8080" } return ":" + port } ``` ### capycoding-app/src-tauri/src/main.rs ```rust // Prevents additional console window on Windows in release, DO NOT REMOVE!! #![cfg_attr(not(debug_assertions), windows_subsystem = "windows")] fn main() { capycoding_app_lib::run() } ``` ### capycoding-esp/src/bin/main.rs ```rust #![no_std] #![no_main] #![deny( clippy::mem_forget, reason = "mem::forget is generally not safe to do with esp_hal types, especially those \ holding buffers for the duration of a data transfer." )] use capycoding_esp::ble::ble_task; use capycoding_esp::wifi::{connection, net_task, wifi_task}; use capycoding_esp::{CapyConfig, WeactTermInitPins, ui_task}; use embassy_executor::Spawner; use embassy_net::dns::DnsSocket; use embassy_net::tcp::client::{self, TcpClient, TcpClientState}; use embassy_net::{Config, DhcpConfig, Runner, Stack, StackResources}; use embassy_sync::blocking_mutex::raw::{CriticalSectionRawMutex, NoopRawMutex}; use embassy_sync::pubsub::PubSubChannel; use embassy_time::{Duration, Timer}; use esp_hal::clock::CpuClock; use esp_hal::rng::Rng; use esp_hal::spi::{self, master::Spi}; use esp_hal::{time::Rate, timer::timg::TimerGroup}; use esp_radio::wifi::{ self, ClientConfig, Config as WifiConfig, ModeConfig, ScanConfig, WifiController, WifiDevice, WifiEvent, WifiStaState, }; use esp_storage::FlashStorage; use esp_backtrace as _; use capycoding_esp::alloc::string::String; use log::info; use reqwless::client::{HttpClient, TlsConfig}; use static_cell::StaticCell; // This creates a default app-descriptor required by the esp-idf bootloader. // For more information see: esp_bootloader_esp_idf::esp_app_desc!(); use embassy_sync::mutex::Mutex; const SSID: &str = "Surendra S21"; const PASSWORD: &str = "Surendra2006"; pub static CONFIG: StaticCell>> = StaticCell::new(); static RADIO: StaticCell> = StaticCell::new(); static PUB_SUB_CHANNEL: static_cell::StaticCell> = static_cell::StaticCell::new(); #[derive(Copy, Clone)] pub enum Message { Connected, } macro_rules! mk_static { ($t:ty,$val:expr) => {{ static STATIC_CELL: static_cell::StaticCell<$t> = static_cell::StaticCell::new(); #[deny(unused_attributes)] let x = STATIC_CELL.uninit().write(($val)); x }}; } #[esp_rtos::main] async fn main(spawner: Spawner) -> () { esp_println::logger::init_logger_from_env(); let config = esp_hal::Config::default().with_cpu_clock(CpuClock::max()); let peripherals = esp_hal::init(config); // shit ton of memory alloc esp_alloc::heap_allocator!(#[unsafe(link_section = ".dram2_uninit")] size: 66320); esp_alloc::heap_allocator!(size: 170* 1024); // rtos init let timg0 = TimerGroup::new(peripherals.TIMG0); let sw_interrupt = esp_hal::interrupt::software::SoftwareInterruptControl::new(peripherals.SW_INTERRUPT); esp_rtos::start(timg0.timer0, sw_interrupt.software_interrupt0); info!("Embassy initialized!"); // init radio let radio_init: esp_radio::Controller<'_> = esp_radio::init().expect("Failed to initialize Wi-Fi/BLE controller"); let radio = RADIO.init(radio_init); // init for term let mosi_pin = peripherals.GPIO5; let sclk_pin = peripherals.GPIO4; let spi_bus = Spi::new( peripherals.SPI2, spi::master::Config::default() .with_frequency(Rate::from_khz(100)) .with_mode(spi::Mode::_0), ) .unwrap() .with_sck(sclk_pin) .with_mosi(mosi_pin); let term_init_pins = WeactTermInitPins { cs_pin: peripherals.GPIO3, dc_pin: peripherals.GPIO2, rst_pin: peripherals.GPIO1, busy_pin: peripherals.GPIO0, }; let mut flash = FlashStorage::new(peripherals.FLASH); let state = CapyConfig::load(&mut flash); // spawn tasks let capyconfig = CONFIG.init(Mutex::new(state)); let capy_ref = &*capyconfig; let (wifi_controller, ifaces) = esp_radio::wifi::new(radio, peripherals.WIFI, WifiConfig::default()).unwrap(); let wifi_interface = ifaces.sta; let rng = Rng::new(); let net_seed = rng.random() as u64 | ((rng.random() as u64) << 32); let tls_seed = rng.random() as u64 | ((rng.random() as u64) << 32); let dhcp_config = DhcpConfig::default(); let config = embassy_net::Config::dhcpv4(dhcp_config); // Init network stack let (stack, runner) = embassy_net::new( wifi_interface, config, mk_static!(StackResources<3>, StackResources::<3>::new()), net_seed, ); // BLE handler spawner .spawn(ble_task(radio, peripherals.BT, capy_ref)) .unwrap(); // UI handler spawner .spawn(ui_task(spi_bus, term_init_pins, capy_ref)) .unwrap(); // wifi util tasks spawner.spawn(connection(wifi_controller)).unwrap(); spawner.spawn(net_task(runner)).unwrap(); // main wifi task spawner.spawn(wifi_task(stack, tls_seed)).unwrap(); } ``` [403 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]