# Project export: VisionArena This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: An AI-native boss fight where your body is the controller: computer vision reads your gestures, agents adapt the boss, and a live voice narrator reacts like GTA mission control. - Devpost: https://devpost.com/software/visionarena - GitHub: https://github.com/behzad-janjua/VIsionArena - Video: https://www.youtube.com/embed/K78cdJTatXo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Behzad (31 commits), Claude Sonnet 4.6 (8 commits) ## Devpost submission (written by the team) No Devpost description available. ## README (from the GitHub repository) # Vision Arena A real-time 2.5D anime boss fight controlled by **computer-vision hand gestures**, a **MYO EMG armband**, and a swarm of **AI agents** — built for the Fetch.ai / ASI:One hackathon. Open your palm to move, close your fist to punch (longer windup = heavier hit), and an adaptive boss learns to counter you between exchanges. **Live agent (ASI:One):** https://asi1.ai/chat/a8e8512f-aac3-40ee-949b-4b7dbf310f3e **Agentverse address:** `agent1q0x73mhcy54lj0efh3eus5zxxqgkjkzascm8syy222c2ypx2zzkmy0gtez6` ## Highlights - **Embodied control** — MediaPipe webcam gestures and a MYO armband drive movement and punch tiers, with a seamless keyboard fallback. - **Adaptive boss** — every hit is traced (Arize) and stored as a 5-dim style vector in Redis; a KNN query recalls the most similar past player and reuses the strategy that beat them. - **Agent-run match** — a `GameMasterAgent` orchestrates enemy, narrator, and recap agents, demoable through ASI:One via Fetch.ai Agentverse. - **Pre-fight boss call** — the boss phones the player (Vapi) to taunt them before the match. - **Post-fight recap** — Pika generates a cinematic boxing-style recap from real match telemetry; Deepgram voices live commentary. ## Architecture A Unity client and a FastAPI backend talk over a WebSocket (real-time events) and HTTP (agent decisions). A single `KiForgeArenaBootstrap` builds the whole arena at runtime. ![Vision Arena architecture](docs/architecture-diagram.svg) **See [`docs/architecture.md`](docs/architecture.md) for the detailed data flows and a text (Mermaid) version of this diagram.** ## Project layout | Path | Responsibility | | --- | --- | | `Assets/Scripts/Bootstrap` | `KiForgeArenaBootstrap` — runtime wiring of the entire arena | | `Assets/Scripts/Input` | CV aim, MYO charge tiers, keyboard fallback, WebSocket sources | | `Assets/Scripts/Combat` | Punch tiers, health, guard timing, boss agent link, strategy weights | | `Assets/Scripts/UI` | HUD, charge/health bars, narration, captions, Redis & Fight Lab panels, boss-call, recap | | `Assets/Scripts/Telemetry` | Match event recorder, Arize-style coach feedback, mock agent client | | `Assets/Scripts/Effects` `/Scene` `/Animation` | Charge aura, impact FX, camera, fighter animation | | `backend/` | FastAPI WebSocket service, agent workflow, Redis wrapper, Arize tracing, Pika recap, Vapi/Deepgram adapters | ## Run the Unity demo 1. Open this folder in **Unity 2022.3** or newer. 2. Open `Assets/Scenes/VisionArena.unity` (it contains a `KiForgeArenaBootstrap` GameObject that builds everything at runtime). 3. Press **Play**. A phone-number entry screen appears first (the boss call — press **skip** to go straight to the arena), then the fight loads. The scene is fully playable **without the backend** — gestures and agents simply fall back to keyboard + deterministic local behavior. ### Controls (keyboard fallback) | Input | Action | | --- | --- | | `A` / `D` | Move left / right | | `J` / `K` | Left / right punch | | `U` / `I` | Heavy / very-heavy punch | | `;` / `'` | Hold to guard left / right | | `B` | Hold to charge a punch (simulates MYO fist contraction) | | `R` | Toggle the Redis sponsor panel | | `Tab` | Toggle the AI Fight Lab panel | When CV is connected: **open palm → walk forward**, **closed fist → punch** (longer contraction = heavier tier). ## Run the backend ```bash python3 -m venv .venv source .venv/bin/activate pip install -r backend/requirements.txt uvicorn backend.main:app --reload --port 8000 ``` - Health check: `http://127.0.0.1:8000/health` - Unity/backend event socket: `ws://127.0.0.1:8000/ws/unity` - See [`backend/README.md`](backend/README.md) for the full endpoint list, Redis schema, and Agentverse/ASI:One setup. ### Configuration (`.env`) Everything runs in mock mode with no keys. Set only what you want to light up — the backend loads `.env` at startup via `python-dotenv`, so **restart it after edits**. | Variable(s) | Enables | | --- | --- | | `REDIS_URL` | Persistent player memory + vector recall (Redis Stack; in-memory fallback otherwise) | | `ARIZE_API_KEY`, `ARIZE_SPACE_ID` | Arize fight-trace export (local Fight Lab works without it) | | `ASI_ONE_API_KEY` / `OPENAI_API_KEY` (+ `ASI_ONE_BASE_URL`, `ASI_ONE_MODEL`) | Live LLM agents (deterministic fallback otherwise) | | `FETCH_AI_AGENT_SEED`, `AGENT_PORT` | Stable Fetch.ai Agentverse address for `backend/uagents_app.py` | | `PIKA_API_KEY`, `PIKA_RECAP_QUEUE` | Pika recap-video generation | | `DEEPGRAM_API_KEY`, `DEEPGRAM_COMMENTATOR_VOICE`, `DEEPGRAM_BOSS_VOICE` | Live commentator / boss TTS | | `VAPI_API_KEY`, `VAPI_PHONE_NUMBER_ID`, `VAPI_VOICE_ID` | Outbound pre-fight boss phone call | | `CV_CAMERA_INDEX`, `CV_TARGET_FPS`, `CV_PUNCH_DEPTH`, … | MediaPipe webcam tuning | | `MYO_ENABLED`, `MYO_EMG_THRESHOLD`, `MYO_DEBOUNCE`, … | MYO armband driver tuning | > **Vapi note:** outbound calls need a real telephony number **imported into Vapi** > (e.g. a Twilio number with your Account SID/Auth Token). Vapi's free built-in > numbers register a call but frequently never reach a real handset (`silence-timed-out`) > and can't dial international numbers. The voice provider must be `"11labs"`. ### Mock MYO / vision bridges ```bash python -m backend.myo_listener --ws ws://127.0.0.1:8000/ws/unity --repeat 3 ``` ## Tests ```bash pytest # backend (tests/test_backend.py) ``` Unity: open the **Test Runner** and run the EditMode tests under `Assets/Tests/EditMode` (e.g. `CombatRulesTests`). ## Reliability philosophy MYO, MediaPipe, Fetch.ai, Arize, Redis, Pika, Deepgram, and Vapi all have mock/fallback seams. The intended flow: make the local Unity fight fun first, then progressively connect live hardware and sponsor APIs without risking the core demo. ## Detected evidence (automated analysis) Indexed codebase: 261 recognized source files, 2070 KB. - C# (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Redis (technology) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 2820) ``` .agents/skills/app-sizzle/references/liquid-glass.md .agents/skills/app-sizzle/references/template-a-cinematic.md .agents/skills/app-sizzle/SKILL.md .agents/skills/app-store-screens/references/default-layout.md .agents/skills/app-store-screens/references/layout-archetypes.md .agents/skills/app-store-screens/references/render-pipeline.md .agents/skills/app-store-screens/SKILL.md .agents/skills/arize-admin/references/ax-profiles.md .agents/skills/arize-admin/references/ax-setup.md .agents/skills/arize-admin/references/REFERENCE.md .agents/skills/arize-admin/SKILL.md .agents/skills/arize-ai-provider-integration/references/ax-profiles.md .agents/skills/arize-ai-provider-integration/references/ax-setup.md .agents/skills/arize-ai-provider-integration/SKILL.md .agents/skills/arize-annotation/references/ax-profiles.md .agents/skills/arize-annotation/references/ax-setup.md .agents/skills/arize-annotation/SKILL.md .agents/skills/arize-compliance-audit/references/compliance-checklist-template.md .agents/skills/arize-compliance-audit/references/eu-ai-act-gpai.md .agents/skills/arize-compliance-audit/references/iso-42001.md .agents/skills/arize-compliance-audit/references/us-ai-compliance.md .agents/skills/arize-compliance-audit/SKILL.md .agents/skills/arize-dataset/references/ax-profiles.md .agents/skills/arize-dataset/references/ax-setup.md .agents/skills/arize-dataset/SKILL.md .agents/skills/arize-evaluator/references/ax-profiles.md .agents/skills/arize-evaluator/references/ax-setup.md .agents/skills/arize-evaluator/SKILL.md .agents/skills/arize-experiment/references/ax-profiles.md .agents/skills/arize-experiment/references/ax-setup.md .agents/skills/arize-experiment/SKILL.md .agents/skills/arize-instrumentation/references/ax-profiles.md .agents/skills/arize-instrumentation/references/integration-routing.md .agents/skills/arize-instrumentation/references/manual-spans.md .agents/skills/arize-instrumentation/references/tracing-assistant-mcp.md .agents/skills/arize-instrumentation/SKILL.md .agents/skills/arize-link/references/EXAMPLES.md .agents/skills/arize-link/SKILL.md .agents/skills/arize-prompt-optimization/references/ax-profiles.md .agents/skills/arize-prompt-optimization/references/ax-setup.md .agents/skills/arize-prompt-optimization/SKILL.md .agents/skills/arize-prompts/references/ax-profiles.md .agents/skills/arize-prompts/references/ax-setup.md .agents/skills/arize-prompts/references/cli-prompts.md .agents/skills/arize-prompts/SKILL.md .agents/skills/arize-trace/references/ax-profiles.md .agents/skills/arize-trace/references/ax-setup.md .agents/skills/arize-trace/SKILL.md .agents/skills/baseball-trend/SKILL.md .agents/skills/build-a-brand/references/brand-directions.md .agents/skills/build-a-brand/references/brand-guidelines.md .agents/skills/build-a-brand/references/brand-identity.md .agents/skills/build-a-brand/references/brand-md-template.md .agents/skills/build-a-brand/SKILL.md .agents/skills/content-director/formats/dance.md .agents/skills/content-director/formats/duet.md .agents/skills/content-director/formats/pov.md .agents/skills/content-director/formats/talking.md .agents/skills/content-director/formats/teleprompter.html .agents/skills/content-director/formats/teleprompter.md .agents/skills/content-director/README.md .agents/skills/content-director/SKILL.md .agents/skills/explainer/SKILL.md .agents/skills/fix-my-look/SKILL.md .agents/skills/founder-product-video/references/ops-notes.md .agents/skills/founder-product-video/SKILL.md .agents/skills/kiss-cam/SKILL.md .agents/skills/language-swap/references/language-coverage.md .agents/skills/language-swap/SKILL.md .agents/skills/persona-builder/references/aesthetic-prompts.md .agents/skills/persona-builder/references/persona-md-template.md .agents/skills/persona-builder/references/templates/mood-board-no-header.template.html .agents/skills/persona-builder/references/templates/mood-board.template.html .agents/skills/persona-builder/references/templates/pdf-about.template.html .agents/skills/persona-builder/references/templates/pdf-content-categories.template.html .agents/skills/persona-builder/references/templates/pdf-cover.template.html .agents/skills/persona-builder/references/templates/pdf-do-dont.template.html .agents/skills/persona-builder/references/templates/pdf-hooks-dm.template.html .agents/skills/persona-builder/references/templates/pdf-moodboard.template.html .agents/skills/persona-builder/references/templates/pdf-next-steps.template.html .agents/skills/persona-builder/references/templates/pdf-shared-head.template.html .agents/skills/persona-builder/references/templates/pdf-voice-mode.template.html .agents/skills/persona-builder/SKILL.md .agents/skills/podcast/SKILL.md .agents/skills/ugc-ads/SKILL.md .agents/skills/viral-hook/SKILL.md .claude/skills/app-sizzle .claude/skills/app-store-screens .claude/skills/arize-admin .claude/skills/arize-ai-provider-integration .claude/skills/arize-annotation .claude/skills/arize-compliance-audit .claude/skills/arize-dataset .claude/skills/arize-evaluator .claude/skills/arize-experiment .claude/skills/arize-instrumentation .claude/skills/arize-link .claude/skills/arize-prompt-optimization .claude/skills/arize-prompts .claude/skills/arize-trace .claude/skills/baseball-trend .claude/skills/build-a-brand .claude/skills/content-director .claude/skills/explainer .claude/skills/fix-my-look .claude/skills/founder-product-video .claude/skills/kiss-cam .claude/skills/language-swap .claude/skills/persona-builder .claude/skills/podcast .claude/skills/ugc-ads .claude/skills/viral-hook .env.example .gitignore Assets/AeryeonjeongComplex.meta Assets/AeryeonjeongComplex/Aeryeonjeong.meta Assets/AeryeonjeongComplex/Aeryeonjeong/Material.meta Assets/AeryeonjeongComplex/Aeryeonjeong/Material/MI_AngleRafter01C.mat Assets/AeryeonjeongComplex/Aeryeonjeong/Material/MI_AngleRafter01C.mat.meta Assets/AeryeonjeongComplex/Aeryeonjeong/Material/MI_Blackmetal01A.mat [2700 more files omitted for size] ``` ### Dependencies - backend/requirements.txt: arize-otel@>=0.3.0, fastapi@>=0.111.0, mediapipe@>=0.10.0, numpy@>=1.24.0, openai@>=1.30.0, opencv-python-headless@>=4.9.0, openinference-instrumentation-openai@>=0.1.0, pytest@>=8.0.0, python-dotenv@>=1.0.0, redis@>=5.0.0, requests@>=2.31.0, uagents@>=0.22.0, uvicorn[standard]@>=0.30.0, websockets@>=12.0 ### Recent commits (newest first) - chore: architecture - chore: updating read me - chore: updating read me and architecture diagram - fix: updating pika - feat: pika - feat: fixing vapi, redis, and deepgram implementations - chore: fixing health bar colours - fixes - feat: chore: changing colours and characters - chore: changing colours and characters - feat: adding mob boss call - fixing CV movement, myo connectivity, and adding a pre boss scene - chore: disable Pika submission until demo is ready - feat: Pika highlight-based cinematic recap from real fight data - docs: add ASI:One chat link and Agentverse address to README - fix: remove fight_lab dependency, fix learning_enabled, update tests - feat: wire Arize OTel tracing for fight lab - feat: damage numbers, AI fight lab panel, Agentverse wiring - feat: adding commentator agent and narration bar - fix: remove stale evaluation reference and clean up coach import ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/devpost_notes.md ```markdown # Vision Arena Devpost Notes ## Track Story Vision Arena is a 2.5D Unity boss fight controlled by body input. Unity handles the real-time combat loop while backend agents handle strategy, narration, memory, evaluation, and recap generation. ## Sponsor Mapping - Fetch.ai: GameMaster, Enemy, Narrator, and Recap agent boundaries. - Arize: deterministic fight-lab evaluator now, trace/export integration later. - Redis: player profile, boss strategy, match history, generated move names. - Pika: recap prompt generation from real match telemetry. ## Demo Path 1. Start Unity scene with `KiForgeArenaBootstrap`. 2. Use keyboard/mouse fallback to charge, aim, slash, shield, and ultimate. 3. Show mock agent narration and fight-lab adaptation panel. 4. Optionally start FastAPI backend and route combat telemetry through `/ws/unity`. 5. Show generated Pika recap prompt after a fight. ``` ### plan.md ```markdown # Vision Arena — Hackathon Build Plan ## Project Summary **Title:** Vision Arena **Main Track:** Ddoski’s Playground **Core Pitch:** A real-time anime boss-fight game where your computer-vision hand gestures control movement and punch timing, and AI agents run the enemy, narrator, coach, and post-fight recap. **Core Interaction:** - Computer vision detects open palm and fist gestures. - Open palm drives player movement. - Closed fist throws a punch. - Longer punch windups create heavier punch tiers. - Fetch.ai agents run the game master, enemy behavior, narration, coaching, and recap workflow. - Arize traces and evaluates enemy decisions so the boss gets better at fighting the player. - Redis stores player style, boss memory, match history, and generated punch names. - Pika generates post-fight boxing-style recap videos from real match telemetry. --- ## Strategic Positioning ### Main Track **Ddoski’s Playground** This project is a game, an interactive experience, and an experimental AI interface. It is not just a chatbot or dashboard; it is an embodied AI battle system. ### Sponsor Tracks to Target | Sponsor | Why It Fits | |---|---| | **Fetch AI** | Core agent system: GameMasterAgent, EnemyAgent, NarratorAgent, CoachAgent, RecapAgent. Agents should be registered on Agentverse and demoable through ASI:One. | | **Arize** | Traces and evaluates boss decisions, then uses those evaluations to improve the EnemyAgent’s strategy between rounds. | | **Redis** | Stores persistent player style, match history, boss strategy memory, cooldowns, generated move names, and recap data. | | **Pika** | Generates post-fight cinematic recap videos from match telemetry. | | **Band** | Optional: host EnemyAgent, RefereeAgent, NarratorAgent, and CoachAgent in a shared multi-agent room. | | **Deepgram** | Optional: voice commands such as “start duel,” “guard,” or AI announcer narration. | | **Sentry** | Optional: reliability/error monitoring for Unity, backend, and the agent pipeline. | --- ## Core Demo Moment The ideal judge demo: 1. You stand in front of the webcam. 2. The game tracks your wrist and body position. 3. You open your palm to move forward. 4. You close your fist to throw a punch. 5. Longer windup creates a heavier punch. 6. The boss takes damage if the punch lands in range. 7. The NarratorAgent names the punch. 8. Arize/Fight Lab shows the boss decision trace and evaluation. 9. The boss starts with a bad baseline policy. 10. Arize/Fight Lab catches the bad counter. 11. Battle Agent adapts and chooses the correct punch counter on the next exchange. 12. At the end, the RecapAgent creates a Pika prompt for a boxing-style recap. --- ## Input System ### Computer Vision Gesture Map | CV Input | Game Action | |---|---| | **Open palm** | Walk / move forward | | **Closed fist** | Throw punch | | **Short fist / quick tap** | Normal punch | | **Longer fist windup** | Heavy punch | | **Longest fist windup** | Very-heavy punch | | **Keyboard fall [truncated — 20974 more characters] ``` ### backend/requirements.txt ``` fastapi>=0.111.0 python-dotenv>=1.0.0 uvicorn[standard]>=0.30.0 redis>=5.0.0 numpy>=1.24.0 pytest>=8.0.0 mediapipe>=0.10.0 opencv-python-headless>=4.9.0 websockets>=12.0 requests>=2.31.0 # Fetch.ai uAgents — registers the GameMasterAgent on Agentverse (Mailbox) and # exposes the Chat Protocol for ASI:One. See backend/uagents_app.py. uagents>=0.22.0 # Optional: enables live ASI:One / OpenAI-compatible LLM agents. The backend # runs fine without it (agents fall back to deterministic output). openai>=1.30.0 # Arize fight-lab tracing. Set ARIZE_API_KEY + ARIZE_SPACE_ID in .env to activate. arize-otel>=0.3.0 openinference-instrumentation-openai>=0.1.0 # Optional: MYO armband real hardware driver. # pip install pyomyo ``` ### backend/main.py ```python from __future__ import annotations import asyncio import json import logging import os from dotenv import load_dotenv load_dotenv() from contextlib import asynccontextmanager from typing import Any from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect from pydantic import BaseModel, Field from backend.agents import GameMasterAgent from backend.agents.game_master_agent import _TRACKED_ABILITIES from backend.agentverse_adapter import respond_to_text as respond_to_agent_text from backend.commentary_adapter import respond_to_commentary_text from backend.fight_tracing import setup_tracing from backend.models import CombatTelemetry, EventType, NormalizedEvent from backend.player_memory import style_vector from backend.vision_bridge import vision_bridge_stream log = logging.getLogger(__name__) class _ConnectionManager: """Tracks connected Unity clients and broadcasts pose events to all of them.""" def __init__(self) -> None: self._queues: list[asyncio.Queue[dict[str, Any]]] = [] def connect(self) -> asyncio.Queue[dict[str, Any]]: q: asyncio.Queue[dict[str, Any]] = asyncio.Queue(maxsize=60) self._queues.append(q) return q def disconnect(self, q: asyncio.Queue[dict[str, Any]]) -> None: if q in self._queues: self._queues.remove(q) def broadcast(self, payload: dict[str, Any]) -> None: """Non-blocking push to all client queues; drops frames when a queue is full.""" for q in self._queues: try: q.put_nowait(payload) except asyncio.QueueFull: pass _manager = _ConnectionManager() _game_master = GameMasterAgent() class CombatTelemetryRequest(BaseModel): round: int = Field(..., description="Current combat round.") player_action: str = Field(..., description="Player action, e.g. left_punch, right_punch, heavy_punch, very_heavy_punch, guard.") charge_time: float = Field(..., description="Seconds the player charged before releasing.") accuracy: float = Field(..., ge=0.0, le=1.0, description="Player aim/pose accuracy from 0 to 1.") damage_dealt_by_player: int damage_dealt_by_boss: int boss_action: str = Field("unknown", description="Previous or proposed boss action, overwritten by the agent.") boss_health_after: int player_health_after: int outcome: str = Field(..., description="Outcome label, e.g. boss_staggered, boss_ko, player_blocked.") class AgentChatRequest(BaseModel): message: str = Field(..., description="Natural language request or CombatTelemetry JSON.") player_id: str = Field("agentverse_player", description="Stable player/session id for memory.") class AgentChatResponse(BaseModel): reply: str player_id: str audio_b64: str | None = None def _handle_agent_payload(payload: dict[str, Any], player_id: str) -> dict[str, Any]: telemetry = CombatTelemetry(**payload) return _game_master.handle_combat_event(telemetry, player_id=player_id).to_event().payload def _model_dict(model: BaseModel) -> dict[str, Any]: if hasattr(model, "model_dump"): return model.model_dump() return model.dict() async def _vision_task() -> None: """Background task: reads CV frames and broadcasts POSE_UPDATE to all Unity clients.""" async for event in vision_bridge_stream(): _manager.broadcast(event.to_dict()) async def _narration_pubsub_task() -> None: """Background task: subscribes to the Redis narration channel and broadcasts to Unity. This makes Pub/Sub the actual real-time pipe for narration rather than dead code. Falls back silently when Redis isn't configured. """ url = _game_master.store.url if not url: return try: import redis.asyncio as aioredis client = aioredis.from_url(url, decode_responses=True) pubsub = client.pubsub() await pubsub.subscribe("arena:narration") async for message in pubsub.listen(): if message["type"] == "message": try: data = json.loads(message["data"]) _manager.broadcast({"type": "NARRATION", **data}) except Exception: pass except asyncio.CancelledError: raise except Exception as exc: log.warning("[PubSub] narration subscriber error: %s", exc) @asynccontextmanager async def lifespan(app: FastAPI): setup_tracing() _game_master.store.seed_ghost_players() vision_task = asyncio.create_task(_vision_task()) pubsub_task = asyncio.create_task(_narration_pubsub_task()) try: yield finally: vision_task.cancel() pubsub_task.cancel() for t in (vision_task, pubsub_task): try: await t except asyncio.CancelledError: pass app = FastAPI(title="Vision Arena Backend", lifespan=lifespan) @app.get("/health") def health() -> dict[str, str]: return {"status": "ok", "cv": "live"} @app.get("/demo/state") def demo_state(player_id: str = "demo_player") -> dict[str, Any]: return _game_master.demo_state(player_id) @app.get("/demo/player-profile") def demo_player_profile(player_id: str = "demo_player") -> dict[str, Any]: return _game_master.store.get_json(f"player:{player_id}:profile") @app.get("/demo/fight-lab") def demo_fight_lab(player_id: str = "demo_player") -> dict[str, Any]: state = _game_master.demo_state(player_id) profile = state["player_profile"] latest_response = state["latest_response"] return { "player_style": profile.get("style", "unknown"), "boss_counter_success_before": profile.get("boss_counter_success_before", 0.0), "boss_counter_success_after": profile.get("boss_counter_success_after", 0.0), "most_common_player_move": profile.get("favorite_move", "none"), "boss_adaptation": profile.get("boss_adaptation", latest_response.get("next_strategy", "") [truncated — 11478 more characters] ``` ### register_commentator_agent.py ```python import os from uagents_core.utils.registration import ( register_chat_agent, RegistrationRequestCredentials, ) register_chat_agent( "Commentator Agent", "https://proxy-abide-proofs.ngrok-free.dev/agent/commentary", active=True, credentials=RegistrationRequestCredentials( agentverse_api_key=os.environ["COMMENTATOR_AGENTVERSE_KEY"], agent_seed_phrase=os.environ["COMMENTATOR_AGENT_SEED_PHRASE"], ), ) ``` ### backend/__init__.py ```python """Vision Arena backend package.""" ``` ### backend/deepgram_tts.py ```python from __future__ import annotations import base64 import logging import os import requests as http log = logging.getLogger(__name__) _DG_TTS_URL = "https://api.deepgram.com/v1/speak" def synthesize_b64(text: str, voice: str) -> str | None: """Call Deepgram Aura TTS and return base64-encoded WAV, or None if unavailable. WAV (linear16) is requested so Unity can decode PCM samples without a third-party MP3 library — AudioClip.Create() + SetData() handles it directly. """ api_key = os.getenv("DEEPGRAM_API_KEY") if not api_key: log.debug("[Deepgram] DEEPGRAM_API_KEY not set — skipping TTS") return None try: resp = http.post( f"{_DG_TTS_URL}?model={voice}&encoding=linear16&container=wav", headers={"Authorization": f"Token {api_key}", "Content-Type": "application/json"}, json={"text": text}, timeout=10, ) resp.raise_for_status() return base64.b64encode(resp.content).decode() except Exception as exc: log.warning("[Deepgram] TTS failed: %s", exc) return None ``` ### backend/commentary_adapter.py ```python from __future__ import annotations import json from backend.agents.narrator_agent import NarratorAgent from backend.agentverse_adapter import _COMBAT_KEYS, demo_combat_payload from backend.models import CombatTelemetry def format_commentary_reply(move_name: str, narration: str) -> str: return f"{move_name}\n{narration}" def respond_to_commentary_text(text: str) -> str: stripped = text.strip() narrator = NarratorAgent() try: data = json.loads(stripped) if isinstance(data, dict) and _COMBAT_KEYS.issubset(data.keys()): return format_commentary_reply(*narrator.narrate(CombatTelemetry(**data))) except (json.JSONDecodeError, ValueError, TypeError): pass lowered = stripped.lower() if any(word in lowered for word in ("start", "demo", "commentate", "narrate", "punch", "fight", "duel")): return format_commentary_reply(*narrator.narrate(CombatTelemetry(**demo_combat_payload()))) return ( "I am Commentator Agent.\n" "Say 'start commentary' to narrate a demo punch, or paste CombatTelemetry JSON " "and I will return a move name plus a real narration line." ) ``` ### backend/recap_queue.py ```python from __future__ import annotations import json import os from pathlib import Path from time import time from typing import Any from uuid import uuid4 from backend.pika_recap import submit_recap_to_pika _DEFAULT_QUEUE_PATH = Path(__file__).with_name("recap_jobs.jsonl") class RecapQueue: def __init__(self, path: str | os.PathLike[str] | None = None) -> None: self.path = Path(path or os.getenv("PIKA_RECAP_QUEUE", _DEFAULT_QUEUE_PATH)) def enqueue(self, prompt: str, metadata: dict[str, Any] | None = None) -> dict[str, Any]: self.path.parent.mkdir(parents=True, exist_ok=True) # Attempt live Pika video generation when the API key is present. pika_result = submit_recap_to_pika(prompt, metadata) pika_status = pika_result.get("status", "skipped") pika_job_id = pika_result.get("job_id", f"recap_{uuid4().hex[:12]}") local_status = "submitted" if pika_status == "submitted" else "queued" job = { "job_id": pika_job_id if pika_status == "submitted" else f"recap_{uuid4().hex[:12]}", "status": local_status, "provider": "pika", "prompt": prompt, "metadata": metadata or {}, "pika_result": pika_result, "created_at": time(), } with self.path.open("a", encoding="utf-8") as fh: fh.write(json.dumps(job) + "\n") return job def list_jobs(self, limit: int = 20) -> list[dict[str, Any]]: if not self.path.exists(): return [] lines = self.path.read_text(encoding="utf-8").splitlines() jobs = [json.loads(line) for line in lines if line.strip()] return jobs[-limit:] ``` ### backend/cv_check.py ```python """Standalone body CV smoke test — prints live body gestures from the webcam. Run from the repo root (venv active): python -m backend.cv_check Stand in front of your camera and try: • Extend right arm toward camera (jab) → RIGHT_PUNCH • Extend left arm toward camera → LEFT_PUNCH • Extend both arms simultaneously → HEAVY_PUNCH • Raise both hands above nose level → GUARD • Lean body to your right → WALK_RIGHT • Lean body to your left → WALK_LEFT Ctrl-C stops. [MOCK] means the camera or model couldn't be opened. Check System Settings > Privacy & Security > Camera for Terminal permissions. Tune sensitivity via env vars if needed: CV_PUNCH_DEPTH=0.22 metres wrist must extend past shoulder (default 0.22) CV_GUARD_PAD=0.04 wrist clearance above nose (default 0.04) CV_WALK_LEAN=0.10 hip-centre offset from 0.5 to trigger walk (default 0.10) """ from __future__ import annotations import asyncio from backend.vision_bridge import _CAMERA_INDEX, vision_bridge_stream async def main() -> None: print( f"[cv_check] camera {_CAMERA_INDEX} — full-body tracker\n" " jab toward camera = punch | raise hands = guard | lean = walk\n" ) n = 0 async for evt in vision_bridge_stream(): p = evt.payload n += 1 gesture = p["gesture"].upper().ljust(13) hip_x = p.get("bodyCenter", {}).get("x", 0.5) mock = " [MOCK]" if p.get("mock") else "" print(f"#{n:04d} {gesture} conf={p['confidence']:.2f} hip_x={hip_x:.2f}{mock}") if __name__ == "__main__": try: asyncio.run(main()) except KeyboardInterrupt: print("\n[cv_check] stopped") ``` [240 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]