# Project export: Norbel Arena This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: A Deterministic, Replayable Benchmark for Social Reasoning in Multi-Agent AI - Devpost: https://devpost.com/software/norbel-arena - GitHub: https://github.com/anshgandhi4/norbel-arena - Video: https://www.youtube.com/embed/IhoPVIGmsts?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Ansh Gandhi (3 commits) ## Devpost submission (written by the team) ### Inspiration Most AI demos are difficult to compare in any meaningful way. Different prompts, hidden system instructions, and cherry-picked outputs make it hard to tell whether a model actually performed better or just benefited from setup advantages. There is rarely replayable evidence and rarely a way to audit how a result was produced. We built Norbel Arena to make AI vs. AI and human vs. AI evaluation transparent, deterministic, and competitive. We focused on building infrastructure for fair benchmarking. We wanted something that strictly enforces rules, clearly exposes outcomes, and allows anyone to replay a match and verify what happened. We started with Codenames and then implemented Wavelength to demonstrate that this was not a one-off game implementation, but a reusable framework for evaluating social reasoning across multiple environments. What It Does Norbel Arena is a state-based multi-agent competition platform that runs complete matches autonomously and produces structured, replayable results. It enforces legal moves, validates model outputs against strict JSON schemas, handles malformed or invalid responses gracefully, and records turn-by-turn events for later inspection. Every match produces a winner, a termination reason, and detailed statistics, all of which can be replayed through our interface. We currently support Codenames and Wavelength, two games that stress different aspects of social reasoning. Codenames tests hidden information and role asymmetry, where a spymaster has access to a key that operatives do not. The system ensures that only the correct role sees private information and that all moves conform to the rules of the game. Wavelength introduces asymmetric information and multi-round estimation. A “psychic” agent communicates a clue about a hidden position on a spectrum, and a “guesser” agent attempts to infer that position. This stresses calibration, communication clarity, and probabilistic reasoning across multiple rounds. All matches are exposed through a FastAPI backend and a React frontend that support live play, replay controls, transcripts, leaderboards, and persistent report cards with role-aware Elo tracking. The platform supports AI vs. AI competitions as well as human participation. How We Built It Under the hood, Norbel Arena is built around typed, extensible abstractions including Game, State, Move, Observation, Agent, MatchRunner, and Arena. This structure allows us to add new games without rewriting core infrastructure. Deterministic seeded game creation ensures that matches are reproducible. Partial observability is enforced at the state level so that each role only sees what it is allowed to see. We designed strict JSON move contracts for LLM agents and implemented parsing and repair logic to handle imperfect model outputs without breaking game flow. The agent layer is provider-agnostic and supports OpenAI, Anthropic, Perplexity, local models, Nemotron variants, random agents, and human players. This flexibility allows side-by-side comparisons across providers under identical conditions. We also built persistent report cards that track role-specific Elo ratings, since performance can vary significantly depending on whether a model is acting as a clue giver, guesser, or estimator. The system includes robust failure handling for illegal moves, exceptions, and output validation errors. We validated the framework with a comprehensive test suite covering the engine, rules, API, provider integrations, and local model execution paths. Technical Complexity Although the user-facing experience is simple, the underlying system handles deterministic state transitions, strict schema enforcement, multi-provider LLM integration, replayable event logs, and role-aware ranking. Preventing hidden-information leakage while still giving agents enough context to reason correctly required careful design. Ensuring that LLM outputs conform to structured move schemas without constantly breaking gameplay required a layered validation and repair strategy. Supporting both hosted APIs and local models introduced practical runtime and dependency constraints that we had to resolve within a tight time frame. Designing evaluation modes that isolate model quality by role required rethinking traditional Elo approaches to account for asymmetric gameplay. Social Impact As AI systems become more integrated into education, negotiation, customer service, and collaborative decision-making, we need better ways to evaluate how they reason socially and strategically. Many real-world applications involve partial information, role asymmetry, and communication under uncertainty. Hidden-information games provide a compact and controllable way to simulate those dynamics. Norbel Arena provides infrastructure for transparent and reproducible benchmarking of these capabilities. Researchers can compare models fairly under identical conditions. Developers can identify failure modes in communication and coordination. Organizations can demand auditable evaluation before deploying multi-agent systems in sensitive contexts. By focusing on replayability, determinism, and structured evaluation, we aim to raise the standard for how collaborative AI systems are tested and compared. Accomplishments In 36 hours, we designed and implemented a general multi-agent arena framework, shipped two fully integrated social-reasoning games, and delivered an end-to-end product that includes the core engine, API server, and interactive frontend. We built deterministic replayability into the system from the start, implemented role-specific Elo tracking, and created a provider-agnostic agent stack capable of supporting both hosted and local models. The system is backed by a comprehensive test suite to ensure stability and reliability. What’s Next We plan to expand Norbel Arena with additional cooperative and adversarial games that stress different reasoning capabilities. We also want to build large-scale tournament tooling, richer leaderboard analytics, deeper replay diagnostics, and standardized benchmark suites for longitudinal cross-model comparison. Our long-term vision is to use Norbel Arena as infrastructure for safer, more accountable multi-agent AI systems that interact with humans in meaningful, high-stakes environments. ## README (from the GitHub repository) # Codenames Arena (Framework + UI) This repo contains: - A game framework for state-based games - A Codenames implementation with partial observability - A local FastAPI backend for configurable human/AI match sessions - A React frontend for setup, live play, replay, and report cards ## Run Backend ```bash .venv/bin/uvicorn server.main:app --reload --host 0.0.0.0 --port 8000 ``` API endpoints: - `POST /api/match/new` - `GET /api/match/{match_id}/observation?player_id=...` - `POST /api/match/{match_id}/move` - `GET /api/match/{match_id}/events` - `GET /api/report-cards` ## Run Frontend ```bash cd frontend npm install npm run dev ``` Frontend defaults to `http://localhost:8000` for API calls. Override with: ```bash VITE_API_BASE=http://localhost:8000 npm run dev ``` ## Play Flow 1. Open the frontend in your browser. 2. Configure each seat (`RED_SPYMASTER`, `RED_OPERATIVE`, `BLUE_SPYMASTER`, `BLUE_OPERATIVE`) as: - `human` - `random` - `openai` - `anthropic` - `perplexity` - `local` - `nemotron` 3. Choose a viewer perspective and create a match. 4. If your viewer seat is `human`, use controls to submit strict JSON-equivalent moves: - GiveClue: `{ "type": "GiveClue", "clue": "animal", "count": 2 }` - Guess: `{ "type": "Guess", "index": 13 }` - EndTurn: `{ "type": "EndTurn" }` 5. Use replay controls to move backward/forward through turns. 6. Inspect report cards for aggregate performance by agent label and game. ## Tests Run all tests: ```bash .venv/bin/pytest -q ``` Includes API smoke coverage (`tests/test_server_api_smoke.py`) for: - create match - fetch observation - submit legal move - fetch events - run all-AI match to terminal - retrieve persisted report cards ## Notes - Match sessions are stored in-memory and keyed by `match_id`. - Sessions also store immutable `state_history` snapshots so observations can be requested for prior turns (`turn` query param) for replay. - Observations are server-shaped per player role. Operative views do not receive hidden assignments. - Human players are optional. You can run full AI-vs-AI matches and watch/replay from any seat perspective. - Report cards persist to `server/data/report_cards.json`. - Override report-card path with `REPORT_CARD_DB_PATH=/path/to/report_cards.json`. - `.env` can contain API keys for LLM agents and is loaded by the agents utilities. - For `local`/`nemotron` with `backend="transformers"`, install local runtime deps (including `protobuf`, `sentencepiece`, and `tiktoken`) before running matches. - `nvidia/llama-3.1-nemotron-70b-instruct` is intended for served/OpenAI-compatible inference (`backend="openai_compat"`), while `nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1` is the default for in-process `backend="transformers"`. - `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8` may require `mamba-ssm`/`causal-conv1d` for in-process Transformers loading; on CPU-only environments this is commonly unavailable, so prefer `backend="openai_compat"` for Nemotron. ## Detected evidence (automated analysis) Indexed codebase: 67 recognized source files, 339 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (75 of 75) ``` .gitignore codenames/__init__.py codenames/codenames_game.py codenames/codenames_moves.py codenames/codenames_observation.py codenames/codenames_state.py framework/__init__.py framework/agents/__init__.py framework/agents/env_utils.py framework/agents/http_utils.py framework/agents/llm_agent.py framework/agents/provider_agents.py framework/agents/provider_clients.py framework/agents/random_agent.py framework/arena.py framework/errors.py framework/events.py framework/game.py framework/move.py framework/observation.py framework/player.py framework/result.py framework/runner.py framework/serialize.py framework/state.py frontend/index.html frontend/package.json frontend/src/App.tsx frontend/src/components/AppHeader.tsx frontend/src/components/Board.tsx frontend/src/components/ControlsPanel.tsx frontend/src/components/GameInfo.tsx frontend/src/components/MatchSetup.tsx frontend/src/components/MoveHistory.tsx frontend/src/components/ReplayControls.tsx frontend/src/components/ReportCardPanel.tsx frontend/src/components/WavelengthControls.tsx frontend/src/components/WavelengthTranscript.tsx frontend/src/LeaderboardPage.tsx frontend/src/lib/api.ts frontend/src/lib/format.ts frontend/src/lib/types.ts frontend/src/main.tsx frontend/src/styles.css frontend/src/vite-env.d.ts frontend/tsconfig.app.json frontend/tsconfig.app.tsbuildinfo frontend/tsconfig.json frontend/tsconfig.node.json frontend/tsconfig.node.tsbuildinfo frontend/vite.config.d.ts frontend/vite.config.js frontend/vite.config.ts README.md requirements.txt server/__init__.py server/agent_factory.py server/main.py server/report_cards.py server/schemas.py server/session.py tests/conftest.py tests/test_codenames_rules.py tests/test_llm_agent.py tests/test_llm_provider_clients.py tests/test_local_model_smoke.py tests/test_report_cards.py tests/test_runner_smoke.py tests/test_server_api_smoke.py tests/test_wavelength_rules.py wavelength/__init__.py wavelength/wavelength_game.py wavelength/wavelength_moves.py wavelength/wavelength_observation.py wavelength/wavelength_state.py ``` ### Dependencies - frontend/package.json: @types/react@^18.3.18, @types/react-dom@^18.3.5, @vitejs/plugin-react@^4.3.4, react@^18.3.1, react-dom@^18.3.1, typescript@^5.7.2, vite@^5.4.11 - requirements.txt: fastapi@>=0.129.0, httpx@>=0.28.0, protobuf@>=5.0.0, pytest@>=9.0.0, sentencepiece@>=0.2.0, tiktoken@>=0.9.0, uvicorn@>=0.40.0 ### Recent commits (newest first) - fein - update readme - ch0nky ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` fastapi>=0.129.0 uvicorn>=0.40.0 httpx>=0.28.0 pytest>=9.0.0 protobuf>=5.0.0 sentencepiece>=0.2.0 tiktoken>=0.9.0 ``` ### frontend/package.json ``` { "name": "codenames-frontend", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "preview": "vite preview" }, "dependencies": { "react": "^18.3.1", "react-dom": "^18.3.1" }, "devDependencies": { "@types/react": "^18.3.18", "@types/react-dom": "^18.3.5", "@vitejs/plugin-react": "^4.3.4", "typescript": "^5.7.2", "vite": "^5.4.11" } } ``` ### server/main.py ```python """FastAPI server exposing a local match API for human and/or AI play.""" from __future__ import annotations import time from typing import Any from fastapi import FastAPI, HTTPException, Query from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import PlainTextResponse from framework.serialize import json_dumps from server.schemas import CreateMatchRequest, SubmitMoveRequest from server.session import SessionStore app = FastAPI(title="State Games Local API", version="0.1.0") store = SessionStore() app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:5173", "http://127.0.0.1:5173"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) @app.get("/api/health") def health() -> dict[str, str]: """Healthcheck endpoint.""" return {"status": "ok"} def _time_based_seed() -> int: """Generate a positive time-derived seed when client does not provide one.""" seed = int(time.time_ns() & 0x7FFFFFFF) return seed if seed != 0 else 1 @app.post("/api/match/new") def new_match(request: CreateMatchRequest) -> dict: """Create a new in-memory match session.""" seed = request.seed if request.seed is not None else _time_based_seed() try: session = store.create_match( game=request.game, seed=seed, config=request.config, players=request.players, human_player_id=request.human_player_id, viewer_player_id=request.viewer_player_id, ) except Exception as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc player_id = session.selected_player(request.viewer_player_id or request.human_player_id) return session.view(player_id) @app.get("/api/match/{match_id}/observation") def get_observation( match_id: str, player_id: str = Query(...), turn: int | None = Query(default=None, ge=0), ) -> dict: """Get latest observation and legal moves for one player.""" try: session = store.get(match_id) return session.view(player_id, turn=turn) except KeyError as exc: raise HTTPException(status_code=404, detail=f"Unknown match_id: {match_id}") from exc except Exception as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc @app.post("/api/match/{match_id}/move") def submit_move(match_id: str, request: SubmitMoveRequest) -> dict: """Submit a strict JSON move for a human player and advance session.""" try: session = store.get(match_id) except KeyError as exc: raise HTTPException(status_code=404, detail=f"Unknown match_id: {match_id}") from exc try: return session.submit_human_move(player_id=request.player_id, move_payload=request.move) except PermissionError as exc: raise HTTPException(status_code=403, detail=str(exc)) from exc except ValueError as exc: # Include refreshed state for convenient UI recovery. payload = session.view(request.player_id) payload["error"] = str(exc) raise HTTPException(status_code=400, detail=payload) from exc except Exception as exc: raise HTTPException(status_code=500, detail=str(exc)) from exc @app.get("/api/match/{match_id}/events", response_model=None) def get_events(match_id: str, format: str = Query(default="array")) -> Any: """Return full event history as array (default) or JSONL text.""" try: events = store.all_events(match_id) except KeyError as exc: raise HTTPException(status_code=404, detail=f"Unknown match_id: {match_id}") from exc if format == "jsonl": text = "\n".join(json_dumps(event) for event in events) return PlainTextResponse(content=text, media_type="application/jsonl") return events @app.get("/api/report-cards") def get_report_cards() -> dict[str, Any]: """Return persisted report cards for all games and agents.""" return store.report_cards() if __name__ == "__main__": import uvicorn uvicorn.run("server.main:app", host="0.0.0.0", port=8000, reload=True) ``` ### frontend/src/main.tsx ```typescript import React from 'react' import ReactDOM from 'react-dom/client' import App from './App' import LeaderboardPage from './LeaderboardPage' import './styles.css' function normalizePath(pathname: string): string { if (pathname.length > 1 && pathname.endsWith('/')) { return pathname.slice(0, -1) } return pathname } const path = normalizePath(window.location.pathname) const RootComponent = path === '/leaderboard' ? LeaderboardPage : App ReactDOM.createRoot(document.getElementById('root')!).render( ) ``` ### frontend/src/App.tsx ```typescript import { useEffect, useMemo, useState } from 'react' import Board from './components/Board' import ControlsPanel from './components/ControlsPanel' import GameInfo from './components/GameInfo' import MatchSetup from './components/MatchSetup' import type { SetupSelections } from './components/MatchSetup' import MoveHistory from './components/MoveHistory' import ReplayControls from './components/ReplayControls' import AppHeader, { type Theme } from './components/AppHeader' import WavelengthControls from './components/WavelengthControls' import WavelengthTranscript from './components/WavelengthTranscript' import { createMatch, fetchEvents, fetchObservation, submitMove } from './lib/api' import type { CodenamesObservation, EvaluationMode, GameName, MatchEvent, MatchView, Move, PlayerConfig, WavelengthObservation } from './lib/types' const THEME_STORAGE_KEY = 'treehacks-ui-theme' function resolveInitialTheme(): Theme { if (typeof window === 'undefined') { return 'dark' } const storedTheme = window.localStorage.getItem(THEME_STORAGE_KEY) return storedTheme === 'light' ? 'light' : 'dark' } export default function App() { const [view, setView] = useState(null) const [events, setEvents] = useState([]) const [loading, setLoading] = useState(false) const [error, setError] = useState(null) const [theme, setTheme] = useState(() => resolveInitialTheme()) const [lastSetupSelections, setLastSetupSelections] = useState(null) const matchId = view?.match_id const playerId = view?.player_id useEffect(() => { document.documentElement.setAttribute('data-theme', theme) document.documentElement.style.colorScheme = theme window.localStorage.setItem(THEME_STORAGE_KEY, theme) }, [theme]) useEffect(() => { if (!matchId || !playerId || !view?.meta.is_live) { return } let cancelled = false const poll = async () => { if (!view || view.meta.terminal || view.meta.is_human_turn || !view.meta.is_live) { return } try { const latest = await fetchObservation(matchId, playerId) if (!cancelled) { setView(latest) const ev = await fetchEvents(matchId) if (!cancelled) { setEvents(ev) } } } catch (err) { if (!cancelled) { setError(err instanceof Error ? err.message : String(err)) } } } const timer = window.setInterval(poll, 900) return () => { cancelled = true window.clearInterval(timer) } }, [matchId, playerId, view]) const onCreateMatch = async (params: { game: GameName startingTeam?: 'RED' | 'BLUE' | 'RANDOM' evaluationMode: EvaluationMode players: Record viewerPlayerId: string }) => { setLoading(true) setError(null) setLastSetupSelections({ game: params.game, startingTeam: params.startingTeam, evaluationMode: params.evaluationMode, players: Object.fromEntries( Object.entries(params.players).map(([playerId, config]) => [playerId, { ...config }]) ), viewerPlayerId: params.viewerPlayerId }) try { const config: Record = { evaluation_mode: params.evaluationMode } if (params.game === 'codenames' && params.startingTeam && params.startingTeam !== 'RANDOM') { config.starting_team = params.startingTeam } const humanPlayers = Object.entries(params.players) .filter(([, playerConfig]) => playerConfig.type === 'human') .map(([playerIdValue]) => playerIdValue) const created = await createMatch({ game: params.game, config, players: params.players, human_player_id: humanPlayers.length === 1 ? humanPlayers[0] : undefined, viewer_player_id: params.viewerPlayerId }) setView(created) const ev = await fetchEvents(created.match_id) setEvents(ev) } catch (err) { setError(err instanceof Error ? err.message : String(err)) } finally { setLoading(false) } } const refreshView = async (targetTurn?: number) => { if (!view) { return } const refreshed = await fetchObservation(view.match_id, view.player_id, targetTurn) setView(refreshed) } const submit = async (move: Move) => { if (!view) { return } setLoading(true) setError(null) try { const updated = await submitMove(view.match_id, view.player_id, move) setView(updated) const ev = await fetchEvents(updated.match_id) setEvents(ev) } catch (err) { setError(err instanceof Error ? err.message : String(err)) try { await refreshView() } catch { // no-op: original error is enough } } finally { setLoading(false) } } const goToTurn = async (turn: number) => { if (!view) { return } setLoading(true) setError(null) try { await refreshView(turn) } catch (err) { setError(err instanceof Error ? err.message : String(err)) } finally { setLoading(false) } } const goLive = async () => { if (!view) { return } setLoading(true) setError(null) try { await refreshView() } catch (err) { setError(err instanceof Error ? err.message : String(err)) } finally { setLoading(false) } } const canRenderGame = useMemo(() => !!view, [view]) const canAct = !!view && view.meta.is_human_turn && view.meta.is_live && !view.meta.terminal && !loading const toggleTheme = () => setTheme((prev) => (prev === 'dark' ? 'light' : 'dark')) if (!canRenderGame || !view) { return (