# Project export: RLX-ray This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Observability for RL Environments. A tool that helps you measure the quality of your RL environments and quickly understand agent trajectories. - Devpost: https://devpost.com/software/rlx-ray - GitHub: https://github.com/MittelmanDaniel/agent-observability - Demo: https://agent-observability.vercel.app/ - Video: https://www.youtube.com/embed/goQdR0KWj8w?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Daniel Mittelman (13 commits), Cursor (12 commits) ## Devpost submission (written by the team) ### Inspiration I cofounded Appacella, an AI mobile app developer. While building that, I manually read hundreds of AI coding traces to find issues and improve the agent. Now I work on RL environments and hit the same wall: I need to read hundreds of transcripts of the model attempting a task. I can't trust the model to read the whole thing itself so I wanted an LLM to help me read transcripts quickly, plus a tool to group and keep track of model runs. That's how the idea for this project started. ### What it does RLX-ray is a dashboard for agent run observability. You ingest RL runs from Hugging Face datasets or by triggering SWE-agent on Modal. SWE-agent is the standard harness for SWE-bench. Then each run gets analyzed by an LLM agent that segments the trajectory into sections (good / warning / failure) and summarizes what went wrong. You can search over massive transcripts with Elasticsearch, compare runs within a task with embedding-backed similarity, and see runs with similar failure modes so you don't have to read hundreds of traces by hand. The goal is to better understand how the agent interacts with the environment (where it gets stuck, what it tries, where it goes wrong) without reading every transcript yourself. How I built it Frontend: Next.js. It's a dashboard for displaying runs and sections; server-side rendering works well for that. Storage & search: Elasticsearch. I needed to store runs and search over massive transcripts; Elasticsearch made that tractable. Comparing runs: I wanted embeddings to compare runs within a task. I use Jina (via Elastic) for embeddings and keep everything in Elastic, so I went with Elastic Agent Builder for the trajectory-analyzer agent. It integrates cleanly with the same DB and lets the agent query via Elasticsearch instead of stuffing the whole transcript into context (transcripts can be near the context limit or larger with autocompaction). User-supplied runs: I ingest runs from Hugging Face, but the product only really works if users can run their own. So I use Modal's sandbox to run the SWE-agent harness on a SWE-bench task: the app triggers a Modal endpoint that runs SWE-agent and writes results back to Elasticsearch. Analysis design: I didn't want to dump the entire transcript into the model. I use Elasticsearch so the agent can query the run (e.g. by event type or step) and do sectioning without blowing context limits. Challenges I ran into Section splits: Getting the LLM to produce good section boundaries instead of splitting arbitrarily on errors was hard. I had to tune prompts and tooling so sections reflect meaningful phases (e.g. "got stuck here," "repeated attempt") rather than every failure. Grouping similar runs: With embeddings, runs that feel similar (e.g. "error + loop") should cluster together. But if the error text differs, embeddings put them in different buckets. I needed a pseudo re-ranking approach (not classic search re-ranking) to surface runs that are behaviorally similar even when the surface text differs. Modal + SWE-agent: SWE-agent was rough to wire up for a few reasons. It isn't a simple pip-and-run library: it expects a full repo layout (config/, tools/, trajectories/), so I had to build a Modal image that clones the repo and does an editable install, then runs the CLI. SWE-agent also uses Modal internally for code execution (via swe-rex), so I ended up with Modal-in-Modal: my orchestrator runs on Modal and the CLI spawns its own Modal sandboxes for each run. I hit API compatibility issues too: SWE-agent uses litellm, and some models (e.g. GPT-5, Claude) error on unsupported params like top_p, so I added a custom entrypoint that sets litellm.drop_params=True before invoking the real runner. On top of that, I had to discover and parse the trajectory output (.traj files), normalize "history" vs "trajectory" (schema varies by version), and write events plus run status back to Elasticsearch. Getting the whole chain reliable took a lot of iteration. Accomplishments that I'm proud of End-to-end pipeline: From raw runs (Hugging Face or live Modal) to searchable, sectioned trajectories and "similar runs" in one dashboard. LLM that queries instead of ingesting: The trajectory-analyzer agent uses Elasticsearch as its source of truth and queries by step/event type instead of stuffing full transcripts into context, so I can handle long runs without blowing context limits. Sectioning that surfaces mild failures: Sections don't just split on hard errors; they highlight "got stuck," "repeated attempt," and partial failures so you see where a run went off the rails, not just pass/fail. Custom similar-run grouping: I built a pseudo re-ranking approach so runs that are behaviorally similar (e.g. same failure mode) cluster together even when the literal error text differs, beyond off-the-shelf embedding similarity. Modal + SWE-agent integration: Getting the app → Modal → swe-rex sandbox chain working so users can launch and track SWE-agent runs from the UI. What I learned Formats: LLM trajectories are stored in many formats; normalizing and ingesting them is a real integration challenge. Elasticsearch: I learned Elasticsearch properly: indices, k-NN, and wiring an agent to query it instead of relying on raw context. Beyond re-ranking: I built a custom approach to group "similar" runs when standard embedding similarity wasn't enough (different error text, same failure mode). Product: Focus on value. The core value is analyzing trajectories: sections, similar runs, "where did it go wrong." The Modal "run SWE-agent from the UI" flow is cool but secondary; the dashboard and analysis are what actually help when reading hundreds of runs. ### What's next Environment versioning: Track versions of environments so you can compare runs across env changes and know exactly what code/config a run used. Custom RL environments: Let people upload their own RL environments (e.g. via Docker plus a standardized interface) and run tasks against them, instead of being limited to the tasks already in the system. More trajectory formats: Support additional agent/RL run formats so more teams can ingest their runs. Smarter similar-run logic: Improve the pseudo re-ranking so "same failure mode, different text" clusters even better. Built with TypeScript, React, Next.js, Python, Elasticsearch, Kibana, Elastic Agent Builder, Jina, Hugging Face, Modal, Vercel, SWE-agent, swe-rex, Tailwind CSS, uv Languages & frameworks TypeScript, React 19, Next.js 16 Python (workers, scripts) Infrastructure & data Elasticsearch (Elastic Cloud): store runs, events, sections; full-text and k-NN search Kibana / Elastic Agent Builder: trajectory-analyzer agent (converse API, tools, query over ES) Jina: section and run-summary embeddings (768-dim), similarity for "find similar runs/sections" Hugging Face: dataset ingestion for runs Modal: run SWE-agent in sandboxes (swe-rex); app triggers Modal endpoint, worker writes to Elasticsearch Vercel: host the Next.js app APIs & tooling Elasticsearch API, Kibana Agent Builder Converse API Jina Embeddings API SWE-agent / swe-rex (execution harness) Other Tailwind CSS, react-markdown uv (Python envs and scripts) ## README (from the GitHub repository) This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app). ## Environment (.env.local) | Variable | Purpose | |----------|---------| | `ELASTICSEARCH_URL` | Elasticsearch cluster for index/search (runs, events, sections). Example: `https://….es.us-west1.gcp.elastic.cloud:443` | | `ELASTICSEARCH_API_KEY` | API key for Elasticsearch (and for Kibana Agent Builder API). | | `KIBANA_URL` | Kibana instance **only** used for the Agent Builder REST API (converse, tools, agents). Not used for dashboards or inference. Example: `https://….kb.us-west1.gcp.elastic.cloud:443` | | `ELASTIC_AGENT_CONNECTOR_ID` | Optional. Connector ID for the LLM used by the trajectory-analyzer agent (converse API). If unset, Kibana uses its **default** connector (often set in Stack Management → Connectors or Agent Builder / GenAI settings). Set this to pin the run analyzer to a specific model. | | `JINA_API_KEY` | **Required** for run analysis. Section embeddings use [Jina AI’s API](https://jina.ai/embeddings) (`https://api.jina.ai/v1/embeddings`). If missing, analysis fails with a clear error. Get a key at [jina.ai](https://jina.ai/?sui=apikey). | To list Kibana connectors (for `ELASTIC_AGENT_CONNECTOR_ID`): `npx tsx scripts/list-connectors.ts` ## Getting Started First, run the development server: ```bash npm run dev # or yarn dev # or pnpm dev # or bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file. This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel. ## Learn More To learn more about Next.js, take a look at the following resources: - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome! ## Deploy on Vercel The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details. ## Detected evidence (automated analysis) Indexed codebase: 50 recognized source files, 201 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Hugging Face (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (57 of 57) ``` .cursor/rules/python-uv.mdc .gitignore app/api/clusters/route.ts app/api/runs/[id]/analyze/route.ts app/api/runs/[id]/route.ts app/api/runs/[id]/similar/route.ts app/api/runs/route.ts app/api/sections/[id]/similar/route.ts app/api/swe-agent/run/route.ts app/globals.css app/layout.tsx app/page.tsx app/projects/[source]/page.tsx app/projects/[source]/tasks/[task]/new-run-button.tsx app/projects/[source]/tasks/[task]/page.tsx app/projects/[source]/tasks/[task]/runs-table.tsx app/runs/[id]/analyze-button.tsx app/runs/[id]/message-list.tsx app/runs/[id]/page.tsx app/runs/[id]/run-viewer.tsx app/runs/[id]/timeline.tsx app/tasks/[task]/page.tsx docs/DEMO_NOTES.md docs/IMPLEMENTATION.md docs/PROJECT_PLAN.md docs/TRAJECTORY_DATASETS.md eslint.config.mjs lib/analyze.ts lib/db.ts lib/github.ts lib/projects.ts lib/store.ts lib/types.ts next.config.ts package.json postcss.config.mjs README.md scripts/analyze-task.ts scripts/bulk-update-runs.py scripts/check-reproduction.ts scripts/list-connectors.ts scripts/list-tasks-by-length.ts scripts/list-tasks-by-run-count.ts scripts/load_trace.py scripts/load_trajectories.py scripts/load-env.ts scripts/mark-running-failed.ts scripts/migrate.ts scripts/requirements.txt scripts/seed-db.ts scripts/seed-trace.ts scripts/setup-agent.ts scripts/show-sections.ts scripts/similarity-distribution.ts tsconfig.json workers/run_with_drop_params.py workers/swe_agent_runner.py ``` ### Dependencies - package.json: @elastic/elasticsearch@^9.3.1, @tailwindcss/postcss@^4, @tailwindcss/typography@^0.5.19, @types/node@^20, @types/react@^19, @types/react-dom@^19, dotenv@^17.3.1, eslint@^9, eslint-config-next@16.1.6, next@16.1.6, react@19.2.3, react-dom@19.2.3, react-markdown@^10.1.0, tailwindcss@^4, tsx@^4.21.0, typescript@^5 - scripts/requirements.txt: datasets@>=2.14.0, huggingface_hub@>=0.20.0, requests@>=2.28.0 ### Recent commits (newest first) - Run viewer: Context/Observation labels, first system prompt styling, action/thought extraction - fix: ES mapping types for Vercel build (as const) - SWE-agent on Modal: run button, model picker, worker with clone+editable install, swe-rex[modal], GITHUB_TOKEN; mark-running-failed script; running status badge - Cluster summaries + merge by same summary (reranker-style) - Clustering threshold 0.95, outcome in embedding, Find similar runs, scripts - Run clustering, Jina embeddings, Agent Builder analyzer, scripts - Add GitHub PR link for SWE-bench tasks on task page - Integrate Elastic Agent Builder for trajectory analysis - LLM-driven section boundaries, multi-model data, compact table UI - Redesign run detail page with section-based timeline UI - Switch to Elasticsearch + add run analysis pipeline - Agent observability platform - Phase 1 & 2 - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/DEMO_NOTES.md ```markdown # Demo notes ## Sections: mild error vs full error **Why we like sections:** The trajectory analyzer surfaces **mild / partial failures** (e.g. “got stuck here”, “repeated attempt”, “import/typo”) in addition to hard failures. So you see *where* a run went off the rails, not just “succeeded” or “failed”. **Example run (sections find a mild error, not a full crash):** https://agent-observability-dldc9626d-mittelmandaniels-projects.vercel.app/runs/nebius-MicroPyramid__forex-python-27-357?source=custom Use this in the demo to show that sections add interpretability beyond the final status. --- ## Modal: track run progress **Sandboxes** (where swe-rex runs agent commands): https://modal.com/apps/mittelmandaniel/main/deployed/swe-rex?activeTab=sandboxes&live=true **SWE-agent runner app** (orchestrator that runs `sweagent run` and writes to ES): https://modal.com/apps/mittelmandaniel/main/deployed/swe-agent-runner Use these to watch live logs and sandbox activity while a run is in progress. ``` ### docs/TRAJECTORY_DATASETS.md ```markdown # Agent Trajectory Datasets for Demo/Test This file tracks practical datasets we can use to demo sectioned trajectory diagnosis. ## Selection criteria (how these judgments were made) We prioritized datasets that are: 1. **Actual agent traces** (not just final answers), 2. **Long-horizon / multi-step** enough for sectioning, 3. **Failure-rich** so diagnostics are meaningful, 4. **Easy to access quickly** during hackathon development. --- ## 1) TRAIL (PatronusAI) - best for quality diagnostics - Dataset: `PatronusAI/TRAIL` (Hugging Face) - Paper: "TRAIL: Trace Reasoning and Agentic Issue Localization" (arXiv:2505.08638) - Repo: `patronus-ai/trail-benchmark` ### What it is Human-annotated benchmark for trace debugging: - **148** annotated agent traces - **841** labeled errors - Error taxonomy across reasoning, execution, and planning - Built from GAIA and SWE-Bench style tasks ### Run counts - **Total traces (runs): 148** - Breakdown from dataset card: **118 GAIA + 30 SWE-Bench** - Additional structure detail from card: 1,987 spans total, 575 spans with at least one error ### Notes - Strongest dataset for validating your section labels (`verdict`, `root_cause_guess`, `fix_suggestion`) - Access is gated on Hugging Face (you must accept terms) --- ## 2) Nebius SWE-agent trajectories - best for scale - Dataset: `nebius/SWE-agent-trajectories` ### What it is Large corpus of SWE-agent-style coding traces with reasoning/actions/observations and patch/eval metadata. ### Run counts - **Total trajectories (runs): 80,036** - Dataset card split by issue outcome: - Resolved: 13,389 - Not resolved: 66,647 ### Notes - Great for cross-run analytics ("top repeated failure sections") - Larger/noisier than TRAIL, but excellent for stress testing --- ## 3) SWE-smith trajectories - large and strong coding baseline - Dataset: `SWE-bench/SWE-smith-trajectories` ### What it is Trajectory dataset generated from SWE-agent + Claude 3.7 Sonnet over SWE-smith tasks. ### Run counts - Dataset description says: **5,017 trajectories used for fine-tuning** - Hugging Face row count shows: **76,002 rows** ### Notes on count interpretation For this dataset, "rows" may not equal one full run depending on serialization format. Plan to inspect one sample before assuming "76,002 independent full runs." --- ## Suggested usage in our demo 1. Use **TRAIL** for "gold" examples in the live demo. 2. Use **Nebius SWE-agent trajectories** for "many runs" analytics views. 3. Add **SWE-smith** if we need extra volume or Claude-oriented coding traces. --- ## Possible trajectory types to support in our schema We should normalize all sources into one internal schema (`run`, `events`, `sections`) and support: - `coding_agent_run` (SWE-agent traces) - `research_agent_run` (GAIA-like open-web retrieval traces) - `conversation_agent_run` (assistant turn/tool traces) - `browser_agent_run` (Stagehand/browser automation traces) - `multi_agent_run` (handoff/delegati [truncated — 183 more characters] ``` ### package.json ``` { "name": "agent-observability", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@elastic/elasticsearch": "^9.3.1", "@tailwindcss/typography": "^0.5.19", "dotenv": "^17.3.1", "next": "16.1.6", "react": "19.2.3", "react-dom": "19.2.3", "react-markdown": "^10.1.0", "tsx": "^4.21.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### scripts/requirements.txt ``` # Load Hugging Face trajectory datasets and normalize to our schema. datasets>=2.14.0 huggingface_hub>=0.20.0 requests>=2.28.0 # SWE-agent runner (deployed separately via: modal deploy workers/swe_agent_runner.py) # sweagent, elasticsearch are installed in the Modal image, not locally ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "Agent Observability", description: "Sectioned trajectory diagnosis for multi-step agents", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### app/page.tsx ```typescript import Link from "next/link"; import { listProjects } from "@/lib/store"; import { getProjectMeta } from "@/lib/projects"; export default async function Home() { const projects = await listProjects(); return (

Agent Observability

Select a project to explore agent trajectories.

{projects.length === 0 ? (
No projects yet. Seed some data to get started.
) : (
{projects.map((project) => { const meta = getProjectMeta(project.source); return ( {/* Icon + name */}
{meta.icon}

{meta.displayName}

{/* Description */}

{meta.description}

{/* Stats row */}
{project.tasks} tasks {project.runs} runs {project.succeeded > 0 && ( {project.succeeded} succeeded )} {project.failed > 0 && ( {project.failed} failed )} {project.reward_hacking > 0 && ( {project.reward_hacking} reward hacking )}
{/* Arrow */} → ); })}
)}
); } ``` ### app/api/clusters/route.ts ```typescript import { getRunClusters } from "@/lib/analyze"; import { NextResponse } from "next/server"; export async function GET(request: Request) { const { searchParams } = new URL(request.url); const task = searchParams.get("task") ?? undefined; const source = searchParams.get("source") ?? undefined; const threshold = searchParams.get("threshold"); const similarityThreshold = threshold ? Number(threshold) : undefined; try { const clusters = await getRunClusters({ task, source, similarityThreshold, }); return NextResponse.json({ clusters }); } catch (err) { console.error("Clusters failed:", err); return NextResponse.json( { error: "Clusters failed", detail: String(err) }, { status: 500 } ); } } ``` ### app/api/runs/route.ts ```typescript import { createRun, listRuns } from "@/lib/store"; import type { EventType, Run } from "@/lib/types"; import { NextResponse } from "next/server"; const EVENT_TYPES: EventType[] = [ "llm_call", "tool_call", "tool_result", "thought", "error", "user_feedback", ]; function toEventType(s: string): EventType { return EVENT_TYPES.includes(s as EventType) ? (s as EventType) : "thought"; } export async function GET() { const runs = await listRuns(); return NextResponse.json(runs); } export async function POST(request: Request) { const body = await request.json(); const { task, source = "custom", events = [] } = body as { task?: string; source?: Run["source"]; events?: Array<{ idx: number; ts: string; type: string; actor: string; content: string; metadata?: Record; }>; }; const id = crypto.randomUUID(); const now = new Date().toISOString(); const run: Run = { id, source: source ?? "custom", task: task ?? "Untitled", status: "completed", started_at: now, ended_at: now, }; const eventList = Array.isArray(events) ? events.map((e) => ({ idx: e.idx, ts: e.ts ?? now, type: toEventType(e.type ?? "thought"), actor: e.actor ?? "agent", content: e.content ?? "", metadata: e.metadata, })) : []; await createRun(run, eventList); return NextResponse.json({ id, run }); } ``` ### app/tasks/[task]/page.tsx ```typescript import Link from "next/link"; import { notFound } from "next/navigation"; import { getRunsByTask } from "@/lib/store"; export default async function TaskPage({ params, }: { params: Promise<{ task: string }>; }) { const { task: rawTask } = await params; const task = decodeURIComponent(rawTask); const runs = await getRunsByTask(task); if (runs.length === 0) notFound(); return (
← Tasks

{task}

{runs.length} runs for this task

    {runs.map((run) => (
  • {run.id} {run.status}

    {run.source} · {run.started_at}

    • ))}
); } ``` ### app/runs/[id]/page.tsx ```typescript import Link from "next/link"; import { notFound } from "next/navigation"; import { getEvents, getRun, getSections } from "@/lib/store"; import { getProjectMeta } from "@/lib/projects"; import { findSimilarRuns } from "@/lib/analyze"; import { RunViewer } from "./run-viewer"; export default async function RunPage({ params, searchParams, }: { params: Promise<{ id: string }>; searchParams: Promise<{ source?: string }>; }) { const { id } = await params; const { source: sourceParam } = await searchParams; const run = await getRun(id); if (!run) notFound(); const [events, sections, similarRuns] = await Promise.all([ getEvents(id), getSections(id), findSimilarRuns(id, 8), ]); const source = sourceParam ?? run.source; const meta = getProjectMeta(source); return (
{/* Breadcrumbs */}

{run.task}

{meta.displayName} {run.model_name && <> · {run.model_name}} {" · "}{run.status} {run.exit_status && <> · {run.exit_status}}

{run.source === "trace" && run.trace_label_description && (
TRACE Label {run.trace_label_description}
)}

Find similar runs

{similarRuns.length > 0 ? ( <>

Runs with similar trajectory summaries (Jina embedding, same task).

    {similarRuns.map((r) => (
  • {r.status} {r.model_name && ( {r.model_name} )} {r.id}
    • ))}
) : (

Analyze this run (Sections block below) to compute its embedding; similar runs will appear here.

)}
); } ``` [41 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]