# Project export: Omni QA This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Automated bug detection and resolution with natural language - Devpost: https://devpost.com/software/omni-qa - GitHub: https://github.com/KevinWu098/omni - Team: 4 GitHub contributor(s) — uno-p-5 (55 commits), Jay Wu (48 commits), Kevin Wu (32 commits), kylebtran (23 commits) ## Devpost submission (written by the team) ### Inspiration With artificial intelligence, software is now being generated and deployed faster than ever. Entire applications are built in hours, not weeks. But here’s the catch: the code works… until it doesn’t. AI-generated code is particularly error-prone and buggy. For the startup that just went viral on social media, or the major corporation rolling out a critical feature—one crash, one broken flow, one bad user experience can mean lost customers, lost trust, and millions in lost revenue. Traditional quality assurance isn’t keeping up with the pace of AI-enabled software development. For users and developers alike, this represents a critical problem in our ecosystem. That's why we built Omni, natural language-based testing that proactively identifies and resolves issues before they reach your users. ### What it does Omni modernizes QA with a fleet of AI agents that test like real users. Using natural language-based goals, our agents don’t just follow scripts—they explore your app with context, memory, and intent. They react to UI changes, latency, and feedback in real time, surfacing bugs that traditional tests miss. No more brittle test suites. Omni’s agents don’t depend on specific DOM selectors or content. They intelligently navigate websites even as the underlying codebase changes around them. We capture the full story when something breaks. Screenshots, DOM diffs, network logs, user intent—everything developers need to understand and fix bugs fast. Then we go a step further: Omni automatically generates context-aware pull requests in your repository to proactively catch and resolve bugs. QA built for the AI era. While AI accelerates code creation, Omni ensures that what ships actually works. No more tradeoff between speed and quality. Omni finds what others miss—so your users never end up as your QA testers. ### How we built it Frontend: Next.js Tailwind CSS WebRTC Framer Motion Backend: AWS EC2 FastAPI FFMPEG Browser Use Playwright Fetch AI Our frontend was built using Next.js and Tailwind. On the backend, we have a FastAPI server operating as an orchestration layer between our frontend (client) and our fleet of QA agents. We then deploy browser use agents on an EC2 instance. Using natural language tests on the frontend, the QA agents navigate websites to identify bugs. It streams realtime video and events back to the client. If the agent detects an issue, it uses FetchAI and the Github API to create and solve PRs using context from the test cases. ### Challenges we ran into Managing many concurrent agent sessions was difficult API rate limits, browser rate limits kept dragging us down ### Accomplishments we're proud of Built a fully-featured testing suite Built an end-to-end testing pipeline that can identify AND solve bugs ### What we learned Working with browser-use APIs Streaming concurrent video footage streams ### What's next for Omni Higher concurrent agent capacity Stronger corroboration amongst agent journeys Consistent deployment for exhaustive coverage ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 69 recognized source files, 547 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (78 of 78) ``` .gitignore client/.gitignore client/components.json client/eslint.config.mjs client/next.config.ts client/package.json client/postcss.config.mjs client/prettier.config.cjs client/README.md client/src/app/(layout)/layout.tsx client/src/app/(layout)/page.tsx client/src/app/actions.ts client/src/app/api/auth/[...nextauth]/route.ts client/src/app/globals.css client/src/app/home/page.tsx client/src/app/landing/page.tsx client/src/app/layout.tsx client/src/app/playground/page.tsx client/src/components/auth-provider.tsx client/src/components/big-text.tsx client/src/components/builder/all-side-panel.tsx client/src/components/builder/builder-sidebar.tsx client/src/components/builder/single/single-side-panel.tsx client/src/components/builder/single/step.tsx client/src/components/builder/test-card.tsx client/src/components/builder/viewer.tsx client/src/components/chip.tsx client/src/components/comment-card.tsx client/src/components/content.tsx client/src/components/create-pr-form.tsx client/src/components/login-button.tsx client/src/components/MaskTextSplit.tsx client/src/components/medium-text.tsx client/src/components/prs-list.tsx client/src/components/repos-list.tsx client/src/components/runner/agent-card.tsx client/src/components/runner/claude-loader.tsx client/src/components/runner/runner-sidebar.tsx client/src/components/runner/runner-viewer.tsx client/src/components/runner/timeline/editor.tsx client/src/components/runner/timeline/timeline.tsx client/src/components/runner/timeline/tracks.tsx client/src/components/runner/timeline/types.ts client/src/components/sidebar-wrapper.tsx client/src/components/timeline.tsx client/src/components/ui/badge.tsx client/src/components/ui/button.tsx client/src/components/ui/collapsible.tsx client/src/components/ui/input.tsx client/src/components/ui/resizable.tsx client/src/components/ui/scroll-area.tsx client/src/components/ui/select.tsx client/src/components/ui/separator.tsx client/src/components/ui/sonner.tsx client/src/components/ui/tabs.tsx client/src/components/ui/video-bad.tsx client/src/components/ui/video.tsx client/src/lib/auth.ts client/src/lib/github.ts client/src/lib/globals.ts client/src/lib/hooks/use-command-stream.ts client/src/lib/utils.ts client/src/types/next-auth.d.ts client/tailwind.config.ts client/tsconfig.json server/agent/.gitignore server/agent/app.py server/agent/app2.py server/agent/edit_agent.py server/agent/fetch.py server/agent/git.py server/agent/service.py server/server/app.py server/server/README.md server/testing/.gitignore server/testing/fetch_ai_edit_test.py server/testing/README.md server/testing/stream.py ``` ### Dependencies - client/package.json: @eslint/eslintrc@^3, @gsap/react@^2.1.2, @ianvs/prettier-plugin-sort-imports@^4.4.1, @radix-ui/react-collapsible@^1.1.11, @radix-ui/react-scroll-area@^1.2.9, @radix-ui/react-select@^2.2.5, @radix-ui/react-separator@^1.1.7, @radix-ui/react-slot@^1.2.3, @radix-ui/react-tabs@^1.1.12, @remotion/player@^4.0.318, @tailwindcss/postcss@^4, @types/hls.js@^1.0.0, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@15.3.4, gsap@^3.13.0, gsap-trial@^3.13.0, hls.js@^1.6.5, locomotive-scroll@5.0.0-beta.21, lucide-react@^0.522.0, motion@^12.18.1, next@15.3.4, next-auth@^4.24.11, next-themes@^0.4.6, nuqs@^2.4.3, prettier@^3.5.3, prettier-plugin-tailwindcss@^0.6.8, react@^19.0.0, react-dom@^19.0.0, react-markdown@^10.1.0, react-resizable-panels@^3.0.3, rehype-raw@^7.0.0, remark-gfm@^4.0.1, remotion@^4.0.318, sonner@^2.0.5, tailwind-merge@^2.6.0, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 ### Recent commits (newest first) - feat: enhance landing page navigation and clean up side panel layout - feat: dev dev - Merge remote-tracking branch 'refs/remotes/origin/main' - feat: dev dev dev - Merge branch 'main' of https://github.com/KevinWu098/omni - fix: force 4 - test - fix: fix - feat: add runner statistics tracking and update UI components - feat: update test cases - Merge branch 'main' of https://github.com/KevinWu098/omni - fix: match with frontend - feat: implement step status tracking and update UI accordingly - fix: stop running command on error - port pr maker to app2.py - Merge branch 'main' of github.com:KevinWu098/omni - logs - Merge pull request #8 from KevinWu098/button - Merge branch 'main' into button - feat: add playbar ## Key source files (fetched from GitHub, selected and truncated for size) ### client/package.json ``` { "name": "client", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@gsap/react": "^2.1.2", "@radix-ui/react-collapsible": "^1.1.11", "@radix-ui/react-scroll-area": "^1.2.9", "@radix-ui/react-select": "^2.2.5", "@radix-ui/react-separator": "^1.1.7", "@radix-ui/react-slot": "^1.2.3", "@radix-ui/react-tabs": "^1.1.12", "@remotion/player": "^4.0.318", "@types/hls.js": "^1.0.0", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "gsap": "^3.13.0", "gsap-trial": "^3.13.0", "hls.js": "^1.6.5", "locomotive-scroll": "5.0.0-beta.21", "lucide-react": "^0.522.0", "motion": "^12.18.1", "next": "15.3.4", "next-auth": "^4.24.11", "next-themes": "^0.4.6", "nuqs": "^2.4.3", "react": "^19.0.0", "react-dom": "^19.0.0", "react-markdown": "^10.1.0", "react-resizable-panels": "^3.0.3", "rehype-raw": "^7.0.0", "remark-gfm": "^4.0.1", "remotion": "^4.0.318", "sonner": "^2.0.5", "tailwind-merge": "^2.6.0", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@eslint/eslintrc": "^3", "@ianvs/prettier-plugin-sort-imports": "^4.4.1", "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.3.4", "prettier": "^3.5.3", "prettier-plugin-tailwindcss": "^0.6.8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### server/server/app.py ```python import asyncio import json import logging import uuid from typing import Any, Dict, List, Optional, Set, Tuple import websockets from websockets.server import WebSocketServerProtocol # ——— Configuration —————————————————————————————————————————————————————————————— HOST = "0.0.0.0" PORT = 8765 # ——— Logging Setup ————————————————————————————————————————————————————————————— logging.basicConfig( level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S", ) # ——— In-Memory State ———————————————————————————————————————————————————————————— clients: Set[WebSocketServerProtocol] = set() vms: Dict[str, WebSocketServerProtocol] = {} # vm_id -> websocket connection test_case_to_vm: Dict[str, str] = {} # test_case_id -> vm_id vm_to_test_case: Dict[str, str] = {} # vm_id -> test_case_id (for tracking usage) # ——— VM Management ———————————————————————————————————————————————————————————— async def assign_vms_to_tests(tests: List[Dict[str, Any]]) -> List[Dict[str, Any]]: """ Assign VMs to test cases. Reuse existing VMs or create new ones as needed. """ result = [] for test in tests: test_case_id = test.get("test_case_id") steps = test.get("steps", []) if not test_case_id: continue # Check if this test case already has a VM assigned if test_case_id in test_case_to_vm: vm_id = test_case_to_vm[test_case_id] logging.info(f"Reusing VM {vm_id} for test case {test_case_id}") else: # Find an unused VM or create a new one available_vm_id = None for vm_id in vms.keys(): if vm_id not in vm_to_test_case: available_vm_id = vm_id break if available_vm_id: vm_id = available_vm_id logging.info( f"Assigning existing VM {vm_id} to test case {test_case_id}" ) else: # Create new VM (placeholder - would be actual IaC in production) vm_id = str(uuid.uuid4()) logging.info(f"Created new VM {vm_id} for test case {test_case_id}") # Update mappings test_case_to_vm[test_case_id] = vm_id vm_to_test_case[vm_id] = test_case_id result.append({"test_case_id": test_case_id, "vm_id": vm_id, "steps": steps}) logging.info(f"Assigned VMs to {len(result)} test cases") return result # ——— Connection Lifecycle ——————————————————————————————————————————————————————— async def register_connection( ws: WebSocketServerProtocol, ) -> Tuple[str, Optional[str]]: """ Perform initial registration handshake. Returns (role, vm_id). Raises ValueError on invalid registration message. """ raw = await ws.recv() msg = json.loads(raw) if msg.get("type") != "register" or msg.get("role") not in ("client", "vm"): raise ValueError( "First message must be {type: 'register', role: 'client'|'vm'}" ) role = msg["role"] if role == "client": clients.add(ws) logging.info("Registered new client: %s", ws.remote_address) return "client", None vm_id = msg.get("vm_id") if not vm_id: raise ValueError("VM registration requires a 'vm_id'") vms[vm_id] = ws logging.info("Registered new VM '%s': %s", vm_id, ws.remote_address) return "vm", vm_id async def unregister( ws: WebSocketServerProtocol, role: str, vm_id: Optional[str] = None ) -> None: """ Clean up on disconnect. """ if role == "client": clients.discard(ws) logging.info("Client disconnected: %s", ws.remote_address) else: removed = vms.pop(vm_id, None) # Clean up test case mappings if vm_id in vm_to_test_case: test_case_id = vm_to_test_case.pop(vm_id) test_case_to_vm.pop(test_case_id, None) logging.info( "VM '%s' disconnected: %s", vm_id, ws.remote_address if removed else "not found", ) # ——— Message Handlers —————————————————————————————————————————————————————————— async def handle_client_message( ws: WebSocketServerProtocol, msg: Dict[str, Any] ) -> None: msg_type = msg.get("type") if msg_type == "spin_up": tests = msg.get("tests", []) assigned_tests = await assign_vms_to_tests(tests) await ws.send(json.dumps({"type": "spin_up_response", "tests": assigned_tests})) elif msg_type == "list_vms": vm_list = [] for vm_id, ws_conn in vms.items(): test_case_id = vm_to_test_case.get(vm_id) vm_list.append( {"vm_id": vm_id, "test_case_id": test_case_id, "status": "connected"} ) await ws.send(json.dumps({"type": "list_vms_response", "vms": vm_list})) elif msg_type == "command": vm_id = msg.get("vm_id") command = msg.get("command", "") if vm_id not in vms: await ws.send( json.dumps({"type": "error", "message": f"Unknown vm_id '{vm_id}'"}) ) return await vms[vm_id].send(json.dumps({"type": "command", "command": command})) elif msg_type == "screenshot": vm_id = msg.get("vm_id") full_page = msg.get("full_page", False) if vm_id not in vms: await ws.send( json.dumps({"type": "error", "message": f"Unknown vm_id '{vm_id}'"}) ) return await vms[vm_id].send( json.dumps({"type": "screenshot", "full_page": full_page}) ) else: await ws.send( json.dumps( {"type": "error", "message": f"Unknown message type '{msg_type}'"} ) ) async def handle_vm_message( ws: WebSocketServerProtocol, vm_id: str, msg: Dict[str, Any] ) -> None: msg_type = msg.get("type") i [truncated — 1907 more characters] ``` ### server/agent/app.py ```python from flask import Flask, request, jsonify, Response, stream_with_context from dotenv import load_dotenv import json import asyncio from uuid import uuid4 import os import sys from pathlib import Path from service import VideoAgentService from flask_cors import CORS from edit_agent import pull_edit_pr_streaming from collections import defaultdict # Mapping from run IDs to VideoAgentService instances agents = {} logs = defaultdict(list) load_dotenv() app = Flask(__name__) CORS(app) @app.route("/diag", methods=["GET"]) def diag(): return jsonify({"agents": list(agents.keys())}) @app.route("/agent_logs/", methods=["GET"]) def agent_logs(run_id): return jsonify(logs[run_id]) @app.route("/run_command", methods=["POST"]) def run_command(): data = request.get_json(force=True) commands = data.get("commands") if not commands: return jsonify({"error": "Missing 'commands' in JSON body"}), 400 run_id = data.get( "run_id", None ) # if no run id specified then generate new browser if not run_id: run_id = str(uuid4()) # Create a new service instance with a unique run ID service = VideoAgentService(run_id) agents[run_id] = service else: if run_id not in agents: # Create a new service instance with a unique run ID service = VideoAgentService(run_id) agents[run_id] = service else: service = agents.get(run_id) if not service: return ( jsonify( { "error": f"Run ID {run_id} not found or session already shut down." } ), 404, ) # Ensure the service is not marked as done if it's being reused service.done = False print("Showing commands: ", commands) def generate(): try: # Stream the unique run ID as the first message yield f"data: {{'type': 'uuid', 'id': '{run_id}'}}\n\n" # Ensure recorder is running asyncio.run_coroutine_threadsafe(service.recorder.start(), service.loop).result() # Stream logs and results for log_data in service.run_command_streaming(commands): if (log_data != "data: {'type': 'keepalive'}\n\n"): logs[run_id].append(log_data) yield log_data # Signal completion to client yield "data: {'type': 'done'}\n\n" # Pause recording asyncio.run_coroutine_threadsafe(service.recorder.stop(), service.loop).result() if data.get("soft_shutdown_on_end", False): service.shutdown() except Exception as e: # Catch exceptions during streaming and send an message yield f"data: {{'type': 'error', 'content': '{str(e)}'}}\n\n" # Pause recording on error asyncio.run_coroutine_threadsafe(service.recorder.stop(), service.loop).result() service.shutdown() # Shutdown on error del agents[run_id] # Remove from active agents finally: pass return Response( stream_with_context(generate()), content_type="text/event-stream", headers={ "Cache-Control": "no-cache", "Connection": "keep-alive", }, ) @app.route("/shutdown_run/", methods=["POST"]) def shutdown_run(run_id): data = request.get_json(force=True) if data.get("delete_video", False): service = agents.pop(run_id, None) else: service = agents.get(run_id) if service: service.shutdown() return jsonify({"message": f"Run ID {run_id} shut down successfully."}) return jsonify({"error": f"Run ID {run_id} not found."}), 404 @app.route("/stream//playlist.m3u8", methods=["PUT", "GET"]) def playlist(run_id): service = agents.get(run_id) if not service: return "", 404 if request.method == "PUT": service.hls.put_playlist(request.data) return "" pl = service.get_playlist() if not pl: return "", 404 return Response(pl, mimetype="application/vnd.apple.mpegurl") @app.route("/stream//segments/", methods=["PUT", "GET"]) def segment(run_id, name): service = agents.get(run_id) if not service: return "", 404 if request.method == "PUT": service.hls.put_segment(name, request.data) return "" data = service.get_segment(name) if not data: return "", 404 return Response(data, mimetype="video/mp2t") @app.route("/stream//offer", methods=["POST"]) def offer(run_id): service = agents.get(run_id) if not service: return jsonify({"error": "Unknown run_id"}), 404 # If WebRTC stream is done, return gone if getattr(service, "done", False): return jsonify({"error": "WebRTC stream has ended"}), 410 params = request.get_json(force=True) try: answer = service.handle_offer(params) return jsonify(answer) except RuntimeError as e: # Handle closed event loop gracefully return jsonify({"error": "Agent not available"}), 410 # Simple UI route for viewing HLS/WebRTC with live/DVR toggle VIEW_HTML = """ Live Stream Viewer