# Project export: FormBridge This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: A Chrome extension helping immigrant families master complex English forms through real-time Spanish translation, simplified AI paraphrasing, and guided, voice-powered inputs. - Devpost: https://devpost.com/software/formbridge - GitHub: https://github.com/anya-07/CalHacksAI26.git - Demo: https://agentverse.ai/agents/details/agent1qvqat2zmsqjw654pg5gq5xjzyl47pgaes7w4npzzya8utgr20erqs05rfkx/profile - Video: https://www.youtube.com/embed/zYaU-9Ni5Zk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Anya Garg (1 commits) ## Devpost submission (written by the team) ### Inspiration Roughly 20% of people who qualify for benefits like CalFresh never get them, not because they're ineligible. Instead, the forms are in English and buried in legal jargon. For a monolingual immigrant, completing this paperwork is a challenge. Our goal was to turn this challenge into a simple, clarifying conversation in their own language. We also aimed to prioritize safety for potentially high-stakes legal documents and avoid damage from overly-confident AI hallucinations. ### What it does FormBridge is a voice-first AI assistant for legal paperwork. The user first opens a government/benefits form, and FormBridge reads each field. It converts the confusing English question into a simple spoken question in the user’s specific language (Spanish). It then listens to your answer by voice or waits for a written response. Then, it fills in the form with the correct English answer. It's fully bilingual (English to Spanish and vice versa), and is switchable mid-form, so if you respond in a different language than your initial language choice, it changes accordingly. A key feature: it never submits the form for you: it instead produces a draft, and allows time for user review before submitting or advancing to the next page. With this, FormBridge scores each answer's confidence level and flags sensitive or unclear fields if it "needs review" with a plain-language explanation. If it doesn't capture your answer clearly, it rephrases its question and asks again. ### How we built it Frontend — A Chrome extension (Manifest V3) injects a panel onto any web form, scrapes the fields, runs a per-field voice loop, fills the answers directly into the page (handling React/Angular controlled inputs), and shows a confidence + "Needs Review" draft. It never submits. Backend — A FastAPI bridge orchestrates the intelligence behind the extension: ASI:One generates the simplified, plain-language questions. Deepgram speaks them and transcribes the spoken answers (with a typing fallback). Claude converts each natural-language answer into the correct English field value and judges confidence. Redis stores the grounded policy definitions Claude reasons over. Multi-agent layer — We also built the reasoning as a true multi-agent society on Fetch.ai's uAgents: FormReader, Interpreter, Dialogue, PolicyRAG, Review, and Orchestrator agents that message each other over the ASI:One Chat Protocol and are discoverable on Agentverse — so the orchestrator can be queried directly on ASI:One. ### Challenges we ran into It was difficult coordinating a six-agent system over the chat protocol, along with the difference between local agent-to-agent messaging and Agentverse mailbox routing. On the front end, capturing microphone audio inside a content script and streaming it to Deepgram (while keeping a typing fallback that races the voice input) took real iteration. The biggest design challenge, however, was making the system refuse to be overconfident, and surfacing uncertainty as a feature rather than hiding it. ### Accomplishments we're proud of A working full stack that uses four sponsor technologies from end to end. The safety model — never auto-submit, read-back verification, per-field confidence, and "Needs Review" flags on sensitive fields — is something most AI demos skip, and it's the thing that makes FormBridge trustworthy enough to put in front of a vulnerable person. We're overall proud that we were able to create something that could truly help immigrant populations. ### What we learned We learned how to build and register a multi-agent system on ASI:One/Agentverse using uAgents and the Chat Protocol, and how to architect a clean separation between an orchestrating "brain" and specialist agents. We got hands-on with Deepgram's TTS/STT, Claude for structured extraction with strict-JSON outputs, and ASI:One's LLM. Most importantly, we learned that for high-stakes AI, the engineering that matters most isn't getting the right answer — it's knowing when not to be confident, and designing the human-in-the-loop around that. ### What's next We want to improve FormBridge and make it applicable worldwide by including more languages beyond Spanish. We also want to include a document upload so FormReader can parse any uploaded form PDF and be able to fill it out. ## README (from the GitHub repository) # ⚖️ FormBridge **A voice-first AI form advocate that helps Spanish-speaking residents complete confusing English-language forms — safely, transparently, and under their control.** FormBridge helps people fill out high-impact public forms — CalFresh/SNAP, rental & housing assistance, utility-bill relief, emergency-aid intake, and basic legal-aid intake. You upload or select a form; FormBridge reads it, turns each confusing English question into a **simple spoken question in Spanish**, listens to your answer by voice, and writes the correct **English** answer into the field. It prepares a reviewable **draft**, reads every answer back to you in Spanish, flags anything risky for review — and **never submits automatically**. > Built in 24h for Hackathons @ Berkeley — **DDOSKI'S WORLD** (technology + social impact). > First version: **Spanish ⇄ English**, a small set of high-impact forms, **draft only — no auto-submit.** --- ## The problem Millions of people who qualify for food, housing, and utility help never get it — not because they don't qualify, but because the forms are in English, full of bureaucratic jargon, and frightening to get wrong. Language, literacy, and technology barriers turn a benefit you're entitled to into a wall. FormBridge turns that wall into a conversation in your own language. ## How it works For each field on the form, FormBridge runs a per-field voice loop, then stops for your review: ``` upload/select form ─▶ [ FormReader ] ─▶ for each field: ├─ [ Interpreter ] simplify EN question → ES ├─ [ Dialogue ] speak ES (TTS) · hear ES (STT) ├─ [ Interpreter ] ES answer → correct EN value ├─ [ PolicyRAG ] ground tricky terms (Redis) └─ [ Review ] confidence score + "Needs Review" ▼ English DRAFT ──▶ read back in Spanish ──▶ you confirm (FormBridge NEVER submits on its own) ``` Several agents, each **registered separately on Agentverse** and speaking the ASI:One **Chat Protocol**, so they're mutually discoverable and message each other directly. That agent-to-agent collaboration is the heart of the system. | Agent | Role | Tech | |---|---|---| | **Orchestrator** | Runs the per-field loop; builds the draft; drives read-back & verify; never submits | Fetch.ai / ASI:One | | **FormReader** | Reads the uploaded form, extracts fields, labels, types | Anthropic Claude | | **Dialogue** | Speaks each Spanish question, transcribes the spoken Spanish answer | Deepgram | | **Interpreter** | Simplifies English questions → Spanish; converts Spanish answers → correct English values | Anthropic Claude | | **PolicyRAG** | Grounds tricky terms in cited definitions; remembers the session's answers | Redis | | **Review** | Scores confidence per field; flags sensitive/risky fields as "Needs Review" | Anthropic Claude | ## Safety first (a headline feature, not a disclaimer) - **Never auto-submits** — produces a reviewable draft and pauses for confirmation. - **Read-back verification** — every answer is spoken back in Spanish so you catch errors in your own language. - **Confidence per field** + **"Needs Review"** flags on uncertain, sensitive, or risky fields (immigration status, income, household members, legal declarations, signatures), each with a plain-Spanish explanation. - **No quasi-legal advice** — FormBridge helps fill what the form asks and flags declarations for human review. ## Front-end: Chrome extension The user-facing front-end is a Chrome extension (`extension/`). It injects a panel onto any web form, reads the fields, asks each one aloud in Spanish, fills the English answers, flags sensitive fields **Needs Review**, and builds a draft — and **never submits**. It runs standalone in a built-in **MOCK mode** (no backend), or talks to the agents through the `bridge/` HTTP gateway. Load it via `chrome://extensions → Developer mode → Load unpacked → select extension/`. ## Repository ``` . ├── FORMBRIDGE_BUILD_DOC.md # full strategy: architecture, safety model, timeline, pitch ├── formbridge/ # the multi-agent backend (uAgents on ASI:One) │ ├── agents/ # orchestrator + 6 specialists (chat-protocol wired) │ ├── client.py # local test client (no ASI:One needed during dev) │ ├── run_all.sh # launch the whole agent society │ └── requirements.txt ├── extension/ # Chrome extension front-end (MV3) — MOCK mode works standalone ├── bridge/ # FastAPI gateway: extension HTTP -> uAgents society └── web/ # calfresh_replica.html — a reliable demo target form ``` ## Quickstart ```bash cd formbridge python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # add your keys bash run_all.sh # boot the agents ``` See [`formbridge/README.md`](formbridge/README.md) to run it and [`FORMBRIDGE_BUILD_DOC.md`](FORMBRIDGE_BUILD_DOC.md) for the full build plan. ## Team Three CS students @ UC Berkeley. ## Detected evidence (automated analysis) Indexed codebase: 29 recognized source files, 152 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Redis (technology) — detected in the code ## Codebase structure (from repository index) ### Files (39 of 39) ``` .gitignore agentverse_readmes/dialogue.md agentverse_readmes/formreader.md agentverse_readmes/interpreter.md agentverse_readmes/orchestrator.md agentverse_readmes/policyrag.md agentverse_readmes/review.md bridge/README.md bridge/requirements.txt bridge/server.py extension.crx extension.pem extension/background.js extension/config.js extension/content.js extension/manifest.json extension/panel.css extension/README.md FORMBRIDGE_BUILD_DOC.md formbridge/.env.example formbridge/.gitignore formbridge/agents/agent1qtdfd96s5q_data.json formbridge/agents/agent1qvqat2zmsq_data.json formbridge/agents/common.py formbridge/agents/dialogue_agent.py formbridge/agents/eligibility_agent.py formbridge/agents/formfiller_agent.py formbridge/agents/formreader_agent.py formbridge/agents/interpreter_agent.py formbridge/agents/orchestrator.py formbridge/agents/policy_rag_agent.py formbridge/agents/review_agent.py formbridge/agents/translator_agent.py formbridge/client.py formbridge/README.md formbridge/requirements.txt formbridge/run_all.sh README.md web/calfresh_replica.html ``` ### Dependencies - bridge/requirements.txt: anthropic, fastapi, httpx, openai, pydantic, python-dotenv, redis, uvicorn[standard] - formbridge/requirements.txt: anthropic, browserbase, httpx, openai, playwright, python-dotenv, redis, uagents, uagents-core ### Recent commits (newest first) - Final Agent and Extension - Final agent and extension code - final version - renamed everything to FormBridge - Add root .gitignore; stop tracking __pycache__ - Add Chrome extension front-end, FastAPI bridge, and CalFresh demo form - Refactor to 6-agent voice-first design with per-field loop and Needs-Review - Merge branch 'main' of https://github.com/anya-07/CalHacksAI26 - Tribunal: multi-agent benefits advocate on ASI:One - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### FORMBRIDGE_BUILD_DOC.md ```markdown # FormBridge — Build Doc **A voice-first AI form advocate that helps Spanish-speaking residents complete confusing English-language forms — safely, transparently, and under their control.** The user uploads or selects a form they need help with (CalFresh/SNAP, rental or housing assistance, utility-bill relief, emergency-aid intake, or basic legal-aid intake). FormBridge reads the form, finds each field, and turns each confusing English question into a simple **spoken question in Spanish**. The user answers naturally by voice; FormBridge transcribes, understands the meaning, and writes the correct **English** answer into the field. It then fills a **draft** of the form in English, reads every answer back in Spanish for verification, and **never submits automatically** — it prepares a reviewable draft and pauses for confirmation. Each field gets a **confidence score**, and uncertain, sensitive, or risky fields (immigration status, income, household members, legal declarations, signatures) are flagged **"Needs Review"** with a plain-Spanish explanation. - **Track:** DDOSKI'S WORLD (technology + social impact) - **Team:** 3 strong CS students, 24 hours - **First-version scope:** Spanish ⇄ English, a small set of high-impact forms, **draft only — no auto-submit** - **Goal:** Fetch.ai/ASI:One prize **+** win the track **+** sweep sponsor prizes **+** grand-prize contention --- ## 1. Why this wins (read this first) Hackathon prizes are won on a **memorable, working demo + a story judges repeat to each other**, and increasingly on **responsible-AI judgment**. FormBridge is engineered so the same architecture that makes the demo emotional also (a) stacks sponsor prizes and (b) tells a safety story most teams ignore. Four things make it award-winning: 1. **It does something real, live.** A real English benefits form on screen. The user *speaks Spanish*, and the English fields fill in, one by one, with the agent's reasoning visible. The "it actually understood her and filled the form" moment is what judges remember at 2am. 2. **It's a genuine multi-agent society, visible on Agentverse.** The Fetch.ai prize specifically rewards multi-agent collaboration. You'll have several separately-registered agents messaging each other over the ASI:One Chat Protocol. Judges can open Agentverse and *see* the network. ~80% of ASI:One submissions are a single agent — yours won't be. 3. **Safety is the feature, not a disclaimer.** Confidence scores + "Needs Review" flags + read-back verification + never-auto-submit is exactly the human-in-the-loop, trust-first design that wins "responsible AI" and social-impact judging. It also disarms the judge's #1 fear: "what if it fills something wrong on a legal document?" 4. **The impact is concrete.** Not "AI for good." It's "a monolingual Spanish-speaking mother completes a housing-assistance form she couldn't read — and FormBridge flags the legal-declaration field so she doesn't sign something she doesn't understand." Specifi [truncated — 16717 more characters] ``` ### agentverse_readmes/dialogue.md ```markdown # FormBridge — Dialogue Agent ## Overview Dialogue is the voice specialist in the FormBridge system. It handles the spoken interaction with the user: speaking each form question aloud in the user's language (text-to-speech) and transcribing the user's spoken answer (speech-to-text). This is what makes FormBridge usable hands-free by people who cannot easily read or type — a core accessibility feature for low-literacy and non-English-speaking residents. ## Key features - Text-to-speech: reads simplified questions aloud in Spanish or English. - Speech-to-text: transcribes the resident's spoken answer, with automatic language detection. - Optimized for short, clear prompts to keep the conversation responsive. - Powers the voice-first experience of FormBridge's per-field form-filling loop. ## Usage instructions The agent receives a question to speak and returns the transcribed spoken answer over the Chat Protocol. ``` Input: Question (text) to ask aloud in the chosen language. Output: The user's spoken answer, transcribed to text. ``` ## Use cases / examples - Letting a monolingual Spanish speaker answer benefit-form questions by voice instead of typing. - Any accessible, voice-driven intake flow where reading or typing is a barrier. ## Limitations and known issues - Quality depends on microphone input and ambient noise. - Sensitive fields such as passwords and email addresses are intentionally typed, not spoken, for accuracy and security. ## Metadata and credits - **Project**: FormBridge (Hackathons @ Berkeley). Specialist agent coordinated by the FormBridge Orchestrator. - **Authors**: UC Berkeley CS students. **Keywords:** voice, speech to text, text to speech, transcription, accessibility, Spanish, bilingual, forms, FormBridge ``` ### bridge/requirements.txt ``` fastapi uvicorn[standard] pydantic python-dotenv httpx openai # ASI:One LLM (OpenAI-compatible endpoint) anthropic # Claude (value extraction + review) redis # grounding definitions + cache ``` ### formbridge/requirements.txt ``` uagents uagents-core openai # ASI:One LLM uses the OpenAI-compatible client (base_url=https://api.asi1.ai/v1) anthropic # Interpreter + Review reasoning (Claude) redis # PolicyRAG grounded definitions / cache httpx # Deepgram REST calls (TTS/STT) browserbase # optional autonomous draft-fill mode playwright # drives the Browserbase session python-dotenv # load .env # arize-phoenix # optional: tracing/eval dashboard ``` ### bridge/server.py ```python """Bridge server — the real backend behind the FormBridge Chrome extension. Bilingual (English / Spanish, switchable per request). Every route takes a `lang` ("es" or "en") so the user can switch the interaction language on the fly. The form output value is ALWAYS English (the forms are English); `lang` only controls the language the user is asked/answered/spoken to in. POST /simplify {label, type, lang} -> {question} (ASI:One LLM) POST /process_field {label, type, sensitive, answer, lang} -> {value_en, confidence, needs_review, reason} (Redis + Claude) POST /tts {text, lang} -> audio/mpeg (Deepgram voice/lang) POST /stt?lang=es (audio body) -> {transcript} (Deepgram lang) GET /health -> integration status Never hard-fails: a missing key or API error falls back to deterministic stubs. Keys load from ../formbridge/.env. Run: uvicorn server:app --port 8088 --reload """ import os import re import json from pathlib import Path from typing import List, Optional import httpx from fastapi import FastAPI, Request from fastapi.responses import Response from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel try: from dotenv import load_dotenv load_dotenv(Path(__file__).resolve().parent.parent / "formbridge" / ".env") except Exception: pass ASI_KEY = os.getenv("ASI_ONE_API_KEY", "") ANTHROPIC_KEY = os.getenv("ANTHROPIC_API_KEY", "") REDIS_URL = os.getenv("REDIS_URL", "") DEEPGRAM_KEY = os.getenv("DEEPGRAM_API_KEY", "") CLAUDE_MODEL = os.getenv("CLAUDE_MODEL", "claude-haiku-4-5-20251001") # Deepgram voices per language (override in .env if you prefer different ones) TTS_VOICE = { "es": os.getenv("DEEPGRAM_TTS_ES", "aura-2-celeste-es"), "en": os.getenv("DEEPGRAM_TTS_EN", "aura-2-thalia-en"), } LANG_NAME = {"es": "Spanish", "en": "English"} def _lang(code: str) -> str: return "en" if (code or "").lower().startswith("en") else "es" app = FastAPI(title="FormBridge bridge") app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]) SENSITIVE_RE = re.compile( r"(income|salary|wage|earn|immigration|citizen|residency|ssn|social security|" r"signature|sign|declare|perjury|household)", re.I) UNCERTAIN_RE = re.compile(r"(no s[eé]|no estoy segur|no entiendo|tal vez|creo que|" r"i don'?t know|not sure|unsure)", re.I) CLARIFY_RE = re.compile(r"(no entiend|qu[eé] significa|no s[eé] qu[eé]|me explica|ay[uú]dame|" r"i don'?t understand|what does (this|that) mean|what do you mean|" r"can you explain|explain|help me|repeat)", re.I) _clients = {} def _asi(): if "asi" not in _clients: from openai import OpenAI _clients["asi"] = OpenAI(base_url="https://api.asi1.ai/v1", api_key=ASI_KEY) if ASI_KEY else None return _clients["asi"] def _claude(): if "claude" not in _clients: from anthropic import Anthropic _clients["claude"] = Anthropic(api_key=ANTHROPIC_KEY) if ANTHROPIC_KEY else None return _clients["claude"] def _redis(): if "redis" not in _clients: try: import redis r = redis.Redis.from_url(REDIS_URL, decode_responses=True) if REDIS_URL else None if r: r.ping() r.hsetnx("formbridge:defs", "income", "Gross monthly income = ALL household earnings before taxes: wages, " "self-employment, Social Security, child support. [CalFresh §63-409]") r.hsetnx("formbridge:defs", "household", "Household = people who live together AND buy/prepare food together. " "[CalFresh §63-402]") _clients["redis"] = r except Exception: _clients["redis"] = None return _clients["redis"] class SimplifyIn(BaseModel): label: str type: str = "text" lang: str = "es" choices: Optional[List[str]] = None # present for multiple-choice fields class ChooseIn(BaseModel): label: str type: str = "text" choices: List[str] = [] answer: str = "" lang: str = "es" sensitive: bool = False multi: bool = False # true for checkbox groups /
Include everyone who lives with you and buys/prepares food together.
Are you a U.S. citizen?
Do you have any dependents?
``` ### extension/panel.css ```css #formbridge-panel { position: fixed; top: 16px; right: 16px; width: 340px; max-height: 80vh; overflow-y: auto; z-index: 2147483647; background: #0e0e12; color: #f4f4f6; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; font-size: 14px; border-radius: 14px; box-shadow: 0 10px 40px rgba(0, 0, 0, 0.5); border: 1px solid #2a2a35; } #formbridge-panel.hidden { display: none; } #formbridge-header { display: flex; justify-content: space-between; align-items: center; padding: 12px 14px; font-weight: 700; font-size: 16px; border-bottom: 1px solid #2a2a35; } #formbridge-close { background: none; border: none; color: #aaa; font-size: 20px; cursor: pointer; line-height: 1; } .formbridge-head-right { display: inline-flex; align-items: center; gap: 10px; } #formbridge-lang.seg { display: inline-flex; align-items: center; gap: 3px; background: #16161d; border: 1px solid #2a2a35; border-radius: 9px; padding: 3px; } #formbridge-lang.seg button { display: inline-flex; align-items: center; gap: 5px; background: transparent; color: #9aa0aa; border: 1px solid transparent; font-size: 12px; font-weight: 600; line-height: 1; padding: 5px 12px; border-radius: 7px; cursor: pointer; outline: none; -webkit-appearance: none; appearance: none; margin: 0; transition: background 0.15s ease, color 0.15s ease, box-shadow 0.15s ease; } #formbridge-lang.seg button:focus, #formbridge-lang.seg button:focus-visible { outline: none; box-shadow: none; } #formbridge-lang.seg button.active:focus, #formbridge-lang.seg button.active:focus-visible { box-shadow: 0 1px 3px rgba(0, 0, 0, 0.45); } #formbridge-lang.seg button:hover { color: #e6e7ea; } #formbridge-lang.seg button.active { background: #2c2c38; color: #ffffff; border-color: #3a3a47; box-shadow: 0 1px 3px rgba(0, 0, 0, 0.45); } #formbridge-lang.seg button.active::before { content: "🌐"; font-size: 11px; line-height: 1; opacity: 0.85; } #formbridge-done { background: #e6a800 !important; color: #111 !important; } #formbridge-question { padding: 14px; font-size: 16px; line-height: 1.4; background: #16161d; } #formbridge-status { padding: 6px 14px; color: #9aa0aa; min-height: 18px; } #formbridge-controls { display: flex; flex-wrap: wrap; gap: 8px; padding: 10px 14px; } #formbridge-controls button { background: #5b3df5; color: #fff; border: none; border-radius: 8px; padding: 8px 12px; font-weight: 600; cursor: pointer; } #formbridge-controls button#formbridge-stop { background: #2a2a35; } #formbridge-answer { flex: 1 1 100%; background: #16161d; border: 1px solid #2a2a35; color: #fff; border-radius: 8px; padding: 8px 10px; } #formbridge-draft { padding: 6px 14px 12px; } .formbridge-row { background: #16161d; border: 1px solid #2a2a35; border-radius: 10px; padding: 8px 10px; margin-top: 8px; } .formbridge-row.flag { border-color: #e6a800; background: #211d12; } .formbridge-row.skipped { opacity: 0.55; } .formbridge-row.skipped .formbridge-val { font-style: italic; color: #9aa0aa; } .formbridge-row-top { display: flex; justify-content: space-between; } .formbridge-label { color: #c7cbd4; font-size: 12px; } .formbridge-conf { color: #7de28a; font-size: 12px; font-weight: 700; } .formbridge-row.flag .formbridge-conf { color: #e6a800; } .formbridge-val { font-weight: 600; margin-top: 2px; word-break: break-word; } .formbridge-review { color: #ffcf5c; font-size: 12px; margin-top: 6px; } .formbridge-note { margin-top: 10px; padding: 10px; border-radius: 8px; background: #12211a; color: #8fe6b0; font-size: 12px; line-height: 1.4; } #formbridge-foot { padding: 8px 14px 12px; color: #6b7280; font-size: 11px; } /* page-side highlights on the actual form fields */ .formbridge-active { outline: 3px solid #5b3df5 !important; outline-offset: 2px; } .formbridge-flag { outline: 3px solid #e6a800 !important; outline-offset: 2px; } ``` [17 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]