# Project export: LangBase This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: There are over 7000 unique languages in the world. Most of them have little to no data. We are here to change that. - Devpost: https://devpost.com/software/langbase - GitHub: https://github.com/GaelGil/ucb_ai_hackathon - Video: https://www.youtube.com/embed/BJvnReQgnKo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Gael Gil (39 commits), Tania Tatis (37 commits), Copilot (19 commits), Claude Opus 4.8 (1M context) (8 commits) ## Devpost submission (written by the team) ### Inspiration I am pursuing my Masters in Computational Linguistics and I've really enjoy learning about low resource languages. I like the idea of any language being learnable by anyone and preserving peoples culture. The hardest part of data collection is the manual work so we can let agents do the tedious work. ### What it does Our main focus was parts of speech tagging, translation and text extraction from images. A user can upload data, do specific research on the task and AI will suggestions for parts of speech of that sentence, translate it or extract the text from the image. A user has final say if the suggestion is good. They can deny, accept and update. ### How we built it We use Claude to do research on the grammar and syntactic rules of a language. Browserbase powers Claude with the correct tools to get clean data from websites. We then summarize those into notes and use them to give suggestions on translations and parts of speech. We use previous user choices to make future decisions. ### Challenges we ran into Wifi. Finding data for our chosen sample languages. We had an idea to simulate low resource languages using a small subset for English but then realized Claude would have a very easy time with any task we gave it. ### Accomplishments we're proud of We got some AI suggestions that seem very promising. It seems that we could scale this to be more research heavy. ### What we learned Learned about browserbase and its many helpful uses cases for giving our agents valuable data. ### What's next Not sure maybe we can implement this further and train some dedicated models for translation, parts of speech tagging and character extraction. At the very least a more clean implementation of this project. ## README (from the GitHub repository) # LangBase **A preservation tool for low-resource languages.** LangBase gives linguists and community researchers a structured workspace to import raw text, annotate it with AI-assisted part-of-speech tagging, review translations, and run web research — all in a human-in-the-loop pipeline designed to keep humans in control of the data quality. Built at the UC Berkeley AI Hackathon. --- ## Architecture ![LangBase System Architecture](./resources/langbase-architecture.png) | Layer | Technology | |---|---| | Frontend | React + TypeScript, Bun | | Backend | Python, Flask | | ORM | SQLModel (SQLAlchemy + Pydantic) | | Database | SQLite (dev) / Supabase Postgres (prod) | | AI / LLM | Anthropic Claude | | Web Research | Browserbase | | File Storage | Supabase Storage | | Observability | Arize / Phoenix (OpenTelemetry) | --- ## Features - **Dataset workspaces** — create and manage named datasets per language - **Data import** — paste text, upload CSVs, PDFs, or images - **OCR** — extract text from images via Claude Vision - **AI-assisted POS tagging** — token-level Universal Dependencies UPOS suggestions powered by Claude - **Translation** — Spanish ↔ Nahuatl via Anthropic or a custom model endpoint - **Web research** — Browserbase fetches and summarises language-specific research to ground annotations - **Human review** — accept, reject, or edit every AI suggestion before it enters the dataset - **Background jobs** — long-running tasks (OCR, POS, research) run in background threads; the UI polls for results - **Observability** — optional Arize / Phoenix tracing for every LLM call --- ## Project Structure ``` ucb_ai_hackathon/ ├── backend/ Flask API │ ├── main.py Entry point │ ├── app/ │ │ ├── __init__.py Flask app factory (create_app) │ │ ├── config.py Settings (pydantic-settings) │ │ ├── schemas.py Pydantic request / response schemas │ │ ├── exceptions.py Custom errors │ │ ├── api/ Domain modules — routes + services per feature │ │ │ ├── data/ Text & file import │ │ │ ├── dataset/ Dataset CRUD + dashboard │ │ │ ├── labels/ POS / translation / OCR review │ │ │ ├── language/ Language translation endpoint │ │ │ └── research/ Browserbase research jobs │ │ ├── database/ │ │ │ ├── models/ SQLModel table definitions │ │ │ └── session.py Engine + request-scoped session │ │ ├── clients/ External service wrappers │ │ │ ├── anthropic.py │ │ │ ├── browserbase.py │ │ │ ├── part_of_speech.py │ │ │ ├── image_reader.py │ │ │ ├── translation.py │ │ │ ├── storage.py │ │ │ └── tracing.py │ │ └── utils/ │ │ ├── job_runner.py Background job state management │ │ ├── mappers.py DB model → API schema converters │ │ └── parsing.py File type detection + CSV parsing │ ├── migrations/ Alembic DB migrations │ ├── tests/ pytest integration tests │ └── scripts/ One-off data utility scripts │ ├── frontend/ React + TypeScript UI │ └── src/ │ ├── features/ Feature modules (upload, labels, research, …) │ ├── components/ Shared layout + UI primitives │ ├── hooks/ Data-fetching and workspace state hooks │ ├── lib/ API client, constants, formatters │ └── types/ Shared domain types │ └── sample_data/ Demo CSV files for local seeding └── resources/ Diagrams and other project assets ``` --- ## Getting Started ### Prerequisites | Tool | Version | |---|---| | Python | 3.11+ | | [uv](https://docs.astral.sh/uv/) | latest | | [Bun](https://bun.sh) | 1.x | ### 1 — Clone ```bash git clone https://github.com/GaelGil/ucb_ai_hackathon.git cd ucb_ai_hackathon ``` ### 2 — Backend ```bash cd backend # Install dependencies uv sync # Copy the example env file and fill in your keys cp .env.example .env # Run the server (SQLite, with demo data seeded) CREATE_DB_ON_STARTUP=true SEED_DEMO_DATA=true uv run python main.py ``` The API is now available at **http://localhost:8000**. ### 3 — Frontend ```bash cd frontend # Install dependencies bun install # Start the dev server bun dev ``` Open **http://localhost:3000** in your browser. --- ## Configuration All settings are loaded from `backend/.env` (or real environment variables). Copy `.env.example` and fill in what you need. | Variable | Default | Description | |---|---|---| | `DATABASE_URL` | `sqlite:///./langbase.db` | SQLAlchemy URL — use Supabase Postgres in prod | | `CREATE_DB_ON_STARTUP` | `false` | Auto-create tables on boot (useful for local SQLite) | | `SEED_DEMO_DATA` | `false` | Seed a Nahuatl demo dataset on first boot | | `ANTHROPIC_API_KEY` | — | Enables Claude-powered POS, OCR, translation, and research | | `BROWSERBASE_API_KEY` | — | Enables live web research | | `SUPABASE_URL` | — | Required for cloud file storage | | `SUPABASE_SERVICE_ROLE_KEY` | — | Supabase service key | | `SUPABASE_STORAGE_BUCKET` | `langbase-uploads` | Bucket name | | `ANTHROPIC_MODEL` | `claude-sonnet-4-5` | Claude model to use | | `NAHUATL_MODEL_ENDPOINT_URL` | — | Custom translation endpoint (falls back to Claude) | | `PHOENIX_ENABLED` | `false` | Enable Phoenix / Arize tracing | | `PHOENIX_OTEL_ENDPOINT` | `http://localhost:6006/v1/traces` | OTLP trace endpoint | > **Missing keys are safe.** All external providers fall back to demo mode when credentials are absent so the UI stays usable. --- ## Running Tests ```bash cd backend uv run pytest ``` Tests run fully offline — no real API keys, no network, no Postgres. A fresh in-memory SQLite database is created for each test session. --- ## API Overview | Method | Path | Description | |---|---|---| | `GET` | `/health` | Health check | | `GET/POST` | `/datasets` | List / create datasets | | `GET/PATCH/DELETE` | `/datasets/{id}` | Get / update / delete a dataset | | `GET` | `/datasets/{id}/dashboard` | Aggregated stats for a dataset | | `POST` | `/datasets/{id}/import/text` | Import plain text rows | | `POST` | `/datasets/{id}/import/csv` | Import a CSV file | | `GET` | `/datasets/{id}/data` | Paginated data rows | | `GET` | `/datasets/{id}/labels` | Paginated label suggestions | | `POST` | `/datasets/{id}/labels/{row_id}/accept` | Accept a suggestion | | `POST` | `/datasets/{id}/labels/{row_id}/reject` | Reject a suggestion | | `POST` | `/datasets/{id}/pos` | Trigger POS annotation job | | `POST` | `/datasets/{id}/translate` | Trigger translation job | | `POST` | `/datasets/{id}/research` | Trigger web research job | | `GET` | `/datasets/{id}/jobs` | List background jobs | --- ## Database Migrations After changing a model in `app/database/models/`, generate and apply a migration: ```bash cd backend # Generate uv run alembic revision --autogenerate -m "describe your change" # Apply uv run alembic upgrade head ``` Other useful commands: ```bash uv run alembic current # Show current revision uv run alembic history # Full migration history uv run alembic downgrade -1 # Roll back one step ``` ## Detected evidence (automated analysis) Indexed codebase: 143 recognized source files, 901 KB. - Anthropic (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code - Supabase (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: GitHub Copilot — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 161) ``` .agents/skills/arize-admin/references/ax-profiles.md .agents/skills/arize-admin/references/ax-setup.md .agents/skills/arize-admin/references/REFERENCE.md .agents/skills/arize-admin/SKILL.md .agents/skills/arize-ai-provider-integration/references/ax-profiles.md .agents/skills/arize-ai-provider-integration/references/ax-setup.md .agents/skills/arize-ai-provider-integration/SKILL.md .agents/skills/arize-annotation/references/ax-profiles.md .agents/skills/arize-annotation/references/ax-setup.md .agents/skills/arize-annotation/SKILL.md .agents/skills/arize-compliance-audit/references/compliance-checklist-template.md .agents/skills/arize-compliance-audit/references/eu-ai-act-gpai.md .agents/skills/arize-compliance-audit/references/iso-42001.md .agents/skills/arize-compliance-audit/references/us-ai-compliance.md .agents/skills/arize-compliance-audit/SKILL.md .agents/skills/arize-dataset/references/ax-profiles.md .agents/skills/arize-dataset/references/ax-setup.md .agents/skills/arize-dataset/SKILL.md .agents/skills/arize-evaluator/references/ax-profiles.md .agents/skills/arize-evaluator/references/ax-setup.md .agents/skills/arize-evaluator/SKILL.md .agents/skills/arize-experiment/references/ax-profiles.md .agents/skills/arize-experiment/references/ax-setup.md .agents/skills/arize-experiment/SKILL.md .agents/skills/arize-instrumentation/references/ax-profiles.md .agents/skills/arize-instrumentation/references/integration-routing.md .agents/skills/arize-instrumentation/references/manual-spans.md .agents/skills/arize-instrumentation/references/tracing-assistant-mcp.md .agents/skills/arize-instrumentation/SKILL.md .agents/skills/arize-link/references/EXAMPLES.md .agents/skills/arize-link/SKILL.md .agents/skills/arize-prompt-optimization/references/ax-profiles.md .agents/skills/arize-prompt-optimization/references/ax-setup.md .agents/skills/arize-prompt-optimization/SKILL.md .agents/skills/arize-prompts/references/ax-profiles.md .agents/skills/arize-prompts/references/ax-setup.md .agents/skills/arize-prompts/references/cli-prompts.md .agents/skills/arize-prompts/SKILL.md .agents/skills/arize-trace/references/ax-profiles.md .agents/skills/arize-trace/references/ax-setup.md .agents/skills/arize-trace/SKILL.md .gitignore backend/.env.example backend/.gitignore backend/.python-version backend/alembic.ini backend/app/__init__.py backend/app/api/__init__.py backend/app/api/container.py backend/app/api/data/__init__.py backend/app/api/data/routes.py backend/app/api/data/service.py backend/app/api/dataset/__init__.py backend/app/api/dataset/routes.py backend/app/api/dataset/service.py backend/app/api/labels/__init__.py backend/app/api/labels/routes.py backend/app/api/labels/service.py backend/app/api/language/__init__.py backend/app/api/language/routes.py backend/app/api/language/service.py backend/app/api/research/__init__.py backend/app/api/research/routes.py backend/app/api/research/service.py backend/app/api/responses.py backend/app/clients/__init__.py backend/app/clients/anthropic.py backend/app/clients/browserbase.py backend/app/clients/image_reader.py backend/app/clients/part_of_speech.py backend/app/clients/storage.py backend/app/clients/tracing.py backend/app/clients/translation.py backend/app/config.py backend/app/database/__init__.py backend/app/database/models/__init__.py backend/app/database/models/data_row.py backend/app/database/models/dataset.py backend/app/database/models/import_record.py backend/app/database/models/job.py backend/app/database/models/label.py backend/app/database/models/language.py backend/app/database/models/research.py backend/app/database/models/suggestion.py backend/app/database/session.py backend/app/exceptions.py backend/app/schemas.py backend/app/src/api/data/service.py backend/app/src/api/labels/controller.py backend/app/src/providers.py backend/app/utils/__init__.py backend/app/utils/job_runner.py backend/app/utils/mappers.py backend/app/utils/parsing.py backend/main.py backend/migrations/env.py backend/migrations/README backend/migrations/script.py.mako backend/migrations/versions/20260620_0001_initial_langbase_schema.py backend/pyproject.toml backend/README.md backend/requirements.txt backend/scripts/blank_translation_labels.py backend/scripts/create_pos_rows_from_translation.py backend/scripts/trim_language_rows.py backend/tests/__init__.py backend/tests/conftest.py backend/tests/test_api.py backend/tests/test_browserbase_provider.py backend/tests/test_config.py backend/tests/test_pos_seed.py backend/tests/test_tracing.py backend/tests/test_translation_split.py backend/uv.lock frontend/.gitignore frontend/build.ts frontend/bun-env.d.ts frontend/bun.lock frontend/bunfig.toml frontend/package.json [41 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: alembic@>=1.18.4, anthropic@>=0.111.0, arize-otel@>=0.13.0, flask@>=3.0.0, flask-cors@>=4.0.0, httpx@>=0.27.0, openinference-instrumentation-anthropic@>=1.0.6, opentelemetry-api@>=1.25.0, opentelemetry-exporter-otlp@>=1.25.0, opentelemetry-sdk@>=1.25.0, psycopg2-binary@>=2.9.12, pydantic@>=2.8.0, pydantic-settings@>=2.14.2, python-dotenv@>=1.2.2, sqlmodel@>=0.0.38 - backend/requirements.txt: alembic@==1.18.4, annotated-types@==0.7.0, anthropic@==0.111.0, anyio@==4.14.0, arize-otel@==0.13.0, blinker@==1.9.0, certifi@==2026.6.17, charset-normalizer@==3.4.7, click@==8.4.1, distro@==1.9.0, docstring-parser@==0.18.0, flask@==3.1.3, flask-cors@==6.0.5, googleapis-common-protos@==1.75.0, grpcio@==1.81.1, h11@==0.16.0, httpcore@==1.0.9, httpx@==0.28.1, idna@==3.18, itsdangerous@==2.2.0, jinja2@==3.1.6, jiter@==0.15.0, mako@==1.3.12, markupsafe@==3.0.3, openinference-instrumentation@==0.1.53, openinference-instrumentation-anthropic@==1.0.6, openinference-semantic-conventions@==0.1.30, opentelemetry-api@==1.42.1, opentelemetry-exporter-otlp@==1.42.1, opentelemetry-exporter-otlp-proto-common@==1.42.1, opentelemetry-exporter-otlp-proto-grpc@==1.42.1, opentelemetry-exporter-otlp-proto-http@==1.42.1, opentelemetry-instrumentation@==0.63b1, opentelemetry-proto@==1.42.1, opentelemetry-sdk@==1.42.1, opentelemetry-semantic-conventions@==0.63b1, packaging@==26.2, protobuf@==6.33.6, psycopg2-binary@==2.9.12, pydantic@==2.13.4, pydantic-core@==2.46.4, pydantic-settings@==2.14.2, python-dotenv@==1.2.2, requests@==2.34.2, sniffio@==1.3.1, sqlalchemy@==2.0.51, sqlmodel@==0.0.38, typing-extensions@==4.15.0, typing-inspection@==0.4.2, urllib3@==2.7.0, werkzeug@==3.1.8, wrapt@==2.2.2 - frontend/package.json: @mantine/core@^9.3.2, @mantine/hooks@^9.3.2, @tanstack/react-query@^5.101.0, @tanstack/react-table@^8.21.3, @types/bun@latest, @types/react@^19, @types/react-dom@^19, react@^19, react-dom@^19, react-icons@^5.6.0 ### Recent commits (newest first) - fixing merge conflicts - stuff - updates - docs: update README to reflect routes/ → api/ rename - refactor: rename routes/ → api/ to better reflect folder contents - chore: add pinned requirements.txt - docs: add README and move architecture diagram to resources/ - Split clients/ai.py into one file per client - Rename integrations/ → clients/, providers.py → ai.py - Rename files to match their actual contents - Move utility files into app/utils/ - Group external service wrappers into app/integrations/ - Rename app/api/ → app/routes/ + move create_app() to app/__init__.py - Rename data/ → sample_data/ - Rename api/dependencies.py → api/context.py - Rename controller.py → routes.py in all API domains - Rename app/repositories.py → app/exceptions.py - Rename app/models.py → app/schemas.py - Clean up stale artifacts after restructure - Flatten app/src/ → app/ and update all imports ## Key source files (fetched from GitHub, selected and truncated for size) ### .agents/skills/arize-link/SKILL.md ```markdown --- name: arize-link description: Generates deep links to the Arize UI for traces, spans, sessions, datasets, labeling queues, evaluators, and annotation configs. Produces clickable URLs for sharing Arize resources with team members. Use when the user wants to link to or open a trace, span, session, dataset, evaluator, or annotation config in the Arize UI. metadata: author: arize version: "1.0" --- # Arize Link Generate deep links to the Arize UI for traces, spans, sessions, datasets, labeling queues, evaluators, and annotation configs. ## When to Use - User wants a link to a trace, span, session, dataset, labeling queue, evaluator, or annotation config - You have IDs from exported data or logs and need to link back to the UI - User asks to "open" or "view" any of the above in Arize ## Required Inputs Collect from the user or context (exported trace data, parsed URLs): | Always required | Resource-specific | |---|---| | `org_id` (base64) | `project_id` + `trace_id` [+ `span_id`] — trace/span | | `space_id` (base64) | `project_id` + `session_id` — session | | | `dataset_id` — dataset | | | `queue_id` — specific queue (omit for list) | | | `evaluator_id` [+ `version`] — evaluator | **All path IDs must be base64-encoded** (characters: `A-Za-z0-9+/=`). A raw numeric ID produces a valid-looking URL that 404s. If the user provides a number, ask them to copy the ID directly from their Arize browser URL (`https://app.arize.com/organizations/{org_id}/spaces/{space_id}/…`). If you have a raw internal ID (e.g. `Organization:1:abC1`), base64-encode it before inserting into the URL. ## URL Templates Base URL: `https://app.arize.com` (override for on-prem) **Trace** (add `&selectedSpanId={span_id}` to highlight a specific span): ``` {base_url}/organizations/{org_id}/spaces/{space_id}/projects/{project_id}?selectedTraceId={trace_id}&queryFilterA=&selectedTab=llmTracing&timeZoneA=America%2FLos_Angeles&startA={start_ms}&endA={end_ms}&envA=tracing&modelType=generative_llm ``` **Session:** ``` {base_url}/organizations/{org_id}/spaces/{space_id}/projects/{project_id}?selectedSessionId={session_id}&queryFilterA=&selectedTab=llmTracing&timeZoneA=America%2FLos_Angeles&startA={start_ms}&endA={end_ms}&envA=tracing&modelType=generative_llm ``` **Dataset** (`selectedTab`: `examples` or `experiments`): ``` {base_url}/organizations/{org_id}/spaces/{space_id}/datasets/{dataset_id}?selectedTab=examples ``` **Queue list / specific queue:** ``` {base_url}/organizations/{org_id}/spaces/{space_id}/queues {base_url}/organizations/{org_id}/spaces/{space_id}/queues/{queue_id} ``` **Evaluator** (omit `?version=…` for latest): ``` {base_url}/organizations/{org_id}/spaces/{space_id}/evaluators/{evaluator_id} {base_url}/organizations/{org_id}/spaces/{space_id}/evaluators/{evaluator_id}?version={version_url_encoded} ``` The `version` value must be URL-encoded (e.g., trailing `=` → `%3D`). **Annotation configs:** ``` {base_url}/organizations/{org_id}/spaces/{space_id}/a [truncated — 1481 more characters] ``` ### .agents/skills/arize-ai-provider-integration/SKILL.md ```markdown --- name: arize-ai-provider-integration description: Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize. metadata: author: arize version: "1.0" compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize AI Integration Skill > **`SPACE`** — Most `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`. > **Note:** `ai-integrations create` does **not** accept `--space` — AI integrations are account-scoped. Use `--space` only with `list`, `get`, `update`, and `delete`. ## Concepts - **AI Integration** = stored LLM provider credentials registered in Arize; used by evaluators to call a judge model and by other Arize features that need to invoke an LLM on your behalf - **Provider** = the LLM service backing the integration (e.g., `openAI`, `anthropic`, `awsBedrock`) - **Integration ID** = a base64-encoded global identifier for an integration (e.g., `TGxtSW50ZWdyYXRpb246MTI6YUJjRA==`); required for evaluator creation and other downstream operations - **Scoping** = visibility rules controlling which spaces or users can use an integration - **Auth type** = how Arize authenticates with the provider: `default` (provider API key), `proxy_with_headers` (proxy via custom headers), or `bearer_token` (bearer token auth) ## Prerequisites Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront. If an `ax` command fails, troubleshoot based on the error: - `command not found` or version error → see references/ax-setup.md - `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys - Space unknown → run `ax spaces list` to pick by name, or ask the user - LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the **arize-ai-provider-integration** skill - **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user. --- ## List AI Integrations List all integrations accessible in a space: ```bash ax ai-integ [truncated — 7824 more characters] ``` ### frontend/package.json ``` { "name": "bun-react-template", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "bun --hot src/index.ts", "start": "NODE_ENV=production bun src/index.ts", "build": "bun run build.ts" }, "dependencies": { "@mantine/core": "^9.3.2", "@mantine/hooks": "^9.3.2", "@tanstack/react-query": "^5.101.0", "@tanstack/react-table": "^8.21.3", "react": "^19", "react-dom": "^19", "react-icons": "^5.6.0" }, "devDependencies": { "@types/react": "^19", "@types/react-dom": "^19", "@types/bun": "latest" } } ``` ### backend/pyproject.toml ``` [project] name = "backend" version = "0.1.0" description = "Low-resource language dataset preservation backend" readme = "README.md" requires-python = ">=3.13" dependencies = [ "alembic>=1.18.4", "anthropic>=0.111.0", "arize-otel>=0.13.0", "flask>=3.0.0", "flask-cors>=4.0.0", "httpx>=0.27.0", "openinference-instrumentation-anthropic>=1.0.6", "opentelemetry-api>=1.25.0", "opentelemetry-exporter-otlp>=1.25.0", "opentelemetry-sdk>=1.25.0", "psycopg2-binary>=2.9.12", "pydantic>=2.8.0", "pydantic-settings>=2.14.2", "python-dotenv>=1.2.2", "sqlmodel>=0.0.38", ] [dependency-groups] dev = [ "pytest>=8.3.0", ] [tool.pytest.ini_options] pythonpath = ["."] testpaths = ["tests"] ``` ### backend/requirements.txt ``` # This file was autogenerated by uv via the following command: # uv pip compile pyproject.toml -o requirements.txt alembic==1.18.4 # via backend (pyproject.toml) annotated-types==0.7.0 # via pydantic anthropic==0.111.0 # via backend (pyproject.toml) anyio==4.14.0 # via # anthropic # httpx arize-otel==0.13.0 # via backend (pyproject.toml) blinker==1.9.0 # via flask certifi==2026.6.17 # via # httpcore # httpx # requests charset-normalizer==3.4.7 # via requests click==8.4.1 # via flask distro==1.9.0 # via anthropic docstring-parser==0.18.0 # via anthropic flask==3.1.3 # via # backend (pyproject.toml) # flask-cors flask-cors==6.0.5 # via backend (pyproject.toml) googleapis-common-protos==1.75.0 # via # opentelemetry-exporter-otlp-proto-grpc # opentelemetry-exporter-otlp-proto-http grpcio==1.81.1 # via opentelemetry-exporter-otlp-proto-grpc h11==0.16.0 # via httpcore httpcore==1.0.9 # via httpx httpx==0.28.1 # via # backend (pyproject.toml) # anthropic idna==3.18 # via # anyio # httpx # requests itsdangerous==2.2.0 # via flask jinja2==3.1.6 # via flask jiter==0.15.0 # via anthropic mako==1.3.12 # via alembic markupsafe==3.0.3 # via # flask # jinja2 # mako # werkzeug openinference-instrumentation==0.1.53 # via # arize-otel # openinference-instrumentation-anthropic openinference-instrumentation-anthropic==1.0.6 # via backend (pyproject.toml) openinference-semantic-conventions==0.1.30 # via # arize-otel # openinference-instrumentation # openinference-instrumentation-anthropic opentelemetry-api==1.42.1 # via # backend (pyproject.toml) # openinference-instrumentation # openinference-instrumentation-anthropic # opentelemetry-exporter-otlp-proto-grpc # opentelemetry-exporter-otlp-proto-http # opentelemetry-instrumentation # opentelemetry-sdk # opentelemetry-semantic-conventions opentelemetry-exporter-otlp==1.42.1 # via # backend (pyproject.toml) # arize-otel opentelemetry-exporter-otlp-proto-common==1.42.1 # via # opentelemetry-exporter-otlp-proto-grpc # opentelemetry-exporter-otlp-proto-http opentelemetry-exporter-otlp-proto-grpc==1.42.1 # via opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-http==1.42.1 # via opentelemetry-exporter-otlp opentelemetry-instrumentation==0.63b1 # via openinference-instrumentation-anthropic opentelemetry-proto==1.42.1 # via # arize-otel # opentelemetry-exporter-otlp-proto-common # opentelemetry-exporter-otlp-proto-grpc # opentelemetry-exporter-otlp-proto-http opentelemetry-sdk==1.42.1 # via # backend (pyproject.toml) # arize-otel # openinference-instrumentation # opentelemetry-exporter-otlp-proto-grpc # opentelemetry-exporter-otlp-proto-http opentelemetry-semantic-conventions==0.63b1 # via # openinference-instrumentation-anthropic # opentelemetry-instrumentation # opentelemetry-sdk packaging==26.2 # via opentelemetry-instrumentation protobuf==6.33.6 # via # googleapis-common-protos # opentelemetry-proto psycopg2-binary==2.9.12 # via backend (pyproject.toml) pydantic==2.13.4 # via # backend (pyproject.toml) # anthropic # pydantic-settings # sqlmodel pydantic-core==2.46.4 # via pydantic pydantic-settings==2.14.2 # via backend (pyproject.toml) python-dotenv==1.2.2 # via # backend (pyproject.toml) # pydantic-settings requests==2.34.2 # via opentelemetry-exporter-otlp-proto-http sniffio==1.3.1 # via anthropic sqlalchemy==2.0.51 # via # alembic # sqlmodel sqlmodel==0.0.38 # via backend (pyproject.toml) typing-extensions==4.15.0 # via # alembic # anthropic # grpcio # openinference-instrumentation-anthropic # opentelemetry-api # opentelemetry-exporter-otlp-proto-grpc # opentelemetry-exporter-otlp-proto-http # opentelemetry-sdk # opentelemetry-semantic-conventions # pydantic # pydantic-core # sqlalchemy # sqlmodel # typing-inspection typing-inspection==0.4.2 # via # pydantic # pydantic-settings urllib3==2.7.0 # via requests werkzeug==3.1.8 # via # flask # flask-cors wrapt==2.2.2 # via # openinference-instrumentation # openinference-instrumentation-anthropic # opentelemetry-instrumentation ``` ### backend/main.py ```python from app import create_app app = create_app() def main() -> None: app.run(host="0.0.0.0", port=8000, debug=True) if __name__ == "__main__": main() ``` ### frontend/src/index.ts ```typescript import { serve } from "bun"; import index from "./index.html"; const server = serve({ routes: { // Serve index.html for all unmatched routes. "/*": index, "/api/hello": { async GET(req) { return Response.json({ message: "Hello, world!", method: "GET", }); }, async PUT(req) { return Response.json({ message: "Hello, world!", method: "PUT", }); }, }, "/api/hello/:name": async req => { const name = req.params.name; return Response.json({ message: `Hello, ${name}!`, }); }, }, development: process.env.NODE_ENV !== "production" && { // Enable browser hot reloading in development hmr: true, // Echo console logs from the browser to the server console: true, }, }); console.log(`🚀 Server running at ${server.url}`); ``` ### frontend/src/App.tsx ```typescript import { Grid, Paper, Tabs } from "@mantine/core"; import { TbDatabase, TbFileText, TbTags, TbUpload, TbWand, } from "react-icons/tb"; import { DeleteDatasetModal } from "@/components/layout/DeleteDatasetModal"; import { DetailModal } from "@/components/layout/DetailModal"; import { Sidebar } from "@/components/layout/Sidebar"; import { ToastBanner } from "@/components/layout/ToastBanner"; import { WorkspaceShell } from "@/components/layout/WorkspaceShell"; import { EmptyState } from "@/components/ui/EmptyState"; import { LoadingBlock } from "@/components/ui/LoadingBlock"; import { UI } from "@/lib/constants"; import { useWorkspaceController } from "@/hooks/useWorkspaceController"; import { JobsPanel } from "@/features/jobs/JobsPanel"; import { OcrSuggestionsTable } from "@/features/labels/OcrSuggestionsTable"; import { PosSuggestionsTable } from "@/features/labels/PosSuggestionsTable"; import { TranslationTable } from "@/features/labels/TranslationTable"; import { ModelsPanel } from "@/features/models/ModelsPanel"; import { ResearchPanel } from "@/features/research/ResearchPanel"; import { UploadTab } from "@/features/upload/UploadTab"; export function App() { const workspace = useWorkspaceController(); return ( <> workspace.setDeleteTarget(null)} onDelete={dataset => void workspace.deleteDataset(dataset)} /> workspace.setSelectedDetail(null)} /> void workspace.createDataset()} onDatasetNameChange={workspace.setDatasetName} onLanguageCodeChange={workspace.setLanguageCode} onLanguageNameChange={workspace.setLanguageName} onSelectDataset={workspace.setSelectedDatasetId} onSetAddLanguageFormOpen={workspace.setAddLanguageFormOpen} onSetDeleteTarget={workspace.setDeleteTarget} onSetSidebarCollapsed={workspace.setSidebarCollapsed} /> } > {workspace.toast ? ( workspace.setToast(null)} /> ) : null} {workspace.selectedDataset ? ( <> {workspace.activeTab === "pos" || workspace.activeTab === "translate" ? ( void workspace.runResearch(false)} onRefreshResearch={() => void workspace.runResearch(true)} onTypeChange={workspace.setActiveResearchType} /> ) : null} void workspace.generatePosSuggestions()} onOpenDetail={workspace.openDetail} onPageChange={workspace.setPosRowsPage} onReviewFilterChange={workspace.setPosReviewFilter} onReview={(suggestion, action) => void workspace.reviewSuggestion(suggestion, action)} onTokenChange={workspace.updateTokenDraft} />