# Project export: Fetch Health This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: AI assistant for organ transplant - Devpost: https://devpost.com/software/fetch-health - GitHub: https://github.com/abhinavprkash/Fetch-Health.git - Video: https://www.youtube.com/embed/1Fm5qSy5vFM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Best Use of The Agentverse by Fetch AI) - Team: 2 GitHub contributor(s) — Abhinav (8 commits), Claude Opus 4.8 (1M context) (3 commits) ## Devpost submission (written by the team) ### Inspiration Organ matching is one of the highest-stakes workflows in healthcare, but the process is hard to understand, hard to simulate, and difficult to explain clearly to non-specialists. Hospital staff often deal with fragmented medical documents, imaging files, reports, and patient history while trying to make sense of compatibility, urgency, logistics, and risk. We wanted to build something that makes this workflow more transparent, educational, and easier to reason about. Fetch Health was inspired by the idea of combining agentic AI, medical document understanding, imaging analysis, and explainable scoring into one simple hospital-facing workflow. Our goal was not to replace doctors or clinical systems. Our goal was to create a safe educational and simulation platform that shows how donor-recipient matching could be analyzed, explained, and visualized with modern AI infrastructure to save lives. ### What it does Fetch Health is a hospital-staff-facing donor and recipient intake and matching simulator. The interface is intentionally simple. Staff choose whether they are uploading information for a patient/recipient or a donor, upload medical files, and let the system process the case. For a patient/recipient case, Fetch Health analyzes uploaded documents, imaging, videos, and structured records, then returns the top three donor candidates with: compatibility percentage organ type estimated transport time transportation and medical risk missing information warnings collapsible candidate reports agent-generated reasoning summaries For a donor case, Fetch Health evaluates whether the donor profile can be added to the donor database. If accepted, the system confirms the donor profile was added and shows a respectful donor-status message. If not eligible, it explains that the donor is not currently eligible based on the available information. Behind the scenes, the system uses local agents for document analysis, image/video processing, chunking, profile creation, Redis-backed memory, candidate retrieval, compatibility scoring, ranking, transport risk, benchmarking, reporting, and voice debrief generation. Fetch.ai agents act as the public agent layer, while the heavy processing runs on our local backend infrastructure. ### How we built it We built Fetch Health as a modular agentic healthcare simulation system. The frontend is a simple hospital-staff interface with three screens: intake upload screen loading/processing screen result screen The backend is built around a local HPC-first architecture. Uploaded files are processed by a Master Orchestrator Agent, which coordinates local agents for documents, images, videos, profile building, compatibility scoring, ranking, benchmarking, and reporting. We used Redis as the core state and memory layer. Redis stores profile state, run status, agentic memory, stream events, rankings, document chunks, embeddings, and traceable agent outputs. Every agent writes a structured memory record so the system can explain what happened during a run. We designed a context-compacting layer for agent-to-agent handoff. Instead of passing huge raw context between agents, each agent emits a compact “context capsule” containing hard facts, findings, risks, missing fields, and references. Long narrative context can be compressed before report generation, while critical facts like IDs, scores, rankings, organ type, and risk values are preserved exactly. We used Fetch.ai as the public capability layer. The Fetch agents are lightweight routers that expose capabilities like intake, document intelligence, image intelligence, candidate retrieval, compatibility scoring, ranking, reports, and health checks. These agents route requests to the local backend instead of trying to run heavy medical processing themselves. We also integrated sponsor tools into the workflow: Fetch.ai for public agent orchestration and mailbox-style routing Redis for state, memory, rankings, traces, and retrieval Anthropic for final reasoning and report generation The Token Company conceptually for context compacting and token efficiency Sentry for monitoring backend, agent, and tool failures Deepgram for final voice debriefs MidJourney for UI/UX visual direction and design inspiration The system is designed as an educational and simulation workflow, not a clinical decision engine. ### Challenges we ran into One major challenge was balancing ambition with usability. Organ matching is complex, but hospital staff need a simple interface. We kept the frontend minimal and moved complexity into the backend agents. Another challenge was data structure. We needed profiles that could support documents, images, videos, reports, segmentation outputs, and matching metadata without pretending composite data represented a real patient. We solved this by separating public profile summaries, private artifacts, provenance, and agent memory. Agent orchestration was also difficult. If every agent receives every file and every trace, the system becomes noisy and inefficient. We solved this with Redis-backed context capsules and structured handoffs, so each agent receives only the information it needs. We also had to handle missing or incomplete medical information responsibly. Instead of fabricating data, the system tracks missing fields, lowers confidence, and surfaces warnings in the final report. Finally, coordinating Fetch.ai agents with local HPC processing required a clean separation: Fetch agents are public routers, while local agents do the heavy processing. ### Accomplishments we're proud of We are proud that Fetch Health turns a complex healthcare workflow into a simple three-screen hospital-facing experience. We built a system where every step is traceable. Each agent writes memory, each run has events, and each result can be explained through structured reports. We are also proud of the context-compacting design. Instead of blindly dumping large documents and traces into an LLM, we separate hard facts from compressible context. This makes the agent pipeline more efficient, more reliable, and easier to debug. Another accomplishment is the modular architecture. Document analysis, imaging, video, retrieval, scoring, ranking, reporting, monitoring, and voice output are all separated into agents that can improve independently. Most importantly, we kept the system safe. Fetch Health is framed as an educational simulation and decision-support prototype, not a replacement for doctors or transplant allocation systems. ### What we learned We learned that good agent systems are not just about having many agents. They are about having the right boundaries between agents. We learned that Redis is extremely useful as an agent memory and state layer because it can store run status, events, rankings, vectors, chunks, traces, and summaries in one fast system. We also learned that healthcare AI needs transparency. A compatibility score alone is not enough. Users need to know why a profile ranked highly, what information was missing, and where the system had uncertainty. We learned that frontend simplicity matters. A powerful backend means nothing if hospital staff cannot use the product quickly and confidently. Finally, we learned that context management is one of the hardest parts of agentic AI. Passing less context, but better context, leads to better outputs. ### What's next Next, we want to expand Fetch Health in several directions. First, we want to improve the profile database with more real open-source medical imaging, reports, and benchmark cases across brain, lung, heart, kidney, and liver workflows. Second, we want to strengthen the compatibility scoring system with better baselines, better evaluation, and clearer confidence scoring. Third, we want to improve the Fetch.ai agent layer so hospital staff and educators can interact with the system through ASI:One-style agent workflows, not only the web interface. Fourth, we want to expand the voice debrief system so doctors, educators, and trainees can ask follow-up questions about a report. Finally, we want to make Fetch Health a stronger educational simulator for medical students and transplant teams, where users can explore “what-if” cases, compare candidate rankings, and understand how different compatibility factors affect the final result. Fetch Health is not trying to replace clinical judgment. It is trying to make complex medical matching workflows more understandable, traceable, and teachable. ## README (from the GitHub repository) # Fetch Health Educational organ donor/recipient **matching simulator**. A swarm of public Fetch.ai capability agents fronts a private FastAPI "HPC" gateway that does the real work (document/image/video analysis, retrieval, transparent rule-based matching, reporting). Six sponsor technologies are integrated honestly — no faked clinical output, ever. > **Educational simulation and research workflow only. Not for clinical > decision-making. No real patient data.** ## Sponsor integration map | Sponsor | Where | What it does | |---|---|---| | **Fetch.ai** | `swarm/` | 80 generated uAgents (mailbox, `publish_agent_details`) + a selective-activation Master Orchestrator harness, discoverable on ASI:One. | | **Redis** | `backend/app/repositories/candidate_store.py` | Local redis-stack RediSearch **vector KNN** for candidate retrieval + edu **semantic cache**; Python-cosine fallback when the module is absent. | | **The Token Company** | `backend/app/integrations/llm.py` | Compresses every prompt before Claude; exposes `tokens_saved` / `compression_ratio`. | | **Anthropic Claude** | `backend/app/integrations/llm.py` | Real field extraction + edu explanations + report narration. Default `claude-opus-4-8` (no sampling params — they 400 on Opus 4.8). | | **Deepgram** | `backend/app/integrations/audio.py` | Transcribes uploaded audio / video narration into the document pipeline. | | **Midjourney** | `backend/app/integrations/imagegen.py` | Edu-mode diagrams via a proxy API; **pre-generate + cache**; degrades to none without a key. | | **Sentry** | `backend/app/integrations/observability.py` | Error monitoring + tracing across gateway, bureau, harness. | Every integration **degrades gracefully without its key** — it returns a truthful "not configured" signal rather than fabricating output. ## Architecture (5 layers) 1. **Public Fetch agents** — `swarm/generated/` (80 thin routers, generated from YAML). 2. **Harness** — `swarm/harness.py` + `swarm/activation_manager.py`: the single agent ASI:One talks to; activates only the agents a task needs (resource allocation). 3. **HPC gateway** — `backend/` FastAPI: run lifecycle, uploads, `/api/agents/{name}`, `/api/edu/explain`. Returns truthful statuses (`not_implemented` names the missing worker). 4. **Local Bureau** — `local_agents/`: ~15 internal workers (`document_parser` real, rest truthful stubs). 5. **Shared integrations** — `backend/app/integrations/` (llm/audio/imagegen/observability/redis_search). ## Setup ```bash cd backend && python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cd .. && cp .env.example .env # fill in keys you have (all optional to start) docker compose up -d redis # local Redis Stack with RediSearch/vector KNN ``` ## Commands ```bash scripts/start_gateway.sh # FastAPI gateway (:8000) scripts/start_local_bureau.sh # local Bureau (needs uagents) scripts/start_harness.sh # Master Orchestrator harness (needs uagents) scripts/start_fetch_agents.sh # run a demo subset of public agents scripts/smoke_test.sh # validate -> generate -> compile -> route (no keys needed) python configs/_build_specs.py # regenerate agent_specs.yaml python swarm/validate_agent_specs.py # validate specs python swarm/generate_agents.py # (re)generate the 80 agents cd backend && pytest # 24 tests ``` ## Example Request → gateway: ```bash curl -X POST localhost:8000/api/agents/master-orchestrator \ -H 'content-type: application/json' \ -d '{"request_id":"r1","task_type":"orchestration.master_orchestrator"}' ``` Response (truthful — a run is queued, not faked complete): ```json {"status":"queued","request_id":"r1","run_id":"run_…","agent_name":"master-orchestrator", "message":"Run run_… queued. Start it via POST /api/runs/run_…/start.", "next_status_url":"/api/runs/run_…/status"} ``` Edu slice (returns `not_implemented` until `ANTHROPIC_API_KEY` is set): ```bash curl -X POST localhost:8000/api/edu/explain -H 'content-type: application/json' \ -d '{"question":"How does HLA matching work?","with_diagram":true}' ``` ## Generated agents 80 agents across 8 categories (intake, document, image, video, retrieval, matching, benchmark, orchestration). See `swarm/generated/manifest.json` and per-agent Agentverse READMEs in `swarm/generated/*.md`. ## Missing implementation (truthful TODO) - Image/video workers return `not_implemented` until the local analysis models are wired. - Midjourney proxy provider/endpoint to confirm (`_submit_and_wait` has a TODO). - Matching beyond blood/organ is rule-based scaffolding; extend in `services/retrieval.py`. - Bureau stub workers (`upload_manager`, `chunking_agent`, …) return `not_implemented`. - uAgents runtime not installed in the default venv — `pip install -r requirements.txt` to run the swarm/bureau/harness (the gateway + factory + smoke test run without it). ## Detected evidence (automated analysis) Indexed codebase: 1421 recognized source files, 1396 KB. - Anthropic (technology) — detected in the code - FastAPI (technology) — detected in the code - Python (language) — detected in the code - Redis (technology) — detected in the code - React (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 8472) ``` .agents/skills/iris-development/.cursor-plugin/plugin.json .agents/skills/iris-development/references/ltm-bulk-create.md .agents/skills/iris-development/references/ltm-organize.md .agents/skills/iris-development/references/ltm-search.md .agents/skills/iris-development/references/promotion-overview.md .agents/skills/iris-development/references/session-add-event.md .agents/skills/iris-development/references/session-retrieval.md .agents/skills/iris-development/references/session-when-to-use.md .agents/skills/iris-development/references/setup-auth-token.md .agents/skills/iris-development/references/setup-cloud-service.md .agents/skills/iris-development/SKILL.md .agents/skills/redis-clustering/.cursor-plugin/plugin.json .agents/skills/redis-clustering/references/hash-tags.md .agents/skills/redis-clustering/references/read-replicas.md .agents/skills/redis-clustering/SKILL.md .agents/skills/redis-connections/.cursor-plugin/plugin.json .agents/skills/redis-connections/references/blocking.md .agents/skills/redis-connections/references/client-cache.md .agents/skills/redis-connections/references/pipelining.md .agents/skills/redis-connections/references/pooling.md .agents/skills/redis-connections/references/timeouts.md .agents/skills/redis-connections/SKILL.md .agents/skills/redis-core/.cursor-plugin/plugin.json .agents/skills/redis-core/evals/core/baselines/aggregate-benchmark.json .agents/skills/redis-core/evals/core/baselines/aggregate-benchmark.md .agents/skills/redis-core/evals/core/baselines/baseline.json .agents/skills/redis-core/evals/core/baselines/model-matrix.json .agents/skills/redis-core/evals/core/baselines/README.md .agents/skills/redis-core/evals/core/evals.json .agents/skills/redis-core/evals/core/model-matrix.json .agents/skills/redis-core/references/choose-data-structure.md .agents/skills/redis-core/references/key-naming.md .agents/skills/redis-core/SKILL.md .agents/skills/redis-observability/.cursor-plugin/plugin.json .agents/skills/redis-observability/references/commands.md .agents/skills/redis-observability/references/metrics.md .agents/skills/redis-observability/SKILL.md .agents/skills/redis-query-engine/.cursor-plugin/plugin.json .agents/skills/redis-query-engine/references/dialect.md .agents/skills/redis-query-engine/references/field-types.md .agents/skills/redis-query-engine/references/index-creation.md .agents/skills/redis-query-engine/references/index-management.md .agents/skills/redis-query-engine/references/query-optimization.md .agents/skills/redis-query-engine/references/skip-initial-scan.md .agents/skills/redis-query-engine/SKILL.md .agents/skills/redis-security/.cursor-plugin/plugin.json .agents/skills/redis-security/references/acls.md .agents/skills/redis-security/references/auth.md .agents/skills/redis-security/references/network.md .agents/skills/redis-security/SKILL.md .agents/skills/redis-semantic-cache/.cursor-plugin/plugin.json .agents/skills/redis-semantic-cache/references/best-practices.md .agents/skills/redis-semantic-cache/references/langcache-usage.md .agents/skills/redis-semantic-cache/SKILL.md .agents/skills/redis-vector-search/.cursor-plugin/plugin.json .agents/skills/redis-vector-search/references/algorithm-choice.md .agents/skills/redis-vector-search/references/hybrid-search.md .agents/skills/redis-vector-search/references/index-creation.md .agents/skills/redis-vector-search/references/rag-pattern.md .agents/skills/redis-vector-search/SKILL.md .env.example .gitattributes .gitignore AGENTS.md backend/app/__init__.py backend/app/config.py backend/app/constants.py backend/app/integrations/__init__.py backend/app/integrations/audio.py backend/app/integrations/imagegen.py backend/app/integrations/llm.py backend/app/integrations/observability.py backend/app/integrations/redis_search.py backend/app/main.py backend/app/processors/__init__.py backend/app/processors/document.py backend/app/processors/image.py backend/app/processors/structured.py backend/app/processors/video.py backend/app/redis_client.py backend/app/repositories/__init__.py backend/app/repositories/candidate_store.py backend/app/repositories/profile_store.py backend/app/repositories/run_store.py backend/app/routes/__init__.py backend/app/routes/agents.py backend/app/routes/edu.py backend/app/routes/profiles.py backend/app/routes/runs.py backend/app/routes/simulate.py backend/app/routes/uploads.py backend/app/schemas.py backend/app/services/__init__.py backend/app/services/deidentify.py backend/app/services/donor_matching.py backend/app/services/edu_service.py backend/app/services/llm.py backend/app/services/retrieval.py backend/app/services/run_service.py backend/Dockerfile backend/pyproject.toml backend/README.md backend/requirements.txt backend/scripts/__init__.py backend/scripts/load_factory_data.py backend/scripts/seed_candidates.py backend/storage/parsed/.gitkeep backend/storage/results/.gitkeep backend/storage/uploads/.gitkeep backend/tests/__init__.py backend/tests/test_candidate_store.py backend/tests/test_donor_matching.py backend/tests/test_edu_llm_failures.py backend/tests/test_profiles.py backend/tests/test_runs.py backend/tests/test_uploads.py configs/_build_specs.py configs/agent_specs.yaml data/data_pipeline/.gitignore data/data_pipeline/configs/artifact_schema.yaml [8352 more files omitted for size] ``` ### Dependencies - backend/requirements.txt: anthropic@>=0.40.0, fastapi@>=0.110.0, httpx@>=0.27.0, jinja2@>=3.1.0, pydantic@>=2.6.0, pydantic-settings@>=2.2.0, pypdf@>=4.0.0, pytest@>=8.0.0, pytest-asyncio@>=0.23.0, python-docx@>=1.1.0, python-multipart@>=0.0.9, PyYAML@>=6.0.0, redis@>=5.0.0, sentry-sdk@>=2.0.0, the-token-company@>=0.1.0, uagents@>=0.12.0, uvicorn[standard]@>=0.29.0 - data/data_pipeline/requirements.txt: pytest@>=8.0, PyYAML@>=6.0 - data/transplant_data_factory/requirements.txt: pytest@>=8.0, PyYAML@>=6.0 ### Recent commits (newest first) - feat: add donor matching service, swarm agent updates, and backend enhancements - Add simulate match endpoint, factory loader, Redis hardening - Use ASI One for Fetch Health LLM - Add Fetch Health Agentverse chat profile - Update README.md - Merge chetas-dev into main - Merge pull request #1 from abhinavprkash/Abhinav_Work - Add sponsor integrations and Fetch swarm - feat(data): deep profile-first transplant data factory + earlier data pipeline - feat(backend): donor/recipient profiles, agent memory, AI integration stubs - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENTS.md ```markdown # AGENTS.md Drop-in operating instructions for coding agents. Read this file before every task. **Working code only. Finish the job. Plausibility is not correctness.** This file follows the [AGENTS.md](https://agents.md) open standard (Linux Foundation / Agentic AI Foundation). Claude Code, Codex, Cursor, Windsurf, Copilot, Aider, Devin, Amp read it natively. For tools that look elsewhere, symlink: ```bash ln -s AGENTS.md CLAUDE.md ln -s AGENTS.md GEMINI.md ``` --- ## 0. Non-negotiables These rules override everything else in this file when in conflict: 1. **No flattery, no filler.** Skip openers like "Great question", "You're absolutely right", "Excellent idea", "I'd be happy to". Start with the answer or the action. 2. **Disagree when you disagree.** If the user's premise is wrong, say so before doing the work. Agreeing with false premises to be polite is the single worst failure mode in coding agents. 3. **Never fabricate.** Not file paths, not commit hashes, not API names, not test results, not library functions. If you don't know, read the file, run the command, or say "I don't know, let me check." 4. **Stop when confused.** If the task has two plausible interpretations, ask. Do not pick silently and proceed. 5. **Touch only what you must.** Every changed line must trace directly to the user's request. No drive-by refactors, reformatting, or "while I was in there" cleanups. --- ## 1. Before writing code **Goal: understand the problem and the codebase before producing a diff.** - State your plan in one or two sentences before editing. For anything non-trivial, produce a numbered list of steps with a verification check for each. - Read the files you will touch. Read the files that call the files you will touch. Claude Code: use subagents for exploration so the main context stays clean. - Match existing patterns in the codebase. If the project uses pattern X, use pattern X, even if you'd do it differently in a greenfield repo. - Surface assumptions out loud: "I'm assuming you want X, Y, Z. If that's wrong, say so." Do not bury assumptions inside the implementation. - If two approaches exist, present both with tradeoffs. Do not pick one silently. Exception: trivial tasks (typo, rename, log line) where the diff fits in one sentence. --- ## 2. Writing code: simplicity first **Goal: the minimum code that solves the stated problem. Nothing speculative.** - No features beyond what was asked. - No abstractions for single-use code. No configurability, flexibility, or hooks that were not requested. - No error handling for impossible scenarios. Handle the failures that can actually happen. - If the solution runs 200 lines and could be 50, rewrite it before showing it. - If you find yourself adding "for future extensibility", stop. Future extensibility is a future decision. - Bias toward deleting code over adding code. Shipping less is almost always better. The test: would a senior engineer reading the diff call this overcomplicated? If y [truncated — 9084 more characters] ``` ### deploy/local.md ```markdown # Local deploy (laptop runs the gateway; agent is laptop-free on Agentverse) The gateway runs on your laptop and is exposed to the internet via a tunnel so the Agentverse **hosted** agent (in Fetch.ai cloud) can reach it. No cloud host needed. ``` ASI:One → Agentverse hosted agent → tunnel (https) → local gateway (:8000) → Redis + Claude/TTC/Deepgram/... ``` ## Steps 1. **Gateway** (laptop): ```bash scripts/start_gateway.sh # serves on :8000 curl http://localhost:8000/ # expect {"status":"ok",...} ``` Put real keys in `.env` for real output (else truthful `not_implemented`). 2. **Tunnel** the local gateway to a public HTTPS URL: ```bash ngrok http 8000 # or: cloudflared tunnel --url http://localhost:8000 ``` Copy the `https://...` URL. 3. **Hosted agent** (`swarm/agentverse_hosted_agent.py`): set `GATEWAY_URL` to that tunnel URL (edit the constant or add an Agentverse Secret `GATEWAY_URL`), paste into Agentverse → Hosted → Run. Chat from ASI:One. ## Optional: run the gateway in local Docker instead of uvicorn ```bash docker build -t fetch-health-gateway backend/ docker run -p 8000:8000 --env-file .env fetch-health-gateway ``` ## Notes - Laptop must stay on while the gateway runs (that's the "local for now" tradeoff). - Free ngrok URL changes each restart → update `GATEWAY_URL`. A reserved ngrok domain or named cloudflared tunnel gives a stable URL. - Going fully laptop-free later = run this same container on any host; re-point `GATEWAY_URL` at its public URL. ``` ### docker-compose.yml ```yaml # Local Redis Stack with the RediSearch module for vector KNN. # This is the default Fetch Health matching store for development. services: redis: image: redis/redis-stack:latest ports: - "6379:6379" # Redis - "8001:8001" # RedisInsight UI volumes: - redis_data:/data volumes: redis_data: ``` ### backend/pyproject.toml ``` [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["tests"] ``` ### backend/requirements.txt ``` fastapi>=0.110.0 uvicorn[standard]>=0.29.0 pydantic>=2.6.0 pydantic-settings>=2.2.0 redis>=5.0.0 python-multipart>=0.0.9 httpx>=0.27.0 pytest>=8.0.0 pytest-asyncio>=0.23.0 # --- Sponsor integrations + Fetch agent layer --- uagents>=0.12.0 anthropic>=0.40.0 the-token-company>=0.1.0 sentry-sdk>=2.0.0 jinja2>=3.1.0 PyYAML>=6.0.0 # --- Real document parsing --- pypdf>=4.0.0 python-docx>=1.1.0 ``` ### backend/Dockerfile ``` # Gateway container (optional). Run locally or on any container host. # Build context = backend/. Build: docker build -t fetch-health-gateway backend/ # docker run -p 8000:8000 --env-file ../.env fetch-health-gateway FROM python:3.12-slim WORKDIR /app # System deps for pypdf/python-docx are pure-Python; nothing extra needed. COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . ENV STORAGE_ROOT=/data/storage EXPOSE 8000 # 0.0.0.0 so RunPod's HTTP proxy can reach it. CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] ``` ### data/transplant_data_factory/requirements.txt ``` # Core (required) PyYAML>=6.0 # Optional — used only when real datasets are present locally. # pydicom>=2.4 # DICOM metadata # nibabel>=5.0 # NIfTI metadata # Pillow>=10.0 # image previews/thumbnails # opencv-python>=4.8 # video thumbnails (EchoNet) # Tests pytest>=8.0 ``` ### data/data_pipeline/requirements.txt ``` # Core (required) PyYAML>=6.0 # Optional — parsers degrade gracefully ("dataset_not_available") when absent. # Install only what you need for the real datasets you have locally. # pydicom>=2.4 # DICOM parsing (CT/MR/CXR) # nibabel>=5.0 # NIfTI parsing (MSD/AMOS/KiTS) # Pillow>=10.0 # image thumbnails / metadata # opencv-python>=4.8 # video thumbnails (EchoNet) # pandas>=2.0 # large CSV/STAR ingestion # Tests pytest>=8.0 ``` ### backend/app/main.py ```python from __future__ import annotations import asyncio from contextlib import asynccontextmanager from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from app.config import settings from app.integrations.observability import init_sentry from app.repositories.candidate_store import InMemoryCandidateStore from app.repositories.profile_store import InMemoryProfileStore from app.repositories.run_store import InMemoryRunStore from app.routes import agents, edu, profiles, runs, simulate, uploads def _use_in_memory_stores(app: FastAPI) -> None: app.state.redis = None app.state.run_store = InMemoryRunStore() app.state.candidate_store = InMemoryCandidateStore() app.state.profile_store = InMemoryProfileStore() @asynccontextmanager async def lifespan(app: FastAPI): """Startup / shutdown lifecycle.""" # --- Startup --- init_sentry("gateway") if settings.USE_REDIS: import logging from app.redis_client import get_redis from app.repositories.run_store import RedisRunStore from app.repositories.candidate_store import RedisCandidateStore from app.repositories.profile_store import RedisProfileStore try: redis = await get_redis() await redis.ping() app.state.redis = redis app.state.run_store = RedisRunStore(redis) app.state.candidate_store = RedisCandidateStore(redis) app.state.profile_store = RedisProfileStore(redis) if settings.USE_REDISEARCH: await app.state.candidate_store.ensure_index() except Exception as exc: # noqa: BLE001 - local Redis can be off during dev logging.getLogger("gateway").warning( "Local Redis unavailable at %s (%s); using in-memory stores", settings.REDIS_URL, exc, ) from app.redis_client import close_redis await close_redis() _use_in_memory_stores(app) else: _use_in_memory_stores(app) # Guards the check-then-set in POST /start against concurrent double-starts. app.state.start_lock = asyncio.Lock() if settings.SEED_ON_STARTUP: from scripts.seed_candidates import seed await seed(app.state.candidate_store) if settings.LOAD_FACTORY_ON_STARTUP: # Warm the candidate store from real factory data so matches are instant # and reliable, independent of remote-Redis latency. Never crash on failure. import logging from scripts.load_factory_data import DEFAULT_SOURCE, FALLBACK_SOURCE, load_candidates try: source = DEFAULT_SOURCE if DEFAULT_SOURCE.exists() else FALLBACK_SOURCE limit = settings.FACTORY_DATA_LIMIT or None n = await load_candidates(app.state.candidate_store, source, limit=limit) logging.getLogger("gateway").info("Loaded %d factory candidates on startup", n) except Exception: # noqa: BLE001 - startup warm-load is best-effort logging.getLogger("gateway").exception("Factory warm-load failed; continuing") yield # --- Shutdown --- if settings.USE_REDIS: from app.redis_client import close_redis await close_redis() app = FastAPI( title="Fetch Health Matching Gateway", description="Async donor-recipient matching backend with run lifecycle, file ingestion, Redis/vector retrieval, and clinician reports.", version="0.1.0", lifespan=lifespan, ) app.add_middleware( CORSMiddleware, allow_origins=["*"], # Wildcard origin is incompatible with credentials per the CORS spec; keep # credentials off so the wildcard actually works in browsers. allow_credentials=False, allow_methods=["*"], allow_headers=["*"], ) app.include_router(runs.router) app.include_router(uploads.router) app.include_router(edu.router) app.include_router(agents.router) app.include_router(profiles.router) app.include_router(simulate.router) @app.get("/", tags=["health"]) async def health(): """Health check endpoint.""" return {"status": "ok", "service": "fetch-health-matching-gateway"} ``` ### data/data_pipeline/src/main.py ```python """TransplantTwin AI data pipeline CLI. Commands (run from the data_pipeline/ directory): python -m src.main discover-sources python -m src.main ingest --config configs/datasets.yaml python -m src.main build-profiles --target-total 2000 --donors 1000 --recipients 1000 python -m src.main build-test-profiles --count 500 --require-artifacts true python -m src.main validate python -m src.main export-redis python -m src.main report Each command persists to output/*.jsonl so commands compose across invocations. """ from __future__ import annotations import argparse import json import sys from pathlib import Path from src.common import ( INPUT, MANIFESTS_DIR, OUTPUT, load_yaml, read_jsonl, write_jsonl, ) from src.normalizers import artifact_linker from src.normalizers.account_builder import assign_accounts from src.normalizers.profile_builder import build_profile from src.normalizers.synthetic_source import generate_records # Output file locations ----------------------------------------------------- ACCOUNTS = OUTPUT / "accounts.jsonl" PROFILES = OUTPUT / "profiles.jsonl" DONORS = OUTPUT / "donor_profiles.jsonl" RECIPIENTS = OUTPUT / "recipient_profiles.jsonl" TEST_PROFILES = OUTPUT / "test_profiles.jsonl" ARTIFACTS = OUTPUT / "artifacts.jsonl" DOWNLOADER_MODULES = [ "srtr_loader", "optn_star_loader", "synthea_downloader", "mimic_loader", "kits_loader", "lidc_loader", "msd_downloader", "amos_downloader", "nih_chestxray_loader", "mimic_cxr_loader", "echonet_loader", ] # --- helpers --------------------------------------------------------------- def _load(path: Path) -> list[dict]: return list(read_jsonl(path)) def _split_and_write(profiles: list[dict]) -> None: write_jsonl(PROFILES, profiles) write_jsonl(DONORS, [p for p in profiles if p["profile_type"] == "donor"]) write_jsonl(RECIPIENTS, [p for p in profiles if p["profile_type"] == "recipient"]) def _synthetic_source_meta() -> dict: cfg = load_yaml("datasets.yaml") bs = cfg.get("builtin_synthetic", {}) return { "source_type": "synthetic_research_dataset", "source_dataset": "builtin_synthetic_generator", "source_url": bs.get("url_or_reference", "internal://builtin_synthetic_generator"), "license": bs.get("license_or_access", "Generated locally, demo/research use only"), "role_assignment_method": "synthetic_generator_rule", "composite_profile": False, "synthetic": True, "source_file": "builtin_synthetic_generator", } def _source_status() -> dict: cfg = load_yaml("datasets.yaml") status = {} for key, d in cfg.get("datasets", {}).items(): local = Path(d.get("local_path") or (INPUT / key)) has_files = local.exists() and any(local.rglob("*")) status[key] = { "name": d.get("name", key), "declared_available": bool(d.get("available")), "has_local_files": bool(has_files), "usable": bool(d.get("available")) and bool(has_files), "access": d.get("access"), "license_or_access": d.get("license_or_access"), } return status # --- commands -------------------------------------------------------------- def cmd_discover_sources(_args) -> int: status = _source_status() MANIFESTS_DIR.mkdir(parents=True, exist_ok=True) (MANIFESTS_DIR / "source_status.json").write_text(json.dumps(status, indent=2)) print("Source availability:") for key, s in status.items(): flag = "USABLE" if s["usable"] else "dataset_not_available" print(f" {key:18s} {flag:22s} ({s['access']})") print(" builtin_synthetic USABLE (open, synthetic fallback)") return 0 def cmd_ingest(args) -> int: config = load_yaml(args.config) if args.config else load_yaml("datasets.yaml") import importlib summary = {} for mod_name in DOWNLOADER_MODULES: mod = importlib.import_module(f"src.downloaders.{mod_name}") result = mod.load(config) summary[mod_name] = {"status": result.get("status"), "dataset": result.get("dataset"), "records": len(result.get("records", []))} print(f" {mod_name:22s} -> {result.get('status')}") (MANIFESTS_DIR / "ingest_summary.json").write_text(json.dumps(summary, indent=2)) print("No real datasets available -> profiles will come from the builtin synthetic " "generator (clearly labelled). Drop licensed data into input/ to activate real sources.") return 0 def cmd_build_profiles(args) -> int: meta = _synthetic_source_meta() profiles: list[dict] = [] idx = 0 for role, organ, record in generate_records(args.donors, args.recipients, seed=args.seed): idx += 1 profiles.append(build_profile( index=idx, account_id="account_pending", role=role, organ=organ, record=record, source_meta=meta, )) accounts = assign_accounts(profiles, seed=args.seed) _split_and_write(profiles) write_jsonl(ACCOUNTS, accounts) if not ARTIFACTS.exists(): write_jsonl(ARTIFACTS, []) total = len(profiles) print(f"Built {total} profiles " f"({sum(p['profile_type'] == 'donor' for p in profiles)} donor, " f"{sum(p['profile_type'] == 'recipient' for p in profiles)} recipient) " f"in {len(accounts)} accounts.") if args.target_total and total != args.target_total: print(f" note: total {total} != --target-total {args.target_total} " f"(total = donors + recipients).") return 0 def cmd_build_test_profiles(args) -> int: profiles = _load(PROFILES) if not profiles: print("No profiles found. Run build-profiles first.", file=sys.stderr) return 1 by_id = {p["profile_id"]: p for p in profiles} subset = profiles[:args.count] artifacts: list[dict] = [] art_idx = 0 [truncated — 5137 more characters] ``` [892 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]