Project Info

AuData - AI Copilot for Scientific Integrity

Inspiration

Science runs on trust, but that trust is getting harder to verify by hand. Some of the most consequential errors of the last two decades hid in plain sight: the 2006 Nature paper that helped anchor the amyloid hypothesis of Alzheimer's disease was found in 2022 to contain manipulated figure panels, after steering years of work and enormous funding. Stanford's president stepped down in 2023 over image-integrity problems in his labs' papers, and in 2024 Dana-Farber moved to correct or retract dozens of studies flagged for duplicated images. These are not one-offs. The automated tool statcheck found that roughly half of psychology papers contain at least one statistical reporting inconsistency, and Retraction Watch now tracks more than 50,000 retractions. At the same time, the volume of research has outrun our ability to read it. Millions of papers appear every year, PubMed alone adds over a million, and generative AI is accelerating both legitimate writing and outright fabrication, including industrial-scale paper mills. Peer reviewers are unpaid and overstretched, and no human can realistically recompute every p-value, re-derive every meta-analysis, or visually compare every figure against the prior literature. The community is starting to respond: venues like ICLR and NeurIPS have begun piloting AI-assisted peer review, and research shows large language models can already give useful, structured feedback on manuscripts. AuData was built for exactly this gap. Small inconsistencies, a wrong p-value, an impossible sample size, a misleading citation, a duplicated figure panel, quietly corrode the scientific record, and catching them at the scale science now operates demands machines working alongside human reviewers.

What it does

AuData is a multimodal research-integrity auditor for biomedical papers and preprints. Point it at a study and a team of detection agents go to work: they recompute reported statistics from their test values, recreate meta-analyses, cross-check internal numbers (sums, percentages, table-vs-text), screen figures for cloning, splicing, and cross-paper reuse, test whether the conclusions are actually supported by the methods, and verify every citation for retractions and errors. Each finding comes back prioritized by severity and linked to exact evidence, with a one-click jump to the highlighted spot in the PDF, so a reviewer can see precisely what was flagged and why. Everything consolidates into a downloadable audit report. The point is to turn research integrity from a manual spot-check into a structured, repeatable audit.

How we built it

AuData is a modular multi-agent pipeline behind a standalone FastAPI service. It uses Claude for reasoning-heavy checks and routes lighter and vision work to local models, with Redis powering session-aware caching, semantic caching, vector search for cross-paper figure matching, and long-term agent memory. Paper ingest handles PDF parsing, full-text extraction, section/table/figure detection, and clean metadata resolution, with Browserbase as a fallback fetcher for hard-to-reach sources. On top of that sit the specialized detectors: a statistics engine for p-value recomputation and meta-analysis recreation, numerical-consistency checks, reference integrity, methods-versus-claims analysis, and image forensics (ELA, copy-move detection, splice detection, perceptual hashing, and CLIP-embedding cross-paper comparison). A persistence layer ties every flag back to exact paper evidence so nothing is a black box, and we exposed the auditor as a Band agent so it can collaborate with other agents in a shared room.

Challenges we ran into

Research integrity is not one problem, it is many. Statistical recomputation, citation verification, figure comparison, and methods-claims review each fail in different ways and demand different evidence and different confidence thresholds, so there is no single model or metric that covers them. The harder challenge was trust: every flag had to be inspectable, grounded in the source paper, and framed as reviewer assistance rather than an automated accusation. We spent a lot of time and effort making findings point to a precise line, figure, or reference instead of an opaque score.

Accomplishments we're proud of

We brought five distinct integrity checks into one coherent, multimodal workflow instead of five disconnected scripts, end to end from ingest to a downloadable report. The image forensics surfaces the actual figure with the suspicious region highlighted, the statistical checks show the full recomputation, and every flag carries a locator back into the PDF. Most of all, the whole system is built around transparency and human judgment, which is the only responsible way to ship a research-integrity tool.

What we learned

Building AuData showed us that research integrity is really a systems problem. It requires combining document parsing, retrieval, statistical validation, image analysis, and semantic reasoning in a way that is precise, practical, and understandable. We also learned that the most important part is not just detecting a possible issue, but presenting it with enough evidence and context for a reviewer to make a defensible judgment.

What's next

for AuData We want to deepen each agent and close the loop. That means stronger statistical and numerical recomputation, richer cross-paper analysis using related-work and same-author figure comparison, and reviewer-decision calibration, where the system learns from past accept/dismiss decisions (via agent memory) to reduce noise over time. We also want to harden the reporting layer so any flag can be exported with clear severity, confidence, and evidence links for researchers, reviewers, and journals.

AuData — Biomedical Research-Integrity Auditor

AuData audits a single paper or preprint for statistical errors, numerical inconsistencies, figure manipulation, methods-vs-claim mismatches, and citation/reference problems — then surfaces prioritized, calibrated, evidence-linked flags through a human review surface. The framing is reviewer-assist, never automated accusation: a human stays in the loop on every flag.

AuData is built on the Evidence Engine systematic-review platform as a template. The reusable parts (LLM dispatcher, literature APIs, statistics engine, session store, SSE streaming, decision/report UI) are kept intact; the review workflow is repurposed into an audit pipeline.

Audit pipeline

Manage → Ingest → Detect → Reliability → Report

Each tab in the app is currently a placeholder carrying the spec for the feature it will become (inputs, outputs, the template modules it reuses, and what's left to build). We build them one at a time.

Stage	Tab	Feature
Manage	Dashboard	Pipeline overview
Manage	Audits	Paper-under-audit projects, versions, shared review
Ingest	Ingest	Parse structure, stats, tables, figures, references; version diff
Detect	Statistical Recompute	Recompute reported statistics, flag mismatches
Detect	Numerical Consistency	Internal-number / total / percentage / table checks
Detect	Image Forensics	Figure manipulation, duplication, AI-generation
Detect	Methods ↔ Claims	Conclusions vs. methods/results support
Detect	Reference Integrity	Resolve, verify, retraction-check citations
Reliability	Reliability Layer	Per-flag calibration, abstention, conclusion-impact triage
Reliability	Flag Review	Human-in-the-loop accept / dismiss / needs-human
Report	Audit Report	Structured report (PDF + JSON) with severity/confidence/evidence

Reuse / Adapt / Add (against the actual codebase)

Reuse directly

LLM dispatcher — AIService.get_model* in Backend/utils.py (provider-agnostic: Claude, OpenAI, Gemini, Ollama via LangChain).
Literature APIs — Backend/data_services.py (Crossref, Semantic Scholar, OpenAlex, Europe PMC, PubMed, arXiv/bioRxiv/medRxiv, …) for citation resolution, reference checks, and version fetch.
Statistics engine — Backend/meta_analysis.py (pure-numpy effect sizes, pooling, heterogeneity, Egger/Begg, etc.) for stats-recompute.
Frontend shell + SSE + decision UI — src/app/ session store, the /api/simulation/agentic/stream SSE pattern, and the screening-decision table → per-flag triage.

Adapt

Extraction service (AITableExtractor, /api/extract/text) → extractor of reported stats / Ns / claims.
Session model (PICO, papers, decisions, extractions) → (paper-under-audit + versions, flags, decisions, labels).
Supabase KV store (supabase/) → extend namespaces with flags, labels, versions, reports.
PRISMA/export UI (PrismaFlow.tsx, docx) → structured audit report.

Add (new builds — not present in the template)

Detection agents: stats-recompute, numerical-consistency, image-forensics, methods-claims, reference-integrity.
Reliability layer: per-flag calibration (Platt/isotonic — note: no calibration code or sklearn exists in the template yet) + abstention + conclusion-impact triage.
Full-PDF parsing (GROBID + PyMuPDF), figure + coordinate maps, table parsing (pdfplumber/Camelot).
Image forensics (OpenCV / ELA / perceptual hashing / embeddings / AI-figure detection).
Recompute sandbox (E2B / Modal), scipy/statsmodels.
Retraction Watch checks + preprint version diff.
Sponsor integrations: Fetch uAgents, Redis (vector/memory/cache/queue), Terac (labeling + calibration fine-tune), Arize (tracing/evals/calibration curves), Browserbase (web fetch), Sentry (errors).

Stack

Frontend: React + TypeScript + Vite, shadcn/Radix + Tailwind, REST + SSE.
Backend: a standalone AuData FastAPI service (Backend/audata/), separate from the legacy Evidence Engine app (Backend/api.py).
LLM serving: Ollama local models + cloud (Claude/GPT/Gemini) for reasoning-heavy steps.
Storage: Redis for short-term session storage/cache (set REDIS_URL; falls back to in-memory if unset), SQLite for long-term persistence (Backend/audata.db) — both separate from Evidence Engine's Supabase. Browserbase for web fetch; biomedical literature APIs (Crossref/OpenAlex/Unpaywall/…).

Quick start

One command installs an isolated environment (Backend Python venv + frontend node_modules), configures Backend/.env, and launches both services:

./setup.sh                 # frontend :5173, backend :8010 (health-checked)
./setup.sh --with-models   # also install Ollama + pull local LLMs (~9 GB)
./teardown.sh              # stop everything it started

Open http://localhost:5173. API docs at http://localhost:8010/docs.
Re-running ./setup.sh is idempotent. Override ports with BACKEND_PORT=8011 FRONTEND_PORT=5174 ./setup.sh.
Local LLMs are opt-in — the placeholder app runs without them. To enable AI, either add a cloud key (ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY) to Backend/.env and pick the model in the sidebar, or run ./setup.sh --with-models.

Develop (manual)

# Frontend
pnpm install
pnpm dev          # Vite dev server (http://localhost:5173 → proxies /api to :8010)
pnpm typecheck
pnpm build

# Backend — the AuData service (in its venv)
cd Backend && python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn audata.main:app --port 8010

Copy Backend/.env.example → Backend/.env and fill what you need: ENTREZ_EMAIL (polite pool for Crossref/OpenAlex/Unpaywall), BROWSERBASE_API_KEY / BROWSERBASE_PROJECT_ID (URL fetch), REDIS_URL (short-term storage; optional — falls back to in-memory), and a model key (ANTHROPIC_API_KEY / …) for AI steps.

The legacy Evidence Engine FastAPI app still lives at Backend/api.py for reference but is not started by AuData.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
FastAPIIn code
HTMLIn code
LangChainIn code
OllamaIn code
OpenAIIn code
PythonIn code
ReactIn code
RedisIn code
SQLIn code
StreamlitIn code
SupabaseIn code
Tailwind CSSIn code
TypeScriptIn code
Google GeminiClaimed

15 of 16 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

Claude CodeCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

1.7 MB

Source files

154

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.