# Project export: RepliCAT: Multi-Agent Framework for Reproducing Research
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: Ever read a research paper and thought, “But does this actually work?” RepliCAT puts it to the test as a multi-agent “CI/CD for papers,” automatically verifying if results can be reliably reproduced.
- Devpost: https://devpost.com/software/broshengpt
- GitHub: https://github.com/JJ-Xie/repliCAT-frontend
- Demo: https://github.com/Poarthan/RepliCAT
- Video: https://www.youtube.com/embed/zn7NMgRwpPY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Justin Xie (10 commits), gpt-engineer-app[bot] (9 commits)
## Devpost submission (written by the team)
### Inspiration
Reproducibility in research is often harder than it should be. Not because the ideas are unclear, but because the operational details are scattered. Many existing research repos contain undocumented dependencies, ambiguous experiment commands, missing datasets, environment mismatches, or small inconsistencies between the paper & the repo. In our own research (and in conversations with other researchers), we’ve repeatedly seen similar issues: a paper looks promising, the repo exists, but reproducing the results requires hours (or days) of detective work. We wanted to treat research reproduction like a software engineering workflow problem rather than simply an academic one. RepliCAT was inspired by this. What if reproducing a paper felt more like running CI/CD on a codebase (structured, verified, and easy to use) instead of manually reverse-engineering instructions?
### What it does
RepliCAT is an agentic system that turns "I want to reproduce this paper" into a very specific replication plan, and all it needs is a paper title, url link, file upload, or citation. Given these inputs, RepliCAT can: Understand the paper and repository, generating all the code you need as a researcher to reproduce their experiments Generate a step-by-step replication plan (e.g. git clone for the original research repo, environment setup, dataset & artifact acquisition, and exact experiment commands) Verify the plan against the paper and codebase. It also estimates feasibility: suggesting required Hardware (CPU/GPU), Tools and APIs, estimated cost, and estimated time Allow human to interrupt and reprompt the model if needed in a multi-step agentic architecture Produce a structured replication report with rubric-based scores: Scientific Methodology (focusing on statistical rigor, while flagging data poisoning & cherry-picking results), Transparency, Fact / Source Reliability, Repeatability, and Clarity as well as any necessary logged code modifications to the official research repo.
### How we built it
RepliCAT is an agentic driven system built on two distinct stacks: the Agent Stack and the Tech Stack. The Agent Stack is driven by a Lab Supervisor which orchestrates a multi-stage agent pipeline with typed, machine-checkable outputs. The Agent Stack was powered by Anthropic's Claude SDK Agents and hosted in Modal's Sandbox environments for execution. Agent Stack Stage 1: Review A ResearchAgent performs paper analysis, codebase analysis, and fact-check analysis on the research paper (condensing these findings using a structured rubric into a markdown file and a set of reproducibility scores). Stage 2: Plan + Verify + Gate Given the work done by the ResearchAgent, the ReplicationAgent produces a runnable "ReplicationPlan". A separated VerifierAgent checks the validity of the code, process, and alignment with the paper / repo. This stage also generates a feasibility estimate (required tools, cost, and time), allowing a human to stay in the loop and approve the code changes as necessary. Stage 3: Replication Director A ReplicationDirector agent spawns child agents designed to execute experiments either locally (Docker) or remotely (Modal sandbox backends) depending on the complexity of the research experiment. Stage 4: Report Generates a structured "Replication Report" in JSON + Markdown that includes a detailed and comprehensive report of the RepliCAT metrics. All inter-agent communication is enforced through typed Pydantic models to ensure structured outputs and reduce parsing errors (with additional processing as necessary). Prompts are centralized for consistency. Tech Stack The Agent Stack was integrated into a larger Tech Stack which consisted of a traditional front-end and back-end as well as a database, all making extensive use CloudFlare's Development Platform. The front-end was designed to target two main functionalities. To display important information such as metrics of reproducibility, clarity, detailed replication steps, artifacts, and other crucial components of a solid scientific process. As a community, researchers can use the RepliCAT dashboard as a resource for collecting and standardizing analysis about the validity and value of research from a wide range of fields. To offer an interface where users can upload the papers of their choice for detailed analysis by the agentic RepliCAT system. This allows researchers to continually increase the catalog of valuable information and meta-information pertaining to the most niche and most popular research fields. The main job of the backend was to act as the bridge between all data streams, bringing together the Agent Stack, front-end, and database. For the backend, we made extensive use of CloudFlare Workers. For example, they were crucial in both initial interaction with the Modal Sandboxes to instantiate agent action as well as the intermediary for streaming status and progress information between the front-end and Agent Stack during active paper review. We also made use of CloudFlare's D1 SQL Database to store the growing list of research with RepliCAT analysis.
### Challenges we ran into
Minimal modification policy: We enforced a strict rule: never modify the original repository unless explicitly required. Every change must be logged. This ensures we balance between the 2 extremes (not deviating too far from the original premise / repo of the research paper, while making enough changes to successfully reproduce experiments). Plan validity vs. hallucination risk: LLM-generated plans can drift from the official research repo structure. The VerifierAgent cross-checks these plans against real files and documentation. Execution safety: Running arbitrary research code could be risky and also expensive. We designed a mandatory human approval gate and structured command preview before any execution. Computation Limitations: We faced limited compute power and were unable to recreate every single paper in the given time constraints. We built a minifier, to find minimum experiment to prove quality of paper.
### Accomplishments we're proud of
Designed a full multi-step multi-agent pipeline making use of Agentic AI, Cloudflare Dev Platforms, and Sandbox Execution Environments that performs much-needed analysis of the research quality of research across fields Created a database and dashboard for collecting / compiling an increasing list of audited research Compiled an extensive rubric for assessing research replication, transparency, clarity, and quality
### What we learned
We learned how to design multi-step Agentic systems for a wide range of tasks aimed at completing a goal of analyzing research quality We learned how to use tools such as Modal, Cloudflare Dev, Claude SDK
### What's next
for RepliCat: Multi-Agent Framework for Reproducing Research Our long-term vision is to make reproducibility and replicability measurable, comparable, and automated (starting from just a paper and its repository). Through the RepliCAT platform, and given both time and compute resources, we hope to create an extensive database of research, both big and small, that allows for detailed analysis of the findings of one paper, patterns in research at a specific conference, or the trends of an entire research field. We would love to see RepliCAT become an important tool for improving ease of research replication and a symbol for promoting ethical research with reproducibility, clarity, reliability, and transparency.
## README (from the GitHub repository)
# repliCAT Frontend
## Local Development
```bash
npm install
npm run dev
```
## Build
```bash
npm run build
```
## Environment Variables
- `VITE_DB_API_URL`
- `VITE_PROCESS_API_URL`
- `VITE_API_URL` (fallback)
## Detected evidence (automated analysis)
Indexed codebase: 93 recognized source files, 243 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Docker (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (101 of 101)
```
.gitignore
components.json
eslint.config.js
index.html
package.json
postcss.config.js
public/robots.txt
README.md
src/App.css
src/App.tsx
src/components/ArtifactsSection.tsx
src/components/CitationGraph.tsx
src/components/GradeBadge.tsx
src/components/Navbar.tsx
src/components/NavLink.tsx
src/components/PaperCard.tsx
src/components/RadarScoreChart.tsx
src/components/ReplicationBreakdown.tsx
src/components/ReplicationProcess.tsx
src/components/ReplicationTimeline.tsx
src/components/ReviewerCard.tsx
src/components/ScoreCircle.tsx
src/components/SearchBar.tsx
src/components/SectionHeader.tsx
src/components/SectionModal.tsx
src/components/SimilarPapers.tsx
src/components/Sparkline.tsx
src/components/ui/accordion.tsx
src/components/ui/alert-dialog.tsx
src/components/ui/alert.tsx
src/components/ui/aspect-ratio.tsx
src/components/ui/avatar.tsx
src/components/ui/badge.tsx
src/components/ui/breadcrumb.tsx
src/components/ui/button.tsx
src/components/ui/calendar.tsx
src/components/ui/card.tsx
src/components/ui/carousel.tsx
src/components/ui/chart.tsx
src/components/ui/checkbox.tsx
src/components/ui/collapsible.tsx
src/components/ui/command.tsx
src/components/ui/context-menu.tsx
src/components/ui/dialog.tsx
src/components/ui/drawer.tsx
src/components/ui/dropdown-menu.tsx
src/components/ui/form.tsx
src/components/ui/hover-card.tsx
src/components/ui/input-otp.tsx
src/components/ui/input.tsx
src/components/ui/label.tsx
src/components/ui/menubar.tsx
src/components/ui/navigation-menu.tsx
src/components/ui/pagination.tsx
src/components/ui/popover.tsx
src/components/ui/progress.tsx
src/components/ui/radio-group.tsx
src/components/ui/resizable.tsx
src/components/ui/scroll-area.tsx
src/components/ui/select.tsx
src/components/ui/separator.tsx
src/components/ui/sheet.tsx
src/components/ui/sidebar.tsx
src/components/ui/skeleton.tsx
src/components/ui/slider.tsx
src/components/ui/sonner.tsx
src/components/ui/switch.tsx
src/components/ui/table.tsx
src/components/ui/tabs.tsx
src/components/ui/textarea.tsx
src/components/ui/toast.tsx
src/components/ui/toaster.tsx
src/components/ui/toggle-group.tsx
src/components/ui/toggle.tsx
src/components/ui/tooltip.tsx
src/components/ui/use-toast.ts
src/data/mockData.ts
src/hooks/use-mobile.tsx
src/hooks/use-toast.ts
src/index.css
src/index.ts
src/lib/gradeUtils.ts
src/lib/utils.ts
src/main.tsx
src/pages/Index.tsx
src/pages/Launch.tsx
src/pages/NotFound.tsx
src/pages/PaperDetail.tsx
src/pages/SubmitPaper.tsx
src/services/api.ts
src/test/example.test.ts
src/test/setup.ts
src/vite-env.d.ts
tailwind.config.ts
TEST_API.md
tsconfig.app.json
tsconfig.json
tsconfig.node.json
vite.config.ts
vitest.config.ts
wrangler.toml
```
### Dependencies
- package.json: @eslint/js@^9.32.0, @hookform/resolvers@^3.10.0, @radix-ui/react-accordion@^1.2.11, @radix-ui/react-alert-dialog@^1.1.14, @radix-ui/react-aspect-ratio@^1.1.7, @radix-ui/react-avatar@^1.1.10, @radix-ui/react-checkbox@^1.3.2, @radix-ui/react-collapsible@^1.1.11, @radix-ui/react-context-menu@^2.2.15, @radix-ui/react-dialog@^1.1.14, @radix-ui/react-dropdown-menu@^2.1.15, @radix-ui/react-hover-card@^1.1.14, @radix-ui/react-label@^2.1.7, @radix-ui/react-menubar@^1.1.15, @radix-ui/react-navigation-menu@^1.2.13, @radix-ui/react-popover@^1.1.14, @radix-ui/react-progress@^1.1.7, @radix-ui/react-radio-group@^1.3.7, @radix-ui/react-scroll-area@^1.2.9, @radix-ui/react-select@^2.2.5, @radix-ui/react-separator@^1.1.7, @radix-ui/react-slider@^1.3.5, @radix-ui/react-slot@^1.2.3, @radix-ui/react-switch@^1.2.5, @radix-ui/react-tabs@^1.1.12, @radix-ui/react-toast@^1.2.14, @radix-ui/react-toggle@^1.1.9, @radix-ui/react-toggle-group@^1.1.10, @radix-ui/react-tooltip@^1.2.7, @tailwindcss/typography@^0.5.16, @tanstack/react-query@^5.83.0, @testing-library/jest-dom@^6.6.0, @testing-library/react@^16.0.0, @types/node@^22.16.5, @types/react@^18.3.23, @types/react-dom@^18.3.7, @vitejs/plugin-react-swc@^3.11.0, autoprefixer@^10.4.21, axios@^1.13.5, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@^1.1.1, date-fns@^3.6.0, embla-carousel-react@^8.6.0, eslint@^9.32.0, eslint-plugin-react-hooks@^5.2.0, eslint-plugin-react-refresh@^0.4.20, globals@^15.15.0, input-otp@^1.4.2, jsdom@^20.0.3, lucide-react@^0.462.0, next-themes@^0.3.0, postcss@^8.5.6, react@^18.3.1, react-day-picker@^8.10.1, react-dom@^18.3.1, react-hook-form@^7.61.1, react-resizable-panels@^2.1.9, react-router-dom@^6.30.1, recharts@^2.15.4, sonner@^1.7.4, tailwind-merge@^2.6.0, tailwindcss@^3.4.17, tailwindcss-animate@^1.0.7, typescript@^5.8.3, typescript-eslint@^8.38.0, vaul@^0.9.9, vite@^5.4.19, vitest@^3.2.4, zod@^3.25.76
### Recent commits (newest first)
- finalizing frontend
- fixed some display bugs
- wiring to backend prayge
- another deploy attempt
- please deploy with env vars
- hopefully worker vars
- removed bun
- wrangler for frontend deployment
- frontend iteration for first deployment
- first iteration for layout
- Manually updated project theme
- Recolor grade badges to original colors
- Changes
- Adjust grade badge styling
- Changes
- Remove borders from score circle
- Changes
- Tan background and border added
- Changes
- Applied theme Brutalist
## Key source files (fetched from GitHub, selected and truncated for size)
### TEST_API.md
```markdown
# Testing Your API Connection
## ✅ Backend is Deployed
Your backend is live at: https://replicat-backend.justinjxie.workers.dev
## Test API Endpoints
### 1. Get All Papers
```bash
curl https://replicat-backend.justinjxie.workers.dev/api/papers
```
### 2. Get Single Paper
```bash
curl https://replicat-backend.justinjxie.workers.dev/api/papers/1
```
### 3. Get Reviewers
```bash
curl https://replicat-backend.justinjxie.workers.dev/api/papers/1/reviewers
```
## Start Frontend Locally
```bash
npm run dev
```
Then visit http://localhost:5173
## What Was Updated
✅ Frontend now fetches from your deployed API
✅ Loading states added
✅ Error handling implemented
✅ Both Index and PaperDetail pages use live data
✅ CORS configured to allow all origins
## Next Steps
### Option 1: Test Locally
Just run `npm run dev` in your frontend directory and everything should work!
### Option 2: Deploy Frontend
#### Deploy to Cloudflare Pages
```bash
npm run build
npx wrangler pages deploy dist --project-name=replicat-frontend
```
After deployment, update backend CORS:
```toml
# In backend/wrangler.toml
[vars]
ALLOWED_ORIGIN = "https://replicat-frontend.pages.dev"
```
Then redeploy backend:
```bash
cd ../backend
npm run deploy
```
#### Deploy to Vercel
1. Push to GitHub
2. Import on vercel.com
3. Add env var: `VITE_API_URL=https://replicat-backend.justinjxie.workers.dev`
4. Deploy
#### Deploy to Netlify
1. Push to GitHub
2. Import on netlify.com
3. Build: `npm run build`
4. Publish: `dist`
5. Add env var: `VITE_API_URL=https://replicat-backend.justinjxie.workers.dev`
## Verification
When your frontend loads, check:
- [ ] Papers display on homepage
- [ ] Search works
- [ ] Filters work
- [ ] Can click on paper to see details
- [ ] Radar chart displays
- [ ] Citations graph shows
- [ ] No CORS errors in console
## Troubleshooting
### Issue: "Failed to fetch" error
- Check browser console for CORS errors
- Verify API URL in .env.local is correct
- Make sure backend is deployed and accessible
### Issue: Papers not loading
- Open Network tab in dev tools
- Check if requests are going to the right URL
- Verify API is returning data with the curl commands above
### Issue: Blank page
- Check console for errors
- Make sure you ran `npm install` after adding axios
- Try clearing cache and reloading
## Success!
If you see papers loading on your homepage, congratulations! 🎉
Your full-stack app is now:
- ✅ Backend on Cloudflare Workers
- ✅ Database on Cloudflare D1
- ✅ Frontend connected and working
- ✅ Ready to deploy anywhere!
```
### package.json
```
{
"name": "vite_react_shadcn_ts",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"build:dev": "vite build --mode development",
"lint": "eslint .",
"preview": "vite preview",
"test": "vitest run",
"test:watch": "vitest"
},
"dependencies": {
"@hookform/resolvers": "^3.10.0",
"@radix-ui/react-accordion": "^1.2.11",
"@radix-ui/react-alert-dialog": "^1.1.14",
"@radix-ui/react-aspect-ratio": "^1.1.7",
"@radix-ui/react-avatar": "^1.1.10",
"@radix-ui/react-checkbox": "^1.3.2",
"@radix-ui/react-collapsible": "^1.1.11",
"@radix-ui/react-context-menu": "^2.2.15",
"@radix-ui/react-dialog": "^1.1.14",
"@radix-ui/react-dropdown-menu": "^2.1.15",
"@radix-ui/react-hover-card": "^1.1.14",
"@radix-ui/react-label": "^2.1.7",
"@radix-ui/react-menubar": "^1.1.15",
"@radix-ui/react-navigation-menu": "^1.2.13",
"@radix-ui/react-popover": "^1.1.14",
"@radix-ui/react-progress": "^1.1.7",
"@radix-ui/react-radio-group": "^1.3.7",
"@radix-ui/react-scroll-area": "^1.2.9",
"@radix-ui/react-select": "^2.2.5",
"@radix-ui/react-separator": "^1.1.7",
"@radix-ui/react-slider": "^1.3.5",
"@radix-ui/react-slot": "^1.2.3",
"@radix-ui/react-switch": "^1.2.5",
"@radix-ui/react-tabs": "^1.1.12",
"@radix-ui/react-toast": "^1.2.14",
"@radix-ui/react-toggle": "^1.1.9",
"@radix-ui/react-toggle-group": "^1.1.10",
"@radix-ui/react-tooltip": "^1.2.7",
"@tanstack/react-query": "^5.83.0",
"axios": "^1.13.5",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"cmdk": "^1.1.1",
"date-fns": "^3.6.0",
"embla-carousel-react": "^8.6.0",
"input-otp": "^1.4.2",
"lucide-react": "^0.462.0",
"next-themes": "^0.3.0",
"react": "^18.3.1",
"react-day-picker": "^8.10.1",
"react-dom": "^18.3.1",
"react-hook-form": "^7.61.1",
"react-resizable-panels": "^2.1.9",
"react-router-dom": "^6.30.1",
"recharts": "^2.15.4",
"sonner": "^1.7.4",
"tailwind-merge": "^2.6.0",
"tailwindcss-animate": "^1.0.7",
"vaul": "^0.9.9",
"zod": "^3.25.76"
},
"devDependencies": {
"@eslint/js": "^9.32.0",
"@tailwindcss/typography": "^0.5.16",
"@testing-library/jest-dom": "^6.6.0",
"@testing-library/react": "^16.0.0",
"@types/node": "^22.16.5",
"@types/react": "^18.3.23",
"@types/react-dom": "^18.3.7",
"@vitejs/plugin-react-swc": "^3.11.0",
"autoprefixer": "^10.4.21",
"eslint": "^9.32.0",
"eslint-plugin-react-hooks": "^5.2.0",
"eslint-plugin-react-refresh": "^0.4.20",
"globals": "^15.15.0",
"jsdom": "^20.0.3",
"postcss": "^8.5.6",
"tailwindcss": "^3.4.17",
"typescript": "^5.8.3",
"typescript-eslint": "^8.38.0",
"vite": "^5.4.19",
"vitest": "^3.2.4"
}
}
```
### src/index.ts
```typescript
export default {
async fetch(request: Request, env: any) {
return new Response(`API Key is: ${env.MY_SECRET}`);
}
};
```
### src/main.tsx
```typescript
import { createRoot } from "react-dom/client";
import App from "./App.tsx";
import "./index.css";
createRoot(document.getElementById("root")!).render();
```
### src/App.tsx
```typescript
import { Toaster } from "@/components/ui/toaster";
import { Toaster as Sonner } from "@/components/ui/sonner";
import { TooltipProvider } from "@/components/ui/tooltip";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { BrowserRouter, Routes, Route } from "react-router-dom";
import Launch from "./pages/Launch";
import Index from "./pages/Index";
import PaperDetail from "./pages/PaperDetail";
import SubmitPaper from "./pages/SubmitPaper";
import NotFound from "./pages/NotFound";
const queryClient = new QueryClient();
const App = () => (
} />
} />
} />
} />
{/* ADD ALL CUSTOM ROUTES ABOVE THE CATCH-ALL "*" ROUTE */}
} />
);
export default App;
```
### src/pages/Index.tsx
```typescript
import { useState } from "react";
import { useQuery } from "@tanstack/react-query";
import { Navbar } from "@/components/Navbar";
import { SearchBar } from "@/components/SearchBar";
import { PaperCard } from "@/components/PaperCard";
import { paperAPI } from "@/services/api";
const Index = () => {
const [search, setSearch] = useState("");
// Fetch papers from API
const { data: papers = [], isLoading } = useQuery({
queryKey: ['papers', search],
queryFn: () => paperAPI.getAll({ search: search || undefined }),
});
const sortedPapers = [...papers].sort((a, b) => {
const aDate = Date.parse((a as { createdAt?: string; created_at?: string }).createdAt || (a as { createdAt?: string; created_at?: string }).created_at || "");
const bDate = Date.parse((b as { createdAt?: string; created_at?: string }).createdAt || (b as { createdAt?: string; created_at?: string }).created_at || "");
if (!Number.isNaN(aDate) && !Number.isNaN(bDate)) return bDate - aDate;
if (!Number.isNaN(aDate)) return -1;
if (!Number.isNaN(bDate)) return 1;
const aYear = a.year ?? 0;
const bYear = b.year ?? 0;
if (aYear !== bYear) return bYear - aYear;
const aIdNum = Number(a.id);
const bIdNum = Number(b.id);
if (!Number.isNaN(aIdNum) && !Number.isNaN(bIdNum)) return bIdNum - aIdNum;
return String(b.id).localeCompare(String(a.id));
});
return (
{/* Hero / Search */}
Research Reproducibility Tracker
See which academic papers are reproducible and transparent.
{/* Main content */}
{/* Loading State */}
{isLoading && (