Autolab Labs

Autonomous end-to-end 2D materials discovery -- from exfoliation to characterization, no human in the loop.

Built at TreeHacks 2026.

---

What It Does

AI Materials Researcher is a multi-agent platform that autonomously executes the entire 2D materials research pipeline. Describe an experiment in plain English -- the system plans it, controls the hardware, detects flakes, runs spectroscopy, searches literature, and generates a full report.

The platform connects three layers:

1. Custom hardware -- A graphene stamping/straining jig with stepper motors controlled by a Raspberry Pi running a local VLM. The researcher talks to the jig through natural language prompts.
2. Intelligent vision -- A two-stage CV + Claude Vision hybrid pipeline for real-time flake detection. Local computer vision optimizes contrast and finds candidates in ~30ms, then Claude Sonnet 4 verifies and classifies each one in ~3 seconds.
3. Autonomous agents -- An orchestrator agent coordinates synthesis, characterization, and theory sub-agents. When the jig finishes exfoliating, it signals the platform to begin the next step automatically. Raman spectra are analyzed, peaks are fitted, and results are compared to literature -- all without human intervention.

---

Project Structure

TreeHacks_26/
├── backend/                    # FastAPI backend -- agents, CV pipeline, WebSocket server
│   ├── main.py                 # FastAPI app + WebSocket endpoint + session management
│   ├── video_feed.py           # Video/image feed worker with CV + VLM flake detection
│   ├── agents/
│   │   ├── base.py             # Base agent class (Claude API, tool use, streaming)
│   │   ├── orchestrator.py     # Master agent -- plans experiments, dispatches sub-agents
│   │   ├── synthesis.py        # Controls hardware for sample preparation
│   │   ├── characterization.py # Runs Raman spectroscopy analysis
│   │   └── theory.py           # Theoretical models + academic paper search
│   ├── tools/
│   │   ├── flake_finder.py     # Core CV + VLM flake detection logic
│   │   ├── hardware.py         # Raspberry Pi motor control interface
│   │   ├── microscope.py       # Microscope image capture
│   │   ├── raman.py            # Raman spectroscopy data tools
│   │   └── research.py         # Academic paper search (Semantic Scholar)
│   ├── sources/                # Input images/video for the microscope feed
│   └── requirements.txt
│
├── frontend/                   # React + TypeScript + Vite frontend
│   └── src/
│       ├── App.tsx             # Main app -- WebSocket state, layout
│       ├── components/
│       │   ├── ExperimentInput.tsx   # Natural language experiment input
│       │   ├── PlanView.tsx          # Orchestrator plan with approval workflow
│       │   ├── AgentBox.tsx          # Live agent thinking + feedback
│       │   ├── VideoFeed.tsx         # Microscope feed with detection overlays
│       │   ├── SampleWorkbench.tsx   # Sample management
│       │   └── FinalReport.tsx       # Generated research report
│       └── hooks/
│           └── useWebSocket.ts      # WebSocket connection management
│
├── flake_finder/               # Standalone desktop flake detection tool
│   ├── main.py                 # Screen-capture + VLM flake finder (keyboard-driven)
│   ├── main_qwen.py            # Local VLM variant (Qwen on RPi)
│   ├── config.json             # Detection parameters
│   └── Stepper Motor Stuff/
│       └── ThorLabs.py         # ThorLabs KDC101 motor control via pyftdi (APT protocol)
│
├── raman-agent/                # Standalone Raman spectroscopy analysis tool
│   ├── backend/
│   │   ├── main.py             # FastAPI endpoints for spectrum upload + analysis
│   │   ├── raman.py            # Baseline correction, peak detection, Gaussian fitting
│   │   └── llm_parser.py       # LLM-powered spectrum interpretation
│   └── frontend/               # Next.js UI for spectrum visualization
│
├── slides/                     # Presentation deck (Next.js + Framer Motion)
├── Makefile                    # Build and run commands
└── camera.py                   # Live camera feed utility

---

How It Works

1. Experiment Design (Frontend)

The researcher types an experiment description in plain English:

"Prepare 5 graphene samples at 0-4% strain and characterize each with Raman spectroscopy."

The orchestrator agent parses this, generates a step-by-step plan, and presents it for approval.

2. Multi-Agent Execution (Backend)

Once approved, the orchestrator dispatches tasks to specialized sub-agents:

Agent	Role	Tools
Orchestrator	Plans experiments, coordinates sub-agents, tracks progress	All sub-agent dispatch
Synthesis	Controls hardware for sample preparation (exfoliation, stamping)	Motor control, hardware interface
Characterization	Runs and analyzes Raman spectroscopy	Spectrum fitting, peak detection, material ID
Theory	Searches literature, builds theoretical models	Semantic Scholar API, calculations

All agents use Claude Sonnet 4 with tool use and stream their thinking in real-time to the frontend via WebSocket.

3. Flake Detection (CV + VLM Pipeline)

The microscope feed runs through a two-stage detection pipeline:

Stage 1 -- Local CV (~30ms):

• Automatic contrast optimization (alpha/beta sweep to maximize flake-substrate separation)
• CLAHE enhancement + Otsu thresholding
• Contour analysis with area and edge-density filtering
• Generates candidate bounding boxes

Stage 2 -- Claude Vision (~2-3s):

• Sends contrast-optimized image (512px, 70% JPEG) to Claude Sonnet 4
• Claude independently detects flakes using its own vision
• CV candidates are provided as optional hints, not hard constraints
• Returns verified detections with confidence scores, bounding boxes, and reasoning

4. Hardware Control (Raspberry Pi + Stepper Motors)

A custom graphene stamping/straining jig with:

• ThorLabs KDC101 motor controller + MTS25-Z8 linear translation stage
• Direct USB communication via pyftdi using the APT binary protocol (bypasses macOS FTDI VCP driver issues)
• Precision: 0.001mm (34,304 encoder counts/mm)
• Raspberry Pi runs a local VLM that accepts natural language commands ("exfoliate at position 3", "apply 2% strain")
• The RPi agent reports completion back to the web platform, triggering the next pipeline step automatically

5. Raman Spectroscopy Analysis

Automated spectral analysis pipeline:

• Asymmetric Least Squares (ALS) baseline correction
• scipy.signal.find_peaks for peak detection
• Multi-Gaussian fitting with scipy.optimize.curve_fit
• Material-specific labeling (Graphene D/G/2D bands, MoS2 E2g/A1g)
• LLM-powered interpretation and comparison to literature values

---

Getting Started

Prerequisites

• Python 3.11+
• Node.js 18+
• An Anthropic API key

Installation

# Clone the repo
git clone https://github.com/your-org/TreeHacks_26.git
cd TreeHacks_26

# Install all dependencies
make install

Or manually:

# Backend
cd backend
pip install -r requirements.txt

# Frontend
cd ../frontend
npm install

Environment Variables

Create backend/.env:

ANTHROPIC_API_KEY=sk-ant-...

For the standalone flake finder, create flake_finder/.env:

ANTHROPIC_API_KEY=sk-ant-...

Running

Main application (requires two terminals):

# Terminal 1: Backend (FastAPI on port 8000)
make backend

# Terminal 2: Frontend (Vite on port 5173)
make frontend

Then open http://localhost:5173.

Standalone flake finder:

cd flake_finder
python main.py

Keyboard controls: v = scan, c = cycle contrast, e = edges, r = ROI, s = save, p = pause, q = quit.

Standalone Raman analysis tool:

cd raman-agent
./start_backend.sh   # FastAPI on port 8000
./start_frontend.sh  # Next.js on port 3000

ThorLabs motor control:

cd flake_finder/Stepper\ Motor\ Stuff
pip install pyftdi pyusb
brew install libusb  # macOS only
python ThorLabs.py

---

Tech Stack

Layer	Technology
Frontend	React 19, TypeScript, Vite, Framer Motion, React Markdown, KaTeX
Backend	FastAPI, Uvicorn, WebSockets, asyncio
AI	Claude Sonnet 4 (Anthropic SDK) -- agents + vision
Computer Vision	OpenCV, NumPy, SciPy (contrast optimization, CLAHE, contour analysis)
Spectroscopy	SciPy (ALS baseline, peak detection, Gaussian fitting), Matplotlib
Hardware	pyftdi (ThorLabs APT protocol), pyserial (Raspberry Pi), stepper motors
Research	Semantic Scholar API

---

Architecture

┌────────────────────────────────────────────────────────────────────┐
│                    ORCHESTRATOR AGENT (Web Platform)               │
│   "Prepare 5 graphene samples at 0-4% strain, characterize each"   │
└─────────┬───────────────┬──────────────────┬───────────────┬───────┘
          │               │                  │               │
          ▼               ▼                  ▼               ▼
   ┌─────────────┐ ┌─────────────┐  ┌──────────────┐ ┌───────────┐
   │ STAMPING JIG│ │ FLAKE FINDER│  │CHARACTERIZE  │ │  THEORY   │
   │ (RPi + VLM) │ │ (CV + VLM)  │  │(Raman Agent) │ │  AGENT    │
   │             │ │             │  │              │ │           │
   │ Exfoliate   │ │ Detect      │  │ Peak fit     │ │ Search    │
   │ Strain      │ │ Verify      │  │ Material ID  │ │ papers    │
   │ Position    │ │ Locate      │  │ Compare      │ │ Compare   │
   └──────┬──────┘ └──────┬──────┘  └──────┬───────┘ └─────┬─────┘
          │               │                │               │
          ▼               ▼                ▼               ▼
    Stepper Motors   Microscope Feed   Raman Spectra   Semantic Scholar
    (ThorLabs KDC101) (OpenCV + Claude) (scipy fitting) (API)

The RPi jig agent and the web platform agents communicate bidirectionally. When the jig finishes exfoliating, it triggers flake detection. When good flakes are found, it triggers Raman. When strain is needed, the platform tells the jig to apply it. Then Raman again. Then compare. No human in the loop.

---

Real-Time Communication

All communication between frontend and backend uses a single WebSocket connection. Message types include:

• experiment_design -- user submits experiment description
• orchestrator_thought -- live agent thinking stream
• plan -- generated experiment plan for approval
• agent_update -- sub-agent progress and results
• video_frame -- encoded microscope frame with detection overlays
• video_command -- UI controls (scan, contrast, presets, edges)
• final_report -- completed research report with figures

---

Video Feed Controls

The microscope feed UI supports:

• Contrast presets -- Raw, Low, Med, High, V.Hi, Ext (toggle on/off)
• Manual contrast/brightness -- Real-time slider adjustment
• Edge overlay -- Canny edge detection toggle
• Scan -- Triggers VLM flake detection (freezes frame during analysis)
• Optimized view -- Auto-switches to contrast settings that maximize flake visibility (computed by the CV pipeline)

Works on both live video streams and static microscope images.

---

License

Built at TreeHacks 2026.