# Project export: AutoLab Labs This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Autonomous lab to exfoliate, strain, and measure nanometer-thick graphene flakes, powered by an agentic ecosystem where research agents access tools (hardware+software) to run end-to-end experiments - Devpost: https://devpost.com/software/auto-lab - GitHub: https://github.com/nlee1126/TreeHacks_26 - Video: https://www.youtube.com/embed/b60iOnSo2t4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Rally Lin (16 commits), nlee1126 (11 commits) ## Devpost submission (written by the team) ### Inspiration We believe for AI to truly enable human flourishing, it needs to go beyond software and consumer products. Accelerated science could compress a century’s worth of discoveries and quality of life gains into a decade. Our team is composed of EE, physics, and materials majors interested in next-generation compute, energy storage, therapeutics—and how AI can help get us there. Graphene is an incredibly important material both for use in advanced nanoelectronics and quantum materials research (2D electron system). We knew we wanted to build an autonomous lab for Treehacks, and we realized we could achieve state-of-the-art synthesis methods of graphene flakes with simple hardware since the state-of-the-art is using scotch tape to peel apart the layers. Prof. Mannix and Goldhaber-Gordon in the materials/physics departments pointed us towards Raman spectroscopy and gave us graphite chips to exfoliate. ### What it does AutoLab is an autonomous end-to-end 2D materials discovery solution — from exfoliation to characterization, no human in the loop. Our system exfoliates graphene, identifies candidate flakes, applies controlled strain, and characterizes the resulting material properties using Raman spectroscopy. This is powered by an agentic ecosystem where research agents access SoTA analysis tools and real-world hardware to run end-to-end experiments. The platform connects three layers: Custom hardware -- A graphene stamping/straining jig with stepper motors controlled by a Raspberry Pi running a local VLM. The researcher talks to the jig through natural language prompts. Intelligent vision -- A two-stage CV + Claude Vision hybrid pipeline for real-time flake detection. Local computer vision optimizes contrast and finds candidates in ~30ms, then Claude Sonnet 4 verifies and classifies each one in ~3 seconds. Autonomous orchestration -- Rather than simply automating steps, AutoLab reasons about experiments. Agents can trigger measurements, analyze spectra, detect anomalies, and iterate on experimental parameters—transforming a traditional lab workflow into a closed-loop intelligent system. ### How we built it 1. Experiment Design (Frontend) The researcher types an experiment description in plain English: "Prepare 5 graphene samples at 0-4% strain and characterize each with Raman spectroscopy." The orchestrator agent parses this, generates a step-by-step plan, and presents it for approval. 2. Multi-Agent Execution (Backend) Once approved, the orchestrator dispatches tasks to specialized sub-agents: Orchestrator plans experiments, coordinates sub-agents, tracks progress. All sub-agent dispatch. Synthesis controls hardware for sample preparation (exfoliation, stamping). Motor control, hardware interface. Characterization runs and analyzes Raman spectroscopy. Spectrum fitting, peak detection, material ID. Theory Searches literature, builds theoretical models. Semantic Scholar API, calculations. All agents use Claude Sonnet 4 with tool use and stream their thinking in real-time to the frontend via WebSocket. 3. Flake Detection (CV + VLM Pipeline) The microscope feed runs through a two-stage detection pipeline: Stage 1 -- Local CV (~30ms): Automatic contrast optimization (alpha/beta sweep to maximize flake-substrate separation) CLAHE enhancement + Otsu thresholding Contour analysis with area and edge-density filtering Generates candidate bounding boxes Stage 2 -- Claude Vision (~2-3s): Sends contrast-optimized image (512px, 70% JPEG) to Claude Sonnet 4 Claude independently detects flakes using its own vision CV candidates are provided as optional hints, not hard constraints Returns verified detections with confidence scores, bounding boxes, and reasoning 4. Hardware Control (Raspberry Pi + Stepper Motors) A custom graphene stamping/straining jig with: ThorLabs KDC101 motor controller + MTS25-Z8 linear translation stage Direct USB communication via pyftdi using the APT binary protocol (bypasses macOS FTDI VCP driver issues) Precision: 0.001mm (34,304 encoder counts/mm) Raspberry Pi runs a local VLM that accepts natural language commands ("exfoliate at position 3", "apply 2% strain") The RPi agent reports completion back to the web platform, triggering the next pipeline step automatically 5. Raman Spectroscopy Analysis Automated spectral analysis pipeline: Asymmetric Least Squares (ALS) baseline correction scipy.signal.find_peaks for peak detection Multi-Gaussian fitting with scipy.optimize.curve_fit Material-specific labeling (Graphene D/G/2D bands, MoS2 E2g/A1g) LLM-powered interpretation and comparison to literature values ### Challenges we ran into -Strain control: Applying repeatable strain without tearing ultrathin flakes required precise mechanical design. -Signal-to-noise in Raman: Distinguishing meaningful peak shifts (G and 2D bands) from noise required calibration and careful preprocessing. -Detecting outliers (e.g., “Sample 3 looks wrong”) and triggering resynthesis required building a feedback loop rather than a linear pipeline. -Deciding when to stop iterating vs. gather more samples was a core scientific design challenge. ### Accomplishments we're proud of Built a fully closed-loop autonomous materials discovery system. Achieved reliable graphene flake detection in real time. Implemented automated Raman peak fitting and strain quantification. Created an agent architecture capable of iterative experimental reasoning. Successfully integrated real-world hardware with AI orchestration. ### What we learned Autonomy is primarily a systems engineering challenge, not just an AI problem. Scientific workflows are loops, not pipelines. Structured data exchange between agents dramatically improves reliability. Grounding AI reasoning in physics and experimental constraints is essential. ### What's next for AutoLab: Closed-loop optimization of strain parameters using adaptive experimental design. Expansion to additional 2D materials (e.g., MoS2, WS2, heterostructures). Real-time experiment visualization dashboard. Higher-throughput parallel exfoliation modules. Moving toward a fully autonomous self-driving materials lab. ## README (from the GitHub repository) # Autolab Labs **Autonomous end-to-end 2D materials discovery -- from exfoliation to characterization, no human in the loop.** Built at TreeHacks 2026. --- ## What It Does AI Materials Researcher is a multi-agent platform that autonomously executes the entire 2D materials research pipeline. Describe an experiment in plain English -- the system plans it, controls the hardware, detects flakes, runs spectroscopy, searches literature, and generates a full report. The platform connects three layers: 1. **Custom hardware** -- A graphene stamping/straining jig with stepper motors controlled by a Raspberry Pi running a local VLM. The researcher talks to the jig through natural language prompts. 2. **Intelligent vision** -- A two-stage CV + Claude Vision hybrid pipeline for real-time flake detection. Local computer vision optimizes contrast and finds candidates in ~30ms, then Claude Sonnet 4 verifies and classifies each one in ~3 seconds. 3. **Autonomous agents** -- An orchestrator agent coordinates synthesis, characterization, and theory sub-agents. When the jig finishes exfoliating, it signals the platform to begin the next step automatically. Raman spectra are analyzed, peaks are fitted, and results are compared to literature -- all without human intervention. --- ## Project Structure ``` TreeHacks_26/ ├── backend/ # FastAPI backend -- agents, CV pipeline, WebSocket server │ ├── main.py # FastAPI app + WebSocket endpoint + session management │ ├── video_feed.py # Video/image feed worker with CV + VLM flake detection │ ├── agents/ │ │ ├── base.py # Base agent class (Claude API, tool use, streaming) │ │ ├── orchestrator.py # Master agent -- plans experiments, dispatches sub-agents │ │ ├── synthesis.py # Controls hardware for sample preparation │ │ ├── characterization.py # Runs Raman spectroscopy analysis │ │ └── theory.py # Theoretical models + academic paper search │ ├── tools/ │ │ ├── flake_finder.py # Core CV + VLM flake detection logic │ │ ├── hardware.py # Raspberry Pi motor control interface │ │ ├── microscope.py # Microscope image capture │ │ ├── raman.py # Raman spectroscopy data tools │ │ └── research.py # Academic paper search (Semantic Scholar) │ ├── sources/ # Input images/video for the microscope feed │ └── requirements.txt │ ├── frontend/ # React + TypeScript + Vite frontend │ └── src/ │ ├── App.tsx # Main app -- WebSocket state, layout │ ├── components/ │ │ ├── ExperimentInput.tsx # Natural language experiment input │ │ ├── PlanView.tsx # Orchestrator plan with approval workflow │ │ ├── AgentBox.tsx # Live agent thinking + feedback │ │ ├── VideoFeed.tsx # Microscope feed with detection overlays │ │ ├── SampleWorkbench.tsx # Sample management │ │ └── FinalReport.tsx # Generated research report │ └── hooks/ │ └── useWebSocket.ts # WebSocket connection management │ ├── flake_finder/ # Standalone desktop flake detection tool │ ├── main.py # Screen-capture + VLM flake finder (keyboard-driven) │ ├── main_qwen.py # Local VLM variant (Qwen on RPi) │ ├── config.json # Detection parameters │ └── Stepper Motor Stuff/ │ └── ThorLabs.py # ThorLabs KDC101 motor control via pyftdi (APT protocol) │ ├── raman-agent/ # Standalone Raman spectroscopy analysis tool │ ├── backend/ │ │ ├── main.py # FastAPI endpoints for spectrum upload + analysis │ │ ├── raman.py # Baseline correction, peak detection, Gaussian fitting │ │ └── llm_parser.py # LLM-powered spectrum interpretation │ └── frontend/ # Next.js UI for spectrum visualization │ ├── slides/ # Presentation deck (Next.js + Framer Motion) ├── Makefile # Build and run commands └── camera.py # Live camera feed utility ``` --- ## How It Works ### 1. Experiment Design (Frontend) The researcher types an experiment description in plain English: > "Prepare 5 graphene samples at 0-4% strain and characterize each with Raman spectroscopy." The orchestrator agent parses this, generates a step-by-step plan, and presents it for approval. ### 2. Multi-Agent Execution (Backend) Once approved, the orchestrator dispatches tasks to specialized sub-agents: | Agent | Role | Tools | |-------|------|-------| | **Orchestrator** | Plans experiments, coordinates sub-agents, tracks progress | All sub-agent dispatch | | **Synthesis** | Controls hardware for sample preparation (exfoliation, stamping) | Motor control, hardware interface | | **Characterization** | Runs and analyzes Raman spectroscopy | Spectrum fitting, peak detection, material ID | | **Theory** | Searches literature, builds theoretical models | Semantic Scholar API, calculations | All agents use Claude Sonnet 4 with tool use and stream their thinking in real-time to the frontend via WebSocket. ### 3. Flake Detection (CV + VLM Pipeline) The microscope feed runs through a two-stage detection pipeline: **Stage 1 -- Local CV (~30ms):** - Automatic contrast optimization (alpha/beta sweep to maximize flake-substrate separation) - CLAHE enhancement + Otsu thresholding - Contour analysis with area and edge-density filtering - Generates candidate bounding boxes **Stage 2 -- Claude Vision (~2-3s):** - Sends contrast-optimized image (512px, 70% JPEG) to Claude Sonnet 4 - Claude independently detects flakes using its own vision - CV candidates are provided as optional hints, not hard constraints - Returns verified detections with confidence scores, bounding boxes, and reasoning ### 4. Hardware Control (Raspberry Pi + Stepper Motors) A custom graphene stamping/straining jig with: - **ThorLabs KDC101** motor controller + **MTS25-Z8** linear translation stage - Direct USB communication via `pyftdi` using the APT binary protocol (bypasses macOS FTDI VCP driver issues) - Precision: 0.001mm (34,304 encoder counts/mm) - **Raspberry Pi** runs a local VLM that accepts natural language commands ("exfoliate at position 3", "apply 2% strain") - The RPi agent reports completion back to the web platform, triggering the next pipeline step automatically ### 5. Raman Spectroscopy Analysis Automated spectral analysis pipeline: - Asymmetric Least Squares (ALS) baseline correction - `scipy.signal.find_peaks` for peak detection - Multi-Gaussian fitting with `scipy.optimize.curve_fit` - Material-specific labeling (Graphene D/G/2D bands, MoS2 E2g/A1g) - LLM-powered interpretation and comparison to literature values --- ## Getting Started ### Prerequisites - Python 3.11+ - Node.js 18+ - An [Anthropic API key](https://console.anthropic.com/) ### Installation ```bash # Clone the repo git clone https://github.com/your-org/TreeHacks_26.git cd TreeHacks_26 # Install all dependencies make install ``` Or manually: ```bash # Backend cd backend pip install -r requirements.txt # Frontend cd ../frontend npm install ``` ### Environment Variables Create `backend/.env`: ```env ANTHROPIC_API_KEY=sk-ant-... ``` For the standalone flake finder, create `flake_finder/.env`: ```env ANTHROPIC_API_KEY=sk-ant-... ``` ### Running **Main application** (requires two terminals): ```bash # Terminal 1: Backend (FastAPI on port 8000) make backend # Terminal 2: Frontend (Vite on port 5173) make frontend ``` Then open [http://localhost:5173](http://localhost:5173). **Standalone flake finder:** ```bash cd flake_finder python main.py ``` Keyboard controls: `v` = scan, `c` = cycle contrast, `e` = edges, `r` = ROI, `s` = save, `p` = pause, `q` = quit. **Standalone Raman analysis tool:** ```bash cd raman-agent ./start_backend.sh # FastAPI on port 8000 ./start_f [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 60 recognized source files, 412 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (89 of 89) ``` .claude/settings.local.json .gitignore backend/.env.example backend/agents/__init__.py backend/agents/base.py backend/agents/characterization.py backend/agents/orchestrator.py backend/agents/synthesis.py backend/agents/theory.py backend/main.py backend/requirements.txt backend/sources/time_17.txt backend/sources/time_18.txt backend/sources/time_19.txt backend/sources/time_20.txt backend/sources/time_21.txt backend/sources/time_22.txt backend/tools/__init__.py backend/tools/flake_finder.py backend/tools/hardware.py backend/tools/microscope.py backend/tools/raman.py backend/tools/research.py backend/video_feed.py camera_density_algorithm.py camera.py CLAUDE.md flake_finder/.env.example flake_finder/config_qwen.json flake_finder/config.json flake_finder/main_qwen.py flake_finder/main.py flake_finder/README.md flake_finder/requirements.txt flake_finder/Stepper Motor Stuff/ThorLabs.py frontend/.gitignore frontend/eslint.config.js frontend/index.html frontend/package.json frontend/README.md frontend/src/App.css frontend/src/App.tsx frontend/src/components/AgentBox.tsx frontend/src/components/ExperimentInput.tsx frontend/src/components/FinalReport.tsx frontend/src/components/PlanView.tsx frontend/src/components/SampleWorkbench.tsx frontend/src/components/VideoFeed.tsx frontend/src/hooks/useWebSocket.ts frontend/src/index.css frontend/src/main.tsx frontend/tsconfig.app.json frontend/tsconfig.json frontend/tsconfig.node.json frontend/vite.config.ts Makefile raman-agent/.gitignore raman-agent/ARCHITECTURE.md raman-agent/backend/.env.example raman-agent/backend/llm_parser.py raman-agent/backend/main.py raman-agent/backend/raman_parser_enhanced.py raman-agent/backend/raman.py raman-agent/backend/requirements.txt raman-agent/frontend/app/globals.css raman-agent/frontend/app/layout.tsx raman-agent/frontend/app/page.tsx raman-agent/frontend/next-env.d.ts raman-agent/frontend/next.config.js raman-agent/frontend/package.json raman-agent/frontend/postcss.config.js raman-agent/frontend/tailwind.config.js raman-agent/frontend/tsconfig.json raman-agent/IMPLEMENTATION_CHECKLIST.md raman-agent/INDEX.md raman-agent/METADATA_EXTRACTION.md raman-agent/PROJECT_STRUCTURE.md raman-agent/QUICKSTART.md raman-agent/README.md raman-agent/samples/metadata.csv raman-agent/samples/sample_spectrum_2.txt raman-agent/samples/sample_spectrum_3.txt raman-agent/samples/sample_spectrum.txt raman-agent/SHOWCASE.md raman-agent/start_backend.sh raman-agent/start_frontend.sh raman-agent/SUMMARY.md raman-agent/test_api.sh README.md ``` ### Dependencies - backend/requirements.txt: anthropic@==0.43.0, fastapi@==0.109.0, httpx@==0.26.0, matplotlib@>=3.7.0, mss@>=9.0.0, numpy@>=1.24.0, opencv-python@>=4.8.0, python-multipart@==0.0.6, scipy@>=1.10.0, uvicorn[standard]@==0.27.0, websockets@==12.0 - flake_finder/requirements.txt: anthropic@>=0.39, mss@>=9.0, numpy@>=1.24, openai@>=1.40, opencv-python@>=4.8, Pillow@>=10.0 - frontend/package.json: @eslint/js@^9.39.1, @types/node@^24.10.1, @types/react@^19.2.7, @types/react-dom@^19.2.3, @vitejs/plugin-react@^5.1.1, eslint@^9.39.1, eslint-plugin-react-hooks@^7.0.1, eslint-plugin-react-refresh@^0.4.24, framer-motion@^12.34.0, globals@^16.5.0, katex@^0.16.28, react@^19.2.0, react-dom@^19.2.0, react-markdown@^10.1.0, rehype-katex@^7.0.1, remark-math@^6.0.0, typescript@~5.9.3, typescript-eslint@^8.48.0, vite@^7.3.1 - raman-agent/backend/requirements.txt: fastapi@==0.109.0, httpx@==0.27.0, matplotlib@==3.8.2, numpy@==1.26.3, pandas@==2.1.4, python-dotenv@==1.0.1, python-multipart@==0.0.6, scipy@==1.11.4, uvicorn[standard]@==0.27.0 - raman-agent/frontend/package.json: @types/node@^20.11.5, @types/react@^18.2.48, @types/react-dom@^18.2.18, autoprefixer@^10.4.17, next@14.1.0, postcss@^8.4.33, react@^18.2.0, react-dom@^18.2.0, tailwindcss@^3.4.1, typescript@^5.3.3 ### Recent commits (newest first) - Correct project name from 'Auto Lab' to 'Autolab Labs' - Merge branch 'main' of github.com:nlee1126/TreeHacks_26 - final changes - Fix formatting in Architecture section of README - Rename project from AI Materials Researcher to Auto Lab - Merge branch 'main' of https://github.com/nlee1126/TreeHacks_26 - Update README.md - description - ui - title - UI - ea - Merge branch 'main' of github.com:nlee1126/TreeHacks_26 - video feed - Merge branch 'main' of https://github.com/nlee1126/TreeHacks_26 - stepper motor - clear for resynth - restore old flake finder - workbench animgst - workbench anim ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Development Commands ```bash make install # Install all dependencies make backend # Run backend (localhost:8000) make frontend # Run frontend (localhost:5173) ``` Requires `ANTHROPIC_API_KEY` environment variable. ## Architecture AI Materials Researcher - a multi-agent system for automated materials science experiments. ### Components **Backend (Python/FastAPI)** - `backend/main.py` - FastAPI app with WebSocket endpoint for real-time communication - `backend/agents/` - Agent implementations using Claude SDK - `orchestrator.py` - Master agent that creates plans and coordinates sub-agents - `synthesis.py` - Controls hardware for sample preparation - `characterization.py` - Performs Raman spectroscopy analysis - `theory.py` - Creates theoretical models, searches academic papers - `base.py` - Base agent class with Claude API integration - `backend/tools/` - Dummy tool implementations - `hardware.py` - Raspberry Pi motor control simulation - `microscope.py` - Microscope image capture simulation - `raman.py` - Raman spectroscopy data generation - `research.py` - Academic paper search **Frontend (React/TypeScript/Vite)** - `frontend/src/App.tsx` - Main app with WebSocket state management - `frontend/src/components/` - UI components - `ExperimentInput.tsx` - Experiment design input - `PlanView.tsx` - Shows orchestrator plan with approval workflow - `AgentBox.tsx` - Displays agent thinking and allows feedback - `FinalReport.tsx` - Shows final research report - `frontend/src/hooks/useWebSocket.ts` - WebSocket connection hook ### Agent Workflow 1. User submits experiment design 2. Orchestrator parses design, creates plan, waits for user approval 3. Synthesis Agent prepares each sample (exfoliate, microscope check, apply strain) 4. Characterization Agent measures Raman spectra, outputs observations 5. Theory Agent analyzes data, searches papers, creates model 6. If outliers detected, re-synthesis can be triggered 7. Final report generated ### WebSocket Message Types - `start_experiment` - User submits experiment design - `plan` - Orchestrator sends plan for approval - `approve_plan` - User approves plan - `agent_start/thought/complete` - Agent lifecycle events - `tool_use/tool_result` - Tool invocation events - `experiment_complete` - Final report ready ``` ### raman-agent/PROJECT_STRUCTURE.md ```markdown # Raman Agent Project Structure ``` raman-agent/ ├── README.md ├── backend/ │ ├── main.py # FastAPI application │ ├── raman.py # Core analysis algorithms │ └── requirements.txt # Python dependencies ├── frontend/ │ ├── app/ │ │ ├── globals.css # Global styles │ │ ├── layout.tsx # Root layout │ │ └── page.tsx # Main UI page │ ├── package.json # Node dependencies │ ├── tsconfig.json # TypeScript config │ ├── tailwind.config.js # Tailwind config │ ├── postcss.config.js # PostCSS config │ └── next.config.js # Next.js config └── samples/ ├── sample_spectrum.txt # Test spectrum 1 ├── sample_spectrum_2.txt # Test spectrum 2 ├── sample_spectrum_3.txt # Test spectrum 3 └── metadata.csv # Test metadata for trends ``` ## Quick Test 1. **Start Backend** (Terminal 1): ```bash cd backend python3 -m venv venv source venv/bin/activate pip install -r requirements.txt uvicorn main:app --reload ``` 2. **Start Frontend** (Terminal 2): ```bash cd frontend npm install npm run dev ``` 3. **Open Browser**: http://localhost:3000 4. **Test with Samples**: Upload files from `samples/` directory ## API Testing Test single file: ```bash curl -X POST http://localhost:8000/analyze_single \ -F "file=@samples/sample_spectrum.txt" ``` Test batch with trends: ```bash curl -X POST http://localhost:8000/analyze_batch \ -F "files=@samples/sample_spectrum.txt" \ -F "files=@samples/sample_spectrum_2.txt" \ -F "files=@samples/sample_spectrum_3.txt" \ -F "metadata_csv=@samples/metadata.csv" ``` ``` ### backend/requirements.txt ``` fastapi==0.109.0 uvicorn[standard]==0.27.0 anthropic==0.43.0 websockets==12.0 python-multipart==0.0.6 httpx==0.26.0 numpy>=1.24.0 scipy>=1.10.0 matplotlib>=3.7.0 opencv-python>=4.8.0 mss>=9.0.0 ``` ### flake_finder/requirements.txt ``` # Core opencv-python>=4.8 numpy>=1.24 mss>=9.0 Pillow>=10.0 # API VLM (main.py — Claude or OpenAI) anthropic>=0.39 openai>=1.40 # Local VLM (main_qwen.py — no API key needed) # pip install torch transformers accelerate ``` ### frontend/package.json ``` { "name": "frontend", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "framer-motion": "^12.34.0", "katex": "^0.16.28", "react": "^19.2.0", "react-dom": "^19.2.0", "react-markdown": "^10.1.0", "rehype-katex": "^7.0.1", "remark-math": "^6.0.0" }, "devDependencies": { "@eslint/js": "^9.39.1", "@types/node": "^24.10.1", "@types/react": "^19.2.7", "@types/react-dom": "^19.2.3", "@vitejs/plugin-react": "^5.1.1", "eslint": "^9.39.1", "eslint-plugin-react-hooks": "^7.0.1", "eslint-plugin-react-refresh": "^0.4.24", "globals": "^16.5.0", "typescript": "~5.9.3", "typescript-eslint": "^8.48.0", "vite": "^7.3.1" } } ``` ### raman-agent/backend/requirements.txt ``` fastapi==0.109.0 uvicorn[standard]==0.27.0 python-multipart==0.0.6 numpy==1.26.3 scipy==1.11.4 matplotlib==3.8.2 pandas==2.1.4 python-dotenv==1.0.1 httpx==0.27.0 ``` ### raman-agent/frontend/package.json ``` { "name": "raman-agent-frontend", "version": "1.0.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "react": "^18.2.0", "react-dom": "^18.2.0", "next": "14.1.0" }, "devDependencies": { "typescript": "^5.3.3", "@types/node": "^20.11.5", "@types/react": "^18.2.48", "@types/react-dom": "^18.2.18", "tailwindcss": "^3.4.1", "postcss": "^8.4.33", "autoprefixer": "^10.4.17" } } ``` ### backend/main.py ```python import asyncio import json from fastapi import FastAPI, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from typing import Optional from agents.orchestrator import OrchestratorAgent from video_feed import VideoFeedWorker app = FastAPI(title="AI Materials Researcher") # CORS for frontend app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:5173", "http://127.0.0.1:5173"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) class ExperimentSession: """Manages a single experiment session.""" def __init__(self, websocket: WebSocket): self.websocket = websocket self.orchestrator: Optional[OrchestratorAgent] = None self.is_running = False self.awaiting_approval = False self.message_queue = asyncio.Queue() self.current_task: Optional[asyncio.Task] = None # File upload waiting mechanism self._file_upload_event: asyncio.Event = asyncio.Event() self._file_upload_data: Optional[dict] = None # Video feed self.video_feed: Optional[VideoFeedWorker] = None async def send_message(self, message: dict): """Send message to frontend via WebSocket.""" msg_type = message.get("type", "unknown") if msg_type == "video_frame": return await self.websocket.send_json(message) agent = message.get("agent", "") prefix = f"[WS OUT] {msg_type}" if agent: prefix += f" ({agent})" # Print condensed info per message type if msg_type == "orchestrator_thought": print(f"{prefix}: {message.get('content', '')[:100]}") elif msg_type == "plan": steps = message.get("plan", []) print(f"{prefix}: {len(steps)} steps, current_step={message.get('current_step')}") elif msg_type == "agent_thought": print(f"{prefix}: {message.get('content', '')[:100]}") elif msg_type == "tool_use": print(f"{prefix}: {message.get('tool', '')} -> {json.dumps(message.get('input', {}))[:100]}") elif msg_type == "tool_result": result_str = json.dumps(message.get("result", {})) print(f"{prefix}: {result_str[:150]}") elif msg_type in ("agent_start", "agent_complete"): print(f"{prefix}: {message.get('context', message.get('result', ''))[:100]}") elif msg_type == "experiment_complete": print(f"{prefix}: iterations={message.get('iterations')}") else: print(f"{prefix}") await self.websocket.send_json(message) async def request_file_upload(self, sample_id: str, strain_percent: float) -> Optional[dict]: """Request a file upload from the frontend and wait for it.""" self._file_upload_event.clear() self._file_upload_data = None await self.send_message({ "type": "file_upload_needed", "agent": "characterization", "sample_id": sample_id, "strain_percent": strain_percent, }) # Wait for upload with timeout (5 minutes) try: await asyncio.wait_for(self._file_upload_event.wait(), timeout=300) except asyncio.TimeoutError: print(f"[Session] File upload timed out for sample {sample_id}") return None return self._file_upload_data async def start_experiment(self, design: str): """Start a new experiment with the given design.""" self.orchestrator = OrchestratorAgent( self.send_message, request_file_upload=self.request_file_upload, ) self.is_running = True self.awaiting_approval = True # Create and send plan for approval await self.orchestrator.run_experiment(design) async def approve_plan(self): """User approved the plan, execute it.""" print(f"[Session] approve_plan called: orchestrator={self.orchestrator is not None}, awaiting_approval={self.awaiting_approval}") if self.orchestrator and self.awaiting_approval: self.awaiting_approval = False try: await self.orchestrator.execute_approved_plan() except asyncio.CancelledError: print("[Session] execute_approved_plan was cancelled (restart in progress)") except Exception as e: print(f"[Session] ERROR in execute_approved_plan: {e}") import traceback traceback.print_exc() self.is_running = False else: print("[Session] approve_plan skipped — no orchestrator or not awaiting approval") async def send_feedback(self, agent: str, feedback: str): """Send user feedback to a specific agent via the orchestrator.""" if not self.orchestrator: await self.send_message({ "type": "error", "message": "No active experiment to send feedback to" }) return await self.send_message({ "type": "user_feedback_received", "agent": agent, "feedback": feedback }) if agent == "orchestrator": # Cancel current work, regenerate plan if self.current_task and not self.current_task.done(): self.current_task.cancel() try: await self.current_task except asyncio.CancelledError: pass self.awaiting_approval = True self.current_task = asyncio.create_task( self.orchestrator.handle_orchestrator_feedback(feedback) ) elif self.orchestrator.current_agent == agent: # Agent is currently running — inject feedback await self.orchestrator.handle_feedback(agent, feedback) else: # Agent is not running — cancel current task if any, then restart [truncated — 3743 more characters] ``` ### flake_finder/main.py ```python #!/usr/bin/env python3 """Flake Finder — fast CV contrast + Claude detect. v=scan c=cycle e=edges h=toggle 1-6=preset r=ROI s=save p=pause q=quit""" import base64, json, os, re, sys, time, threading from datetime import datetime from pathlib import Path _env = Path(__file__).resolve().parent / ".env" if _env.exists(): for ln in open(_env): ln = ln.strip() if ln and not ln.startswith("#") and "=" in ln: k, v = ln.split("=", 1) os.environ.setdefault(k.strip(), v.strip()) import cv2, mss, numpy as np ROOT = Path(__file__).resolve().parent CFG_PATH = ROOT / "config.json" OUT_DIR = ROOT / "outputs" OUT_DIR.mkdir(exist_ok=True) DEF_CFG = {"vlm_dim": 256, "jpeg_q": 55, "monitor": 1, "roi": None} CONF_MIN = 0.40 PRESETS = [(1.0, 0), (1.5, -15), (2.0, -35), (2.5, -55), (3.0, -75), (3.5, -95)] PNAMES = ["Raw", "Low", "Med", "High", "V.Hi", "Ext"] # Pre-compiled regex for JSON extraction _JSON_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)```") # Pre-allocated morph kernel _KERNEL = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)) def load_cfg(): if CFG_PATH.exists(): with open(CFG_PATH) as f: return {**DEF_CFG, **json.load(f)} return dict(DEF_CFG) def save_cfg(c): with open(CFG_PATH, "w") as f: json.dump(c, f, indent=2) # ── Helpers ────────────────────────────────────────────────────────────────── def hc_gray(frame, a, b): g = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) return cv2.cvtColor(cv2.convertScaleAbs(g, alpha=a, beta=b), cv2.COLOR_GRAY2BGR) def proc(frame, a, b, clahe_clip): """Contrast + optional CLAHE → (BGR, gray).""" g = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) hc = cv2.convertScaleAbs(g, alpha=a, beta=b) if clahe_clip > 0: hc = cv2.createCLAHE(clahe_clip, (8, 8)).apply(hc) return cv2.cvtColor(hc, cv2.COLOR_GRAY2BGR), hc def enc(img, dim, q): """JPEG base64 encode, resize if needed. Returns (b64, w, h).""" h, w = img.shape[:2] if max(h, w) > dim: s = dim / max(h, w) img = cv2.resize(img, (int(w * s), int(h * s)), cv2.INTER_LINEAR) sh, sw = img.shape[:2] _, buf = cv2.imencode(".jpg", img, [cv2.IMWRITE_JPEG_QUALITY, q]) return base64.b64encode(buf.tobytes()).decode(), sw, sh def pj(text): """Parse JSON from VLM response.""" m = _JSON_RE.search(text) if m: text = m.group(1) i, j = text.find("{"), text.rfind("}") if i >= 0 and j > i: try: return json.loads(text[i:j + 1]) except json.JSONDecodeError: pass return None def clamp(bb, w, h): if not bb or len(bb) != 4: return None x1, y1, x2, y2 = (int(round(v)) for v in bb) x1, y1 = max(0, min(w - 1, x1)), max(0, min(h - 1, y1)) x2, y2 = max(0, min(w - 1, x2)), max(0, min(h - 1, y2)) return (x1, y1, x2, y2) if x2 > x1 and y2 > y1 else None # ── CV tools ───────────────────────────────────────────────────────────────── def opt_contrast(gray): """Sweep alpha, auto-beta, score edges, try CLAHE. Returns (a, b, clip, score). Input: single-channel gray. ~15-25ms.""" n = gray.size mu = float(np.mean(gray)) best_a, best_b, best_sc = 2.0, -35, 0.0 for a10 in range(10, 35, 4): # 1.0 1.4 1.8 2.2 2.6 3.0 3.4 a = a10 / 10.0 b = max(-100, min(0, int(120 - a * mu))) hc = cv2.convertScaleAbs(gray, alpha=a, beta=b) if (np.count_nonzero(hc > 250) + np.count_nonzero(hc < 5)) / n > 0.30: continue sc = np.count_nonzero(cv2.Canny(hc, 50, 150)) / n if sc > best_sc: best_sc, best_a, best_b = sc, a, b # CLAHE on winner hc = cv2.convertScaleAbs(gray, alpha=best_a, beta=best_b) best_clip = 0.0 for cl in (2.5, 4.0): eq = cv2.createCLAHE(cl, (8, 8)).apply(hc) if (np.count_nonzero(eq > 250) + np.count_nonzero(eq < 5)) / n > 0.30: continue sc = np.count_nonzero(cv2.Canny(eq, 50, 150)) / n if sc > best_sc: best_sc, best_clip = sc, cl return best_a, best_b, best_clip, best_sc def find_cands(gray): """Candidates from Otsu + contours on preprocessed gray. ~15ms.""" ih, iw = gray.shape iarea = ih * iw _, bw = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU) bw = cv2.morphologyEx(cv2.morphologyEx(bw, cv2.MORPH_CLOSE, _KERNEL), cv2.MORPH_OPEN, _KERNEL) cnts, _ = cv2.findContours(bw, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) edges = cv2.Canny(gray, 50, 150) mu = float(np.mean(gray)) out = [] for c in cnts: a = cv2.contourArea(c) if a < 600: continue x, y, w, h = cv2.boundingRect(c) if w < 12 or h < 12 or (w * h) / iarea > 0.60: continue ed = np.count_nonzero(edges[y:y+h, x:x+w]) / max(1, w * h) if ed < 0.015: continue # Fast mean from bounding rect (avoids expensive mask) rm = float(np.mean(gray[y:y+h, x:x+w])) if abs(rm - mu) < 8: continue out.append({"bbox": [x, y, x+w, y+h], "area": int(a), "edge_score": round(ed, 3)}) out.sort(key=lambda c: c["edge_score"] * c["area"], reverse=True) return out[:8] def edge_overlay(gray, alpha, beta, clip): hc = cv2.convertScaleAbs(gray, alpha=alpha, beta=beta) if clip > 0: hc = cv2.createCLAHE(clip, (8, 8)).apply(hc) ov = np.zeros((*gray.shape, 3), dtype=np.uint8) ov[:, :, 1] = cv2.Canny(hc, 50, 150) return ov # ── Claude ─────────────────────────────────────────────────────────────────── SYS = ( "Microscope flake detector. ONLY JSON output.\n" "IGNORE overlays/UI/text/scale bars.\n" "GOOD flake: contiguous polygon, CONTRASTS with substrate, Sometimes the best isnt the biggest. Look for the most uniform" "uniform interior, SHARP outer edges. Minimal interior texture and features. Size 5-10% of image.\n" "NOT flake: dust, scratche [truncated — 10312 more characters] ``` ### frontend/src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import './index.css' import App from './App.tsx' createRoot(document.getElementById('root')!).render( , ) ``` [51 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]