BioPrep-AI

A CLI tool and Streamlit web UI that takes a raw biosensor file (signal or image) plus a plain-English description of the sensor and goal, audits it (deterministic math for signals, vision model + CV fallback for images), then generates, self-tests, and containerizes a custom preprocessing pipeline — with a self-healing retry loop if generated code fails.

Built with Codex & GPT-5.6

Built for OpenAI Build Week 2026, using OpenAI's models in two distinct roles:

Development — Codex was used throughout development to generate and refine project files and code, including scaffolding the CLI structure and generated- pipeline templates, refactoring the sandboxing and self-test generation flow, and sharpening ideas during iteration (tightening error handling, fixing test isolation issues, improving the audit-report generator).

Runtime — GPT-5.6 is supported as one of several interchangeable, OpenAI- compatible LLM providers (see "LLM providers" below). When configured, it is used for interpreting biosensor files during audit (vision-based image analysis, signal reasoning), selecting preprocessing stages based on the sensor description and audit results, and explaining quality-impact metrics in plain language. If no LLM is configured, or a call fails/times out, the tool falls back to deterministic, offline-safe logic automatically.

---

Requirements

Requirement	Version
Python	3.11 or higher
OS	Windows, macOS, or Linux
Disk space	~50 MB (dependencies)
Internet	Only needed for LLM providers (optional)

Python packages (installed automatically)

Core (always installed):

• numpy >= 1.26 — array math
• scipy >= 1.11 — signal processing (bandpass, PSD, Savitzky-Golay)
• opencv-python-headless >= 4.8 — image processing
• pillow >= 10.0 — image I/O
• typer >= 0.12 — CLI framework
• pytest >= 8.0 — test runner
• rich >= 13.0 — styled terminal output

Optional — LLM support:

• openai >= 1.40 — enables LLM-powered stage selection and metric annotations

Optional — Web UI:

• streamlit >= 1.30 — browser-based interface with file upload, preview, and downloads

Optional — file watcher:

• watchdog >= 4.0 — efficient file change detection for watch mode

---

Installation (step by step)

1. Clone the repo

git clone https://github.com/KharfiIslam/BioPrep-AI.git
cd BioPrep-AI-main

2. Check your Python version

python --version
# Must show 3.11 or higher

If you don't have Python 3.11+, download it from https://www.python.org/downloads/

3. Install all dependencies

Windows (double-click run.bat): A launcher menu will appear — pick option 3 to install, then 1 or 2 to run.

Or run from terminal:

# Install everything (core + LLM + web UI):
python -m pip install -e ".[ui,llm]"

# Or install only core (no LLM, no web UI):
python -m pip install -e .

# Or use requirements.txt directly:
python -m pip install -r requirements.txt

4. Generate sample files (optional)

python scripts/make_samples.py

This creates demo files in samples/ (EEG CSV, ECG NPY, lateral-flow JPG).

5. Verify installation

python -m pytest tests/ -v
# Should show all tests passing

---

How to run

Option A: Launcher script (easiest)

Windows: Double-click run.bat — a menu appears:

============================================
   preprocess-ops launcher
============================================

  Choose an option:

    1) CLI menu (terminal)
    2) Web UI (browser)
    3) Install / update dependencies
    4) Run tests
    5) Quit

  Enter number (1-5):

Pick 1 for the terminal menu, 2 for the browser UI.

Linux / macOS:

chmod +x run.sh
./run.sh

Option B: CLI menu (interactive)

python -m preprocess_ops

A rich terminal menu guides you through:

1. Audit a file
2. Generate a pipeline (audit + sandbox + self-heal)
3. Full demo (generate → test → build → report)
4. Run tests
5. Build Dockerfile
6. Configure LLM
7. Launch web UI
8. Quit

Option C: CLI one-shot commands

# Audit a signal file:
python -m preprocess_ops audit samples/eeg_sample.csv \
  --describe "3-electrode dry EEG, want it clean for seizure classification"

# Generate a pipeline (offline, no LLM):
python -m preprocess_ops generate samples/eeg_sample.csv \
  --describe "dry EEG cleanup" --offline

# Generate with LLM + full report:
python -m preprocess_ops generate samples/lateral_flow_strip.jpg \
  --describe "lateral flow strip quantification" --offline

# Run all tests:
python -m preprocess_ops test

# Build Docker image:
python -m preprocess_ops build --run

# Batch process a directory:
python -m preprocess_ops batch samples/ --describe "batch cleanup" --offline

# Watch for file changes:
python -m preprocess_ops watch samples/eeg_sample.csv --poll 2

Option D: Web UI (browser)

python -m streamlit run preprocess_ops/ui/streamlit_app.py

Opens http://localhost:8501 with four pages:

Page	What it does
Home	Overview, feature summary, LLM status
Pipeline	Upload file → preview → configure → run → download artifacts
LLM Config	Set up provider, API key, model — save for reuse
Batch	Upload multiple files, process all at once

---

What the tool produces

Every pipeline run creates these files in output/:

File	Description
profile.json	Audit results (shape, sampling rate, PSD peaks, lighting, blur, ROI)
pipeline.py	Generated, inspectable, editable preprocessing script
cleaned.npy / cleaned.png	Cleaned output (signal or image)
cleaned.metrics.json	ROI coordinates or intensity metrics (images only)
quality_comparison.json	Before/after quality deltas with percent changes
audit_report.html	One-page HTML report bundling everything
test_generated_pipeline.py	Pytest contract test for the generated pipeline
test_result.json	Test pass/fail status and pytest output
generation.json	Pipeline metadata (attempts, fallback status, sandbox transcript)

---

Quality metrics

Every run computes before/after quality deltas:

╭──────────── QUALITY IMPACT ────────────╮
│  Noise (raw)      0.412                │
│  Noise (cleaned)  0.087   ▼ 78.9%      │
│  Sharpness (raw)  152.7                │
│  Sharpness (clean) 210.3   ▲ 37.7%     │
╰─────────────────────────────────────────╯

Metric annotations explain why each metric changed:

• LLM-generated — configured model explains each delta in plain English
• Rule-based — deterministic fallback using a stage-effect knowledge base
• Mixed — LLM explains some metrics, rule-based covers the rest

---

Audit report

The HTML report bundles everything into a single shareable document:

1. Audit summary — modality, profiler, shape, sampling rate / lighting / blur / ROI
2. Quality impact — before/after metrics with percent deltas
3. Why metrics changed — LLM or rule-based annotations per metric
4. Sandbox result — pipeline source, attempts, transcript
5. Generated pipeline — full syntax-highlighted code
6. Test results — PASSED/FAILED with pytest output
7. Artifacts — file paths

Generate it with:

python -m preprocess_ops report --output-dir output/

---

LLM providers (optional)

Install LLM support:

python -m pip install -e ".[llm]"

Provider	Setup
OpenAI	PREPROCESS_OPS_PROVIDER=openai + OPENAI_API_KEY=...
Groq	PREPROCESS_OPS_PROVIDER=groq + GROQ_API_KEY=...
OpenRouter	openrouter + key — Kimi, GLM, and many others
Kimi / Moonshot	PREPROCESS_OPS_PROVIDER=kimi + API key
GLM / Zhipu	PREPROCESS_OPS_PROVIDER=glm + API key
DeepSeek	PREPROCESS_OPS_PROVIDER=deepseek + API key
Together	together + TOGETHER_API_KEY=...
Ollama (local)	PREPROCESS_OPS_PROVIDER=ollama
LM Studio (local)	PREPROCESS_OPS_PROVIDER=lmstudio
Other	PREPROCESS_OPS_PROVIDER=other + PREPROCESS_OPS_BASE_URL=...

Configure via CLI menu (option 6), web UI (LLM Config page), or environment variables. Settings can be saved to .preprocess_ops_llm.json for reuse.

---

CLI reference

Command	Purpose
python -m preprocess_ops / menu	Interactive numbered menu
audit FILE	Write output/profile.json, print audit summary
generate FILE	Audit + synthesize + sandbox + cleaned data + test + report
report	Bundle artifacts into audit_report.html
batch DIR	Process every supported file in a directory
watch FILE	Re-run on file changes (Ctrl-C to stop)
test	Run unit tests + generated contract test
build [--run]	Render Dockerfile, optionally docker build
web	Launch Streamlit UI

Common flags: --describe/-d, --sampling-rate/-r, --offline, --output-dir/-o, --run-id, --roi "x,y,w,h", --json, -v/--verbose.

---

Architecture

User input: file (CSV/NumPy or PNG/JPG) + free-text description
                    ↓
        [Modality Dispatcher] — routes by file type
        ↓                                    ↓
  [Signal Profiler]                   [Image Profiler]
  deterministic (scipy)               vision model call
  sampling rate, PSD peaks,           → on fail/timeout →
  amplitude, noise floor              CV fallback (OpenCV)
        ↓                                    ↓
              profile.json (unified schema)
                    ↓
        [Pipeline Synthesizer] (any OpenAI-compatible LLM)
        constrained stage plan inside fixed class template
                    ↓
        [Sandbox Execution + Self-Heal]
        run on sample → error/NaN? → retry once → trusted fallback
                    ↓
        [Quality Metrics] — before/after noise, RMS, contrast, sharpness
                    ↓
        [Metric Annotations] — LLM or rule-based explanations
                    ↓
        [Test Generator] → pytest contract test
                    ↓
        [Audit Report] → single HTML page
                    ↓
        [Containerizer] → Dockerfile

---

Repo layout

preprocess-ops/
├── run.bat                        # Windows launcher (double-click)
├── run.sh                         # Linux/macOS launcher
├── requirements.txt               # Dependency list
├── preprocess_ops/                # Python package
│   ├── cli.py                     # Typer CLI (menu + subcommands)
│   ├── service.py                 # Shared orchestration (run_pipeline)
│   ├── metrics.py                 # Before/after quality proxies
│   ├── annotations.py             # LLM + rule-based metric explanations
│   ├── report.py                  # HTML audit report renderer
│   ├── present.py                 # Rich terminal UI helpers
│   ├── llm.py                     # OpenAI-compatible LLM client
│   ├── audit/                     # Input file profiling
│   ├── filters/                   # Trusted preprocessing stages
│   ├── synth/                     # Pipeline code generation
│   ├── testgen/                   # Contract test generation
│   └── ui/                        # Streamlit web UI
│       └── streamlit_app.py       # 4-page interface
├── tests/                         # Test suite
├── samples/                       # Demo input files
├── docker/Dockerfile.template
├── scripts/make_samples.py
├── output/                        # Generated artifacts
└── pyproject.toml

---

Design principles

• Deterministic where precision matters. Signal audit is pure scipy; never LLM-guessed numbers.
• Scaffolded generation. The model chooses stages from a trusted whitelist; method bodies stay inside a fixed class template.
• Self-healing. Sandbox-run every pipeline → one retry with error context → trusted fallback.
• Same contract across modalities. Both profilers emit a shared profile.json shape.
• Quantified proof. Every run produces before/after quality metrics with percent deltas.
• Transparent explanations. Metric annotations show source (LLM / rule-based / mixed).
• Provenance by default. HTML report bundles everything into a single shareable document.

---

Quick troubleshooting

Problem	Fix
python not found	Use python3 instead, or add Python to PATH during install
pip not found	Use python -m pip instead
ModuleNotFoundError	Run python -m pip install -e ".[ui,llm]"
Streamlit won't start	Run python -m pip install streamlit>=1.30
LLM not connecting	Check provider URL and API key, or use --offline
Tests fail	Run python -m pip install -e . first to install the package