# Project export: Auto Chem This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Manage and measure your chem experiment with TLC easier - Devpost: https://devpost.com/software/auto-chem - GitHub: https://github.com/KaiyueLi-Bruce/hackathon - Team: 2 GitHub contributor(s) — KaiyueLi-Bruce (44 commits), Claude Opus 4.8 (1M context) (16 commits) ## Devpost submission (written by the team) ### Overview As a biology student, chemistry laboratory courses are an unavoidable part of my curriculum. Many experiments rely heavily on thin-layer chromatography (TLC), whether it's monitoring reaction progress in Suzuki coupling reactions or checking fractions during column chromatography purification. In a typical lab session, I often need to analyze five or more TLC plates. The process involves taking pictures, measuring solvent fronts manually, calculating Rf values by hand, and recording everything in lab notebooks. These repetitive tasks are time-consuming, prone to human error, and often produce inconsistent results. I wanted to build a tool that transforms TLC analysis from a manual process into a smart and automated workflow. Auto Chem is an AI-assisted TLC analysis and documentation platform. The software can: Automatically detect and crop TLC plate images from photos. Correct perspective distortion to standardize plate orientation. Identify solvent fronts, baseline positions, and spot locations. Calculate Rf values automatically. Use third-party AI APIs together with computer vision to improve spot recognition. Continuously improve detection accuracy through YOLO-based training. Save experiments with one click and maintain an archive of previous TLC plates. Generate AI-powered experimental reports automatically, reducing manual documentation work. Ultimately, Auto Chem aims to bridge computer vision and chemical laboratory workflows. I built Auto Chem using: SwiftUI for a native macOS interface. A canvas-first workflow that allows manual adjustment when automatic detection is uncertain. SQLite for experiment management and archiving. OpenCV for image preprocessing, perspective correction, and plate standardization. YOLO models for spot detection and future machine-learning improvements. Third-party AI APIs to assist machine vision and generate experiment reports. By combining traditional image processing with AI models, I created a system that is both accurate and user-friendly. One of the biggest challenges was balancing automation with reliability. TLC plates vary significantly between laboratories. Differences in lighting conditions, plate quality, UV intensity, camera angles, and spot appearance make automatic recognition difficult. Some spots are faint, overlapping, or partially invisible, making detection inconsistent. Another challenge was designing a workflow that integrates computer vision, machine learning models, local storage, and external AI services while maintaining a smooth user experience. Finally, because scientific applications require accuracy, I had to ensure users could always manually adjust results whenever automatic detection was uncertain. I'm proud that Auto Chem successfully turns a tedious laboratory task into a streamlined digital workflow. Some achievements I'm especially excited about include: Building a complete end-to-end TLC analysis pipeline. Combining AI with classical computer vision rather than relying on only one approach. Providing both automatic recognition and manual correction for higher reliability. Creating a searchable archive of experiments instead of leaving results scattered across notebooks and photos. Demonstrating how AI can genuinely improve productivity in scientific research rather than simply acting as a chatbot. Through this project, I learned that scientific software requires a different mindset from traditional applications. Accuracy and reproducibility are often more important than full automation. I also learned that machine learning alone is rarely enough; combining domain knowledge with classical image-processing techniques produces much better results. Most importantly, I gained experience integrating UI design, computer vision, AI APIs, and data management into a single product while keeping the workflow intuitive for real laboratory users. My next steps are: Expanding the YOLO training dataset with more annotated TLC images to improve detection accuracy. Supporting more chemical workflows and reaction types. Developing Auto Chem into a full-featured digital chemistry lab notebook. Recording additional experimental metadata such as solvents, reagents, yields, and reaction conditions. Generating more comprehensive AI-powered reports. Building better search and archival tools for long-term experiment management. My long-term vision is to create an intelligent laboratory assistant that helps researchers spend less time on repetitive documentation and more time on science. ## README (from the GitHub repository) # Auto Chem A macOS app for TLC (Thin-Layer Chromatography) plate analysis. Drop in a photo, get Rf values, AI interpretation, and a searchable experiment archive — automatically. --- ## What it does TLC is a daily routine in organic chemistry labs: develop a plate, hold it under UV, manually measure distances, hand-calculate Rf. Auto Chem automates that process: 1. **Import** a TLC plate photo 2. **Auto-detect** baseline, solvent front, and spots (OpenCV pipeline) 3. **Calculate Rf** values instantly 4. **Generate an AI report** — reaction status, spot interpretation, next-step suggestions 5. **Save** to a searchable local archive Manual adjustment is always available — drag lines and spots to correct anything the auto-detection got wrong. --- ## Requirements - macOS 14 (Sonoma) or later - Python 3.10+ (for the CV sidecar) - An [Anthropic API key](https://console.anthropic.com/) (for AI reports) - Xcode 15+ (to build from source) --- ## Setup ### 1. Clone the repo ```bash git clone https://github.com/KaiyueLi-Bruce/hackathon.git cd hackathon ``` ### 2. Set up the Python sidecar ```bash cd cv python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` ### 3. Start the sidecar ```bash # From the cv/ directory, with .venv activated python run.py ``` The sidecar runs at `http://localhost:8765` and handles all image processing. Keep this terminal open while using the app. ### 4. Build and run the app ```bash cd ../App swift build # or open in Xcode and press ▶ ``` Or open `App/` as a Swift package in Xcode and run the `Auto Chem` target. ### 5. Add your API key In the app: click the **gear icon** (top-right) → paste your Anthropic API key → Save. --- ## Basic workflow ### Analyze a plate 1. **Import a photo** — drag and drop a TLC plate image onto the canvas, or click **Import** in the floating toolbar. 2. **Auto-detect** — click the **Auto-detect** button (highlighted in the toolbar). The sidecar detects: - Baseline and solvent front lines - All spots with their Rf values - Lane assignments 3. **Review** — Rf values appear in the **Results** tab on the right. Drag the baseline or solvent front lines to fine-tune if needed. Click spots to add labels (SM / Product / By-product / Standard). 4. **Generate AI report** — switch to the **AI** tab → click **Generate AI report**. The report covers: - Reaction status (complete / incomplete / inconclusive) - Spot-by-spot interpretation - Next-step suggestions 5. **Save** — press **⌘S** or click the save button. The plate is stored in the local archive with its photo, Rf data, and report. ### Browse the archive Click the **grid icon** in the left rail to open the archive. Search by experiment name, date, or Rf range. --- ## Teaching the detector (online learning) The spot detector improves as you correct it: 1. Run auto-detect on a plate. 2. **Add missed spots** by clicking on the plate. **Remove false positives** by double-clicking a spot. 3. Save the plate — corrections are immediately fed back to the classifier. After a few plates the detector learns your typical plate appearance and needs fewer corrections. The inspector shows **"Learned from N corrections"** to track progress. --- ## YOLO model (optional, higher accuracy) A YOLOv8-based detector is available as a higher-accuracy fallback. It activates automatically once trained. ### Train the YOLO model ```bash cd cv source .venv/bin/activate pip install ultralytics # one-time python train_yolo.py --epochs 50 --n-synth 2000 ``` Training takes ~60–90 minutes on Apple Silicon (MPS). The script: 1. Generates 2000 synthetic TLC images from photos in `training_pictures/` 2. Trains YOLOv8n for 50 epochs 3. Exports to `cv/models/yolo_spot.onnx` You can also trigger training from the app: **Settings → YOLO Spot Detector → Re-train**. Once the model is ready, the status dot turns green and YOLO is used automatically when the standard detector finds zero spots. --- ## Project structure ``` hackathon/ ├── App/ # SwiftUI macOS app (Swift Package) │ └── Sources/ChromaLog/ │ ├── AppStore.swift # Central state │ ├── CVClient.swift # HTTP client for the sidecar │ └── Views/ # UI components ├── cv/ # Python sidecar (FastAPI + OpenCV) │ ├── chromalog_cv/ # Detection pipeline │ │ ├── pipeline.py # Main pipeline entry point │ │ ├── spots.py # Spot detection & Rf calculation │ │ ├── rectify.py # Perspective correction │ │ ├── learn.py # Online incremental classifier (SGD) │ │ └── yolo.py # YOLO ONNX inference │ ├── train_yolo.py # YOLO training script │ ├── models/ # ONNX model files (gitignored) │ └── tests/ # pytest test suite (45 tests) ├── training_pictures/ # Real TLC photos used for YOLO training └── docs/ # Design specs ``` --- ## Detection pipeline ``` Photo → Perspective correction (OpenCV contour → homography) → CLAHE illumination normalization → Auto-polarity binarization (minority class = spots) → Hough line detection (baseline + solvent front) → Connected-component spot candidates → Lane assignment (x-projection histogram) → SGD patch classifier (if trained, improves with corrections) → YOLO fallback (if model exists and classifier finds 0 spots) → Rf = (baselineY − spotY) / (baselineY − frontY) ``` --- ## Running tests ```bash cd cv source .venv/bin/activate pytest tests/ -q ``` 45 tests covering detection, learning, YOLO inference, and the FastAPI endpoints. --- ## Keyboard shortcuts | Action | Shortcut | |--------|----------| | Save plate | ⌘S | | Toggle left rail | toolbar sidebar button | | Toggle inspector | toolbar right-sidebar button | --- ## Tech stack | Layer | Technology | |-------|------------| | macOS UI | SwiftUI (macOS 14+) | | Local storage | SQLite via GRDB.swift | | Image processing | Python · OpenCV · FastAPI | | Spot classification | scikit-learn SGDClassifier (online learning) | | YOLO detection | Ultralytics YOLOv8n → ONNX Runtime | | AI reports | Anthropic Claude API | ## Detected evidence (automated analysis) Indexed codebase: 53 recognized source files, 413 KB. - FastAPI (technology) — detected in the code - Python (language) — detected in the code - Swift (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (57 of 57) ``` .claude/settings.local.json .gitignore App/Package.resolved App/Package.swift App/README.md App/Sources/ChromaLog/AppStore.swift App/Sources/ChromaLog/ChromaLogApp.swift App/Sources/ChromaLog/CVClient.swift App/Sources/ChromaLog/KeychainHelper.swift App/Sources/ChromaLog/Models.swift App/Sources/ChromaLog/Persistence/AppDatabase.swift App/Sources/ChromaLog/Persistence/Records.swift App/Sources/ChromaLog/SidecarManager.swift App/Sources/ChromaLog/Theme.swift App/Sources/ChromaLog/Views/ArchiveView.swift App/Sources/ChromaLog/Views/CanvasView.swift App/Sources/ChromaLog/Views/ContentView.swift App/Sources/ChromaLog/Views/DigitalPlateView.swift App/Sources/ChromaLog/Views/FilmstripView.swift App/Sources/ChromaLog/Views/FloatingToolbar.swift App/Sources/ChromaLog/Views/IconRail.swift App/Sources/ChromaLog/Views/InspectorView.swift App/Sources/ChromaLog/Views/LabelPicker.swift App/Sources/ChromaLog/Views/PlateCanvas.swift App/Sources/ChromaLog/Views/PlateExportView.swift App/Sources/ChromaLog/Views/SettingsView.swift ChromaLog-Spec.md cv/chromalog_cv/__init__.py cv/chromalog_cv/binarize.py cv/chromalog_cv/config.py cv/chromalog_cv/enhance.py cv/chromalog_cv/learn.py cv/chromalog_cv/lines.py cv/chromalog_cv/llm_detect.py cv/chromalog_cv/pipeline.py cv/chromalog_cv/preprocess.py cv/chromalog_cv/rectify.py cv/chromalog_cv/report.py cv/chromalog_cv/server.py cv/chromalog_cv/spots.py cv/chromalog_cv/yolo.py cv/README.md cv/requirements.txt cv/run.py cv/tests/__init__.py cv/tests/test_learn.py cv/tests/test_train_yolo.py cv/tests/test_yolo_infer.py cv/tests/test_yolo_pipeline.py cv/tests/test_yolo_server.py cv/tests/test_yolo_synth.py cv/train_yolo.py docs/superpowers/plans/2026-06-20-spot-classifier-online-learning.md docs/superpowers/plans/2026-06-21-yolo-spot-detection.md docs/superpowers/specs/2026-06-20-spot-classifier-online-learning-design.md docs/superpowers/specs/2026-06-21-yolo-spot-detection-design.md README.md ``` ### Dependencies - cv/requirements.txt: fastapi@>=0.110, numpy@>=1.24, opencv-python-headless@>=4.8, pillow@>=10.0, python-multipart@>=0.0.9, scikit-learn@>=1.4, uvicorn[standard]@>=0.27 ### Recent commits (newest first) - fix(ui): correct magnifier offset formula - feat(ui): magnifier loupe above spot while dragging - feat(ui): follow system dark/light mode (NSApp.appearance=nil + preferredColorScheme) - fix(ui): remove duplicate settings button from toolbar - fix(ui): widen inspector panel to 280px for Rf display - feat: rename app to Auto Chem - Rename project from ChromaLog to Auto Chem - docs: add README with setup and usage guide - Merge pull request #1 from KaiyueLi-Bruce/feat/yolo-spot-detection - Debug and better logic for openCV - fix(test): relax test_model_info_untrained assertion for extra line-clf fields - fix(app): YOLO GroupBox, trained_at date display, Task cancellation handle - feat(app): YOLO status polling + Re-train button in Settings - test(yolo): mock subprocess.Popen in train-yolo endpoint test - feat(yolo): /detect use_yolo param + POST /train-yolo + GET /yolo-model - feat(yolo): pipeline.py YOLO fallback when sklearn returns 0 spots - feat(yolo): training script train_yolo.py (MPS, synthetic→ONNX export) - feat(yolo): ONNX inference (detect_yolo + NMS); no model → empty result - fix(yolo): str|Path signature + remove dead imports - feat(yolo): synthetic TLC dataset generator + .gitignore entries ## Key source files (fetched from GitHub, selected and truncated for size) ### ChromaLog-Spec.md ```markdown # ChromaLog — TLC 视觉识别与化学实验档案（macOS） > 工作名 **ChromaLog**（备选名见附录 C，待定稿）。 > 本文档是交给 **Claude Code** 的构建规格：定义目标、技术栈、功能、数据模型、UI 方向、里程碑。 > 平台核心：**macOS 原生**。原则：**本地优先、注重 UI、手动优先 + 自动增强**。 --- ## 1. 一句话定位 拍一张 TLC（薄层色谱）板照片，自动标定、计算 Rf 值、生成可解读的实验结论，并把结果沉淀为可搜索的本地化学实验档案。 --- ## 2. 背景与痛点 TLC 是有机 / 药化 / 天然产物实验室每天要做几十次的基础操作，用来监控反应进度、判断纯度、比对化合物。但现状很原始： - 在 UV 灯下肉眼看板，拿尺子手量斑点距离，手算 `Rf = 斑点迁移距离 / 溶剂前沿距离`。 - 结果要么画进纸质实验本，要么拍张手机照丢进相册——零散、不一致、无法检索。 - Rf 只有配上溶剂体系 / 板型 / 显色方式才有意义，而这些条件常常没被记下来，导致难以复现。 **价值主张（面向非化学评委也讲得通）**：把一件繁琐的手工测量工作自动化，并把记录变成可搜索的电子实验档案。"乱糟糟的照片 → 干净的数字化平板 + 自动 Rf + 可检索档案"这个转变，谁都看得懂。 --- ## 3. 目标用户 有机 / 药物化学 / 天然产物方向的研究生、研究员、QC 人员；个人或小型实验室。 --- ## 4. 30 秒 Demo 脚本（北极星，团队和 Claude Code 都对齐这条线） 1. 拖入一张真实 TLC 板照片（多泳道、UV 下拍摄）。 2. 拖动两条参考线：基线（原点）和溶剂前沿；点选几个斑点。 3. 右侧即时出现 Rf 表，中间生成一块标准化重绘的"数字平板"。 4. 点"AI 分析"，自动生成一份带结论的实验小报告（反应是否完成、产物在哪、下一步建议）。 5. 保存 → 进入档案；演示一次搜索（"找产物 Rf≈0.4 的实验"）。 **安全网**：全流程用手动点选就能 100% 跑通，自动识别只是锦上添花——现场绝不翻车。 --- ## 5. 技术栈与架构（已定） | 层 | 选型 | 说明 | |---|---|---| | 前端 / 外壳 | **SwiftUI**（macOS 14+） | 原生、漂亮、贴合 HIG；Claude Code 擅长生成 SwiftUI | | 视觉处理 | **Python sidecar + OpenCV** | 作为本地子进程；MVP 阶段纯 SwiftUI/Core Graphics 手动点选即可，不依赖它 | | Swift ↔ Python 通信 | **本地 HTTP（FastAPI）**，JSON 往返 | 简单、易调试；备选：标准输入输出 / 进程管道 | | 本地存储 | **SQLite**（建议 GRDB.swift）+ 文件系统存原图与标注 | 本地优先，数据不出本机 | | AI 分析 | **Anthropic API** | 生成结果分析与报告；用户填 API key，存 macOS Keychain | | 打包 | Xcode app bundle | 黑客松阶段可先要求本机装 Python 环境；后续再用 PyInstaller / embedded venv 打进 bundle | **架构示意** ``` ┌─────────────────────────────────────────────┐ │ SwiftUI App (macOS) │ │ Sidebar │ Workspace 画布 │ Inspector │ │ 导航 │ 图像+标注+Rf │ 结果/AI │ └───────┬───────────────┬───────────────┬──────┘ │ GRDB │ 本地 HTTP │ HTTPS ▼ ▼ ▼ SQLite + 文件 Python sidecar Anthropic (本地档案) (OpenCV 斑点检测) (分析/报告) ``` **为什么这么选**：SwiftUI 给最"Mac"的漂亮原生体验；OpenCV 留在 Python 生态最成熟、文档最全（开发者无需 CV 经验，由 Claude Code 生成）；手动 MVP 让 app 在 CV 接好之前就能完整跑通和演示。备选栈（Tauri + React + Python）见附录 A。 --- ## 6. 信息架构与界面布局（画布优先 Canvas-first） > **设计原则更新**：放弃通用的「三等分三栏」，改为**画布优先**——导航瘦成图标栏把空间让给板子，工具条浮在画布上，检视器做成上下文卡片栈。所有 app 内文案为**英文**；所有界面背景使用系统色,**跟随系统浅色 / 深色自动切换**。 - **左侧 Icon Rail（窄图标导航，~54px）** - 图标项：Experiments（实验树）、All Plates、Compounds（指纹库）、Search；底部 Settings - 选中项高亮（强调色填充）；hover 展开浮层显示项目/实验列表，不常驻占宽 - 可完全折叠，进入「Focus / 纯画布标定」模式 - **中栏 Canvas（主舞台，占据最大空间）** - 当前板图像 + 标注画布：Baseline、Solvent front 两条可拖拽参考线，斑点标记 - **浮动玻璃工具条**（悬浮在画布底部居中，非顶部整行）：Import · Spot · **Auto-detect**（高亮为主操作）· Redraw - **底部 Reaction time course 胶片条**：多块板按时间排成 filmstrip，当前板高亮边框，一眼看出原料消失 / 产物长出；末尾「＋」加板 - **右侧 Inspector（上下文卡片栈，~208px）** - 顶部 **Segmented control** 切换：Results / Conditions / AI（不是常驻三 Tab 面板） - **Results**：Rf values 卡（tabular 数字对齐）、泳道与斑点列表、Co-spot check 判定卡 - **Conditions**：solvent system/ratio、stationary phase、visualization、plate type - **AI**：Generate AI report 按钮 + 报告预览 + 对板问答（chat） - 可隐藏，配合左栏折叠进入纯画布模式 --- ## 7. 核心 [truncated — 10989 more characters] ``` ### docs/superpowers/specs/2026-06-20-spot-classifier-online-learning-design.md ```markdown # 设计:斑点分类器在线增量学习 + 标注闭环(YOLO 留 seam) - 日期:2026-06-20 - 状态:已批准,待转实现计划 - 关联:`ChromaLog-Spec.md` 附录 D.3(模型 2/3)、D.5(标注闭环)、D.6(架构) ## 1. 背景与目标 用户诉求:「先把 YOLO 接进来,然后能通过手动标定不断学习」。 经核对 spec 与现状,确认一个核心矛盾并据此调整方向: - spec 中「通过手动标定不断学习/越标越准」的**真正机制是模型 2:sklearn `SGDClassifier.partial_fit` 在线增量 patch 分类器**——小数据即起效,标一条立刻更准(定位「M5 起步即可接」)。 - **YOLO 是模型 3**,明确是「数据攒够后最后上」的精度天花板,走**离线 fine-tune→ONNX**, **本质上不做在线增量**。现状无模型、无数据(3–7 张图)、venv 未装 ML 依赖。 **结论(用户已确认)**:本轮做 sklearn 在线增量这条轨道——标注闭环 + patch 二分类「越标越准」, 并为 YOLO 留下清晰的离线热替换接口(seam);**本轮不接 YOLO 推理/训练**。 ### 目标 1. 用户手动矫正斑点后,引擎能从矫正中学习,**下次检测对同类板更准**(抑制板角/反光/背景等误检)。 2. 学习闭环平台无关(Python sidecar),app 改动最小。 3. 留好 YOLO 热替换边界,下游(泳道/Rf/渲染)接口不变。 ### 非目标(本轮明确不做) YOLO 实际推理/训练;板分割 U-Net(正畸来源 B);ONNX 导出;主动学习 UI;显式「确认/否决」交互。 ## 2. 总体数据流 ``` 检测: OpenCV 候选 ──► [若已训练] sklearn 打分(P≥阈值保留) ──► 斑点 [未训练] 退回 AI 框过滤 / 面积拐点(现状) ▲ 保存矫正: app POST /learn 「正畸图 + 最终斑点 + 原始自动候选」 │ sidecar 派生样本: │ 保留的候选 / 用户新增 = 正样本 │ 被删的候选 = 硬负样本 (板角/反光… 信息量最大) │ 随机背景 = 易负样本 ▼ SGDClassifier.partial_fit → 落盘 pkl + 累积样本 npz → 下次检测立即更准 (越标越准) ``` ## 3. Sidecar(Python)新增 ### 3.1 `chromalog_cv/learn.py` - **特征提取** `patch_features(bgr, bbox_norm) -> np.ndarray`:按归一化 bbox 从正畸图裁 patch, resize 到定长(如 24×24)灰度并标准化,展平为像素向量 + 少量手工统计(均值/对比度/Sobel 梯度能量)。 定长输出,保证 `partial_fit` 维度稳定。 - **`SpotClassifier`**:封装 `StandardScaler`(用 `partial_fit` 在线更新均值方差)+ `SGDClassifier(loss="log_loss")`(输出概率,支持 `partial_fit(classes=[0,1])`); 方法 `update(X, y)`、`proba(X) -> p_real`、`save(path)`、`load(path)`、`is_trained`。 - **样本派生** `derive_samples(bgr, final_spots, auto_candidates) -> (X, y)`: - 候选 ↔ 最终斑点 用质心/IoU 匹配:被保留的候选 → 正;被删的候选(未匹配上)→ 硬负。 - 用户新增(最终斑点里匹配不到任何候选的)→ 正。 - 在板内随机采若干不与任何最终斑点重叠的位置 → 易负(数量与正样本数挂钩,平衡类别)。 - **持久化**:`cv/models/spot_clf.pkl`(分类器+scaler)、`cv/models/spot_samples.npz` (累积 X,y,供将来全量重训 / ONNX 导出)。 ### 3.2 端点 - **`POST /learn`**:multipart 接收正畸图 + JSON `{final_spots:[bbox_norm…], auto_candidates:[bbox_norm…]}`; `derive_samples` → `SpotClassifier.update`(partial_fit)→ 落盘; 返回 `{trained_total:int, batch:{pos:int, neg:int}, ok:true}`。任何异常返回 `{ok:false, error}`,不抛 500。 - **`GET /model`**:返回 `{trained:bool, n_samples:int, updated_at:str|null}`,供 app 显示学习状态。 ## 4. 检测集成与优先级(三级热替换) `pipeline.run_pipeline` 加载 `SpotClassifier`(若存在),把一个打分器传入 `spots.detect_spots`; 在现有候选(形状/线/边缘过滤后)之后增加一个过滤阶段,优先级: 1. **已训练分类器** → 对每个候选 patch 打分,`P(real) ≥ 0.5`(可配 `spot_clf_thresh`)保留 —— **替代面积拐点**。 2. **否则 AI 框**(若开 AI)→ 现有 `_filter_by_regions` + Plan C 兜底(已实现)。 3. **否则** → `_area_knee_cut` 面积拐点压噪(现状)。 `PipelineResult` 扩展:`engine_used` 增加 `+skl` 形态(如 `opencv+skl`、`ai+opencv+skl`); 新增布尔字段 `learned`(本次检测是否用了学习模型)。 ## 5. App(Swift)改动(最小) - **`AppStore`**:检测后把响应里的斑点另存为 `autoCandidates: [Spot]`(用户编辑前的原始候选,只读快照)。 - **`CVClient`**:新增 `learn [truncated — 1424 more characters] ``` ### cv/requirements.txt ``` # ChromaLog CV sidecar — 全部平台无关 (Win/macOS/Linux), 纯 CPU opencv-python-headless>=4.8 numpy>=1.24 fastapi>=0.110 uvicorn[standard]>=0.27 python-multipart>=0.0.9 pillow>=10.0 # 斑点 patch 在线增量分类 (M5+): scikit-learn>=1.4 # 升级项 (未启用): # onnxruntime>=1.17 # 跨平台推理 (板分割 U-Net / YOLO / sklearn->onnx) # skl2onnx # sklearn -> ONNX 导出 # ultralytics>=8.2 # YOLO training only: pip install ultralytics ``` ### cv/chromalog_cv/server.py ```python """跨平台 Python sidecar (FastAPI, 本地 HTTP, JSON)。 SwiftUI 通过本地端口调用 /detect, 引擎本身平台无关 (附录 D.2 / 第 5 节)。 启动: python -m chromalog_cv.server (默认 127.0.0.1:8765) 或: uvicorn chromalog_cv.server:app --host 127.0.0.1 --port 8765 """ from __future__ import annotations import base64 import json import subprocess import sys from typing import Optional from pathlib import Path as _Path import cv2 import numpy as np from fastapi import FastAPI, File, UploadFile, Query, Header, Form from fastapi.responses import JSONResponse from .config import Config from .pipeline import run_pipeline from . import rectify as R from .rectify import rectify as cv_rectify from .enhance import enhance_scan from . import llm_detect as L from . import report as RPT from . import learn as LN app = FastAPI(title="ChromaLog CV sidecar", version="0.1.0") # YOLO model paths and constants _CV_ROOT = _Path(__file__).resolve().parent.parent # cv/ _YOLO_ONNX = _CV_ROOT / "models" / "yolo_spot.onnx" _YOLO_LOCK = _CV_ROOT / "models" / ".yolo_training" _TRAIN_SCRIPT = _CV_ROOT / "train_yolo.py" @app.get("/health") def health(): return {"status": "ok", "engine": "opencv-auto-pipeline", "version": "0.1.0"} @app.get("/config") def get_config(): """返回可实时调节的旋钮及其默认值, 供 Swift UI 初始化滑块。""" d = Config() return {"tunable": {k: getattr(d, k) for k in Config.TUNABLE}} @app.post("/rectify") async def rectify_only( file: UploadFile = File(...), # ---- AI 找板正畸 (AI 粗定位 + OpenCV 精修四角); 失败自动回退纯 OpenCV ---- use_ai: bool = Query(False, description="启用 OpenRouter AI 找板 + OpenCV 精修四角正畸"), or_model: Optional[str] = Query(None, description="OpenRouter 模型 id (视觉)"), x_openrouter_key: Optional[str] = Header(None, description="OpenRouter API key"), ): """只做正畸 (快): 导入时调用, 让画布立即显示正畸后的图。不跑斑点检测。 use_ai+key+model 时让 AI 定位板主体, OpenCV 在该区域内精修四角再拉正; AI 失败绝不报错, 自动回退纯 OpenCV 正畸 (warnings 说明原因)。 """ try: img = _decode(await file.read()) except Exception as e: return JSONResponse(status_code=400, content={"error": str(e)}) cfg = Config() rec, engine, warns = _rectify(img, cfg, use_ai, or_model, x_openrouter_key) disp = enhance_scan(rec.image, cfg) if cfg.enhance_enabled else rec.image payload = { "width": int(disp.shape[1]), "height": int(disp.shape[0]), "rectified": rec.rectified, "rectify_confidence": round(rec.confidence, 3), "note": rec.note, "engine_used": engine, "warnings": warns, } ok, enc = cv2.imencode(".png", disp) if ok: payload["image_b64"] = base64.b64encode(enc.tobytes()).decode("ascii") return payload def _decode(buf: bytes) -> np.ndarray: arr = np.frombuffer(buf, np.uint8) img = cv2.imdecode(arr, cv2.IMREAD_COLOR) if img is None: raise ValueError("无法解码图像") return img def _rectify(img: np.ndarray, cfg: Config, use_ai: bool, or_model: Optional[str], key: Optional[str]): """统一正畸入口: OpenCV 优先, AI 仅在 OpenCV 失败/低置信时介入纠偏。 返回 (RectifyResult, engine_used, warnings)。AI 失败绝不报错, 只降级。 设计意义 (见与 detect_regions 同款的"AI 补 OpenCV 短板"哲学): 纯 OpenCV 对"板坐落在大片均匀背景上 + 透视"这类图已能可靠拉正; AI 预裁切反而会 破坏 OpenCV 找板所依赖的全图背景上下文, 把流程逼入更差的降级分支 (TLC_real_2 即此)。 故先跑 OpenCV, 够可信就采信; 仅当其失败/低置信 (背景杂乱/板非最大均匀区) 才请 AI 定位纠偏。""" warns = [] # ① 先跑纯 OpenCV (便宜, 且对干净背景的图已足够好) rec_cv = cv_rectify(img, cfg) if rec_cv.rectified and rec_cv.confidence >= cfg.rectify_cv_trust: return rec_cv, "opencv", warns # ② OpenCV 失败/低置信 -> 让 AI 找板纠偏 (AI 粗定位 + OpenCV 在该区域内精修四角) if use_ai and key and or_model: try: bbox, quad = L.detect_plate(img, key, or_model) rec_ai = R.rectify_ai(img, cfg, bbox=bbox, quad=quad) if rec_ai.rectified: return rec_ai, "ai+opencv", warns warns.append("AI 正畸亦未成功, 用 OpenCV 结果") except L.LLMError as e: warns.append(f"AI 找板不可用, 用 OpenCV 正畸: {e}") except Exception as e: warns.append(f"AI 找板异常, 用 OpenCV 正畸: {e}") return rec_cv, "opencv", warns @app.post("/learn") async def learn_endpoint( file: UploadFile = File(...), payload: str = Form(...), ): """从一次手动矫正在线增量训练斑点分类器 (设计 §3.2)。 payload: {"final_spots": [[x,y]...], "auto_candidates": [[x,y]...]} 归一化质心。 任何坏输入返回 ok:false, 不抛 500。""" try: img = _decode(await file.read()) data = json.loads(payload) final_pts = [(float(p[0]), float(p[1])) for p in data.get("final_spots", [])] auto_pts = [(float(p[0]), float(p[1])) for p in data.get("auto_candidates", [])] baseline_y = data.get("baseline_y") front_y = data.get("front_y") baseline_y = float(baseline_y) if baseline_y is not None else None front_y = float(front_y) if front_y is not None else None except Exception as e: return JSONResponse(status_code=200, content={"ok": False, "error": str(e)}) try: return LN.apply_correction(img, final_pts, auto_pts, Config(), baseline_y=baseline_y, front_y=front_y) except Exception as e: return JSONResponse(status_code=200, content={"ok": False, "error": str(e)}) @app.get("/model") def model_endpoint(): return LN.model_info(LN.CLF_PATH, LN.SAMPLES_PATH) @app.post("/report") async def report_endpoint( payload: str = Form(...), mode: str = Query("report", description="questions | report"), model: str = Query(...), x_openrouter_key: str = Header(None), ): """AI 实验报告 (spec §10)。 payload: {"data": {...Rf/条件/时程...}, "notebook": "", "answers": ""} mode=questions -> {questions:[...]}; mode=report -> {markdown:"..."}。""" try: body = json.loads(payload) data = body.get("data", {}) notebook = str(body.get("notebook", "") or "") answers = str(body.get("answers", "") or "") except Exception as e: return JSONResponse(status_code=200, content={"ok": False, "error": str(e) [truncated — 4870 more characters] ``` ### App/Package.swift ```swift // swift-tools-version:5.9 import PackageDescription let package = Package( name: "AutoChem", platforms: [ .macOS(.v14) ], dependencies: [ .package(url: "https://github.com/groue/GRDB.swift.git", from: "6.0.0") ], targets: [ .executableTarget( name: "AutoChem", dependencies: [ .product(name: "GRDB", package: "GRDB.swift") ], path: "Sources/ChromaLog" ) ] ) ``` ### cv/run.py ```python #!/usr/bin/env python3 """本地 CLI: 对单张图跑全自动流水线, 打印 JSON, 可选导出调试图。 纯 OpenCV: python run.py [--debug out.png] AI 正畸+检测: python run.py --use-ai --or-model [--debug out.png] (key 取自环境变量 OPENROUTER_API_KEY, 或 cv/.env 里的同名项) """ import argparse import json import os import sys import cv2 from chromalog_cv.config import Config from chromalog_cv.pipeline import run_pipeline from chromalog_cv import llm_detect as L from chromalog_cv.server import _rectify def _load_env_file(): """简易加载脚本同目录下的 .env (KEY=VALUE), 不覆盖已存在的环境变量。""" env_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".env") if not os.path.isfile(env_path): return for line in open(env_path, encoding="utf-8"): line = line.strip() if not line or line.startswith("#") or "=" not in line: continue k, v = line.split("=", 1) k, v = k.strip(), v.strip().strip('"').strip("'") os.environ.setdefault(k, v) def main(): ap = argparse.ArgumentParser() ap.add_argument("image", help="输入图片路径") ap.add_argument("--debug", help="导出调试叠加图路径") ap.add_argument("--use-ai", action="store_true", help="启用 OpenRouter 找板正畸 + 斑点粗框") ap.add_argument("--or-model", default=os.environ.get("OPENROUTER_MODEL"), help="OpenRouter 视觉模型 id, 如 google/gemini-2.0-flash-001") args = ap.parse_args() _load_env_file() key = os.environ.get("OPENROUTER_API_KEY") img = cv2.imread(args.image) if img is None: print(json.dumps({"error": f"无法读取图片: {args.image}"}, ensure_ascii=False)) sys.exit(1) cfg = Config() # ① 正畸: AI 找板(粗) + OpenCV 精修四角; 失败/未启用 -> 纯 OpenCV (与 server 同款逻辑) use_ai = args.use_ai if use_ai and not (key and args.or_model): print("[警告] --use-ai 已开启, 但缺少 OPENROUTER_API_KEY 或 --or-model, " "将回退纯 OpenCV。", file=sys.stderr) rec, engine, warns = _rectify(img, cfg, use_ai, args.or_model, key) # ② AI 斑点粗框 (同一张正畸图); 失败/未启用 -> OpenCV 兜底 llm_regions = None if use_ai and key and args.or_model: try: reg = L.detect_regions(rec.image, key, args.or_model) llm_regions = reg.regions engine = "ai+opencv" except L.LLMError as e: warns.append(f"AI 斑点检测不可用, 回退 OpenCV: {e}") result, debug_img, rect_img = run_pipeline( img, cfg, debug=bool(args.debug), llm_regions=llm_regions, engine_used=engine, rect=rec, ) payload = result.to_json() if isinstance(payload, dict): payload.setdefault("warnings", []) for w in warns: if w not in payload["warnings"]: payload["warnings"].append(w) payload["engine_used"] = engine print(json.dumps(payload, ensure_ascii=False, indent=2)) print(f"[engine_used] {engine}", file=sys.stderr) if args.debug and debug_img is not None: cv2.imwrite(args.debug, debug_img) rect_path = args.debug.rsplit(".", 1)[0] + "_rectified.png" cv2.imwrite(rect_path, rect_img) print(f"[调试图已保存] {args.debug}\n[正畸图已保存] {rect_path}", file=sys.stderr) if __name__ == "__main__": main() ``` ### cv/train_yolo.py ```python #!/usr/bin/env python3 """Standalone YOLO training script (M5+ c). Usage: python train_yolo.py # generate synth data + train + export ONNX python train_yolo.py --skip-synth # skip synth generation (data/synth/ exists) python train_yolo.py --epochs 30 # override epoch count Requires: pip install ultralytics (not in default venv) """ import argparse import sys import time from pathlib import Path ROOT = Path(__file__).resolve().parent # cv/ SYNTH = ROOT / "data" / "synth" MODELS = ROOT / "models" ONNX = MODELS / "yolo_spot.onnx" LOCK = MODELS / ".yolo_training" sys.path.insert(0, str(ROOT)) def main() -> None: parser = argparse.ArgumentParser(description="Train YOLOv8n spot detector") parser.add_argument("--skip-synth", action="store_true", help="Skip synthetic data generation (use existing data/synth/)") parser.add_argument("--epochs", type=int, default=50) parser.add_argument("--n-synth", type=int, default=2000, help="Number of synthetic images to generate") args = parser.parse_args() try: from ultralytics import YOLO except ImportError: print("ERROR: ultralytics not installed. Run: pip install ultralytics", file=sys.stderr) sys.exit(1) MODELS.mkdir(exist_ok=True) # Write lockfile so sidecar can report "training" status LOCK.write_text(str(time.time())) try: _run(args, YOLO) finally: if LOCK.exists(): LOCK.unlink() def _run(args, YOLO) -> None: from chromalog_cv.yolo import generate_synthetic_dataset # ── Step 1: Generate synthetic data ───────────────────────────────────── if not args.skip_synth: print(f"[train_yolo] Generating {args.n_synth} synthetic images → {SYNTH}") real_dir = ROOT.parent / "training_pictures" generate_synthetic_dataset( real_images_dir=str(real_dir), out_dir=str(SYNTH), n=args.n_synth, ) print("[train_yolo] Synthetic data ready.") else: print(f"[train_yolo] --skip-synth: using existing {SYNTH}") dataset_yaml = SYNTH / "dataset.yaml" if not dataset_yaml.exists(): print(f"ERROR: {dataset_yaml} not found. Run without --skip-synth first.", file=sys.stderr) raise SystemExit(1) # ── Step 2: Pre-train on synthetic data ────────────────────────────────── print(f"[train_yolo] Training YOLOv8n for {args.epochs} epochs on synthetic data (device=mps)…") model = YOLO("yolov8n.pt") model.train( data=str(dataset_yaml), epochs=args.epochs, imgsz=640, device="mps", project=str(ROOT / "runs"), name="yolo_spot", exist_ok=True, verbose=False, ) # ── Step 3: Optional fine-tune on real annotated data ──────────────────── real_dataset = ROOT / "data" / "real" / "dataset.yaml" if real_dataset.exists(): print(f"[train_yolo] Fine-tuning on real data ({real_dataset}) for 20 epochs…") model.train( data=str(real_dataset), epochs=20, imgsz=640, device="mps", lr0=0.001, project=str(ROOT / "runs"), name="yolo_spot_finetune", exist_ok=True, verbose=False, ) # ── Step 4: Export to ONNX ─────────────────────────────────────────────── print(f"[train_yolo] Exporting to ONNX → {ONNX}") model.export(format="onnx", opset=12, simplify=True) # ultralytics writes to runs/.../weights/best.onnx — move to models/ import shutil candidates = list((ROOT / "runs").rglob("best.onnx")) if not candidates: raise RuntimeError("ONNX export produced no best.onnx file") shutil.copy2(str(candidates[-1]), str(ONNX)) print(f"[train_yolo] Model saved to {ONNX}") # ── Step 5: Smoke-test with ONNX Runtime ───────────────────────────────── try: import onnxruntime as ort import numpy as np sess = ort.InferenceSession(str(ONNX), providers=["CPUExecutionProvider"]) dummy = np.zeros((1, 3, 640, 640), dtype=np.float32) t0 = time.perf_counter() sess.run(None, {sess.get_inputs()[0].name: dummy}) ms = (time.perf_counter() - t0) * 1000 print(f"[train_yolo] ONNX smoke-test passed. Inference latency: {ms:.1f} ms") except Exception as e: print(f"[train_yolo] WARNING: ONNX smoke-test failed: {e}", file=sys.stderr) print("[train_yolo] Done.") if __name__ == "__main__": main() ``` ### cv/chromalog_cv/preprocess.py ```python """② 光照归一化 (附录 D.2)。 UV 灯下拍摄光照极不均匀 (中间亮、四周暗、有辉光), 直接阈值会被光照干扰。 CLAHE 自适应直方图均衡压掉不均匀光照, 让后续二值化稳定。 """ from __future__ import annotations import cv2 import numpy as np from .config import Config def to_gray_clahe(bgr: np.ndarray, cfg: Config) -> np.ndarray: gray = cv2.cvtColor(bgr, cv2.COLOR_BGR2GRAY) clahe = cv2.createCLAHE(clipLimit=cfg.clahe_clip, tileGridSize=(cfg.clahe_grid, cfg.clahe_grid)) return clahe.apply(gray) ``` ### cv/chromalog_cv/__init__.py ```python """ChromaLog 跨平台 CV 引擎 (M5 全自动 OpenCV 流水线)。 设计原则 (见 ChromaLog-Spec.md 附录 D): - 全自动: 无需手动定位四角 / 基线 / 前沿; 手动只做事后矫正。 - 跨平台: 纯 OpenCV + NumPy, 平台无关; 模型升级走 ONNX。 - 优雅降级: 任一步失败均回退, 绝不让单张难图整链崩溃。 流水线顺序 (附录 D.2): ① 自动找板 + 透视校正 -> rectify.py ② 光照归一化 (CLAHE) -> preprocess.py ④ Hough 铅笔基线/前沿 -> lines.py (在 gray 上先于③, 为③提供 ROI) ③ 自动极性二值化 -> binarize.py ⑤ 斑点候选 + ⑥ 泳道归组 -> spots.py 编排 -> pipeline.py """ from .config import Config from .pipeline import run_pipeline, PipelineResult __all__ = ["Config", "run_pipeline", "PipelineResult"] __version__ = "0.1.0" ``` ### cv/tests/test_yolo_infer.py ```python import numpy as np import pytest def test_detect_yolo_no_model_returns_empty(tmp_path, monkeypatch): """With no .onnx file, detect_yolo must return empty SpotsResult.""" import chromalog_cv.yolo as Y monkeypatch.setattr(Y, "YOLO_ONNX_PATH", tmp_path / "nonexistent.onnx") monkeypatch.setattr(Y, "_YOLO_SESSION", None) from chromalog_cv.config import Config img = np.zeros((200, 400, 3), dtype=np.uint8) result = Y.detect_yolo(img, Config()) assert result.spots == [] def test_nms_removes_overlapping_boxes(): from chromalog_cv.yolo import _nms boxes = np.array([ [10, 10, 50, 50], [12, 12, 52, 52], # heavily overlaps with box 0 → should be suppressed [200, 200, 240, 240], # no overlap → kept ], dtype=np.float32) scores = np.array([0.9, 0.8, 0.7], dtype=np.float32) kept = _nms(boxes, scores, iou_thr=0.45) assert 0 in kept assert 1 not in kept assert 2 in kept def test_nms_empty_input(): from chromalog_cv.yolo import _nms kept = _nms(np.zeros((0, 4), dtype=np.float32), np.zeros(0, dtype=np.float32)) assert kept == [] ``` [40 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]