# Project export: Myko

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2026
- Tagline: Real-time vision language agent for microscopy
- Devpost: https://devpost.com/software/myko
- GitHub: https://github.com/Yammmma/myko-treehacks
- Video: https://www.youtube.com/embed/U53GfTdG3D0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Amy (30 commits), shayhacker (4 commits)

## Devpost submission (written by the team)

### Inspiration

You can buy an optical microscope on Amazon for $80, but the expertise required to interpret what you’re seeing takes years of medical training. Whether it’s a high school student seeing "purple blobs" instead of mitosis, or a rural nurse unable to confirm a diagnosis without a pathologist on-site, the problem is the same: Access to optics is cheap; access to answers is expensive. We built Myko to bridge this gap.

### What it does

Myko is a real-time language agent for optical microscopy. Users clip their phone to any microscope, and Myko streams the feed to our backend, where agentic models run real-time segmentation, detection, and cellular analysis. The system allows you to ask natural language questions such as "Is this tissue healthy?" or make commands such as "Highlight the macrophages," “Count the nuclei,” or “Segment abnormal regions.”

### How we built it

Myko is a hybrid edge-cloud system designed for low latency and high intelligence. Frontend: Native iOS app built with SwiftUI. Uses AVFoundation for real-time camera capture, Apple SpeechAnalyzer framework for on-device speech-to-text, and a persistent WebSocket connection to stream microscope frames to the backend at full frame rate Backend: Python FastAPI server exposing a REST endpoint and a WebSocket. The agent calls a vision-language model via the OpenAI-compatible API for image understanding and tool-use reasoning. A custom OpenCV/watershed segmentation pipeline (with optional SAM2 and Cellpose backends) proposes and renders cell masks in real time, overlaying results onto the streamed frames before returning them to the client. Frames are tunneled to the device via ngrok. When the VLM receives a query, it invokes specific computer vision tools (segmentation and classification models) running on the GX10's Nvidia GB100 GPU. The server processes the frame and sends the segmentation masks back to the iPhone to be overlaid in real-time.

### Challenges we ran into

About halfway through the hackathon, our primary demo microscope fell off the table and shattered into several pieces. We had to duct-tape the optics and realign the lenses to get a clear image again. Hardware is hard!

### Accomplishments we're proud of

Seamless AR Overlay: Seeing the AI draw a perfect bounding box around a microscopic cell in real-time on a phone screen feels like magic. The Architecture: We successfully integrated a complex tool-calling loop (Audio -> Text -> LLM -> CV Tool -> Visual Overlay) that feels instantaneous to the user.

### What we learned

VLMs need tools: Pure vision models are great at describing "a slide of cells," but they struggle with specific tasks like "count exactly 14 cells." The agentic approach (calling a counting tool) is far superior. Microscopy is messy: Real-world slides have dust, bubbles, and bad lighting. We learned a lot about preprocessing images to make them readable for the AI. The power of edge compute: Moving the inference to the ASUS GX10 was critical; the phone simply couldn't handle the heavy segmentation models alongside the AR rendering.

### What's next

We want to distill our models down to run locally on-device for use in areas without internet access.

## README (from the GitHub repository)

# myko-treehacks

## Detected evidence (automated analysis)

Indexed codebase: 22 recognized source files, 149 KB.
- Python (language) — detected in the code
- Swift (language) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (43 of 43)

```
.gitignore
.gitmodules
backend/agent.py
backend/server.py
backend/tracker.py
myko-treehacks.xcodeproj/project.pbxproj
myko-treehacks.xcodeproj/project.xcworkspace/contents.xcworkspacedata
myko-treehacks.xcodeproj/project.xcworkspace/xcshareddata/WorkspaceSettings.xcsettings
myko-treehacks/AppState.swift
myko-treehacks/Assets.xcassets/AccentColor.colorset/Contents.json
myko-treehacks/Assets.xcassets/AppIcon.appiconset/Contents.json
myko-treehacks/Assets.xcassets/cell1.imageset/Contents.json
myko-treehacks/Assets.xcassets/cell2.imageset/Contents.json
myko-treehacks/Assets.xcassets/cell3.imageset/Contents.json
myko-treehacks/Assets.xcassets/cell4.imageset/Contents.json
myko-treehacks/Assets.xcassets/cell5.imageset/Contents.json
myko-treehacks/Assets.xcassets/cell6.imageset/Contents.json
myko-treehacks/Assets.xcassets/Contents.json
myko-treehacks/Assets.xcassets/myko-logo-1.imageset/Contents.json
myko-treehacks/Assets.xcassets/myko-logo-transparent.imageset/Contents.json
myko-treehacks/Assets.xcassets/myko-micro.imageset/Contents.json
myko-treehacks/Assets.xcassets/myko-position1.imageset/Contents.json
myko-treehacks/Assets.xcassets/myko-position2.imageset/Contents.json
myko-treehacks/Assets.xcassets/myko-position3.imageset/Contents.json
myko-treehacks/ChatMessage.swift
myko-treehacks/Extensions/UIImage+Extensions.swift
myko-treehacks/HistoryCardView.swift
myko-treehacks/HistoryItem.swift
myko-treehacks/Info.plist
myko-treehacks/Item.swift
myko-treehacks/myko_treehacksApp.swift
myko-treehacks/OnboardingView.swift
myko-treehacks/SpeechAnalyzerTranscriptionService.swift
myko-treehacks/Styles/MykoColors.swift
myko-treehacks/ViewModels/ChatViewModel.swift
myko-treehacks/ViewModels/EndpointViewModel.swift
myko-treehacks/ViewModels/HandsFreeModeController.swift
myko-treehacks/Views/CameraView.swift
myko-treehacks/Views/ChatPopupView.swift
myko-treehacks/Views/ContentView.swift
myko-treehacks/Views/HistoryView.swift
myko-treehacks/Views/HomeView.swift
README.md
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Implemented endpoint reset flow
- final fixes
- Merge remote-tracking branch 'refs/remotes/origin/camera-feed'
- fixed
- Merge commit '5a996eafc1ab3a6e049e565ceb5fb740de3f188e' into camera-feed
- Merge commit 'edc4abb5ca2bd96edb786901eaa6e8debd3e88ce' into camera-feed
- fixed camera issue
- Mini checkpoint
- change wording in onboarding
- fixed version
- safe 2
- add delete and share in history as possible options
- fixed ui
- fixed ui for history
- add bounding box (slightly glitchy)
- rotate image to be upright
- added basic bounding box
- merged
- Merge remote-tracking branch 'refs/remotes/origin/camera-feed'
- Merge commit '09d6176dcf0c533b22b7144644df62ea7d18d73f' into camera-feed

## Key source files (fetched from GitHub, selected and truncated for size)

### backend/server.py

```python
import asyncio
import json

import uvicorn
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel

from agent import Agent
from tracker import Segmenter

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

segmenter = Segmenter()
agent = Agent(segmenter, debug=True)
print("🚀 Server ready on :8000  (set OPENAI_API_KEY before sending /query requests)")


class ChatRequest(BaseModel):
    prompt: str
    frame: str | None = None


@app.post("/query")
async def query_endpoint(request: ChatRequest):
    response = await agent.query(request.prompt, request.frame)
    return {"response": response}


def _extract_frame(message: dict) -> str | None:
    if "text" in message:
        try:
            data = json.loads(message["text"])
            return data.get("frame")
        except Exception:
            return message["text"]
    if "bytes" in message:
        try:
            data = json.loads(message["bytes"])
            return data.get("frame")
        except Exception:
            return message["bytes"].decode("utf-8")
    return None


@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    print("✅ WS Connected")

    latest_frame: dict = {"data": None, "event": asyncio.Event()}

    async def receiver():
        try:
            while True:
                message = await websocket.receive()
                if message["type"] == "websocket.disconnect":
                    raise WebSocketDisconnect()
                frame = _extract_frame(message)
                if frame:
                    latest_frame["data"] = frame
                    latest_frame["event"].set()
        except (WebSocketDisconnect, Exception):
            latest_frame["event"].set()
            raise

    async def processor():
        try:
            while True:
                await latest_frame["event"].wait()
                latest_frame["event"].clear()

                frame = latest_frame["data"]
                if frame is None:
                    continue

                try:
                    edited_frame = await asyncio.to_thread(segmenter.render_frame, frame)
                except Exception as render_err:
                    print(f"⚠️ Frame render error: {render_err}")
                    edited_frame = frame.split(",", 1)[-1] if "," in frame else frame

                await websocket.send_text(edited_frame)
        except Exception:
            raise

    try:
        await asyncio.gather(receiver(), processor())
    except WebSocketDisconnect:
        print("❌ WS Disconnected")
    except Exception as err:
        print(f"⚠️ WS Error: {err}")


if __name__ == "__main__":
    uvicorn.run("server:app", host="0.0.0.0", port=8000, reload=True, timeout_keep_alive=300000)

```

### myko-treehacks/Item.swift

```swift
//
//  Item.swift
//  myko-treehacks
//
//  Created by Amy Sun Key on 2/13/26.
//


import Foundation
import SwiftData

@Model
final class Item {
    var timestamp: Date
    
    init(timestamp: Date) {
        self.timestamp = timestamp
    }
}


```

### myko-treehacks/ChatMessage.swift

```swift
//
//  ChatMessage.swift
//  myko-treehacks
//
//  Created by Amy Sun Key on 2/13/26.
//


import Foundation

struct ChatMessage: Identifiable, Equatable {
    enum Role: String, Codable {
        case user
        case myko
    }

    let id: UUID
    let role: Role
    let text: String
    let timestamp: Date

    init(id: UUID = UUID(), role: Role, text: String, timestamp: Date = .now) {
        self.id = id
        self.role = role
        self.text = text
        self.timestamp = timestamp
    }
}
```

### myko-treehacks/AppState.swift

```swift
//
//  AppState.swift
//  myko-treehacks
//
//  Created by Amy Sun Key on 2/13/26.
//

import Combine
import Foundation

@MainActor
final class AppState: ObservableObject {
    enum Tab: Hashable {
        case home
        case history
    }
    
    enum HistorySortMode: Hashable {
        case newest
        case favorites
    }

    @Published var selectedTab: Tab = .home
    @Published var historySortMode: HistorySortMode = .newest

    let historyStore = HistoryStore()
    private var cancellables = Set<AnyCancellable>()

    init() {
        historyStore.objectWillChange
            .sink { _ in
                DispatchQueue.main.async { [weak self] in
                    self?.objectWillChange.send()
                }
            }
            .store(in: &cancellables)
    }
}

```

### myko-treehacks/myko_treehacksApp.swift

```swift
//
//  myko_treehacksApp.swift
//  myko-treehacks
//
//  Created by Yuma Soerianto on 2/13/26.
//

import SwiftUI
import SwiftData

@main
struct myko_treehacksApp: App {
    @StateObject private var appState = AppState()
    var sharedModelContainer: ModelContainer = {
        let schema = Schema([
            Item.self,
        ])
        let modelConfiguration = ModelConfiguration(schema: schema, isStoredInMemoryOnly: false)
        
        do {
            return try ModelContainer(for: schema, configurations: [modelConfiguration])
        } catch {
            fatalError("Could not create ModelContainer: \(error)")
        }
    }()
    
    var body: some Scene {
        WindowGroup {
            RootView()
                .environmentObject(appState)
        }
        .modelContainer(sharedModelContainer)
    }
}

struct RootView: View {
    @EnvironmentObject private var appState: AppState
    @State private var isOnboardingPresented = true
    
    var body: some View {
        TabView(selection: $appState.selectedTab) {
            
            HomeView()
                .tabItem { Label("Home", systemImage: "house") }
                .tag(AppState.Tab.home)
            HistoryView()
                .tabItem { Label("History", systemImage: "clock") }
                .tag(AppState.Tab.history)

        }
        .tint(MykoColors.leafBase)
        .fullScreenCover(isPresented: $isOnboardingPresented) {
            OnboardingView(isPresented: $isOnboardingPresented)
        }
    }
    init() {
        UIApplication.shared.isIdleTimerDisabled = true
    }
}

```

### myko-treehacks/HistoryCardView.swift

```swift
//
//  HistoryCardView.swift
//  myko-treehacks
//
//  Created by Amy Sun Key on 2/14/26.
//


import SwiftUI
import ImageIO

struct HistoryCardView: View {
    @EnvironmentObject private var appState: AppState
    let item: HistoryItem
    var onFavoriteTap: (() -> Void)? = nil

    @State private var thumbnail: UIImage?

    var body: some View {
        VStack(alignment: .leading, spacing: 10) {
            ZStack(alignment: .topTrailing) {
                Group {
                    if let thumbnail {
                        Image(uiImage: thumbnail)
                            .resizable()
                            .scaledToFit()
                            .scaleEffect(0.9)
                            .frame(maxWidth: .infinity, maxHeight: .infinity)
                            .background(Color.black)
                    } else {
                        ZStack {
                            Color.gray.opacity(0.12)
                            ProgressView()
                        }
                    }
                }

                Button {
                    onFavoriteTap?()
                } label: {
                    Image(systemName: item.isFavorite ? "star.fill" : "star")
                        .foregroundStyle(item.isFavorite ? .yellow : .white)
                        .padding(8)
                }
                .buttonStyle(.plain)
            }
            .frame(height: 150)
            .clipShape(RoundedRectangle(cornerRadius: 14, style: .continuous))

            Text(item.title)
                .font(.subheadline.weight(.semibold))
                .foregroundStyle(.primary)
                .lineLimit(2)

            Text(item.createdAt.formatted(date: .abbreviated, time: .shortened))
                .font(.caption)
                .foregroundStyle(.secondary)
                .lineLimit(1)
            if !item.notes.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
                Text(item.notes)
                    .font(.caption)
                    .foregroundStyle(.secondary)
                    .lineLimit(2)
            }
            
        }
        .frame(maxWidth: .infinity, minHeight: 230, alignment: .topLeading)
        .frame(maxWidth: .infinity, alignment: .leading)
        .padding(8)
        .background(
            RoundedRectangle(cornerRadius: 18, style: .continuous)
                .fill(Color(.systemBackground))
                .shadow(color: .black.opacity(0.08), radius: 10, y: 4)
        )
        .task {
            if thumbnail == nil {
                thumbnail = downsampledImage(at: appState.historyStore.imageURL(for: item), maxDimension: 500)
            }
        }
    }

    private func downsampledImage(at url: URL, maxDimension: CGFloat) -> UIImage? {
        let options: [CFString: Any] = [kCGImageSourceShouldCache: false]
        guard let imageSource = CGImageSourceCreateWithURL(url as CFURL, options as CFDictionary) else { return nil }

        let downsampleOptions: [CFString: Any] = [
            kCGImageSourceCreateThumbnailFromImageAlways: true,
            kCGImageSourceShouldCacheImmediately: true,
            kCGImageSourceCreateThumbnailWithTransform: true,
            kCGImageSourceThumbnailMaxPixelSize: maxDimension
        ]

        guard let cgImage = CGImageSourceCreateThumbnailAtIndex(imageSource, 0, downsampleOptions as CFDictionary) else {
            return nil
        }

        return UIImage(cgImage: cgImage)
    }
}

```

### myko-treehacks/HistoryItem.swift

```swift
//
//  HistoryItem.swift
//  myko-treehacks
//
//  Created by Amy Sun Key on 2/14/26.
//

import Foundation
import Combine
import UIKit

struct HistoryItem: Codable, Identifiable, Hashable, Equatable {
    let id: UUID
    let createdAt: Date
    let imagePath: String
    var title: String
    var notes: String
    var isFavorite: Bool
        
    init(
        id: UUID,
        createdAt: Date,
        imagePath: String,
        title: String = "Microscope Capture",
        notes: String = "",
        isFavorite: Bool = false
    ) {
        self.id = id
        self.createdAt = createdAt
        self.imagePath = imagePath
        self.title = title
        self.notes = notes
        self.isFavorite = isFavorite
    }
    
    private enum CodingKeys: String, CodingKey {
        case id
        case createdAt
        case imagePath
        case title
        case notes
        case isFavorite
    }
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        id = try container.decode(UUID.self, forKey: .id)
        createdAt = try container.decode(Date.self, forKey: .createdAt)
        imagePath = try container.decode(String.self, forKey: .imagePath)
        title = try container.decodeIfPresent(String.self, forKey: .title) ?? "Microscope Capture"
        notes = try container.decodeIfPresent(String.self, forKey: .notes) ?? ""
        isFavorite = try container.decodeIfPresent(Bool.self, forKey: .isFavorite) ?? false
    }
}

@MainActor
final class HistoryStore: ObservableObject {
    @Published private(set) var items: [HistoryItem] = []

    private let metadataFileName = "history.json"

    init() {
        loadItems()
    }
    @discardableResult
    func save(image: UIImage, title: String = "Microscope Capture") throws -> HistoryItem {
        guard let pngData = image.pngData() else {
            throw CocoaError(.fileWriteUnknown)
        }

        let id = UUID()
        let fileName = "\(id.uuidString).png"
        let fileURL = historyDirectory.appendingPathComponent(fileName)
        try pngData.write(to: fileURL, options: .atomic)

        let newItem = HistoryItem(id: id, createdAt: Date(), imagePath: fileName, title: title)
        items.insert(newItem, at: 0)
        try persistMetadata()
        return newItem
    }

    func imageURL(for item: HistoryItem) -> URL {
        historyDirectory.appendingPathComponent(item.imagePath)
    }

    func toggleFavorite(for item: HistoryItem) {
        setFavorite(!item.isFavorite, for: item.id)
    }

    func setFavorite(_ isFavorite: Bool, for itemID: HistoryItem.ID) {
        guard let index = items.firstIndex(where: { $0.id == itemID }) else { return }
        var updated = items
        updated[index].isFavorite = isFavorite
        items = updated

        do {
            try persistMetadata()
        } catch {
            // If persistence fails, keep in-memory state so UI still reflects user's action.
        }
    }
    
    func updateNotes(for itemID: HistoryItem.ID, notes: String) {
        guard let index = items.firstIndex(where: { $0.id == itemID }) else { return }
        items[index].notes = notes
        items = items

        do {
            try persistMetadata()
        } catch {
            // If persistence fails, keep in-memory state so UI still reflects user's action.
        }
    }
    
    func delete(_ item: HistoryItem) {
        items.removeAll { $0.id == item.id }

        do {
            try FileManager.default.removeItem(at: imageURL(for: item))
        } catch {
            print("Failed to delete image file for item \(item.id): \(error)")
        }

        do {
            try persistMetadata()
        } catch {
            // Keep in-memory state updated even if persistence fails.
        }
    }

    func delete(id: HistoryItem.ID) {
        guard let item = items.first(where: { $0.id == id }) else { return }
        delete(item)
    }
    
    func updateTitle(for itemID: HistoryItem.ID, title: String) {
        guard let index = items.firstIndex(where: { $0.id == itemID }) else { return }
        items[index].title = title
        items = items

        do {
            try persistMetadata()
        } catch {
            // If persistence fails, keep in-memory state so UI still reflects user's action.
        }
    }

    private var historyDirectory: URL {
        let documents = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
        let historyDirectory = documents.appendingPathComponent("History", isDirectory: true)
        if !FileManager.default.fileExists(atPath: historyDirectory.path) {
            try? FileManager.default.createDirectory(at: historyDirectory, withIntermediateDirectories: true)
        }
        return historyDirectory
    }

    private var metadataURL: URL {
        historyDirectory.appendingPathComponent(metadataFileName)
    }

    private func loadItems() {
        let url = metadataURL
        guard let data = try? Data(contentsOf: url) else {
            items = []
            return
        }

        do {
            let decoded = try JSONDecoder().decode([HistoryItem].self, from: data)
            items = decoded.sorted { $0.createdAt > $1.createdAt }
        } catch {
            items = []
            return
        }
    }

    private func persistMetadata() throws {
        let sorted = items.sorted { $0.createdAt > $1.createdAt }
        let data = try JSONEncoder().encode(sorted)
        try data.write(to: metadataURL, options: .atomic)
        items = sorted
    }
}

```

### myko-treehacks/OnboardingView.swift

```swift
import SwiftUI
import AVFoundation

struct OnboardingView: View {
    @Binding var isPresented: Bool
//    @AppStorage("hasSeenOnboarding") private var hasSeenOnboarding = false

    @State private var currentPage = 0
    @State private var cameraStatus = AVCaptureDevice.authorizationStatus(for: .video)
    @State private var micStatus = AVAudioSession.sharedInstance().recordPermission

    var body: some View {
        TabView(selection: $currentPage) {
            OnboardingPage(
                imageName: "myko-micro",
                title: "AI Microscopy in Your Pocket",
                bodyText: "Capture and analyze microscopic samples instantly with on-device intelligence.",
                buttonTitle: "Continue",
                pageIndex: 0,
                currentPage: $currentPage,
                primaryAction: advancePage
            )
            .tag(0)

            OnboardingPage(
                imageName: "myko-position1",
                title: "Camera Access Required",
                bodyText: "Myko uses your camera to capture microscope images for analysis.",
                buttonTitle: cameraButtonTitle,
                pageIndex: 1,
                currentPage: $currentPage,
                primaryAction: handleCameraAction
            )
            .tag(1)

            OnboardingPage(
                imageName: "myko-position2",
                title: "Voice Dictation",
                bodyText: "Use your voice to issue commands for hands-free cell analysis.",
                buttonTitle: micButtonTitle,
                pageIndex: 2,
                currentPage: $currentPage,
                primaryAction: handleMicAction,
                secondaryActionTitle: "Skip for now",
                secondaryAction: advancePage
            )
            .tag(2)

            OnboardingPage(
                imageName: "myko-position3",
                title: "You’re Ready to Scan",
                bodyText: "Place your sample under the microscope and tap Scan to begin.",
                buttonTitle: "Start Using Myko",
                pageIndex: 3,
                currentPage: $currentPage,
                primaryAction: completeOnboarding
            )
            .tag(3)
        }
        .tabViewStyle(.page(indexDisplayMode: .never))
        .indexViewStyle(.page(backgroundDisplayMode: .always))
        .background(Color(.systemBackground))
        .onAppear {
            cameraStatus = AVCaptureDevice.authorizationStatus(for: .video)
            micStatus = AVAudioSession.sharedInstance().recordPermission
        }
    }

    private var cameraButtonTitle: String {
        cameraStatus == .authorized ? "Continue" : "Enable Camera"
    }

    private var micButtonTitle: String {
        micStatus == .granted ? "Continue" : "Enable Microphone"
    }

    private func advancePage() {
        withAnimation(.easeInOut(duration: 0.25)) {
            currentPage = min(currentPage + 1, 3)
        }
    }

    private func handleCameraAction() {
        switch cameraStatus {
        case .authorized:
            advancePage()
        case .notDetermined:
            AVCaptureDevice.requestAccess(for: .video) { granted in
                DispatchQueue.main.async {
                    cameraStatus = granted ? .authorized : .denied
                    if granted {
                        advancePage()
                    }
                }
            }
        case .denied, .restricted:
            openSettings()
        @unknown default:
            break
        }
    }

    private func handleMicAction() {
        switch micStatus {
        case .granted:
            advancePage()
        case .undetermined:
            AVAudioSession.sharedInstance().requestRecordPermission { granted in
                DispatchQueue.main.async {
                    micStatus = granted ? .granted : .denied
                    if granted {
                        advancePage()
                    }
                }
            }
        case .denied:
            openSettings()
        @unknown default:
            break
        }
    }

    private func openSettings() {
        guard let url = URL(string: UIApplication.openSettingsURLString),
              UIApplication.shared.canOpenURL(url) else { return }
        UIApplication.shared.open(url)
    }

    private func completeOnboarding() {
//        hasSeenOnboarding = true
        isPresented = false
    }
}

private struct OnboardingPage: View {
    let imageName: String
    let title: String
    let bodyText: String
    let buttonTitle: String
    let pageIndex: Int
    @Binding var currentPage: Int
    let primaryAction: () -> Void
    var secondaryActionTitle: String?
    var secondaryAction: (() -> Void)?

    @State private var animateImage = false

    var body: some View {
        VStack(spacing: 0) {
            Spacer()

            Image(imageName)
                .resizable()
                .scaledToFit()
                .frame(maxHeight: 260)
                .opacity(animateImage ? 1 : 0)
                .offset(y: animateImage ? 0 : 12)
                .padding(.horizontal, 24)

            Text(title)
                .font(.title2.weight(.semibold))
                .multilineTextAlignment(.center)
                .padding(.top, 24)
                .padding(.horizontal, 24)

            Text(bodyText)
                .font(.body)
                .foregroundStyle(.secondary)
                .multilineTextAlignment(.center)
                .frame(maxWidth: 340)
                .padding(.top, 8)
                .padding(.horizontal, 24)

            Spacer()

            Button(action: primaryAction) {
                Text(buttonTitle)
                    .font(.headline)
                    .frame(maxWidth: .infinity)
                    .frame(height: 54)
                    .background(MykoColors.leafBase)
                    .foregroundStyle(.white)
                    .clipShape(RoundedRectangle(cornerRadius: 14, style:
[truncated — 888 more characters]
```

### backend/agent.py

```python
import os
import json
import time
import re
from openai import AsyncOpenAI


class Agent:
    def __init__(self, segmenter, debug: bool = True):
        self.segmenter = segmenter
        self.debug = debug
        self.client = None  # lazily created on first query

        self.tools = [
            {
                "type": "function",
                "function": {
                    "name": "propose_masks",
                    "description": (
                        "Generate model-agnostic candidate masks from the current image. "
                        "Use this first for segmentation/highlight requests. Returns candidate indices and features."
                    ),
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "query": {
                                "type": "string",
                                "description": "Segmentation intent, such as 'red blood cells', 'all cells', or 'cells at bottom'."
                            },
                            "backend": {
                                "type": "string",
                                "enum": ["auto", "cellpose", "sam2", "opencv"],
                                "description": "Optional backend override for mask proposals. Prefer auto unless user asks otherwise."
                            }
                        },
                        "required": ["query"]
                    }
                }
            },
            {
                "type": "function",
                "function": {
                    "name": "apply_masks",
                    "description": (
                        "Select and render a subset of candidate masks by index after propose_masks has been called."
                    ),
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "indices": {
                                "type": "array",
                                "items": {"type": "integer"},
                                "description": "Indices of candidate masks to render."
                            },
                            "query": {
                                "type": "string",
                                "description": "Optional label for this selected mask set."
                            }
                        },
                        "required": ["indices"]
                    }
                }
            },
            {
                "type": "function",
                "function": {
                    "name": "clear_masks",
                    "description": "Remove all active rendered masks.",
                    "parameters": {
                        "type": "object",
                        "properties": {}
                    }
                }
            }
        ]

        self.messages = [
            {
                "role": "system",
                "content": (
                    "You are Myko, an expert AI histopathology assistant with deep "
                    "knowledge of cell morphology, tissue architecture, and clinical "
                    "pathology. You speak with authority and technical precision. "
                    "You ALWAYS provide a definitive, detailed answer — never say you "
                    "'can't' identify something or need more information. Analyse the "
                    "image you are given and describe what you observe confidently.\n\n"
                    "TOOL USAGE RULES:\n"
                    "• ONLY call tools when the user explicitly asks to "
                    "segment, highlight, show, mark, or overlay structures.\n"
                    "  Examples: 'segment the cells', 'highlight all cells', "
                    "'show me the tissue boundaries'.\n"
                    "• For segmentation requests, ALWAYS do this sequence:\n"
                    "  1) call 'propose_masks' first,\n"
                    "  2) inspect returned candidates,\n"
                    "  3) call 'apply_masks' with selected indices.\n"
                    "• NEVER call any tool for counting or quantitative analysis "
                    "questions like 'how many', 'count', 'number of', or 'total'. "
                    "For those, answer directly from visual analysis only.\n"
                    "• The microscope image is ALWAYS attached to every message. "
                    "You can see it. Never ask for an image or say you need more info.\n"
                    "• To remove overlays, call 'clear_masks'.\n\n"
                    "Be confident and technical in your responses."
                )
            }
        ]

    @staticmethod
    def _is_counting_prompt(prompt: str) -> bool:
        p = (prompt or "").lower()
        patterns = [
            r"\bhow many\b",
            r"\bcount\b",
            r"\bnumber of\b",
            r"\btotal\b",
            r"\bquantity\b",
        ]
        return any(re.search(pattern, p) for pattern in patterns)

    def _get_client(self) -> AsyncOpenAI:
        if self.client is None:
            api_key = os.getenv("OPENAI_API_KEY")
            if not api_key:
                raise ValueError("OPENAI_API_KEY not set — export it before sending a query")
            self.client = AsyncOpenAI(api_key=api_key)
        return self.client

    async def _completion(self, messages: list, tools: list | None = None):
        kwargs: dict = {"model": "gpt-4o", "messages": messages}
        if tools:
            kwargs["tools"] = tools
            kwargs["tool_choice"] = "auto"
        response = await self._get_client().chat.completions.create(**kwargs)
        return response.choices[0].message

    async def query(self, prompt: str, frame_b64: str | None = None) -> str:
        if self.debug:
            print(f"ℹ️ Agent received prompt: {prompt[:100]}{'...' if len(prompt) > 100 else ''}")

        # Auto-attach the 
[truncated — 3545 more characters]
```

### myko-treehacks/SpeechAnalyzerTranscriptionService.swift

```swift
//
//  SpeechAnalyzerTranscriptionService.swift
//  myko-treehacks
//
//  Created by Amy Sun Key on 2/13/26.
//

import AVFoundation
import Combine
import Foundation
import Speech

@MainActor
final class SpeechAnalyzerTranscriptionService: ObservableObject {
    enum TranscriptionError: LocalizedError {
        case localeNotSupported
        case permissionDenied
        case setupFailed
        case invalidAudioDataType
        case alreadyRecording

        var errorDescription: String? {
            switch self {
            case .localeNotSupported:
                return "Speech transcription is not supported for the selected locale."
            case .permissionDenied:
                return "Microphone or speech recognition permission was denied."
            case .setupFailed:
                return "Failed to set up speech transcription."
            case .invalidAudioDataType:
                return "Invalid audio format for SpeechAnalyzer."
            case .alreadyRecording:
                return "Speech transcription is already running."
            }
            
        }
    }

    @Published private(set) var isRecording = false
    
    private var isStartingRecording = false
    private var isTapInstalled = false
    
    private var transcriber: SpeechTranscriber?
    private var analyzer: SpeechAnalyzer?
    private var analyzerFormat: AVAudioFormat?
    private var inputSequence: AsyncStream<AnalyzerInput>?
    private var inputBuilder: AsyncStream<AnalyzerInput>.Continuation?

    private let audioEngine = AVAudioEngine()
    private var recognizerTask: Task<Void, Never>?

    private var finalizedTranscript = ""
    private var volatileTranscript = ""

    func startRecording(locale: Locale = .current, onTranscriptUpdate: @escaping (String) -> Void) async throws {
        guard !isRecording, !isStartingRecording else {
            throw TranscriptionError.alreadyRecording
        }
        isStartingRecording = true

        do {
            guard await isAuthorized() else { throw TranscriptionError.permissionDenied }

            try setUpAudioSession()
            try await setUpTranscriber(locale: locale)
            startRecognitionTask(onTranscriptUpdate: onTranscriptUpdate)
            try startAudioEngineTap()

            isRecording = true
            isStartingRecording = false
        } catch {
            await resetRecordingPipeline()
            isStartingRecording = false
            throw error
        }
    }

    func stopRecording() async {
        guard isRecording || isStartingRecording || isTapInstalled else { return }
        await resetRecordingPipeline()
        isRecording = false
        isStartingRecording = false
    }

    private func setUpTranscriber(locale: Locale) async throws {
        transcriber = SpeechTranscriber(
            locale: locale,
            transcriptionOptions: [],
            reportingOptions: [.volatileResults],
            attributeOptions: [.audioTimeRange]
        )

        guard let transcriber else { throw TranscriptionError.setupFailed }

        let analyzer = SpeechAnalyzer(modules: [transcriber])
        self.analyzer = analyzer
        analyzerFormat = await SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith: [transcriber])

        try await ensureModel(transcriber: transcriber, locale: locale)

        (inputSequence, inputBuilder) = AsyncStream<AnalyzerInput>.makeStream()
        guard let inputSequence else { throw TranscriptionError.setupFailed }

        try await analyzer.start(inputSequence: inputSequence)

        finalizedTranscript = ""
        volatileTranscript = ""
    }

    private func startRecognitionTask(onTranscriptUpdate: @escaping (String) -> Void) {
        recognizerTask?.cancel()

        guard let transcriber else { return }
        let results = transcriber.results
        recognizerTask = Task {
            do {
                for try await result in results {
                    let text = String(result.text.characters)
                    if result.isFinal {
                        finalizedTranscript += text
                        volatileTranscript = ""
                    } else {
                        volatileTranscript = text
                    }
                    onTranscriptUpdate(finalizedTranscript + volatileTranscript)
                }
            } catch {
                // Task cancellation and stream shutdown are expected when stopping.
            }
        }
    }

    private func startAudioEngineTap() throws {
        let inputNode = audioEngine.inputNode
        let inputFormat = inputNode.outputFormat(forBus: 0)
        
        if isTapInstalled {
            inputNode.removeTap(onBus: 0)
            isTapInstalled = false
        }

        inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputFormat) { [weak self] buffer, _ in
            guard let self else { return }
            Task {
                try? await self.streamAudioToTranscriber(buffer)
            }
        }
        isTapInstalled = true
        audioEngine.prepare()
        try audioEngine.start()
    }

    private func resetRecordingPipeline() async {
        recognizerTask?.cancel()
        await recognizerTask?.value
        recognizerTask = nil

        if audioEngine.isRunning {
            audioEngine.stop()
        }

        if isTapInstalled {
            audioEngine.inputNode.removeTap(onBus: 0)
            isTapInstalled = false
        }

        inputBuilder?.finish()
        inputBuilder = nil
        inputSequence = nil

        try? await analyzer?.finalizeAndFinishThroughEndOfInput()
        analyzer = nil
        transcriber = nil
        analyzerFormat = nil

        finalizedTranscript = ""
        volatileTranscript = ""
    }
    
    private func streamAudioToTranscriber(_ buffer: AVAudioPCMBuffer) async throws {
        guard let inputBuilder, let analyzerFormat else {
            throw TranscriptionError.invalidAudioDataType
        
[truncated — 4170 more characters]
```

[11 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]