# Project export: pFOG: Parkinson's Medical Aid Device This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Patients with Parkinson’s often suffer from Freezing of Gait (FOG). Using a 1D-CNN and LSTM, we predict FOG and deliver targeted stimulation via wearable robotics to regain ability - Devpost: https://devpost.com/software/foggy - GitHub: https://github.com/harryyuncheng/parkinsons-fog-device - Video: https://www.youtube.com/embed/l1K1MoSdC4Q?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (GRAND PRIZE - Social Impact Track; Cal Hacks: Hacker's Choice) - Team: 4 GitHub contributor(s) — raphael-64 (22 commits), Harry Cheng (18 commits), shuyuvankerkwijk (15 commits), lewisabel (1 commits) ## Devpost submission (written by the team) ### Inspiration Someone we know has Parkinson’s disease, and like many patients, he suffers from Freezing of Gait (FOG) — a sudden, involuntary inability to walk. Recent research (N.E. Ibrahim, Nature, 2024) has shown that targeted muscle stimulation can effectively eliminate FOG. Inspired by this, his family noticed that gently poking the back of his leg with a stick during an episode helped him move again. This simple action inspired us to develop an automated, predictive solution. ### What it does Our project, pFOG, is a wearable device that predicts Freezing of Gait using sensor data and machine learning. Once FOG is detected, it gives targeted muscle stimulation to the frozen leg, mimicking the effect of a physical "poke" and helping the patient resume walking. We also built a web interface for easy data collection and labeling, which allows users to fine-tune a model using their own walking data—essential for adapting to gait variability across patients. ### How we built it We collected accelerometer and gyroscope data using an Adafruit MPU-6050 IMU mounted on a wearable module. A 1D Convolutional Neural Network (1D-CNN) extracts rich spatial features, which are fed into a Long Short-Term Memory (LSTM) model with learned temporal patterns. Together, these are able to recognize the onset of FOG from real-time motion data. Upon detection, a servo-actuated mechanism delivers localized muscle stimulation to the leg using a robotic arm, replicating the therapeutic “poke.” ### Challenges we ran into Our challenges began before the hackathon even started—we didn’t have a 3D printer! Huge thanks to Nour and Team AR Glasses for letting us run a quick print on theirs! From there: We had to improvise wiring and mechanical assemblies with whatever was available (true hacker spirit!). UART communication was not as “plug and play” as advertised—serial timing issues cost us a lot of debugging hours. Training a machine learning model that didn’t overfit and could generalize across different walking styles was tricky, especially with limited, noisy data. ### Accomplishments we're proud of We are very proud to demonstrate a working prototype that successfully detects FOG episodes with >90% accuracy and responds in real time! ## README (from the GitHub repository) # FOG Detection System A comprehensive real-time system for collecting and annotating IMU data for Freezing of Gait (FOG) detection in Parkinson's patients. ## System Architecture ``` ESP32 → Serial → Backend (ESP32 Connector) → Flask API → Next.js Frontend ↓ ↓ Terminal Keys Real-time WebSocket ↓ ↓ HTTP API ← ← ← ← ← ← ← ← Database (SQLite) ``` ## Components ### 🔧 Backend (`/backend/`) - **Flask API** (`app.py`) - REST endpoints and WebSocket server - **ESP32 Connector** (`esp32_connector.py`) - Serial bridge for real ESP32 data - **SQLite Database** - Stores all IMU data with 3-way classification labels - **PyTorch 1d-CNN + LSTM Hybrid model** - CNN extracts rich spatial features, then feeds those features to the LSTM to learn temporal patterns. ### 🖥️ Frontend (`/frontend/`) - **Next.js App** - Modern React-based web interface - **Real-time Charts** - Live visualization of IMU data (accelerometer & gyroscope) - **State Annotation** - 3-way classification: Walking / Standing / Freezing ## Quick Start ### 1. Backend Setup ```bash # Install backend dependencies cd backend pip install -r backend_requirements.txt # Start Flask server python app.py ``` ### 2. Frontend Setup ```bash # Install frontend dependencies cd frontend npm install # Start Next.js development server npm run dev ``` ### 3. Access the System - **Web Interface**: http://localhost:3000 - **Backend API**: http://localhost:6000 ## Usage Modes ### 🧪 Simulated Data (Testing) 1. Start both backend and frontend 2. Open web interface 3. Click "Start Recording" 4. Use keyboard shortcuts to annotate states: - `W` - Walking - `S` - Standing - `F` - Freezing ### 🔌 Real ESP32 Data 1. Connect ESP32 via USB 2. Update serial port in `backend/esp32_connector.py` 3. Start backend and frontend 4. Start ESP32 connector: ```bash cd backend python esp32_connector.py ``` 5. Real data will appear automatically in web interface ## ESP32 Data Format Your ESP32 should send CSV data over serial: ``` ax,ay,az,gx,gy,gz 1.23,-0.45,9.67,12.34,-5.67,8.90 ``` Where: - `ax, ay, az` = Accelerometer (m/s²) - `gx, gy, gz` = Gyroscope (°/s) ## Features ### 📊 Real-time Visualization - Live accelerometer and gyroscope charts with X,Y,Z legends - Color-coded state indicators - Sample counters for each state ### 🏷️ Data Annotation - **3-way classification**: Walking vs Standing vs Freezing - **Keyboard shortcuts**: W/S/F keys - **Real-time feedback** with immediate visual updates ### 💾 Data Management - **Session recording** with unique IDs - **SQLite storage** with timestamps and labels - **CSV export** for machine learning model training - **Session history** with statistics ### 🔄 Multiple Input Sources - **Simulated data** for testing and development - **Real ESP32 data** via serial connection - **Manual annotation** via web interface or ESP32 connector ## API Endpoints - `GET /` - Health check - `POST /start_session` - Start recording session - `POST /stop_session` - Stop recording session - `POST /annotate_state` - Annotate current state (`{'state': 'walking'|'standing'|'freezing'}`) - `GET /get_sessions` - Get all recording sessions - `GET /get_session_data/` - Get data for specific session ## WebSocket Events - `imu_data` - Real-time IMU data stream - `state_annotation` - State annotation updates - `esp32_status` - ESP32 connector status ## Development ### File Structure ``` parkinsons/ ├── backend/ │ ├── app.py # Flask API server │ ├── esp32_connector.py # ESP32 serial bridge │ ├── backend_requirements.txt # Python dependencies │ └── esp32_requirements.txt # ESP32 connector dependencies ├── frontend/ │ ├── app/ │ │ └── page.tsx # Main Next.js page │ ├── lib/ │ │ └── api.ts # API service layer │ └── package.json # Node.js dependencies └── README.md # This file ``` ### Environment Variables Create `.env.local` in `/frontend/`: ``` NEXT_PUBLIC_BACKEND_URL=http://localhost:5000 ``` ### Database Schema ```sql CREATE TABLE imu_data ( id INTEGER PRIMARY KEY, timestamp TEXT, acc_x REAL, acc_y REAL, acc_z REAL, gyro_x REAL, gyro_y REAL, gyro_z REAL, label TEXT, -- 'walking', 'standing', 'freezing' session_id TEXT ); ``` ## Troubleshooting ### Backend Issues - **Port 6000 in use**: Change port in `app.py` - **Database errors**: Delete `fog_data.db` to reset - **CORS errors**: Ensure Flask-CORS is installed ### ESP32 Issues - **Serial connection failed**: Check port in `esp32_connector.py` - **Data format errors**: Ensure ESP32 sends correct CSV format - **Permission denied**: Run with admin/sudo privileges ### Frontend Issues - **API connection failed**: Check backend is running on port 5000 - **WebSocket disconnects**: Check firewall settings - **Build errors**: Run `npm install` to update dependencies ## Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes 4. Test with both simulated and real data 5. Submit a pull request ## License This project is licensed under the GPL 3.0 License - see the LICENSE file for details. ## Detected evidence (automated analysis) Indexed codebase: 28 recognized source files, 171 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (54 of 54) ``` .gitignore backend/app.py backend/data/isabel/session_20250621_170543.csv backend/data/isabel/session_20250621_170640.csv backend/data/isabel/session_20250621_171423.csv backend/data/isabel/session_20250621_200115.csv backend/data/isabel/session_20250621_200245.csv backend/data/session_20250622_084805.csv backend/data/shuyu/session_20250621_185016.csv backend/data/shuyu/session_20250621_185106.csv backend/data/shuyu/session_20250621_185600.csv backend/data/shuyu/session_20250621_194749.csv backend/esp32_connector.py backend/export_imu_data.py backend/fog_data.db backend/fog_predictor.py backend/models/fog_classifier_20250622_003351.pth backend/models/fog_classifier_20250622_021051.pth backend/models/fog_classifier_20250622_022820.pth backend/models/fog_classifier_20250622_035458.pth backend/models/fog_classifier_20250622_041238.pth backend/models/fog_classifier_20250622_060902.pth backend/requirements.txt backend/serial_control.py backend/serial_controller.py backend/test_serial_backup.py backend/training/model_training.ipynb backend/training/train_new_model.py firmware/esp32_firmware/esp32_firmware.ino frontend/app/api/save-session/route.ts frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/components.json frontend/components/AIMonitoring.tsx frontend/components/ModelPerformance.tsx frontend/components/NotificationToast.tsx frontend/components/SessionHistory.tsx frontend/components/ui/badge.tsx frontend/components/ui/button.tsx frontend/components/ui/card.tsx frontend/components/ui/progress.tsx frontend/components/ui/tabs.tsx frontend/eslint.config.mjs frontend/lib/api.ts frontend/lib/utils.ts frontend/next-env.d.ts frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/tsconfig.json label-key.py README.md ``` ### Dependencies - backend/requirements.txt: Flask@==3.0.0, Flask-CORS@==4.0.0, Flask-SocketIO@==5.3.6, keyboard@==0.13.5, numpy@>=1.26.0, pyserial@==3.5, python-socketio[client]@==5.8.0, requests@==2.31.0, torch@>=2.0.0, torchvision@>=0.15.0 - frontend/package.json: @eslint/eslintrc@^3, @radix-ui/react-slot@^1.2.3, @radix-ui/react-tabs@^1.1.12, @tailwindcss/postcss@^4, @types/node@^20.19.1, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@15.2.4, lucide-react@^0.522.0, next@15.2.4, react@^19.0.0, react-dom@^19.0.0, recharts@^2.15.4, socket.io-client@^4.8.1, tailwind-merge@^3.3.1, tailwindcss@^4, tw-animate-css@^1.3.4, typescript@^5 ### Recent commits (newest first) - removing license - Update README.md - frontend cleanup - shortened title - Update esp32_firmware.ino to no buffer - changed website title/subtitle - Added script to train new model - prediction not AI - changed to work, and repeated animation on performance - added model selection capability - Merge branch 'main' of https://github.com/harryyuncheng/parkinsons - WORKS.2!! buffer on hardware side - Cleaned up repo - Added firmware - changed title to black - reverted app and fog predicor - stick figure ui - stylistic changes - added smooth transition of tabs - fixed bugwith rapid calls in successions ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/requirements.txt ``` Flask==3.0.0 Flask-SocketIO==5.3.6 Flask-CORS==4.0.0 numpy>=1.26.0 torch>=2.0.0 torchvision>=0.15.0 pyserial==3.5 keyboard==0.13.5 requests==2.31.0 python-socketio[client]==5.8.0 ``` ### frontend/package.json ``` { "name": "fog", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@radix-ui/react-slot": "^1.2.3", "@radix-ui/react-tabs": "^1.1.12", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.522.0", "next": "15.2.4", "react": "^19.0.0", "react-dom": "^19.0.0", "recharts": "^2.15.4", "socket.io-client": "^4.8.1", "tailwind-merge": "^3.3.1" }, "devDependencies": { "@eslint/eslintrc": "^3", "@tailwindcss/postcss": "^4", "@types/node": "^20.19.1", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.2.4", "tailwindcss": "^4", "tw-animate-css": "^1.3.4", "typescript": "^5" } } ``` ### backend/app.py ```python from flask import Flask, request, jsonify from flask_socketio import SocketIO, emit from flask_cors import CORS import sqlite3 import json from datetime import datetime import threading import time import csv import os import serial import signal import sys import atexit from fog_predictor import initialize_predictor, get_predictor, cleanup_predictor app = Flask(__name__) app.config['SECRET_KEY'] = 'fog_detection_secret_key' CORS(app) # Enable CORS for Next.js frontend socketio = SocketIO(app, cors_allowed_origins="*") # Database setup def init_db(): conn = sqlite3.connect('fog_data.db') c = conn.cursor() c.execute('''CREATE TABLE IF NOT EXISTS imu_data (id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp TEXT, acc_x REAL, acc_y REAL, acc_z REAL, gyro_x REAL, gyro_y REAL, gyro_z REAL, label TEXT, session_id TEXT)''') conn.commit() conn.close() # Initialize database init_db() # Create data directory for CSV exports DATA_DIR = 'data' if not os.path.exists(DATA_DIR): os.makedirs(DATA_DIR) print(f"Created data directory: {DATA_DIR}") # Global variables for data streaming streaming = False current_session_id = None current_state = 'standing' # Track current state for labeling # Initialize FOG predictor print("🤖 Initializing FOG predictor...") predictor_initialized = initialize_predictor() if predictor_initialized: print("✅ FOG predictor ready for real-time monitoring!") else: print("❌ FOG predictor failed to initialize") def store_imu_data(data, label='standing'): """Store IMU data in database""" conn = sqlite3.connect('fog_data.db') c = conn.cursor() c.execute('''INSERT INTO imu_data (timestamp, acc_x, acc_y, acc_z, gyro_x, gyro_y, gyro_z, label, session_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)''', (data['timestamp'], data['acc_x'], data['acc_y'], data['acc_z'], data['gyro_x'], data['gyro_y'], data['gyro_z'], label, current_session_id)) conn.commit() conn.close() @app.route('/') def index(): return jsonify({'message': 'FOG Detection Backend API', 'status': 'running'}) @app.route('/start_session', methods=['POST']) def start_session(): global streaming, current_session_id if not streaming: current_session_id = datetime.now().strftime("%Y%m%d_%H%M%S") streaming = True print(f"Started session: {current_session_id}") return jsonify({'status': 'started', 'session_id': current_session_id}) return jsonify({'status': 'already_running', 'session_id': current_session_id}) @app.route('/stop_session', methods=['POST']) def stop_session(): global streaming, current_session_id if streaming and current_session_id: streaming = False session_to_return = current_session_id print(f"Session stopped: {current_session_id}") return jsonify({'status': 'stopped', 'session_id': session_to_return}) else: return jsonify({'status': 'no_active_session'}) @app.route('/annotate_state', methods=['POST']) def annotate_state(): """Annotate current time window with activity state""" global current_state data = request.json state = data.get('state', 'standing') # 'walking', 'standing', 'freezing' # Validate state if state not in ['walking', 'standing', 'freezing']: return jsonify({'error': 'Invalid state. Must be walking, standing, or freezing'}), 400 # Update current state current_state = state # Update the last few records in database conn = sqlite3.connect('fog_data.db') c = conn.cursor() # Mark last 2 seconds of data with the new state c.execute('''UPDATE imu_data SET label = ? WHERE session_id = ? AND id IN (SELECT id FROM imu_data WHERE session_id = ? ORDER BY id DESC LIMIT 100)''', (state, current_session_id, current_session_id)) conn.commit() conn.close() print(f"State updated to: {state}") socketio.emit('state_annotation', {'state': state, 'timestamp': datetime.now().isoformat()}) return jsonify({'status': 'annotated', 'state': state}) @app.route('/get_session_data/') def get_session_data(session_id): """Get all data for a specific session""" conn = sqlite3.connect('fog_data.db') c = conn.cursor() c.execute('''SELECT * FROM imu_data WHERE session_id = ? ORDER BY timestamp''', (session_id,)) data = c.fetchall() conn.close() # Convert to list of dictionaries columns = ['id', 'timestamp', 'acc_x', 'acc_y', 'acc_z', 'gyro_x', 'gyro_y', 'gyro_z', 'label', 'session_id'] result = [dict(zip(columns, row)) for row in data] return jsonify(result) @app.route('/get_sessions') def get_sessions(): """Get list of all recording sessions""" conn = sqlite3.connect('fog_data.db') c = conn.cursor() c.execute('''SELECT session_id, COUNT(*) as sample_count, MIN(timestamp) as start_time, MAX(timestamp) as end_time, SUM(CASE WHEN label = 'walking' THEN 1 ELSE 0 END) as walking_count, SUM(CASE WHEN label = 'standing' THEN 1 ELSE 0 END) as standing_count, SUM(CASE WHEN label = 'freezing' THEN 1 ELSE 0 END) as freezing_count FROM imu_data GROUP BY session_id ORDER BY start_time DESC''') sessions = c.fetchall() conn.close() columns = ['session_id', 'sample_count', 'start_time', 'end_time', 'walking_count', 'standing_count', 'freezing_count'] result = [dict(zip(columns, row)) for row in sessions] return jsonify(result) @app.route('/save_session_csv/', methods=['POST']) def save_ses [truncated — 8353 more characters] ``` ### frontend/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "pFOG", description: "A Parkinson's Medical Aid Wearable", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### frontend/app/page.tsx ```typescript "use client"; import { useState, useEffect } from "react"; import { api, socketEvents, IMUData as ApiIMUData, SessionData as ApiSessionData, disconnectSocket, } from "@/lib/api"; import { Button } from "@/components/ui/button"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { Badge } from "@/components/ui/badge"; import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"; import { LineChart, Line, XAxis, YAxis, CartesianGrid, ResponsiveContainer, Legend, } from "recharts"; import { Play, Square, Wifi, WifiOff, Activity, AlertTriangle, User, UserCheck, CheckCircle, X, Info, } from "lucide-react"; import AIMonitoring from "@/components/AIMonitoring"; import ModelPerformance from "@/components/ModelPerformance"; import SessionHistory from "@/components/SessionHistory"; interface DataPoint { time: string; accelX: number; accelY: number; accelZ: number; gyroX: number; gyroY: number; gyroZ: number; state: string; } interface PredictionData { prediction: string; raw_prediction?: string; confidence: number; probabilities: { walking: number; standing: number; freezing: number; }; buffer_size: number; status: string; } export default function FreezeOfGaitMonitor() { const [activeTab, setActiveTab] = useState("performance"); const [isConnected, setIsConnected] = useState(false); const [isRecording, setIsRecording] = useState(false); const [currentState, setCurrentState] = useState("standing"); const [data, setData] = useState([]); const [sessionId, setSessionId] = useState(null); const [sessions, setSessions] = useState([]); const [sampleCount, setSampleCount] = useState(0); const [recordingStartTime, setRecordingStartTime] = useState(0); const [recordingDuration, setRecordingDuration] = useState(0); const [notification, setNotification] = useState<{ type: "success" | "error"; message: string; show: boolean; }>({ type: "success", message: "", show: false }); const [aiPrediction, setAiPrediction] = useState( undefined ); const [showInstructions, setShowInstructions] = useState(false); // Show notification function const showNotification = (type: "success" | "error", message: string) => { setNotification({ type, message, show: true }); // Auto-hide after 5 seconds setTimeout(() => { setNotification((prev) => ({ ...prev, show: false })); }, 5000); }; // Update recording duration useEffect(() => { let interval: NodeJS.Timeout | null = null; if (isRecording && recordingStartTime > 0) { interval = setInterval(() => { setRecordingDuration( Math.floor((Date.now() - recordingStartTime) / 1000) ); }, 1000); } return () => { if (interval) clearInterval(interval); }; }, [isRecording, recordingStartTime]); // Backend connection and real-time data handling useEffect(() => { const initializeConnection = async () => { console.log("Initializing backend connection..."); const isHealthy = await api.checkHealth(); setIsConnected(isHealthy); console.log("Backend health check:", isHealthy); if (isHealthy) { // Define event handlers const handleConnect = () => { console.log("✅ Connected to backend WebSocket"); setIsConnected(true); }; const handleDisconnect = () => { console.log("❌ Disconnected from backend WebSocket"); setIsConnected(false); }; const handleIMUData = (data: ApiIMUData) => { console.log("📡 Real ESP32 data received:", data); console.log("📊 Data details:", { acc_x: data.acc_x, acc_y: data.acc_y, acc_z: data.acc_z, gyro_x: data.gyro_x, gyro_y: data.gyro_y, gyro_z: data.gyro_z, current_state: data.current_state, }); // Create data point for visualization const newPoint: DataPoint = { time: new Date().toLocaleTimeString(), accelX: Number(data.acc_x.toFixed(2)), accelY: Number(data.acc_y.toFixed(2)), accelZ: Number(data.acc_z.toFixed(2)), gyroX: Number(data.gyro_x.toFixed(2)), gyroY: Number(data.gyro_y.toFixed(2)), gyroZ: Number(data.gyro_z.toFixed(2)), state: data.current_state || "standing", }; // Update chart data immediately for smooth streaming setData((prev) => { const newData = [...prev, newPoint].slice(-50); console.log( "📈 Updated chart data, total points:", newData.length, "latest point:", newPoint ); return newData; }); setCurrentState(data.current_state || "standing"); setSampleCount((prev) => prev + 1); // Update AI prediction data if available if (data.ai_prediction) { const newPrediction = data.ai_prediction as PredictionData; setAiPrediction((prev: PredictionData | undefined) => { // Only update if the prediction actually changed to prevent excessive re-renders if ( !prev || prev.prediction !== newPrediction.prediction || prev.confidence !== newPrediction.confidence || prev.status !== newPrediction.status ) { return newPrediction; } return prev; }); } }; const handleStateAnnotation = (data: { state: string; timestamp: string; }) => { console.log("🏷️ State annotation received:", data.state); setCurrentState(data.state); [truncated — 19662 more characters] ``` ### frontend/app/api/save-session/route.ts ```typescript import { type NextRequest, NextResponse } from "next/server" export async function POST(request: NextRequest) { try { const body = await request.json() const { sessionId, type, data, stats, alerts, timestamp } = body // In a real implementation, you would save this to your database // For now, we'll just log it and return success console.log(`Saving ${type} session data:`, { sessionId, type, dataPoints: data.length, stats, alerts: alerts?.length || 0, timestamp, }) // Simulate database save await new Promise((resolve) => setTimeout(resolve, 100)) return NextResponse.json({ success: true, message: `${type} session data saved successfully`, sessionId, }) } catch (error) { console.error("Error saving session data:", error) return NextResponse.json({ success: false, message: "Failed to save session data" }, { status: 500 }) } } ``` ### label-key.py ```python import serial import threading import time import keyboard # Admin access required for non-Windows import csv from collections import deque # === CONFIG === SERIAL_PORT = '/dev/ttyUSB0' # Adjust to your ESP32 port BAUD_RATE = 115200 LABEL_KEYS = { 'w': 'walking', 's': 'standing', 'f': 'freezing' } BUFFER_DURATION = 5 # seconds of rolling data LOG_FILE = 'imu_log.csv' # === Data Structures === data_buffer = deque() # holds (timestamp, ax, ay, az, gx, gy, gz) label_intervals = deque() # holds (start_ts, end_ts, label) # === Thread: Serial Reading === def read_serial(): ser = serial.Serial(SERIAL_PORT, BAUD_RATE, timeout=1) print("[Serial] Connected.") while True: line = ser.readline().decode('utf-8').strip() try: # Assume format: ax,ay,az,gx,gy,gz ax, ay, az, gx, gy, gz = map(float, line.split(',')) timestamp = time.time() data_buffer.append((timestamp, ax, ay, az, gx, gy, gz)) # Trim old data while data_buffer and timestamp - data_buffer[0][0] > BUFFER_DURATION: data_buffer.popleft() except Exception as e: print(f"[Serial] Error parsing line: {line} | {e}") # === Thread: Key Labeling === def key_listener(): print("[Key] Press W (walk), S (stand), F (freeze)") key_states = {key: False for key in LABEL_KEYS} key_start_times = {key: None for key in LABEL_KEYS} while True: for key, label in LABEL_KEYS.items(): if keyboard.is_pressed(key): if not key_states[key]: # Key just pressed key_states[key] = True key_start_times[key] = time.time() print(f"[Key] {label.upper()} START at {key_start_times[key]:.2f}") else: if key_states[key]: # Key just released key_states[key] = False start_ts = key_start_times[key] end_ts = time.time() label_intervals.append((start_ts, end_ts, label)) print(f"[Key] {label.upper()} END at {end_ts:.2f}") time.sleep(0.01) # === Logger: Match labels to data === def logger(): with open(LOG_FILE, 'w', newline='') as f: writer = csv.writer(f) writer.writerow(['timestamp', 'ax', 'ay', 'az', 'gx', 'gy', 'gz', 'label']) while True: time.sleep(0.5) while data_buffer: datapoint = data_buffer.popleft() ts, *imu_data = datapoint label = None for start_ts, end_ts, event_label in list(label_intervals): # Label if data point is in the interval, but not in the last 0.2s before release if start_ts <= ts < end_ts - 0.2: label = event_label break writer.writerow([ts] + imu_data + [label]) f.flush() # === Run Threads === if __name__ == "__main__": threading.Thread(target=read_serial, daemon=True).start() threading.Thread(target=key_listener, daemon=True).start() logger() # run on main thread ``` ### frontend/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ }; export default nextConfig; ``` ### frontend/next-env.d.ts ```typescript /// /// // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. ``` ### backend/export_imu_data.py ```python import sqlite3 import csv import sys from datetime import datetime DB_FILE = 'fog_data.db' CSV_FILE = f'imu_data_export_{datetime.now().strftime("%Y%m%d_%H%M%S")}.csv' def export_to_csv(db_file=DB_FILE, csv_file=CSV_FILE): conn = sqlite3.connect(db_file) c = conn.cursor() c.execute('''SELECT timestamp, acc_x, acc_y, acc_z, gyro_x, gyro_y, gyro_z, label, session_id FROM imu_data ORDER BY timestamp''') rows = c.fetchall() conn.close() with open(csv_file, mode='w', newline='') as file: writer = csv.writer(file) writer.writerow(['timestamp', 'acc_x', 'acc_y', 'acc_z', 'gyro_x', 'gyro_y', 'gyro_z', 'label', 'session_id']) writer.writerows(rows) print(f"Exported {len(rows)} rows to {csv_file}") if __name__ == '__main__': db_file = sys.argv[1] if len(sys.argv) > 1 else DB_FILE csv_file = sys.argv[2] if len(sys.argv) > 2 else CSV_FILE export_to_csv(db_file, csv_file) ``` [18 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]