# Project export: Form Check This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Form Check turns your iPhone into a personal coach for weightlifting. Point your camera at yourself during a lift, and FormCheck uses advanced computer vision to analyze your technique in real-time. - Devpost: https://devpost.com/software/form-check-1j60r7 - GitHub: https://github.com/leojia22/Treehacks26 - Team: 4 GitHub contributor(s) — jacobrljarvis (5 commits), Aaron Nguyen (4 commits), CJ (3 commits), leojia22 (2 commits) ## Devpost submission (written by the team) ### Overview Real-Time Weightlifting Form Analysis ### Inspiration We've all been there, lifting weights at the gym, unsure if our form is correct, risking injury with every rep. Personal trainers are expensive, and watching yourself in a mirror only shows one angle while you're mid-lift. We wanted to democratize access to professional-quality form analysis, making it as simple as pointing your phone's camera at yourself while you train. ### What it does FormCheck is an iOS app that analyzes weightlifting form in real-time using computer vision. Simply select your exercise (bench press, squats, deadlifts, etc), start recording, and lift. The app detects your body pose and barbell position using Apple's Vision framework and a custom-trained YOLOv8 object detection model. It automatically counts your reps, tracks bar path, and flags form issues like knee cave, elbow flare, back rounding, or wrist misalignment, providing instant feedback with severity levels (warning/danger) and educational content explaining why proper form matters. After your set, review the recording with pose skeleton overlays, see a per-rep breakdown of form issues, and track your progress over time with session history. ### How we built it We built FormCheck as a modular iOS app using Swift and SwiftUI, organized into seven local packages for clean architecture: FCPoseEstimation: Apple Vision framework (VNDetectHumanBodyPoseRequest) extracts 19 body keypoints from each frame FCPoseEstimation: Apple Vision framework (VNDetectHumanBodyPoseRequest) extracts 19 body keypoints from each frame FCBarbellDetection: Custom YOLOv8 model trained in Python (using Ultralytics) and exported to Core ML for barbell tracking FCBarbellDetection: Custom YOLOv8 model trained in Python (using Ultralytics) and exported to Core ML for barbell tracking FCFormAnalysis: Rule engine with 22+ exercise-specific form rules using angle calculations and biomechanical thresholds FCFormAnalysis: Rule engine with 22+ exercise-specific form rules using angle calculations and biomechanical thresholds FCCapture: AVFoundation-based camera capture with frame-by-frame analysis FCCapture: AVFoundation-based camera capture with frame-by-frame analysis FCPersistence: Core Data for local storage of sessions and metrics FCPersistence: Core Data for local storage of sessions and metrics FCUI: Design system with modern UI components FCUI: Design system with modern UI components MLTraining/: The ML training pipeline uses Python with PyTorch, Ultralytics YOLOv8, and CoreML Tools to prepare datasets, train the barbell detector, and export to a format optimized for on-device inference MLTraining/: The ML training pipeline uses Python with PyTorch, Ultralytics YOLOv8, and CoreML Tools to prepare datasets, train the barbell detector, and export to a format optimized for on-device inference ### Challenges we ran into Real-time performance: Processing pose estimation and object detection at 30fps on-device required significant optimization — we implemented frame skipping, model quantization, and efficient rendering Real-time performance: Processing pose estimation and object detection at 30fps on-device required significant optimization — we implemented frame skipping, model quantization, and efficient rendering Barbell occlusion: The barbell often gets occluded by the lifter's body, making consistent tracking difficult. We implemented a wrist-based fallback estimator Barbell occlusion: The barbell often gets occluded by the lifter's body, making consistent tracking difficult. We implemented a wrist-based fallback estimator Camera angle detection: Form rules vary based on viewing angle (front vs. side). We built an automatic camera angle classifier using shoulder-hip alignment Camera angle detection: Form rules vary based on viewing angle (front vs. side). We built an automatic camera angle classifier using shoulder-hip alignment Rep counting accuracy: Distinguishing between exercise phases (descending, bottom, ascending, lockout) required hysteresis filtering to avoid false triggers from shaky movements Rep counting accuracy: Distinguishing between exercise phases (descending, bottom, ascending, lockout) required hysteresis filtering to avoid false triggers from shaky movements ### Accomplishments we're proud of Runs entirely on-device with no cloud processing — privacy-first and works offline Runs entirely on-device with no cloud processing — privacy-first and works offline 22+ biomechanically-sound form rules across 6 exercises, each with educational content 22+ biomechanically-sound form rules across 6 exercises, each with educational content Sub-100ms latency from camera frame to form feedback Sub-100ms latency from camera frame to form feedback Smooth 60fps UI even during intensive ML inference Smooth 60fps UI even during intensive ML inference Beautiful, intuitive design that feels like a polished commercial app Beautiful, intuitive design that feels like a polished commercial app Comprehensive architecture with proper separation of concerns and testable components Comprehensive architecture with proper separation of concerns and testable components ### What we learned How to optimize ML models for real-time on-device inference on mobile hardware How to optimize ML models for real-time on-device inference on mobile hardware The complexity of human biomechanics and the nuances of proper lifting technique The complexity of human biomechanics and the nuances of proper lifting technique Apple's Vision framework capabilities and limitations for pose estimation Apple's Vision framework capabilities and limitations for pose estimation Core Data performance optimization for high-frequency writes during recording Core Data performance optimization for high-frequency writes during recording The importance of user feedback timing — too many alerts become noise, too few and issues get missed The importance of user feedback timing — too many alerts become noise, too few and issues get missed Building maintainable iOS codebases using Swift Package Manager for modularization Building maintainable iOS codebases using Swift Package Manager for modularization ### What's next for FormCheck More exercises: Overhead press, rows, pull-ups, dips, and olympic lifts More exercises: Overhead press, rows, pull-ups, dips, and olympic lifts Advanced analytics: Velocity tracking, bar speed zones, fatigue detection across sets Advanced analytics: Velocity tracking, bar speed zones, fatigue detection across sets Social features: Share PRs, compare form with friends, coach-athlete collaboration Social features: Share PRs, compare form with friends, coach-athlete collaboration Apple Watch integration: Heart rate correlation with form breakdown Apple Watch integration: Heart rate correlation with form breakdown 3D visualization: Render full 3D skeleton from multiple camera angles 3D visualization: Render full 3D skeleton from multiple camera angles Form recommendations: AI-powered suggestions for correcting specific issues Form recommendations: AI-powered suggestions for correcting specific issues Gamification: Achievements, streaks, and perfect-form challenges Gamification: Achievements, streaks, and perfect-form challenges Cross-platform: Expand to Android using TensorFlow Lite Cross-platform: Expand to Android using TensorFlow Lite ## README (from the GitHub repository) # FormCheck An iOS app that analyzes weightlifting form in real-time using computer vision. Point your camera at a lift, and FormCheck detects your body pose and barbell position to provide rep-by-rep feedback on technique. ## Supported Exercises - **Bench Press** — bar path, elbow flare, wrist alignment, bar touch point, shoulder rounding - **Squat** — depth, knee cave, forward lean, bar path, heel rise Each issue includes severity levels (warning/danger) and educational content explaining why correct form matters. ## Architecture The project is organized as a main app target with seven local Swift packages: | Package | Purpose | |---|---| | **FCCore** | Shared models (`Session`, `Rep`, `FormIssue`), protocols, angle calculation utilities | | **FCPoseEstimation** | Body pose detection via Apple Vision (`VNDetectHumanBodyPoseRequest`), keypoint interpolation, skeleton building | | **FCBarbellDetection** | Barbell detection using a Core ML YOLOv8 model with wrist-based fallback when no model is available | | **FCFormAnalysis** | Form rule engine (10 rules across 2 exercises), rep counting, camera angle detection, hysteresis filtering | | **FCCapture** | Camera capture service and preview view | | **FCPersistence** | Core Data persistence, session and settings repositories | | **FCUI** | Design system and reusable UI components | ## Screens - **Home** — dashboard and entry point - **Exercise Selection** — choose bench press or squat - **Recording** — live camera feed with real-time form analysis - **Playback** — review recorded sessions with overlay - **Session Detail** — per-rep breakdown and metrics - **History** — browse past sessions - **Settings** — user preferences - **Educational Content** — drill-down explanations for each form issue ## How It Works 1. **Pose Estimation** — Apple Vision framework extracts body joint positions from each camera frame 2. **Barbell Detection** — A Core ML object detection model (or wrist-based fallback) locates the barbell 3. **Rep Counting** — Bar position peaks/valleys and joint angle thresholds detect rep phases (descending, bottom, ascending, lockout) 4. **Form Analysis** — Exercise-specific rules evaluate each frame and flag issues with severity levels 5. **Persistence** — Sessions, reps, and form issues are saved locally via Core Data ## ML Training The `MLTraining/` directory contains a Python pipeline for training the barbell detection model: - `prepare_dataset.py` — dataset preparation - `train_yolov8.py` — YOLOv8 training - `export_coreml.py` — export to Core ML format Dependencies: `ultralytics`, `albumentations`, `opencv-python`, `coremltools` (see `MLTraining/requirements.txt`). ## Requirements - iOS 17.0+ - Xcode 15.0+ - Swift 5.9 ## Getting Started 1. Clone the repository 2. Open `FormCheck.xcodeproj` in Xcode (project generated via [XcodeGen](https://github.com/yonaskolb/XcodeGen) from `project.yml`) 3. Build and run on a physical device (camera required) To train a custom barbell detection model: ```bash cd MLTraining pip install -r requirements.txt python barbell_training/prepare_dataset.py python barbell_training/train_yolov8.py python barbell_training/export_coreml.py ``` Place the exported `BarbellDetector.mlmodelc` in the app bundle. Without it, the app falls back to wrist-based barbell estimation. ## Detected evidence (automated analysis) Indexed codebase: 119 recognized source files, 595 KB. - C (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Swift (language) — detected in the code - TensorFlow (technology) — detected in the code - C++ (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 8377) ``` .DS_Store .gitignore check_joint_names.swift compare_pose_detection.swift create_demo_video.swift FormCheck.xcodeproj/project.pbxproj FormCheck.xcodeproj/project.xcworkspace/contents.xcworkspacedata FormCheck.xcodeproj/project.xcworkspace/xcuserdata/aaronnguyen.xcuserdatad/UserInterfaceState.xcuserstate FormCheck.xcodeproj/project.xcworkspace/xcuserdata/jacobo.xcuserdatad/UserInterfaceState.xcuserstate FormCheck.xcodeproj/xcuserdata/aaronnguyen.xcuserdatad/xcschemes/xcschememanagement.plist FormCheck.xcodeproj/xcuserdata/jacobo.xcuserdatad/xcschemes/xcschememanagement.plist FormCheck/.DS_Store FormCheck/App/DIContainer.swift FormCheck/App/FormCheckApp.swift FormCheck/App/NavigationRouter.swift FormCheck/FormCheck.entitlements FormCheck/Info.plist FormCheck/Resources/BarbellDetector.mlpackage/Data/com.apple.CoreML/model.mlmodel FormCheck/Resources/BarbellDetector.mlpackage/Manifest.json FormCheck/Resources/PoseEstimator.mlpackage/Data/com.apple.CoreML/model.mlmodel FormCheck/Resources/PoseEstimator.mlpackage/Manifest.json FormCheck/Screens/Educational/EducationalContentView.swift FormCheck/Screens/ExerciseSelection/ExerciseSelectionView.swift FormCheck/Screens/History/HistoryView.swift FormCheck/Screens/History/HistoryViewModel.swift FormCheck/Screens/Home/HomeView.swift FormCheck/Screens/Playback/PlaybackView.swift FormCheck/Screens/Playback/PlaybackViewModel.swift FormCheck/Screens/Playback/SkeletonOverlayView.swift FormCheck/Screens/Recording/RecordingView.swift FormCheck/Screens/Recording/RecordingViewModel.swift FormCheck/Screens/SessionDetail/SessionDetailView.swift FormCheck/Screens/SessionSave/SessionSaveView.swift FormCheck/Screens/Settings/SettingsView.swift FormCheckTests/FormCheckTests.swift FormCheckUITests/FormCheckUITests.swift MLTraining/.DS_Store MLTraining/barbell_training/export_coreml.py MLTraining/barbell_training/prepare_dataset.py MLTraining/barbell_training/train_yolov8.py MLTraining/data/barbell_only/data.yaml MLTraining/data/barbell_only/test/labels/mc1_mp4-1_jpg.rf.d58cac845ffcd488d71ef0b82ca36fce.txt MLTraining/data/barbell_only/test/labels/mc10_mp4-0_jpg.rf.a2968d75bcd024698012b449c5a4293b.txt MLTraining/data/barbell_only/test/labels/mc10_mp4-4_jpg.rf.fe4b7c4bc4ad03a6981ff4746d4f8c56.txt MLTraining/data/barbell_only/test/labels/mc13_mp4-5_jpg.rf.0c6158c16269777c3ae2a94f57fe980a.txt MLTraining/data/barbell_only/test/labels/mc16_mp4-3_jpg.rf.89911695ec0960bffe7956adf84b71c8.txt MLTraining/data/barbell_only/test/labels/mc16_mp4-9_jpg.rf.937f69ab04282e338c67ff18dc47f109.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-0_jpg.rf.23f24be6f5667f05f45fd37ce9214aaa.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-12_jpg.rf.8aeae72d62ca42080b32fd2f6f28b80b.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-13_jpg.rf.2a68278160385c35c431d547fb4a468c.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-18_jpg.rf.453fbe97d7310989e8c71714d6497230.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-23_jpg.rf.e2a4d935b0b07dda887e0c41bc56ea02.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-25_jpg.rf.4d312b90a984441e038207de5ea964a4.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-32_jpg.rf.c4b870f9efa5d91e5546680f2ec61cd3.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-35_jpg.rf.5a2de80dd77d69d1d7cc89aa61c69800.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-37_jpg.rf.5350e4c9a223119d301a4fcd4145588e.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-41_jpg.rf.ae4871fd3d21c68665e27f5657729be6.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-52_jpg.rf.b91ee10e9653f59e69f2200e20ca462f.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-58_jpg.rf.8970d1484b57b6f7ead446231f35911f.txt MLTraining/data/barbell_only/test/labels/mc18_mp4-64_jpg.rf.a089663b5d270137f7f75d26ed39fc4e.txt MLTraining/data/barbell_only/test/labels/mc2_mp4-3_jpg.rf.3ca37bfb20467d11699e6ad907be7f52.txt MLTraining/data/barbell_only/test/labels/mc20_mp4-6_jpg.rf.284e452a8438c52f47e74d17b657a3ad.txt MLTraining/data/barbell_only/test/labels/mc21_mp4-0_jpg.rf.cf386aee6b5b65b53d7a1ba4aaf11f9f.txt MLTraining/data/barbell_only/test/labels/mc23_mp4-1_jpg.rf.92809da6360339dd027b12285396863a.txt MLTraining/data/barbell_only/test/labels/mc23_mp4-6_jpg.rf.c06b598fe491b8b11fd18c20ecd08ffd.txt MLTraining/data/barbell_only/test/labels/mc25_mp4-3_jpg.rf.e425ce8cec8b6841c987746064172cce.txt MLTraining/data/barbell_only/test/labels/mc27_mp4-4_jpg.rf.2aa3918c9fe058b7dccfc043b190ffef.txt MLTraining/data/barbell_only/test/labels/mc6_mp4-3_jpg.rf.5ee245d262b73ce69fd82cb50c208181.txt MLTraining/data/barbell_only/test/labels/mc6_mp4-6_jpg.rf.298fc857973859145e46925e5e5099e5.txt MLTraining/data/barbell_only/test/labels/mc6_mp4-7_jpg.rf.0173851b0dbf6e7d60a0f765fbcdf8cd.txt MLTraining/data/barbell_only/test/labels/mc7_mp4-4_jpg.rf.98c7450580d41d92168ed25ddfb22898.txt MLTraining/data/barbell_only/test/labels/mc7_mp4-7_jpg.rf.3f14865c72ad0289713a4c9810ea53e0.txt MLTraining/data/barbell_only/test/labels/mc9_mp4-5_jpg.rf.5454a63c30ee9d43f3a815c057de11ee.txt MLTraining/data/barbell_only/test/labels/wc1_mp4-0_jpg.rf.83441498f49d4c70ec752634f621b928.txt MLTraining/data/barbell_only/test/labels/wc1_mp4-1_jpg.rf.cf94de802a6c6c9c2132258bb738717e.txt MLTraining/data/barbell_only/test/labels/wc11_mp4-2_jpg.rf.ee906b91691717ad9b72500610867b6a.txt MLTraining/data/barbell_only/test/labels/wc12_mp4-4_jpg.rf.6e8995b1a5239efa7707b44f56173f6b.txt MLTraining/data/barbell_only/test/labels/wc13_mp4-10_jpg.rf.7bfdcc449ae44aabedf83cae746615e9.txt MLTraining/data/barbell_only/test/labels/wc13_mp4-4_jpg.rf.ac83cd8c6298a198b247abb4244bbba3.txt MLTraining/data/barbell_only/test/labels/wc14_mp4-7_jpg.rf.1a288420fe7268d9b880d2c84001cbd6.txt MLTraining/data/barbell_only/test/labels/wc2_mp4-14_jpg.rf.7b34611f063d5cb4a7327aff6b1f4239.txt MLTraining/data/barbell_only/test/labels/wc2_mp4-17_jpg.rf.49bee5502a38a22de0306b9154f25821.txt MLTraining/data/barbell_only/test/labels/wc3_mp4-0_jpg.rf.4971ae9be2dbd3a32245824d042d32cf.txt MLTraining/data/barbell_only/test/labels/wc3_mp4-2_jpg.rf.130154e083fcef5347a4e5efc683a241.txt MLTraining/data/barbell_only/test/labels/wc3_mp4-6_jpg.rf.349112cf6ffc5c65744e06bc4dfd022d.txt MLTraining/data/barbell_only/test/labels/wc4_mp4-7_jpg.rf.d24de5078b47420f736adc128dba86bd.txt MLTraining/data/barbell_only/test/labels/wc6_mp4-5_jpg.rf.c306c3e7b4efb97b39e0ad27b36625d3.txt MLTraining/data/barbell_only/test/labels/wc6_mp4-9_jpg.rf.985bb66361f895dc3eec99c729c54972.txt MLTraining/data/barbell_only/test/labels/wc8_mp4-4_jpg.rf.b1d6756a7c6ed40ecf4e5e91a3762309.txt MLTraining/data/barbell_only/test/labels/wc9_mp4-1_jpg.rf.5c6d2948ec9584f624f3b85c25e3f16b.txt MLTraining/data/barbell_only/test/labels/wc9_mp4-10_jpg.rf.c7e6b4d8615b42a5d61c625951b5aaba.txt MLTraining/data/barbell_only/train/labels.cache MLTraining/data/barbell_only/train/labels/mc1_mp4-2_jpg.rf.4d2da542d5bb17ded5c9ed22e1034efd.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-2_jpg.rf.c05a75db6064c36a6a3b62ad3230f9ad.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-2_jpg.rf.efc54c10bad81a703d6bbb6d9b7b508d.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-3_jpg.rf.660eb35ef3e62b2637c908383910a5bc.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-3_jpg.rf.cd1bed212da23c6460b584895cc0ad18.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-5_jpg.rf.1e333a8dad2497cc6970e31bf8098bd9.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-5_jpg.rf.49762a38249b01d75c1eea0e708e03d6.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-6_jpg.rf.1f1e560fbd617f418f23abbe85f300d2.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-6_jpg.rf.a56b4ea65da2a7d98edcc23013e8a380.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-6_jpg.rf.ff19cb6177da9f9bad845c45ce5ba30d.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-7_jpg.rf.bc6071e73d79bb4df82b97785a84e0a8.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-8_jpg.rf.9aae8dc74f975ffa12cf18da438f8fb5.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-8_jpg.rf.c4e63fecda4c6326699aeee2e798cd12.txt MLTraining/data/barbell_only/train/labels/mc1_mp4-8_jpg.rf.e5f523eba60d0ac3fb00f06d0a532767.txt MLTraining/data/barbell_only/train/labels/mc10_mp4-1_jpg.rf.3fe2c6ebfa6d5cb5300ae13082fc3924.txt MLTraining/data/barbell_only/train/labels/mc10_mp4-1_jpg.rf.a6b2bae321775549716e9e7fef7a514d.txt MLTraining/data/barbell_only/train/labels/mc10_mp4-3_jpg.rf.49f7a4c6372c149b5fb6a9e098e52d31.txt MLTraining/data/barbell_only/train/labels/mc10_mp4-3_jpg.rf.4ab506ca1c8acb5a1e19c568ffad1e26.txt MLTraining/data/barbell_only/train/labels/mc10_mp4-3_jpg.rf.87428b3ba62f303f0233ef8aeafed972.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-1_jpg.rf.35d0f88041efc31cc4c606b453cec760.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-1_jpg.rf.5de08bcda43d82bf2d0256aab499ee81.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-1_jpg.rf.bc3addc608b83ca62b1107ebd3566813.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-2_jpg.rf.1162272e262843db4fc7d363d289a374.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-2_jpg.rf.8fbf2a0f509d2097350bc2d2d4a784ec.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-2_jpg.rf.e0c539542cd5e1f357436b302e07ea8b.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-4_jpg.rf.b1f882aebbd29594cc8b11f4c0bb08b3.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-5_jpg.rf.016c9a289ac7b8236cdc9cad5e3ec672.txt MLTraining/data/barbell_only/train/labels/mc11_mp4-5_jpg.rf.9078df47a5128dc4647a30cb44408f1a.txt [8257 more files omitted for size] ``` ### Dependencies - MLTraining/form_classification/requirements.txt: coremltools@>=7.0, keras@>=3.0.0, matplotlib@>=3.7.0, mediapipe@>=0.10.0, numpy@>=1.24.0, opencv-python@>=4.8.0, pandas@>=2.0.0, pyyaml@>=6.0, scikit-learn@>=1.3.0, seaborn@>=0.12.0, tensorflow@>=2.15.0, tqdm@>=4.65.0 - MLTraining/requirements.txt: albumentations@>=1.3.0, coremltools@>=7.0, opencv-python@>=4.8.0, pyyaml@>=6.0, scipy@>=1.10.0, torch@>=2.0.0, torchvision@>=0.15.0, tqdm@>=4.65.0, ultralytics@>=8.0.0 ### Recent commits (newest first) - Merge pull request #2 from leojia22/feature/add-all-exercises - Refine UI components and improve session playback experience - Add support for new exercises (bicep curl, deadlift, lunge, plank) with form analysis rules and skeleton overlay - Add .gitignore to exclude .DS_Store and other system files - aaron ver 4 - aaron version 3 - aaron version 2 - ver aaron - Fixed build errors. - Tracking improvements. - Fixed y-invertion bug, made some improvements to model accuracy. - Merge remote changes - Update FormCheck project with latest changes - readme - Merge pull request #1 from leojia22/aaron - Collected sample videos for model training and conducted first iteration of form classification model training. - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### MLTraining/requirements.txt ``` ultralytics>=8.0.0 albumentations>=1.3.0 opencv-python>=4.8.0 pyyaml>=6.0 coremltools>=7.0 torch>=2.0.0 torchvision>=0.15.0 scipy>=1.10.0 tqdm>=4.65.0 ``` ### MLTraining/form_classification/requirements.txt ``` mediapipe>=0.10.0 opencv-python>=4.8.0 numpy>=1.24.0 pandas>=2.0.0 scikit-learn>=1.3.0 keras>=3.0.0 tensorflow>=2.15.0 coremltools>=7.0 pyyaml>=6.0 matplotlib>=3.7.0 seaborn>=0.12.0 tqdm>=4.65.0 ``` ### check_joint_names.swift ```swift #!/usr/bin/env swift import Foundation import AVFoundation import Vision let url = URL(fileURLWithPath: "openpose_sample.mp4") let asset = AVURLAsset(url: url) let semaphore = DispatchSemaphore(value: 0) var videoTrack: AVAssetTrack? Task { videoTrack = try? await asset.loadTracks(withMediaType: .video).first semaphore.signal() } semaphore.wait() guard let track = videoTrack else { print("No video track") exit(1) } let reader = try! AVAssetReader(asset: asset) let output = AVAssetReaderTrackOutput(track: track, outputSettings: [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA ]) reader.add(output) reader.startReading() // Get one frame guard let sampleBuffer = output.copyNextSampleBuffer(), let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { print("No frame") exit(1) } let request = VNDetectHumanBodyPoseRequest() let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:]) try! handler.perform([request]) if let obs = request.results?.first, let points = try? obs.recognizedPoints(.all) { print("Joint names detected by Vision:") print("─────────────────────────────────────") for (name, point) in points.sorted(by: { $0.key.rawValue.rawValue < $1.key.rawValue.rawValue }) { print(" \(name.rawValue.rawValue): conf=\(String(format: "%.2f", point.confidence))") } } ``` ### project.yml ```yaml name: FormCheck options: bundleIdPrefix: com.formcheck deploymentTarget: iOS: "17.0" xcodeVersion: "15.0" generateEmptyDirectories: true settings: base: SWIFT_VERSION: "5.9" IPHONEOS_DEPLOYMENT_TARGET: "17.0" targets: FormCheck: type: application platform: iOS sources: - path: FormCheck excludes: - "**/*.entitlements" settings: base: INFOPLIST_FILE: FormCheck/Info.plist CODE_SIGN_ENTITLEMENTS: FormCheck/FormCheck.entitlements PRODUCT_BUNDLE_IDENTIFIER: com.formcheck.app MARKETING_VERSION: "1.0.0" CURRENT_PROJECT_VERSION: "1" GENERATE_INFOPLIST_FILE: false ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon dependencies: - package: FCCore - package: FCPoseEstimation - package: FCBarbellDetection - package: FCFormAnalysis - package: FCCapture - package: FCPersistence - package: FCUI entitlements: path: FormCheck/FormCheck.entitlements FormCheckTests: type: bundle.unit-test platform: iOS sources: - path: FormCheckTests dependencies: - target: FormCheck FormCheckUITests: type: bundle.ui-testing platform: iOS sources: - path: FormCheckUITests dependencies: - target: FormCheck packages: FCCore: path: Packages/FCCore FCPoseEstimation: path: Packages/FCPoseEstimation FCBarbellDetection: path: Packages/FCBarbellDetection FCFormAnalysis: path: Packages/FCFormAnalysis FCCapture: path: Packages/FCCapture FCPersistence: path: Packages/FCPersistence FCUI: path: Packages/FCUI ``` ### test_pose_detection.swift ```swift #!/usr/bin/env swift import Foundation import AVFoundation import Vision import CoreImage // MARK: - Test Vision Pose Detection on Video func testPoseDetection(videoPath: String) { let url = URL(fileURLWithPath: videoPath) let asset = AVAsset(url: url) guard let track = asset.tracks(withMediaType: .video).first else { print("❌ No video track found") return } let reader: AVAssetReader do { reader = try AVAssetReader(asset: asset) } catch { print("❌ Failed to create reader: \(error)") return } let outputSettings: [String: Any] = [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA ] let output = AVAssetReaderTrackOutput(track: track, outputSettings: outputSettings) reader.add(output) reader.startReading() var frameCount = 0 var detectedFrames = 0 var totalJoints = 0 print("🎬 Processing video: \(videoPath)") print(" Duration: \(CMTimeGetSeconds(asset.duration))s") print("") // Process every 10th frame to speed up testing let sampleInterval = 10 while let sampleBuffer = output.copyNextSampleBuffer() { frameCount += 1 // Skip frames for faster testing guard frameCount % sampleInterval == 0 else { continue } guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { continue } // Test with different orientations let orientations: [(CGImagePropertyOrientation, String)] = [ (.up, "up"), (.right, "right"), (.down, "down"), (.left, "left"), ] for (orientation, name) in orientations { let request = VNDetectHumanBodyPoseRequest() let handler = VNImageRequestHandler( cvPixelBuffer: pixelBuffer, orientation: orientation, options: [:] ) do { try handler.perform([request]) if let results = request.results, !results.isEmpty { let bestObs = results.max { a, b in let confA = (try? a.recognizedPoints(.all))?.values.reduce(0) { $0 + $1.confidence } ?? 0 let confB = (try? b.recognizedPoints(.all))?.values.reduce(0) { $0 + $1.confidence } ?? 0 return confA < confB } if let obs = bestObs, let points = try? obs.recognizedPoints(.all) { let validPoints = points.filter { $0.value.confidence > 0.1 } if orientation == .right { detectedFrames += 1 totalJoints += validPoints.count } if frameCount % (sampleInterval * 5) == 0 { print("Frame \(frameCount) [orientation: \(name)]:") print(" Bodies detected: \(results.count)") print(" Valid joints: \(validPoints.count)/\(points.count)") // Print some joint positions for (jointName, point) in validPoints.prefix(5) { print(" - \(jointName.rawValue.rawValue): (\(String(format: "%.2f", point.location.x)), \(String(format: "%.2f", point.location.y))) conf: \(String(format: "%.2f", point.confidence))") } print("") } } } } catch { print("❌ Frame \(frameCount) [\(name)]: \(error.localizedDescription)") } } } let processedFrames = frameCount / sampleInterval print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") print("📊 Results Summary:") print(" Total frames: \(frameCount)") print(" Processed frames: \(processedFrames)") print(" Frames with detection: \(detectedFrames)") print(" Detection rate: \(String(format: "%.1f", Double(detectedFrames) / Double(max(processedFrames, 1)) * 100))%") print(" Avg joints per detection: \(detectedFrames > 0 ? totalJoints / detectedFrames : 0)") print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") if detectedFrames == 0 { print("") print("⚠️ No poses detected! Possible issues:") print(" - Video may not contain visible people") print(" - Person may be too small/far from camera") print(" - Unusual camera angle or orientation") } else { print("") print("✅ Pose detection is working!") } } // Main let videoPath = CommandLine.arguments.count > 1 ? CommandLine.arguments[1] : "test_video.mp4" testPoseDetection(videoPath: videoPath) ``` ### create_demo_video.swift ```swift #!/usr/bin/env swift import Foundation import AVFoundation import Vision import CoreImage import AppKit import CoreGraphics // MARK: - Skeleton Drawing Demo Video Generator // Bone connections (matching the app's SkeletonBones) let boneConnections: [(from: String, to: String)] = [ ("neck_1_joint", "head_joint"), ("left_shoulder_1_joint", "right_shoulder_1_joint"), ("left_shoulder_1_joint", "left_forearm_joint"), ("left_forearm_joint", "left_hand_joint"), ("right_shoulder_1_joint", "right_forearm_joint"), ("right_forearm_joint", "right_hand_joint"), ("left_shoulder_1_joint", "left_upLeg_joint"), ("right_shoulder_1_joint", "right_upLeg_joint"), ("left_upLeg_joint", "right_upLeg_joint"), ("left_upLeg_joint", "left_leg_joint"), ("left_leg_joint", "left_foot_joint"), ("right_upLeg_joint", "right_leg_joint"), ("right_leg_joint", "right_foot_joint"), ("neck_1_joint", "root"), ] func createDemoVideo(inputPath: String, outputPath: String) { let inputURL = URL(fileURLWithPath: inputPath) let outputURL = URL(fileURLWithPath: outputPath) // Remove existing output file try? FileManager.default.removeItem(at: outputURL) let asset = AVURLAsset(url: inputURL) // Load video track let semaphore = DispatchSemaphore(value: 0) var videoTrack: AVAssetTrack? var naturalSize: CGSize = .zero var frameRate: Float = 30 Task { if let track = try? await asset.loadTracks(withMediaType: .video).first { videoTrack = track naturalSize = try! await track.load(.naturalSize) frameRate = try! await track.load(.nominalFrameRate) } semaphore.signal() } semaphore.wait() guard let track = videoTrack else { print("❌ No video track found") return } print("🎬 Creating demo video...") print(" Input: \(inputPath)") print(" Size: \(Int(naturalSize.width))x\(Int(naturalSize.height))") print(" Frame rate: \(frameRate) fps") // Setup reader let reader = try! AVAssetReader(asset: asset) let readerOutput = AVAssetReaderTrackOutput(track: track, outputSettings: [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA ]) reader.add(readerOutput) // Setup writer let writer = try! AVAssetWriter(url: outputURL, fileType: .mp4) let videoSettings: [String: Any] = [ AVVideoCodecKey: AVVideoCodecType.h264, AVVideoWidthKey: Int(naturalSize.width), AVVideoHeightKey: Int(naturalSize.height), AVVideoCompressionPropertiesKey: [ AVVideoAverageBitRateKey: 8_000_000, ] ] let writerInput = AVAssetWriterInput(mediaType: .video, outputSettings: videoSettings) writerInput.expectsMediaDataInRealTime = false let adaptor = AVAssetWriterInputPixelBufferAdaptor( assetWriterInput: writerInput, sourcePixelBufferAttributes: [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA, kCVPixelBufferWidthKey as String: Int(naturalSize.width), kCVPixelBufferHeightKey as String: Int(naturalSize.height), ] ) writer.add(writerInput) reader.startReading() writer.startWriting() writer.startSession(atSourceTime: .zero) var frameCount = 0 let maxFrames = 300 // Limit to first 10 seconds at 30fps while let sampleBuffer = readerOutput.copyNextSampleBuffer(), frameCount < maxFrames { frameCount += 1 guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { continue } let presentationTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) // Run pose detection let request = VNDetectHumanBodyPoseRequest() let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:]) try? handler.perform([request]) // Draw skeleton on frame let outputBuffer = drawSkeleton(on: pixelBuffer, poses: request.results ?? []) // Wait for writer to be ready while !writerInput.isReadyForMoreMediaData { Thread.sleep(forTimeInterval: 0.01) } adaptor.append(outputBuffer, withPresentationTime: presentationTime) if frameCount % 30 == 0 { print(" Processed \(frameCount) frames...") } } writerInput.markAsFinished() let finishSemaphore = DispatchSemaphore(value: 0) writer.finishWriting { finishSemaphore.signal() } finishSemaphore.wait() print("✅ Demo video created: \(outputPath)") print(" Total frames: \(frameCount)") } func drawSkeleton(on pixelBuffer: CVPixelBuffer, poses: [VNHumanBodyPoseObservation]) -> CVPixelBuffer { let width = CVPixelBufferGetWidth(pixelBuffer) let height = CVPixelBufferGetHeight(pixelBuffer) // Create CIImage from pixel buffer let ciImage = CIImage(cvPixelBuffer: pixelBuffer) // Create CGContext to draw on let colorSpace = CGColorSpaceCreateDeviceRGB() let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue) guard let context = CGContext( data: nil, width: width, height: height, bitsPerComponent: 8, bytesPerRow: width * 4, space: colorSpace, bitmapInfo: bitmapInfo.rawValue ) else { return pixelBuffer } // Draw original image let ciContext = CIContext() if let cgImage = ciContext.createCGImage(ciImage, from: ciImage.extent) { context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height)) } // Draw skeleton for each pose for pose in poses { guard let points = try? pose.recognizedPoints(.all) else { continue } // Convert to screen coordinates var jointPositions: [String: CGPoint] = [:] var jointConfidences: [String: Float] = [truncated — 3250 more characters] ``` ### compare_pose_detection.swift ```swift #!/usr/bin/env swift import Foundation import AVFoundation import Vision import CoreImage import AppKit // MARK: - Comprehensive Pose Detection Comparison Test struct JointDetection { let name: String let x: CGFloat let y: CGFloat let confidence: Float } struct FrameResult { let frameNumber: Int let bodiesDetected: Int let joints: [JointDetection] let processingTime: Double } func runPoseDetection(videoPath: String, outputDir: String) { let url = URL(fileURLWithPath: videoPath) let asset = AVURLAsset(url: url) // Get video track let semaphore = DispatchSemaphore(value: 0) var videoTrack: AVAssetTrack? Task { videoTrack = try? await asset.loadTracks(withMediaType: .video).first semaphore.signal() } semaphore.wait() guard let track = videoTrack else { print("❌ No video track found") return } let reader: AVAssetReader do { reader = try AVAssetReader(asset: asset) } catch { print("❌ Failed to create reader: \(error)") return } let outputSettings: [String: Any] = [ kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA ] let output = AVAssetReaderTrackOutput(track: track, outputSettings: outputSettings) reader.add(output) reader.startReading() var frameCount = 0 var results: [FrameResult] = [] var totalProcessingTime: Double = 0 // Create output directory try? FileManager.default.createDirectory(atPath: outputDir, withIntermediateDirectories: true) print("🎬 Analyzing: \(videoPath)") print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") // Process frames while let sampleBuffer = output.copyNextSampleBuffer() { frameCount += 1 // Process every frame for accuracy measurement guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { continue } let startTime = CFAbsoluteTimeGetCurrent() let request = VNDetectHumanBodyPoseRequest() let handler = VNImageRequestHandler( cvPixelBuffer: pixelBuffer, orientation: .up, options: [:] ) do { try handler.perform([request]) } catch { continue } let processingTime = CFAbsoluteTimeGetCurrent() - startTime totalProcessingTime += processingTime var joints: [JointDetection] = [] var bodiesDetected = 0 if let observations = request.results, !observations.isEmpty { bodiesDetected = observations.count // Get best observation if let bestObs = observations.max(by: { a, b in let confA = (try? a.recognizedPoints(.all))?.values.reduce(0) { $0 + $1.confidence } ?? 0 let confB = (try? b.recognizedPoints(.all))?.values.reduce(0) { $0 + $1.confidence } ?? 0 return confA < confB }), let points = try? bestObs.recognizedPoints(.all) { for (jointName, point) in points { joints.append(JointDetection( name: jointName.rawValue.rawValue, x: point.location.x, y: 1.0 - point.location.y, confidence: point.confidence )) } } } let result = FrameResult( frameNumber: frameCount, bodiesDetected: bodiesDetected, joints: joints, processingTime: processingTime ) results.append(result) // Progress indicator if frameCount % 30 == 0 { let validJoints = joints.filter { $0.confidence > 0.1 }.count print("Frame \(frameCount): \(bodiesDetected) bodies, \(validJoints) joints, \(String(format: "%.1f", processingTime * 1000))ms") } } // Compute statistics print("") print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") print("📊 DETAILED ANALYSIS") print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") let framesWithDetection = results.filter { $0.bodiesDetected > 0 }.count let detectionRate = Double(framesWithDetection) / Double(results.count) * 100 // Joint-level analysis var jointStats: [String: (total: Int, detected: Int, avgConf: Float)] = [:] let importantJoints = [ "left_shoulder_1_joint", "right_shoulder_1_joint", "left_forearm_joint", "right_forearm_joint", "left_hand_joint", "right_hand_joint", "left_upLeg_joint", "right_upLeg_joint", "left_leg_joint", "right_leg_joint", "left_foot_joint", "right_foot_joint", "neck_1_joint", "head_joint" ] for jointName in importantJoints { jointStats[jointName] = (total: 0, detected: 0, avgConf: 0) } for result in results { for jointName in importantJoints { if let joint = result.joints.first(where: { $0.name == jointName }) { var stats = jointStats[jointName]! stats.total += 1 if joint.confidence > 0.1 { stats.detected += 1 stats.avgConf += joint.confidence } jointStats[jointName] = stats } } } print("") print("Overall Statistics:") print(" Total frames: \(results.count)") print(" Frames with body: \(framesWithDetection) (\(String(format: "%.1f", detectionRate))%)") print(" Avg processing time: \(String(format: "%.1f", totalProcessingTime / Double(results.count) * 1000))ms/frame") print(" Potential FPS: \(String(format: "%.1f", Double(results.count) / totalProcessingTime))") print("") print("Joint Detection Rates (confidence > 0.1):") print(" Joint Detection% Avg Conf") print(" ───────────── [truncated — 3524 more characters] ``` ### FormCheckTests/FormCheckTests.swift ```swift import XCTest final class FormCheckTests: XCTestCase { func testAppLaunches() { // Placeholder — app-level integration tests go here XCTAssertTrue(true) } } ``` ### FormCheckUITests/FormCheckUITests.swift ```swift import XCTest final class FormCheckUITests: XCTestCase { func testNavigationFlow() throws { let app = XCUIApplication() app.launch() // Verify home screen loads XCTAssertTrue(app.navigationBars["FormCheck"].exists) // Navigate to exercise selection let startButton = app.buttons["Start New Session"] if startButton.exists { startButton.tap() XCTAssertTrue(app.staticTexts["Choose an Exercise"].waitForExistence(timeout: 2)) } } func testHistoryNavigation() throws { let app = XCUIApplication() app.launch() let historyButton = app.buttons["clock.arrow.circlepath"] if historyButton.exists { historyButton.tap() XCTAssertTrue(app.navigationBars["History"].waitForExistence(timeout: 2)) } } func testSettingsNavigation() throws { let app = XCUIApplication() app.launch() let settingsButton = app.buttons["gearshape"] if settingsButton.exists { settingsButton.tap() XCTAssertTrue(app.navigationBars["Settings"].waitForExistence(timeout: 2)) } } } ``` ### Packages/FCCore/Package.swift ```swift // swift-tools-version: 5.9 import PackageDescription let package = Package( name: "FCCore", platforms: [.iOS(.v17)], products: [ .library(name: "FCCore", targets: ["FCCore"]), ], targets: [ .target(name: "FCCore"), .testTarget(name: "FCCoreTests", dependencies: ["FCCore"]), ] ) ``` [109 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]