Real-Time Weightlifting Form Analysis

Inspiration

We've all been there, lifting weights at the gym, unsure if our form is correct, risking injury with every rep. Personal trainers are expensive, and watching yourself in a mirror only shows one angle while you're mid-lift. We wanted to democratize access to professional-quality form analysis, making it as simple as pointing your phone's camera at yourself while you train.

What it does

FormCheck is an iOS app that analyzes weightlifting form in real-time using computer vision. Simply select your exercise (bench press, squats, deadlifts, etc), start recording, and lift. The app detects your body pose and barbell position using Apple's Vision framework and a custom-trained YOLOv8 object detection model. It automatically counts your reps, tracks bar path, and flags form issues like knee cave, elbow flare, back rounding, or wrist misalignment, providing instant feedback with severity levels (warning/danger) and educational content explaining why proper form matters. After your set, review the recording with pose skeleton overlays, see a per-rep breakdown of form issues, and track your progress over time with session history.

How we built it

We built FormCheck as a modular iOS app using Swift and SwiftUI, organized into seven local packages for clean architecture: FCPoseEstimation: Apple Vision framework (VNDetectHumanBodyPoseRequest) extracts 19 body keypoints from each frame FCPoseEstimation: Apple Vision framework (VNDetectHumanBodyPoseRequest) extracts 19 body keypoints from each frame FCBarbellDetection: Custom YOLOv8 model trained in Python (using Ultralytics) and exported to Core ML for barbell tracking FCBarbellDetection: Custom YOLOv8 model trained in Python (using Ultralytics) and exported to Core ML for barbell tracking FCFormAnalysis: Rule engine with 22+ exercise-specific form rules using angle calculations and biomechanical thresholds FCFormAnalysis: Rule engine with 22+ exercise-specific form rules using angle calculations and biomechanical thresholds FCCapture: AVFoundation-based camera capture with frame-by-frame analysis FCCapture: AVFoundation-based camera capture with frame-by-frame analysis FCPersistence: Core Data for local storage of sessions and metrics FCPersistence: Core Data for local storage of sessions and metrics FCUI: Design system with modern UI components FCUI: Design system with modern UI components MLTraining/: The ML training pipeline uses Python with PyTorch, Ultralytics YOLOv8, and CoreML Tools to prepare datasets, train the barbell detector, and export to a format optimized for on-device inference MLTraining/: The ML training pipeline uses Python with PyTorch, Ultralytics YOLOv8, and CoreML Tools to prepare datasets, train the barbell detector, and export to a format optimized for on-device inference

Challenges we ran into

Real-time performance: Processing pose estimation and object detection at 30fps on-device required significant optimization — we implemented frame skipping, model quantization, and efficient rendering Real-time performance: Processing pose estimation and object detection at 30fps on-device required significant optimization — we implemented frame skipping, model quantization, and efficient rendering Barbell occlusion: The barbell often gets occluded by the lifter's body, making consistent tracking difficult. We implemented a wrist-based fallback estimator Barbell occlusion: The barbell often gets occluded by the lifter's body, making consistent tracking difficult. We implemented a wrist-based fallback estimator Camera angle detection: Form rules vary based on viewing angle (front vs. side). We built an automatic camera angle classifier using shoulder-hip alignment Camera angle detection: Form rules vary based on viewing angle (front vs. side). We built an automatic camera angle classifier using shoulder-hip alignment Rep counting accuracy: Distinguishing between exercise phases (descending, bottom, ascending, lockout) required hysteresis filtering to avoid false triggers from shaky movements Rep counting accuracy: Distinguishing between exercise phases (descending, bottom, ascending, lockout) required hysteresis filtering to avoid false triggers from shaky movements

Accomplishments we're proud of

Runs entirely on-device with no cloud processing — privacy-first and works offline Runs entirely on-device with no cloud processing — privacy-first and works offline 22+ biomechanically-sound form rules across 6 exercises, each with educational content 22+ biomechanically-sound form rules across 6 exercises, each with educational content Sub-100ms latency from camera frame to form feedback Sub-100ms latency from camera frame to form feedback Smooth 60fps UI even during intensive ML inference Smooth 60fps UI even during intensive ML inference Beautiful, intuitive design that feels like a polished commercial app Beautiful, intuitive design that feels like a polished commercial app Comprehensive architecture with proper separation of concerns and testable components Comprehensive architecture with proper separation of concerns and testable components

What we learned

How to optimize ML models for real-time on-device inference on mobile hardware How to optimize ML models for real-time on-device inference on mobile hardware The complexity of human biomechanics and the nuances of proper lifting technique The complexity of human biomechanics and the nuances of proper lifting technique Apple's Vision framework capabilities and limitations for pose estimation Apple's Vision framework capabilities and limitations for pose estimation Core Data performance optimization for high-frequency writes during recording Core Data performance optimization for high-frequency writes during recording The importance of user feedback timing — too many alerts become noise, too few and issues get missed The importance of user feedback timing — too many alerts become noise, too few and issues get missed Building maintainable iOS codebases using Swift Package Manager for modularization Building maintainable iOS codebases using Swift Package Manager for modularization

What's next

for FormCheck More exercises: Overhead press, rows, pull-ups, dips, and olympic lifts More exercises: Overhead press, rows, pull-ups, dips, and olympic lifts Advanced analytics: Velocity tracking, bar speed zones, fatigue detection across sets Advanced analytics: Velocity tracking, bar speed zones, fatigue detection across sets Social features: Share PRs, compare form with friends, coach-athlete collaboration Social features: Share PRs, compare form with friends, coach-athlete collaboration Apple Watch integration: Heart rate correlation with form breakdown Apple Watch integration: Heart rate correlation with form breakdown 3D visualization: Render full 3D skeleton from multiple camera angles 3D visualization: Render full 3D skeleton from multiple camera angles Form recommendations: AI-powered suggestions for correcting specific issues Form recommendations: AI-powered suggestions for correcting specific issues Gamification: Achievements, streaks, and perfect-form challenges Gamification: Achievements, streaks, and perfect-form challenges Cross-platform: Expand to Android using TensorFlow Lite Cross-platform: Expand to Android using TensorFlow Lite