# Project export: JustChi
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: A VR Tai Chi experience. We use open-sourced computer vision models to transform senior health and wellness through engaging exercises. JustChi is an exciting leap in fitness transformation!
- Devpost: https://devpost.com/software/justchi
- GitHub: https://github.com/nickkbuii/JustChi
- Video: https://www.youtube.com/embed/arl4NgsJtW4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — Rally Lin (13 commits), Daniel Yang (13 commits), nickkbuii (2 commits), yifantong1010 (1 commits)
## Devpost submission (written by the team)
### Inspiration
The idea behind JustChi stemmed from a desire to improve the physical and mental well-being of seniors through accessible and engaging technology. Since everyone on the team has a relative who would benefit from the gentle exercise that is Tai Chi, we thought it would be a wonderful idea to prototype a futuristic VR game targeting the growing senior population. By combining the ancient practice of Tai Chi, known for its health benefits such as improved balance, flexibility, and stress reduction, with modern Virtual Reality (VR) and Computer Vision technology, JustChi aims to provide a unique and immersive experience that is both beneficial and enjoyable for its users.
### What it does
Think "Just Dance" but instead of high-intensity dance, our users are immersed in relaxing and peaceful VR environments where they can engage in the ancient Chinese art of Tai Chi. Once set up on the Meta Quest 2, our users can select from a variety of Tai Chi styles and soothing background environments. They are then sent to these environments where they can follow along with our prerecorded Tai Chi "master" through a game-like experience. Along the way, we provide supportive feedback and calculate the accuracy of their moves, using it to display their Tai Chi score in real-time. Through Just Chi, anyone can learn Tai Chi and reap a myriad of health benefits while being educated on ancient Chinese culture.
### How we built it
We used an open-source computer vision framework called MediaPipe to determine our user's pose by tracking keypoints (and their coordinates) displayed along the user's body, storing and transmitting these coordinates through a Flask server. We then ran these coordinates through our pose comparison Python algorithm, which compares the coordinates of the user's keypoints with the keypoints of our Tai Chi "master" using linear algebra techniques. Then, we used Figma, HTML, CSS, JavaScript, and a VR development framework called A-Frame to design our VR UI/UX and environment that is displayed to the user through the Meta Quest headset.
### Challenges we ran into
Initially, it was difficult to find a way to track keypoints while interfacing with Unity (a game engine we originally thought about using). However, we eventually found A-Frame and never looked back. It was incredibly challenging to write the pose comparison algorithm because we had trouble syncing the Tai Chi "master" recording with the user in order to compare their keypoint coordinates. Even once we implemented several different versions of the algorithm, it wasn't especially accurate at times. It was also difficult for us to design the UI/UX and environment for the Meta Quest. It was all of our first time working with VR so it was quite the learning curve for us.
### Accomplishments we're proud of
We're proud of our functional pose comparison algorithm which we finally got to work after almost an entire day of programming, testing, and debugging. We're also very proud that we were able to have a working, enjoyable, and impactful VR game done in less than 2 days. Additionally, we are proud that we were able to find A-Frame and learn a VR web development framework that was previously completely unknown to us. Finally, we are so proud that we were able to spend the weekend with so many passionate, talented, and friendly people.
### What we learned
We learned a lot about VR development frameworks such as A-Frame, the different kinds of computer vision frameworks such as MediaPipe, the amazing art of Tai Chi, and vector similarity algorithms. Particularly, we learned that there are other VR development platforms out there (besides Unity), how to use MediaPipe, different forms of Tai Chi styles (Chen, Yang, Hao, Wu, Chen, etc), and how we can vectorize information and use linear algebra to determine similarity. Most importantly, we learned more about how prevalent depression and health issues are among the elderly, motivating us to create a fitness program that combats these issues: JustChi.
### What's next
Our roadmap is filled with exciting developments, from introducing multiplayer capabilities to integrating heart rate monitoring for health and fitness insights. We're committed to expanding JustChi's content, offering new Tai Chi styles, and creating more immersive and personalized environments. The potential for JustChi to make a significant impact is boundless, and we're thrilled to continue this journey of innovation and wellness.
## README (from the GitHub repository)
# JustChi
[](LICENSE.md)
A pose-tracking VR application built with Meta Quest 2 for interactive Tai Chi learning. TreeHacks 2024.
## Inspiration
The idea behind JustChi stemmed from a desire to improve the physical and mental well-being of seniors through accessible and engaging technology. Since everyone on the team has a relative who would benefit from the gentle exercise that is Tai Chi, we thought it would be a wonderful idea to prototype a futuristic VR game targeting the growing senior population. By combining the ancient practice of Tai Chi, known for its health benefits such as improved balance, flexibility, and stress reduction, with modern Virtual Reality (VR) and Computer Vision technology, JustChi aims to provide a unique and immersive experience that is both beneficial and enjoyable for its users.
## What it does
Think "Just Dance" but instead of high-intensity dance, our users are immersed in relaxing and peaceful VR environments where they can engage in the ancient Chinese art of Tai Chi. Once set up on the Meta Quest 2, our users can select from a variety of Tai Chi styles and soothing background environments. They are then sent to these environments where they can follow along with our prerecorded Tai Chi "master" through a game-like experience. Along the way, we provide supportive feedback and calculate the accuracy of their moves, using it to display their Tai Chi score in real-time. Through Just Chi, anyone can learn Tai Chi and reap a myriad of health benefits while being educated on ancient Chinese culture.
## How we built it
We used an open-source computer vision framework called MediaPipe to determine our user's pose by tracking keypoints (and their coordinates) displayed along the user's body, storing and transmitting these coordinates through a Flask server. We then ran these coordinates through our pose comparison Python algorithm, which compares the coordinates of the user's keypoints with the keypoints of our Tai Chi "master" using linear algebra techniques. Then, we used Figma, HTML, CSS, JavaScript, and a VR development framework called A-Frame to design our VR UI/UX and environment that is displayed to the user through the Meta Quest headset.
## Installation
Requirements:
* Python (3.11.4)
* mediapipe (0.8.9.1)
* Flask-SocketIO (5.3.6)
* opencv-python (4.5.5.61)
* numpy (1.26.4)
Installation steps:
1. Clone the repository
```
$ git clone https://github.com/nickkbuii/JustChi.git
```
2. Connect host and Meta Quest 2 to a local network
4. Run the `app.py`file
```
$ python app.py
```
5. Open server address on browser in Meta Quest 2
## Our Team
* Rally Lin (rally.lin@duke.edu)
* Daniel Yang (dy5251@princeton.edu)
* Alex Tong (alextong1010@berkeley.edu)
* Nick Bui (nicholasqbui@berkeley.edu)
## Detected evidence (automated analysis)
Indexed codebase: 9 recognized source files, 43 KB.
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- TensorFlow (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (13 of 13)
```
aframe/app.py
aframe/pose_estimation.py
aframe/processnew.py
aframe/static/.DS_Store
aframe/static/js/utils.js
aframe/static/json/calibration_data.json
aframe/static/json/demo_data.json
aframe/static/objects/forest.gltf
aframe/templates/index.html
aframe/templates/test.html
aframe/utils.py
LICENSE.md
README.md
```
### Dependencies
No dependency index available.
### Recent commits (newest first)
- Update README.md
- Rename LICENSE to LICENSE.md
- Create LICENSE
- Update README.md
- Update README.md
- Add files via upload
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Create README.md
- Delete aframe/.DS_Store (cleanup)
## Key source files (fetched from GitHub, selected and truncated for size)
### LICENSE.md
```markdown
MIT License
Copyright (c) 2024 Rally Lin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
### aframe/app.py
```python
from flask import Flask, render_template, request, jsonify
from flask_socketio import SocketIO
import time
import threading
import cv2
import mediapipe as mp
import time
import json
import numpy as np
from utils import *
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret_key'
socketio = SocketIO(app)
angle_key = {
"right_arm_angle": (11, 13, 15),
"left_arm_angle": (12, 14, 16),
"right_torso_angle": (23, 11, 13),
"left_torso_angle": (24, 12, 14),
"right_knee_angle": (23, 25, 27),
"left_knee_angle": (24, 26, 28),
"right_leg_angle": (11, 23, 25),
"left_leg_angle": (12, 24, 26)
}
FPS = 20
VAL_300 = 0.2
VAL_100 = 0.45
VAL_50 = 0.7
def judgement_value(val):
if 0 <= val <= VAL_300:
return 1, "Excellent!"
elif VAL_300 < val <= VAL_100:
return 1/3, "Great!"
elif VAL_100 < val <= VAL_50:
return 1/6, "OK!"
else:
return 0, "Miss :("
with open('./static/json/demo_data.json', 'r') as file:
reference_data = json.load(file)
isTracking = False
start_time = 0
reference_index = 0
score_buffer = []
judgements = []
score = -1
def emit_pose_data():
global isTracking, reference_index, start_time, score_buffer, judgements, score
mp_pose = mp.solutions.pose
pose = mp_pose.Pose()
cap = cv2.VideoCapture(0) # Use the webcam
frame_count = 0
with app.app_context():
while cap.isOpened():
success, image = cap.read()
if not success:
continue
# Convert the BGR image to RGB, flip the image around y-axis for correct handedness output
image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
# Process the image and draw the pose annotation
results = pose.process(image)
if results.pose_landmarks:
pos_loc = []
track_loc = []
for landmark in results.pose_landmarks.landmark:
pos_loc.append({"x": landmark.x, "y": -landmark.y, "z": landmark.z, "visibility": landmark.visibility})
track_loc.append({"x": landmark.x, "y": landmark.y, "z": landmark.z, "visibility": landmark.visibility})
if frame_count % 10 == 0: #every x frames, update score
if score != -1:
print(f"Score: {score}")
buf_size = min(len(score_buffer), 10)
score = -1 if len(score_buffer) == 0 else sum(score_buffer[-buf_size:])/buf_size
judgement_val, judgement_text = judgement_value(score)
judgements.append(judgement_val)
socketio.emit('update_model', {'pose': pos_loc, 'score': str(round(score, 3)),
'final': 'false', 'judgement': judgement_text})
if isTracking:
if not track_loc:
time.sleep(1/FPS)
continue
current_time = time.time() - start_time
# print(current_time, start_time)
while reference_index < len(reference_data) and current_time > float(reference_data[reference_index]['time']):
reference_index = reference_index + 1
if reference_index >= len(reference_data):
print("Reached end of video file.")
isTracking = False
score_buffer = []
final_score = sum(judgements)/len(judgements) * 1_000_000
socketio.emit('update_model', {'pose': pos_loc, 'score': '{:,.0f}'.format(round(final_score)), 'final': 'true'})
time.sleep(1/FPS)
continue
reference_frame = reference_data[reference_index]
frame_angles = {}
for angle_name, indices in angle_key.items():
p1, p2, p3 = [(track_loc[i]['x'], track_loc[i]['y']) for i in indices]
frame_angle = angle(p1, p2, p3)
frame_angles[angle_name] = frame_angle
score_buffer.append(compare_angles(reference_frame['angles'], frame_angles))
frame_count += 1
time.sleep(1/FPS)
@app.route('/')
def index():
return render_template('index.html')
@app.route('/test')
def test():
return render_template('test.html')
@app.route('/video-started', methods=['POST'])
def video_started():
global isTracking, start_time, reference_index, score_buffer, judgements
data = request.json
print("Video Tracking Started")
isTracking = True
start_time = time.time()
reference_index = 0
score_buffer = []
judgements = []
return jsonify({"message": "Received"}), 200
@socketio.on('connect')
def test_connect():
print('Client connected')
threading.Thread(target=emit_pose_data).start() # Start emitting pose data
@socketio.on('disconnect')
def test_disconnect():
print('Client disconnected')
if __name__ == '__main__':
socketio.run(app, debug=True, host='0.0.0.0', ssl_context='adhoc')
# socketio.run(app, debug=True)
```
### aframe/pose_estimation.py
```python
import cv2
import mediapipe as mp
import time
mp_pose = mp.solutions.pose
pose = mp_pose.Pose()
mp_drawing = mp.solutions.drawing_utils
# cap = cv2.VideoCapture('./static/videos/test_calibration_daniel.mov')
# cap = cv2.VideoCapture('./static/videos/demo.mp4')
cap = cv2.VideoCapture(0)
while cap.isOpened():
success, image = cap.read()
if not success:
continue
# Convert the BGR image to RGB, flip the image around y-axis for correct handedness output
image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
# Process the image and draw the pose annotation
results = pose.process(image)
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
if results.pose_landmarks:
mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
cv2.imshow('MediaPipe Pose', image)
if cv2.waitKey(5) & 0xFF == 27: # Press 'ESC' to exit
break
cap.release()
```
### aframe/processnew.py
```python
import cv2
import mediapipe as mp
import numpy as np
import json
mp_pose = mp.solutions.pose
pose = mp_pose.Pose()
mp_drawing = mp.solutions.drawing_utils
cap = cv2.VideoCapture('taichi.mp4')
frame_rate = 30.0
frame_counter = 0
frames_data = []
angle_key = {
"right_arm_angle": (11, 13, 15),
"left_arm_angle": (12, 14, 16),
"right_torso_angle": (23, 11, 13),
"left_torso_angle": (24, 12, 14),
"right_knee_angle": (23, 25, 27),
"left_knee_angle": (24, 26, 28),
"right_leg_angle": (11, 23, 25),
"left_leg_angle": (12, 24, 26)
}
while cap.isOpened():
success, image = cap.read()
if not success:
break # Exit loop if video ends or there's an error
# Calculate the timestamp based on the frame counter
timestamp = frame_counter / frame_rate
image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
results = pose.process(image)
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
if results.pose_landmarks:
mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
def unit_vector(vector):
return vector / np.linalg.norm(vector)
def angle(p1, mid, p2):
p1 = np.array(p1)
mid = np.array(mid)
p2 = np.array(p2)
v1_u = unit_vector(p1 - mid)
v2_u = unit_vector(p2 - mid)
return np.arccos(np.clip(np.dot(v1_u, v2_u), -1.0, 1.0))
landmarks = [{'x': landmark.x, 'y': landmark.y, 'z': landmark.z, 'visibility': landmark.visibility}
for landmark in results.pose_landmarks.landmark]
frame_angles = {}
for angle_name, indices in angle_key.items():
p1, p2, p3 = [(landmarks[i]['x'], landmarks[i]['y']) for i in indices]
frame_angle = angle(p1, p2, p3)
frame_angles[angle_name] = frame_angle
frames_data.append({'time': timestamp, 'landmarks': landmarks, 'angles': frame_angles})
cv2.imshow('MediaPipe Pose', image)
if cv2.waitKey(5) & 0xFF == 27: # Press 'ESC' to exit
break
# Increment frame counter
frame_counter += 1
cap.release()
cv2.destroyAllWindows()
# Write data to JSON file
with open('landmarks_data.json', 'w') as f:
json.dump(frames_data, f, indent=4)
```
### aframe/utils.py
```python
import numpy as np
def unit_vector(vector):
return vector / np.linalg.norm(vector)
def angle(p1, mid, p2):
p1 = np.array(p1)
mid = np.array(mid)
p2 = np.array(p2)
v1_u = unit_vector(p1 - mid)
v2_u = unit_vector(p2 - mid)
return np.arccos(np.clip(np.dot(v1_u, v2_u), -1.0, 1.0))
def compare_angles(reference, frame):
differences = [abs(reference[key] - frame[key]) for key in reference]
return sum(differences) / len(differences)
def objective_function(keypoints1, keypoints2, scale, translation):
transformed_keypoints1 = (keypoints1 * scale) + translation
ssd = np.sum((transformed_keypoints1 - keypoints2)**2)
return ssd
def gradient_descent(keypoints1, keypoints2, learning_rate=0.001, iterations=1000):
scale = 1.0
translation = np.zeros(2)
for i in range(iterations):
scale_grad = 2 * np.sum((keypoints1 * scale + translation - keypoints2) * keypoints1)
translation_grad = 2 * np.sum(keypoints1 * scale + translation - keypoints2, axis=0)
scale -= learning_rate * scale_grad
translation -= learning_rate * translation_grad
if i == iterations-1:
cost = objective_function(keypoints1, keypoints2, scale, translation)
print(f"Iteration {i}: SSD = {cost}")
return scale, translation
def apply_transformation(keypoints, scale, translation):
return (keypoints * scale) + translation
def calculate_similarity(transformed_keypoints, keypoints):
distances = np.sqrt(np.sum((transformed_keypoints - keypoints)**2, axis=1))
similarity_score = np.mean(distances)
return similarity_score
def filter_keypoints_by_visibility(keyframe1, keyframe2):
print(keyframe1[:, 2])
print(np.where(keyframe1[:, 2] > 0.5))
visible_indices = np.where((keyframe1[:, 2] > 0.5) & (keyframe2[:, 2] > 0.5))[0]
print(visible_indices)
filtered_keyframe1 = keyframe1[visible_indices, :2]
filtered_keyframe2 = keyframe2[visible_indices, :2]
return filtered_keyframe1, filtered_keyframe2
if __name__ == "__main__":
import time
import threading
import cv2
import mediapipe as mp
import time
keyframes = []
capture = True
if capture:
mp_pose = mp.solutions.pose
pose = mp_pose.Pose()
mp_drawing = mp.solutions.drawing_utils
time.sleep(2)
max_frames = 3
count = 0
cap = cv2.VideoCapture(0) # Use the webcam
while cap.isOpened():
count += 1
if count > max_frames:
break
success, image = cap.read()
if not success:
continue
# Convert the BGR image to RGB, flip the image around y-axis for correct handedness output
image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
# Process the image and draw the pose annotation
results = pose.process(image)
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
visible = False
if results.pose_landmarks:
pos_loc = []
for landmark in results.pose_landmarks.landmark:
if landmark.visibility > 0.5:
visible = True
pos_loc.append([landmark.x, landmark.y, landmark.visibility])
# pos_loc.append({"x": landmark.x, "y": -landmark.y, "z": landmark.z, "visibility": landmark.visibility})
keyframes.append(pos_loc)
mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
print(visible)
cv2.imshow('MediaPipe Pose', image)
if cv2.waitKey(5) & 0xFF == 27: # Press 'ESC' to exit
break
time.sleep(1)
cap.release()
# Example keypoints (these should be replaced with actual data)
keypoints = np.array(keyframes)
for i in range(len(keypoints)-1):
keypoints1 = keypoints[i]
keypoints2 = keypoints[i+1]
keypoints1, keypoints2 = filter_keypoints_by_visibility(keypoints1, keypoints2)
if len(keypoints1) < 10:
print("Not enough keypoints for comparison")
continue
# Normalize keypoints to have zero mean
keypoints1_centered = keypoints1 - np.mean(keypoints1, axis=0)
keypoints2_centered = keypoints2 - np.mean(keypoints2, axis=0)
# Run gradient descent
scale, translation = gradient_descent(keypoints1_centered, keypoints2_centered)
# Apply the optimal scale and translation to keypoints1
transformed_keypoints1 = apply_transformation(keypoints1_centered, scale, translation)
# Calculate similarity score
similarity_score = calculate_similarity(transformed_keypoints1, keypoints2_centered)
# Calculate similarity score
print(f"Optimal scale: {scale}")
print(f"Optimal translation: {translation}")
print(f"Valid keypoints: {len(keypoints1)}")
print(f"Similarity score (average Euclidean distance): {similarity_score}")
```
### aframe/templates/test.html
```html
AFrame Test
```
### aframe/templates/index.html
```html
Pose Estimation VR {
videoEl.play();
video_player.setAttribute('visible', true);
fetch('/video-started', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ message: 'Video has started' }),
})
.then(response => response.json())
.then(data => console.log(data))
.catch((error) => {
console.error('Error:', error);
});
}, 4000);
}
});
videoEl.addEventListener('ended', function () {
setTimeout(() => {
video_player.setAttribute('visible', false);
player_pose.setAttribute('visible', false);
score_text.setAttribute('visible', false);
display_results();
}, 1000);
// el.setAttribute('class', 'clickable');
// el.setAttribute('visible', true);
});
}
});
AFRAME.registerComponent('hover-image-change', {
schema: {
hoverImage: {type: 'string', default: ''},
originalImage: {type: 'string', default: ''}
},
init: function() {
var el = this.el;
var data = this.data;
var laser_count = 0;
el.setAttribute('material', 'src', data.originalImage);
el.addEventListener('raycaster-intersected', function () {
el.setAttribute('material', 'src', data.hoverImage);
laser_count += 1;
});
el.addEventListener('raycaster-intersected-cleared', function () {
laser_count -= 1;
if (laser_count == 0) el.setAttribute('material', 'src', data.originalImage);
});
}
});
AFRAME.registerComponent('start-countdown', {
init: function () {
this.el.addEventListener('click', () => {
let counter = 3;
const countdownElement = document.querySelector('#countdownText');
countdownElement.setAttribute('text', 'value', counter);
const interval = setInterval(() => {
counter--;
if (counter > 0) {
countdownElement.setAttribute('text', 'value', counter);
} else if (counter == 0) {
countdownElement.setAttribute('text', 'value', "Go!");
} else {
clearInterval(interval);
countdownElement.setAttribute('text', 'value', '');
}
}, 1000);
});
}
});
function start_game() {
setTimeout(() => {
var button = document.querySelector('#startButton_entity');
var pose = document.querySelector('#pose');
button.setAttribute('visible', true);
button.setAttribute('class', 'clickable');
pose.setAttribute('visible', true);
}, 500);
}
AFRAME.registerComponent('open-menu-on-click', {
init: function () {
this.el.addEventListener('click', () => {
var logo = document.querySelector('#logo_entity');
var playButton = document.querySelector('#playButton_entity');
logo.emit('fadeOut');
playButton.emit('fadeOut');
playButton.removeAttribute('class');
var soundEl = document.createElement('a-sound');
soundEl.setAttribute('src', '#intro');
[truncated — 6609 more characters]
```
