# Project export: TherapEase.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Expanding access to physical therapy and critical healthcare services through AI-enabled telehealth. - Devpost: https://devpost.com/software/therapease-ai - GitHub: https://github.com/anshgupta1234/TherapEase/ - Video: https://www.youtube.com/embed/Ciso9Ea6yJA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Health Grand Prize (4x Oura Ring)) - Team: 2 GitHub contributor(s) — Ansh Gupta (11 commits), adarshambati1 (4 commits) ## Devpost submission (written by the team) ### Inspiration One of our team members, Aditya, has been in physical therapy (PT) for the last year after a wrist injury on the tennis court. He describes his experience with PT as expensive and inconvenient. Every session meant a long drive across town, followed by an hour of therapy and then the journey back home. On days he was sick or traveling, he would have to miss his PT sessions. Another team member, Adarsh, saw his mom rushed to the hospital after suffering from a third degree heart block. In the aftermath of her surgery, in which she was fitted with a pacemaker, he noticed how her vital signs monitors, which were supposed to aid in her recovery, inhibited her movement and impacted her mental health. These insights together provided us with the inspiration to create TherapEase.ai. TherapEase.ai uses AI-enabled telehealth to bring affordable and effective PT and contactless vital signs monitoring services to consumers, especially among the elderly and disabled communities. With virtual sessions, individuals can receive effective medical care from home with the power of pose correction technology and built-in heart rate, respiratory, and Sp02 monitoring. This evolution of telehealth flips the traditional narrative of physical development—the trainee can be in more control of their body positioning, granting them greater levels of autonomy. ### What it does The application consists of the following features: Pose Detection and Similarity Tracking Contactless Vital Signs Monitoring Live Video Feed with Trainer Live Assistant Trainer Chatbot Once a PT Trainer or Medical Assistant creates a specific training room, the user is free to join said room. Immediately, the user’s body positioning will be highlighted and compared to that of the trainer. This way the user can directly mimic the actions of the trainer and use visual stimuli to better correct their position. Once the trainer and the trainee are aligned, the body position highlights will turn blue, indicating the correct orientation has been achieved. The application also includes a live assistant trainer chatbot to provide useful tips for the user, especially when the user would like to exercise without the presence of the trainer. Finally, on the side of the video call, the user can monitor their major vital signs: heart rate, respiratory rate, and blood oxygen levels without the need for any physical sensors or wearable devices. All three are estimated using remote Photoplethysmography: a technique in which fluctuations in camera color levels are used to predict physiological markers. ### How we built it We began first with heart rate detection. The remote Photoplethysmography (rPPG) technique at a high level works by analyzing the amount of green light that gets absorbed by the face of the trainee. This serves as a useful proxy as when the heart is expanded, there is less red blood in the face, which means there is less green light absorption. The opposite is true when the heart is contracted. By magnifying these fluctuations using Eulerian Video Magnification, we can then isolate the heart rate by applying a Fast Fourier Transform on the green signal. Once the heart rate detection software was developed, we integrated in PoseNet’s position estimation algorithm, which draws 17 key points on the trainee in the video feed. This lead to the development of two-way video communication using webRTC, which simulates the interaction between the trainer and the trainee. With the trainer’s and the trainee’s poses both being estimated, we built the weighted distance similarity comparison function of our application, which shows clearly when the user matched the position of the trainer. At this stage, we then incorporated the final details of the application: the LLM assistant trainer and the additional vital signs detection algorithms. We integrated Intel’s Prediction Guard, into our chat bot to increase speed and robustness of the LLM. For respiratory rate and blood oxygen levels, we integrated algorithms that built off of rPPG technology to determine these two metrics. ### Challenges we ran into (and solved!) We are particularly proud of being able to implement the two-way video communication that underlies the interaction between a patient and specialist on TherapEase.ai. There were many challenges associated with establishing this communication. We spent many hours building an understanding of webRTC, web sockets, and HTTP protocol. Our biggest ally in this process was the developer tools of Chrome, which we could use to analyze network traffic and ensure the right information is being sent. We are also proud of the cosine similarity algorithm which we use to compare the body pose of a specialist/trainer with that of a patient. A big challenge associated with this was finding a way to prioritize certain points (from posnet) over others (e.g. an elbow joint should be given more importance than an eye point in determining how off two poses are from each other). After hours of mathematical and programming iteration, we devised an algorithm that was able to weight certain joints more than others leading to much more accurate results when comparing poses on the two way video stream. Another challenge was finding a way to efficiently compute and compare two pose vectors in real time (since we are dealing with a live video stream). Rather than having a data store, for this hackathon we compute our cosine similarity in the browser. ### What's next We all are very excited about the development of this application. In terms of future technical developments, we believe that the following next steps would take our application to the next level. Peak Enhancement for Respiratory Rate and SpO2 Blood Pressure Contactless Detection Multi-channel video Calling Increasing Security ## README (from the GitHub repository) ## Inspiration One of our team members, Aditya, has been in physical therapy (PT) for the last year after a back injury on the tennis court. He describes his experience with PT as expensive and inconvenient. Every session meant a long drive across town, followed by an hour of therapy and then the journey back home. On days he was sick or traveling, he would have to miss his PT sessions. Another team member, Adarsh, saw his mom rushed to the hospital after suffering from a third degree heart block. In the aftermath of her surgery, in which she was fitted with a pacemaker, he noticed how her vital signs monitors, which were supposed to aid in her recovery, inhibited her movement and impacted her mental health. These insights together provided us with the inspiration to create TherapEase.ai. TherapEase.ai uses AI-enabled telehealth to bring **affordable and effective PT** and **contactless vital signs monitoring services** to consumers. With virtual sessions, individuals can receive effective medical care from home with the power of pose correction technology and built-in heart rate, respiratory, and Sp02 monitoring. This evolution of telehealth flips the traditional narrative of physical development—the trainee can be in more control of their body positioning, granting them greater levels of autonomy. ## What it does The application consists of the following features: Pose Detection and Similarity Tracking Contactless Vital Signs Monitoring Live Video Feed with Trainer Live Assistant Trainer Chatbot Once a PT Trainer or Medical Assistant creates a specific training room, the user is free to join said room. Immediately, the user’s body positioning will be highlighted and compared to that of the trainer. This way the user can directly mimic the actions of the trainer and use visual stimuli to better correct their position. Once the trainer and the trainee are aligned, the body position highlights will turn blue, indicating the correct orientation has been achieved. The application also includes a live assistant trainer chatbot to provide useful tips for the user, especially when the user would like to exercise without the presence of the trainer. Finally, on the side of the video call, the user can monitor their major vital signs: heart rate, respiratory rate, and blood oxygen levels without the need for any physical sensors or wearable devices. All three are estimated using remote Photoplethysmography: a technique in which fluctuations in camera color levels are used to predict physiological markers. ## How we built it We began first with heart rate detection. The remote Photoplethysmography (rPPG) technique at a high level works by analyzing the amount of green light that gets absorbed by the face of the trainee. This serves as a useful proxy as when the heart is expanded, there is less red blood in the face, which means there is less green light absorption. The opposite is true when the heart is contracted. By magnifying these fluctuations using Eulerian Video Magnification, we can then isolate the heart rate by applying a Fast Fourier Transform on the green signal. Once the heart rate detection software was developed, we integrated in PoseNet’s position estimation algorithm, which draws 17 key points on the trainee in the video feed. This lead to the development of two-way video communication using webRTC, which simulates the interaction between the trainer and the trainee. With the trainer’s and the trainee’s poses both being estimated, we built the weighted distance similarity comparison function of our application, which shows clearly when the user matched the position of the trainer. At this stage, we then incorporated the final details of the application: the LLM assistant trainer and the additional vital signs detection algorithms. We integrated **Intel’s Prediction Guard**, into our chat bot to increase speed and robustness of the LLM. For respiratory rate and blood oxygen levels, we integrated algorithms that built off of rPPG technology to determine these two metrics. ## Challenges we ran into (and solved!) We are particularly proud of being able to implement the two-way video communication that underlies the interaction between a patient and specialist on TherapEase.ai. There were many challenges associated with establishing this communication. We spent many hours building an understanding of webRTC, web sockets, and HTTP protocol. Our biggest ally in this process was the developer tools of Chrome, which we could use to analyze network traffic and ensure the right information is being sent. We are also proud of the cosine similarity algorithm which we use to compare the body pose of a specialist/trainer with that of a patient. A big challenge associated with this was finding a way to prioritize certain points (from posnet) over others (e.g. an elbow joint should be given more importance than an eye point in determining how off two poses are from each other). After hours of mathematical and programming iteration, we devised an algorithm that was able to weight certain joints more than others leading to much more accurate results when comparing poses on the two way video stream. Another challenge was finding a way to efficiently compute and compare two pose vectors in real time (since we are dealing with a live video stream). Rather than having a data store, for this hackathon we compute our cosine similarity in the browser. ## What's next for TherapEase.ai We all are very excited about the development of this application. In terms of future technical developments, we believe that the following next steps would take our application to the next level. Peak Enhancement for Respiratory Rate and SpO2 Blood Pressure Contactless Detection Multi-channel video Calling Increasing Security ## Detected evidence (automated analysis) Indexed codebase: 43 recognized source files, 2527 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (46 of 46) ``` .gitignore backup.html camera.js chat_script.js heartbeat.js index.html index.js instructor.html main.js package.json public/assets/css/bootstrap-datetimepicker.min.css public/assets/css/bootstrap.min.css public/assets/css/dataTables.bootstrap4.min.css public/assets/css/font-awesome.min.css public/assets/css/fullcalendar.min.css public/assets/css/mycss.css public/assets/css/select2.min.css public/assets/css/style.css public/assets/css/tagsinput.css public/assets/js/app.js public/assets/js/bootstrap-datetimepicker.min.js public/assets/js/bootstrap.min.js public/assets/js/Chart.bundle.js public/assets/js/chart.js public/assets/js/dataTables.bootstrap4.min.js public/assets/js/fullcalendar.min.js public/assets/js/html5shiv.min.js public/assets/js/jquery-3.2.1.min.js public/assets/js/jquery.dataTables.min.js public/assets/js/jquery.fullcalendar.js public/assets/js/jquery.slimscroll.js public/assets/js/moment.min.js public/assets/js/popper.min.js public/assets/js/respond.min.js public/assets/js/select2.min.js public/assets/js/tagsinput.js public/assets/plugins/light-gallery/css/lightgallery.min.css public/assets/plugins/light-gallery/js/lightgallery-all.min.js public/haarcascade_frontalface_alt.xml README.md servers/chat_script_middleware.js servers/sender/sender.html servers/sender/sender.js servers/server.cjs servers/style.css style.css ``` ### Dependencies - package.json: @stream-io/video-client@^0.5.10, @tensorflow-models/posenet@^2.2.2, @tensorflow/tfjs@^4.17.0, body-parser@^1.20.2, cors@^2.8.5, express@^4.18.2, node-fetch@^3.3.2, posenet-similarity@^0.4.8, vite@^5.1.3, websocket@^1.0.34 ### Recent commits (newest first) - documents: - Create README.md - similarity update - posenet-similarity - yolo dood - ip address change - Adding servers - Final - Adding spO2-basic - Adding RR-basic - Adding Chatbox - metrics - new website - aditya is cool - your mother - npm yuh ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "monkey", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview" }, "devDependencies": { "vite": "^5.1.3" }, "dependencies": { "@stream-io/video-client": "^0.5.10", "@tensorflow-models/posenet": "^2.2.2", "@tensorflow/tfjs": "^4.17.0", "body-parser": "^1.20.2", "cors": "^2.8.5", "express": "^4.18.2", "node-fetch": "^3.3.2", "posenet-similarity": "^0.4.8", "websocket": "^1.0.34" } } ``` ### main.js ```javascript import './style.css' import javascriptLogo from './javascript.svg' import viteLogo from '/vite.svg' import { setupCounter } from './counter.js' document.querySelector('#app').innerHTML = `

Hello Vite!

Click on the Vite logo to learn more

` setupCounter(document.querySelector('#counter')) ``` ### index.js ```javascript import { min } from "@tensorflow/tfjs"; import { Heartbeat } from "./heartbeat"; const OPENCV_URI = "https://docs.opencv.org/master/opencv.js"; const HAARCASCADE_URI = "haarcascade_frontalface_alt.xml" const RESCAN_INTERVAL = 1000; const DEFAULT_FPS = 30; const LOW_BPM = 42; const HIGH_BPM = 240; const REL_MIN_FACE_SIZE = 0.4; const SEC_PER_MIN = 60; const MSEC_PER_SEC = 1000; const MAX_CORNERS = 10; const MIN_CORNERS = 5; const QUALITY_LEVEL = 0.01; const MIN_DISTANCE = 10; let startTime = new Date().getTime(); // Update Duration setInterval(function() { // Get today's date and time var now = new Date().getTime(); // Find the distance between now and the count down date var distance = startTime - now; var minutes = Math.abs(Math.floor((distance % (1000 * 60 * 60)) / (1000 * 60))) - 1; var seconds = Math.abs(Math.floor((distance % (1000 * 60)) / 1000)); minutes = String(minutes).padStart(2, '0'); seconds = String(seconds).padStart(2, '0'); // Display the result in the element with id="duration" document.getElementById("duration").innerHTML = minutes + ":" + seconds; }, 1000); // Load opencv when needed async function loadOpenCv(uri) { return new Promise(function(resolve, reject) { console.log("starting to load opencv"); var tag = document.createElement('script'); tag.src = uri; tag.async = true; tag.type = 'text/javascript' tag.onload = () => { cv['onRuntimeInitialized'] = () => { console.log("opencv ready"); resolve(); } }; tag.onerror = () => { throw new URIError("opencv didn't load correctly."); }; var firstScriptTag = document.getElementsByTagName('script')[0]; firstScriptTag.parentNode.insertBefore(tag, firstScriptTag); }); } let demo = new Heartbeat("webcam", "canvas", HAARCASCADE_URI, 30, 6, 250); var ready = loadOpenCv(OPENCV_URI); ready.then(function() { demo.init(); }); const webSocket = new WebSocket("ws://10.27.195.70:3000") webSocket.onmessage = (event) => { handleSignallingData(JSON.parse(event.data)) } function handleSignallingData(data) { switch (data.type) { case "offer": peerConn.setRemoteDescription(data.offer) createAndSendAnswer() break case "candidate": peerConn.addIceCandidate(data.candidate) } } function createAndSendAnswer () { peerConn.createAnswer((answer) => { peerConn.setLocalDescription(answer) sendData({ type: "send_answer", answer: answer }) }, error => { console.log(error) }) } function sendData(data) { data.username = username webSocket.send(JSON.stringify(data)) } let localStream let peerConn let username username = "110"; navigator.getUserMedia({ video: { frameRate: 24, width: { min: 480, ideal: 720, max: 1280 }, aspectRatio: 1.33333 }, audio: true }, (stream) => { localStream = stream document.getElementById("webcam").srcObject = localStream let configuration = { iceServers: [ { "urls": ["stun:stun.l.google.com:19302", "stun:stun1.l.google.com:19302", "stun:stun2.l.google.com:19302"] } ] } peerConn = new RTCPeerConnection(configuration) peerConn.addStream(localStream) peerConn.onaddstream = (e) => { document.getElementById("remote-video") .srcObject = e.stream } peerConn.onicecandidate = ((e) => { if (e.candidate == null) return sendData({ type: "send_candidate", candidate: e.candidate }) }) sendData({ type: "join_call" }) }, (error) => { console.log(error) }) ``` ### public/assets/js/app.js ```javascript /* Author : Dreamguys ������ȫ��õ�Bootstrapģ�壺http://www.bootstrapmb.com Version : 1.0 */ $(document).ready(function($) { // Variables declarations var $wrapper = $('.main-wrapper'); var $pageWrapper = $('.page-wrapper'); var $slimScrolls = $('.slimscroll'); var $sidebarOverlay = $('.sidebar-overlay'); // Sidebar var Sidemenu = function() { this.$menuItem = $('#sidebar-menu a'); }; function init() { var $this = Sidemenu; $('#sidebar-menu a').on('click', function(e) { if($(this).parent().hasClass('submenu')) { e.preventDefault(); } if(!$(this).hasClass('subdrop')) { $('ul', $(this).parents('ul:first')).slideUp(350); $('a', $(this).parents('ul:first')).removeClass('subdrop'); $(this).next('ul').slideDown(350); $(this).addClass('subdrop'); } else if($(this).hasClass('subdrop')) { $(this).removeClass('subdrop'); $(this).next('ul').slideUp(350); } }); $('#sidebar-menu ul li.submenu a.active').parents('li:last').children('a:first').addClass('active').trigger('click'); } // Sidebar Initiate init(); // Sidebar overlay function sidebar_overlay($target) { if($target.length) { $target.toggleClass('opened'); $sidebarOverlay.toggleClass('opened'); $('html').toggleClass('menu-opened'); $sidebarOverlay.attr('data-reff', '#' + $target[0].id); } } // Mobile menu sidebar overlay $(document).on('click', '#mobile_btn', function() { var $target = $($(this).attr('href')); sidebar_overlay($target); $wrapper.toggleClass('slide-nav'); $('#chat_sidebar').removeClass('opened'); return false; }); // Chat sidebar overlay $(document).on('click', '#task_chat', function() { var $target = $($(this).attr('href')); console.log($target); sidebar_overlay($target); return false; }); // Sidebar overlay reset $sidebarOverlay.on('click', function() { var $target = $($(this).attr('data-reff')); if($target.length) { $target.removeClass('opened'); $('html').removeClass('menu-opened'); $(this).removeClass('opened'); $wrapper.removeClass('slide-nav'); } return false; }); // Select 2 if($('.select').length > 0) { $('.select').select2({ minimumResultsForSearch: -1, width: '100%' }); } // Floating Label if($('.floating').length > 0) { $('.floating').on('focus blur', function(e) { $(this).parents('.form-focus').toggleClass('focused', (e.type === 'focus' || this.value.length > 0)); }).trigger('blur'); } // Right Sidebar Scroll if($('#msg_list').length > 0) { $('#msg_list').slimscroll({ height: '100%', color: '#878787', disableFadeOut: true, borderRadius: 0, size: '4px', alwaysVisible: false, touchScrollStep: 100 }); var msgHeight = $(window).height() - 124; $('#msg_list').height(msgHeight); $('.msg-sidebar .slimScrollDiv').height(msgHeight); $(window).resize(function() { var msgrHeight = $(window).height() - 124; $('#msg_list').height(msgrHeight); $('.msg-sidebar .slimScrollDiv').height(msgrHeight); }); } // Left Sidebar Scroll if($slimScrolls.length > 0) { $slimScrolls.slimScroll({ height: 'auto', width: '100%', position: 'right', size: '7px', color: '#ccc', wheelStep: 10, touchScrollStep: 100 }); var wHeight = $(window).height() - 60; $slimScrolls.height(wHeight); $('.sidebar .slimScrollDiv').height(wHeight); $(window).resize(function() { var rHeight = $(window).height() - 60; $slimScrolls.height(rHeight); $('.sidebar .slimScrollDiv').height(rHeight); }); } // Page wrapper height var pHeight = $(window).height(); $pageWrapper.css('min-height', pHeight); $(window).resize(function() { var prHeight = $(window).height(); $pageWrapper.css('min-height', prHeight); }); // Datetimepicker if($('.datetimepicker').length > 0) { $('.datetimepicker').datetimepicker({ format: 'DD/MM/YYYY' }); } // Datatable if($('.datatable').length > 0) { $('.datatable').DataTable({ "bFilter": false, }); } // Bootstrap Tooltip if($('[data-toggle="tooltip"]').length > 0) { $('[data-toggle="tooltip"]').tooltip(); } // Mobile Menu $(document).on('click', '#open_msg_box', function() { $wrapper.toggleClass('open-msg-box'); return false; }); // Lightgallery if($('#lightgallery').length > 0) { $('#lightgallery').lightGallery({ thumbnail: true, selector: 'a' }); } // Incoming call popup if($('#incoming_call').length > 0) { $('#incoming_call').modal('show'); } // Summernote if($('.summernote').length > 0) { $('.summernote').summernote({ height: 200, minHeight: null, maxHeight: null, focus: false }); } // Check all email $(document).on('click', '#check_all', function() { $('.checkmail').click(); return false; }); if($('.checkmail').length > 0) { $('.checkmail').each(function() { $(this).on('click', function() { if($(this).closest('tr').hasClass('checked')) { $(this).closest('tr').removeClass('checked'); } else { $(this).closest('tr').addClass('checked'); } }); }); } // Mail important $(document).on('click', '.mail-important', function() { $(this).find('i.fa').toggleClass('fa-star').toggleClass('fa-star-o'); }); // Dropfiles if($('#drop-zone').length > 0) { var dropZone = document.getElementById('drop-zone'); var uploadForm = document.getElementById('js-upload-form'); var startUpload = function(files) { console.log(files); }; uploadForm.addEventListener('submit', function(e) { var uploadFiles = document.getElementById('js-upload-files').files; e.preventDefault(); startUpload(uploadFiles); }); dropZone.ondrop = function(e) { e.preventDefault(); this.className = 'upload-drop-zone'; startUpload(e.dataTransfer.files); }; dropZone.ondragover = function() { this.className = 'upload-drop-zone drop'; return false; }; dropZone.ondragleave = function() { this.className = 'upload-drop-zone'; return false; }; } [truncated — 766 more characters] ``` ### instructor.html ```html Heartbeat

Heartbeat demo


This yeelo runs a simple variant of rPPG directly in your browser to measure your heart rate based on subtle changes in skin color.

For best results, try in a constantly well lit space with minimal device and subject motion.


``` ### style.css ```css #remote-container { position: relative; width: 640px; height: 480px; } #remote_canvas { position: absolute; top: 0; left: 0; width: 640px; height: 480px; } canvas { aspect-ratio: auto 640 / 480; } body { font-family: Arial, sans-serif; } .chat-container { max-width: 400px; margin: 20px auto; padding: 20px; border: 1px solid #ccc; border-radius: 5px; box-shadow: 0 2px 5px rgba(0, 0, 0, 0.1); } .chat-box { height: 300px; overflow-y: scroll; margin-bottom: 10px; padding: 10px; border: 1px solid #ddd; border-radius: 5px; } #user-input { width: calc(100% - 70px); padding: 8px; margin-right: 10px; border: 1px solid #ddd; border-radius: 5px; } #send-btn { padding: 8px 15px; background-color: #007bff; color: #fff; border: none; border-radius: 5px; cursor: pointer; } #send-btn:hover { background-color: #0056b3; } ``` ### backup.html ```html Heartbeat

Heartbeat demo


This demo runs a simple variant of rPPG directly in your browser to measure your heart rate based on subtle changes in skin color.

For best results, try in a constantly well lit space with minimal device and subject motion.


``` ### chat_script.js ```javascript document.addEventListener("DOMContentLoaded", function() { const chatBox = document.getElementById("my-chatbox"); const userInput = document.getElementById("user-input"); const sendBtn = document.getElementById("send-btn"); sendBtn.addEventListener("click", function() { const message = userInput.value; if (message.trim() === "") return; // Display user message in chat box appendMessage("You", message); // Send message to server sendMessageToServer(message) .then(response => { // Display server response in chat box appendMessage("Chatbot", truncateMessage(response)); }) .catch(error => { console.error("Error:", error); }); // Clear user input userInput.value = ""; }); function appendMessage(sender, message) { const messageElement = document.createElement("div"); var side = sender === "Chatbot" ? "chat-left" : "chat-right" const timestamp = new Date().toLocaleTimeString(); messageElement.innerHTML = `
John Doe
${sender} ${timestamp}

${message}

`; chatBox.appendChild(messageElement); chatBox.scrollTop = chatBox.scrollHeight; } async function sendMessageToServer(message) { const response = await fetch("http://10.27.195.70:3001/completions", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt: message, model: "Neural-Chat-7B", }) }); console.log(response) const data = await response.json(); return data.choices[0].text; } function truncateMessage(message) { const lastPeriodIndex = message.lastIndexOf('.'); if (lastPeriodIndex !== -1) { return message.substring(0, lastPeriodIndex + 1); // Include the period } else { return message; // If there's no period, return the original message } } }); ``` ### camera.js ```javascript import '@tensorflow/tfjs' import * as posenet from '@tensorflow-models/posenet' import { poseSimilarity } from 'posenet-similarity'; async function runPoseNet() { // Load PoseNet model const net = await posenet.load(); // Get webcam feed const video = document.getElementById('webcam'); const video2 = document.getElementById('remote-video'); const canvas = document.getElementById('local-canvas'); const remote_canvas = document.getElementById('remote-canvas'); if (navigator.mediaDevices.getUserMedia) { const stream = await navigator.mediaDevices.getUserMedia({ video: true }); video.srcObject = stream; } // Wait for the video to load metadata and play video.onloadedmetadata = function() { video.play(); detectPoseInRealTime(video, net, canvas, false, "positionTrainee"); }; video2.onloadedmetadata = function() { detectPoseInRealTime(video2, net, remote_canvas, true, "positionTrainer"); }; } let positionData = {}; // Function to detect poses in real-time and draw them on the canvas async function detectPoseInRealTime(video, net, canvas, clear, view) { const ctx = canvas.getContext('2d'); // Set canvas size to match video feed canvas.width = video.videoWidth; canvas.height = video.videoHeight; let lastPose = null // Main detection loop async function poseDetectionFrame() { const pose = await net.estimateSinglePose(video, { flipHorizontal: false, imageScaleFactor: 0.4 }); lastPose = pose // Clear canvas ctx.clearRect(0, 0, canvas.width, canvas.height); // Draw detected pose drawPose(pose, ctx); // Call poseDetectionFrame again requestAnimationFrame(poseDetectionFrame); } // Start pose detection loop poseDetectionFrame(); // Set an interval to log the pose every 2 seconds setInterval(() => positionData[view] = lastPose, 200); } // Function to draw the detected pose on the canvas function drawPose(pose, ctx) { let flattened1 = []; let flattened2 = []; let similarity = 0; if (positionData.positionTrainee != null && positionData.positionTrainer != null) { similarity = poseSimilarity(positionData.positionTrainee, positionData.positionTrainer, { strategy: 'weightedDistance', customWeight: { mode: 'multiply', scores: [1, 0, 0, 1, 1, 3, 3, 4, 4, 4, 4, 3, 3, 4, 4, 3, 3] } }); similarity = Math.abs(similarity); console.log(similarity); } // Draw keypoints pose.keypoints.forEach(keypoint => { if (keypoint.score > 0.5 && similarity < 0.08) { ctx.beginPath(); ctx.arc(keypoint.position.x, keypoint.position.y, 8, 0, 2 * Math.PI); ctx.fillStyle = 'blue'; ctx.fill(); } else if (keypoint.score > 0.5) { ctx.beginPath(); ctx.arc(keypoint.position.x, keypoint.position.y, 8, 0, 2 * Math.PI); ctx.fillStyle = 'yellow'; ctx.fill(); } }); // Draw skeleton // let adjacentKeyPoints = posenet.getAdjacentKeyPoints(pose.keypoints, 0.5); // if (similarity < 0.90) { // adjacentKeyPoints.forEach(connection => { // drawSegment(ctx, connection[0].position, connection[1].position, 'yellow'); // }) // } else { // adjacentKeyPoints.forEach(connection => { // drawSegment(ctx, connection[0].position, connection[1].position, 'blue'); // }) // } } function cosineSimilarity(vecA, vecB) { let dotProduct = 0; let normA = 0; let normB = 0; for (let i = 0; i < vecA.length; i++) { dotProduct += vecA[i] * vecB[i]; normA += vecA[i] ** 2; normB += vecB[i] ** 2; } if (normA === 0 || normB === 0) { return 0; } return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB)); } // call through fetch function flattenKeypoints(keypoints) { let acc = [] for (let i = 0; i < keypoints.length; i++) { acc.push(keypoints[i].position.x, keypoints[i].position.y) } return acc; } // Function to draw a line segment between two points function drawSegment(ctx, a, b, color) { ctx.beginPath(); ctx.moveTo(a.x, a.y); ctx.lineTo(b.x, b.y); ctx.strokeStyle = color; ctx.lineWidth = 2; ctx.stroke(); } // Call the function to start PoseNet runPoseNet(); ``` ### heartbeat.js ```javascript const RESCAN_INTERVAL = 1000; const DEFAULT_FPS = 30; const LOW_BPM = 42; const HIGH_BPM = 240; const REL_MIN_FACE_SIZE = 0.4; const SEC_PER_MIN = 60; const MSEC_PER_SEC = 1000; const MAX_CORNERS = 10; const MIN_CORNERS = 5; const QUALITY_LEVEL = 0.01; const MIN_DISTANCE = 10; // Simple rPPG implementation in JavaScript // - Code could be improved given better documentation available for opencv.js export class Heartbeat { constructor(webcamId, canvasId, classifierPath, targetFps, windowSize, rppgInterval) { this.webcamId = webcamId; this.canvasId = canvasId, this.classifierPath = classifierPath; this.streaming = false; this.faceValid = false; this.targetFps = targetFps; this.windowSize = windowSize; this.rppgInterval = rppgInterval; this.display = document.getElementById("heartrate"); this.display2 = document.getElementById("respiratoryrate"); this.display3 = document.getElementById("spO2"); } // Start the video stream async startStreaming() { try { this.stream = await navigator.mediaDevices.getUserMedia({ video: { facingMode: 'user', width: {exact: this.webcamVideoElement.width}, height: {exact: this.webcamVideoElement.height} }, audio: false }); } catch (e) { //console.log(e); } if (!this.stream) { throw new Error('Could not obtain video from webcam.'); } // Set srcObject to the obtained stream this.webcamVideoElement.srcObject = this.stream; // Start the webcam video stream this.webcamVideoElement.play(); this.streaming = true; return new Promise(resolve => { // Add event listener to make sure the webcam has been fully initialized. this.webcamVideoElement.oncanplay = () => { resolve(); }; }); } // Create file from url async createFileFromUrl(path, url) { let request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; request.send(); return new Promise(resolve => { request.onload = () => { if (request.readyState === 4) { if (request.status === 200) { let data = new Uint8Array(request.response); cv.FS_createDataFile('/', path, data, true, false, false); resolve(); } else { console.log('Failed to load ' + url + ' status: ' + request.status); } } }; }); } // Initialise the demo async init() { this.webcamVideoElement = document.getElementById(this.webcamId); try { await this.startStreaming(); this.webcamVideoElement.width = this.webcamVideoElement.videoWidth; this.webcamVideoElement.height = this.webcamVideoElement.videoHeight; this.frameRGB = new cv.Mat(this.webcamVideoElement.height, this.webcamVideoElement.width, cv.CV_8UC4); this.lastFrameGray = new cv.Mat(this.webcamVideoElement.height, this.webcamVideoElement.width, cv.CV_8UC1); this.frameGray = new cv.Mat(this.webcamVideoElement.height, this.webcamVideoElement.width, cv.CV_8UC1); this.overlayMask = new cv.Mat(this.webcamVideoElement.height, this.webcamVideoElement.width, cv.CV_8UC1); this.cap = new cv.VideoCapture(this.webcamVideoElement); // Set variables this.signal = []; // 120 x 3 raw rgb values this.timestamps = []; // 120 x 1 timestamps this.rescan = []; // 120 x 1 rescan bool this.face = new cv.Rect(); // Position of the face // Load face detector this.classifier = new cv.CascadeClassifier(); let faceCascadeFile = "haarcascade_frontalface_alt.xml"; if (!this.classifier.load(faceCascadeFile)) { await this.createFileFromUrl(faceCascadeFile, this.classifierPath); this.classifier.load(faceCascadeFile) } this.scanTimer = setInterval(this.processFrame.bind(this), MSEC_PER_SEC/this.targetFps); this.rppgTimer = setInterval(this.rppg.bind(this), this.rppgInterval); } catch (e) { console.log(e); } } // Add one frame to raw signal processFrame() { try { if (!this.frameGray.empty()) { this.frameGray.copyTo(this.lastFrameGray); // Save last frame } this.cap.read(this.frameRGB); // Save current frame let time = Date.now() let rescanFlag = false; cv.cvtColor(this.frameRGB, this.frameGray, cv.COLOR_RGBA2GRAY); // Need to find the face if (!this.faceValid) { this.lastScanTime = time; this.detectFace(this.frameGray); } // Scheduled face rescan else if (time - this.lastScanTime >= RESCAN_INTERVAL) { this.lastScanTime = time this.detectFace(this.frameGray); rescanFlag = true; } // Track face else { // Disable for now, //this.trackFace(this.lastFrameGray, this.frameGray); } // Update the signal if (this.faceValid) { // Shift signal buffer while (this.signal.length > this.targetFps * this.windowSize) { this.signal.shift(); this.timestamps.shift(); this.rescan.shift(); } // Get mask let mask = new cv.Mat(); mask = this.makeMask(this.frameGray, this.face); // New values let means = cv.mean(this.frameRGB, mask); mask.delete(); // Add new values to raw signal buffer this.signal.push(means.slice(0, 3)); this.timestamps.push(time); this.rescan.push(rescanFlag); } // Draw face cv.rectangle(this.frameRGB, new cv.Point(this.face.x, this.face.y), new cv.Point(this.face.x+this.face.width, this.face.y+this.face.height), [0, 255, 0, 255]); // Apply overlayMask this.frameRGB.setTo([255, 0, 0, 255], this.overlayMask); cv.imshow(this.canvasId, this.frameRGB); } catch (e) { // console.log("Error capturing frame:"); // console.log(e); } } [truncated — 14315 more characters] ``` [22 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]