# Project export: CareSentry: Fall Detection/Alert API This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Real-time fall detection API: Analyzes hospital security camera footage, instantly alerts nearest staff via pager/SMS when a patient falls. - Devpost: https://devpost.com/software/caresentry-fall-detection-alert-api - GitHub: https://github.com/leuneri/CareSentry-Fall-Detection-Alert-API - Video: https://www.youtube.com/embed/lxDbjzwF7sI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — MlsBli (13 commits), Gurik Mangat (8 commits), leuneri (8 commits) ## Devpost submission (written by the team) ### Overview Problem Space Our team member has a personal connection to senior care in hospitals. Her loved one was sent to the hospital for an internal organ issue, but his stay was extended due to a fall injury. Witnessing how devastating a simple fall could be, prolonging hospital stay by months, was eye-opening and served as the inspiration behind this project. In building this hack, our goal was to reduce the impact of fall injuries on senior health. Every year in the US, between 700,000 and 1,000,000 hospital patients experience falls, resulting in approximately 250,000 injuries and up to 11,000 deaths. Shockingly, one in four falls leads to injury, with 10% causing serious harm such as fractures, lacerations, or internal bleeding. This greatly extrapolates to senior citizens who are already fragile. These incidents not only compromise patient well-being but also incur significant additional costs and prolonged hospital stays, diverting resources from addressing patients' primary medical concerns. Compounding the issue, limited hospital personnel creates struggle to effectively monitor each patient continuously. Consequently, patients may remain unattended after a fall, exacerbating their injuries. Clearly, there's a pressing need for innovative solutions to enhance senior patient safety and optimize hospital resources. Existing Solution/Competition Camera/Sensor with Fall Detection Costs $50 The average hospital has 2,000 - 3,000 cameras One time cost of $150,000 for replacements + installation fee Alarm Systems (i.e. Pressure mats, infrared movement detectors, cord-activated alarms, wearable devices) Disruptive, especially to patients with cognitive impairments, potentially exacerbating confusion and agitation Impose limitations on mobility and independence - in US nursing homes, alarms are classified as a form of restraint Sitters According to HospitalView, the average number of total staffed beds for hospitals is 130 beds To hire 130 sitters and pay them hourly is going to be very costly Our solution Fall detection API for $15,000 annual subscription for our API service (revenue from one patient covers our API service for one year) Analyzes security camera footages at hospitals, provides instant alerts to the nearest hospital staff (through a pager or SMS) when a patient falls Impact Efficient Response: Hospital workers can attend to patients faster, minimizing response time Enhanced Patient Safety: Early detection and prompt assistance prevent exacerbation of existing injuries and reduce the risk of further injuries Hospital Efficiency: Optimizes resource allocation and streamlines patient care processes Market The U.S. hospital facilities market size was valued at USD 1,411.7 billion in 2022 and is expected to grow at a compound annual growth rate (CAGR) of 7.7% from 2023 to 2030. According to American Hospital Association, there are 6,120 hospitals in the US → Total Available Market According to the U.S. Department of Health & Human Services, ~20% of the hospitals are critically understaffed → Serviceable Available Market Business Model API licensing sold at $15,000 per year Revenue = 15000 x 6120(0.20) = $1.8 million per year Development Process The dev process comprised of three main parts: 1. Model Training, 2. Model Deployment, and 3. User Experience. For model training, our approach was to develop a data preprocessing pipeline which prepared data before sending it to our training function to train our deep learning model. The preprocessing was crucial as it separates incoming video into batches of 60 frames, resizes each non-null image to be consistent with each other, normalizes its pixel values to a range between 0 and 1, adds it to a list, and then converts this list into a NumPy array. We also implemented data augmentation to quadruple the amount of video we had to train the model. This processed data was used to train our sequential architecture based convolutional LSTM model for the binary classification of "fall detected" and "no fall detected". For improved accuracy and reliability, we employed time-distributed convolutional layers to extract spatial features from each frame, and an LSTM layer to analyze temporal relationships between frames. Furthermore, we were able to overcome a problem of initial overfitting by introducing dropout layers in our model construction, as well as shuffling the indices of the video frames to ensure the model is trained on a randomized dataset, which helped improve generalization. Deploying the model was necessary to implement it into a usable product interface for future customers and healthcare practitioners. After training, the model and its weights were saved as a ".keras" file allowing the model to be exported. Once a connection to a live video feed was established, we would invoke the pre-processing pipeline for the incoming data and then feed it to the ML model for detection of falls. Finally, the user experience portion covered how our users would interact with and use our model. We interfaced with our laptop camera to simulate security cameras and enable a live video stream that got fed into our pre-processing and detection pipeline. When a fall was detected by our pre-trained model, an automatic SMS message using Sinch gets sent to nurse and doctor pagers/phones alerting them that a patient has fallen along with their fall location. Next Steps 1. Enhanced Detection Detect other behaviors besides falling, such as: Bed exit Restless in bed Room & toilet exit 2. Scaling and Expansion Expand our audience to assisted living facilities such as: Nursing/retirement homes Rehab centers ## README (from the GitHub repository) Type `py api_client.py` into the terminal to run the program :) Beware! It takes data from your camera! And make sure not to hurt yourself falling, have fun! https://youtu.be/lxDbjzwF7sI?si=eYElnzT3ymtK4qDq # Problem Space Our team member has a personal connection to senior care in hospitals. Her loved one was sent to the hospital for a simple injury diagnosis, but his stay was extended due to a fall injury. Witnessing how devastating a simple fall could be, prolonging hospital stay by months, was eye-opening and served as the inspiration behind this project. In building this hack, our goal was to reduce the impact of fall injuries on senior health. Every year in the US, between 700,000 and 1,000,000 hospital patients experience falls, resulting in approximately 250,000 injuries and up to 11,000 deaths. Shockingly, one in four falls leads to injury, with 10% causing serious harm such as fractures, lacerations, or internal bleeding. This greatly extrapolates to senior citizens who are already fragile. These incidents not only compromise patient well-being but also incur significant additional costs and prolonged hospital stays, diverting resources from addressing patients' primary medical concerns. Compounding the issue, limited hospital personnel creates struggle to effectively monitor each patient continuously. Consequently, patients may remain unattended after a fall, exacerbating their injuries. Clearly, there's a pressing need for innovative solutions to enhance senior patient safety and optimize hospital resources. # Existing Solution/Competition ## 1. Camera/Sensor with Fall Detection - Costs $50 - The average hospital has 2,000 - 3,000 cameras - One time cost of $150,000 for replacements + installation fee ## 2. Alarm Systems (i.e. Pressure mats, infrared movement detectors, cord-activated alarms, wearable devices) - Alarms are disruptive and may be especially disturbing to cognitively impaired patients, contributing to confusion and agitation. They also restrict mobility and independence; in US nursing homes, alarms are considered a type of restraint. ## 3. Sitters - According to HospitalView, the average number of total staffed beds for hospitals is 130 beds - To hire 130 sitters and pay them hourly is going to be very costly ## Our Solution - Fall detection API for $15,000 annual subscription for our API service (revenue from one patient covers our API service for one year) - Analyzes security camera footages at hospitals, provides instant alerts to the nearest hospital staff (through a pager or SMS) when a patient falls ## Impact - Efficient Response: Hospital workers can attend to patients faster, minimizing response time - Enhanced Patient Safety: Early detection and prompt assistance prevent exacerbation of existing injuries and reduce the risk of further injuries - Hospital Efficiency: Optimizes resource allocation and streamlines patient care processes # Market The U.S. hospital facilities market size was valued at USD 1,411.7 billion in 2022 and is expected to grow at a compound annual growth rate (CAGR) of 7.7% from 2023 to 2030. According to American Hospital Association, there are 6,120 hospitals in the US → Total Available Market According to the U.S. Department of Health & Human Services, ~20% of the hospitals are critically understaffed → Serviceable Available Market # Business Model API licensing sold at $15,000 per year Revenue = 15000 x 6120(0.20) = $1.8 million per year # Development Process The dev process comprised of three main parts: 1. Model Training, 2. Model Deployment, and 3. User Experience. For model training, our approach was to develop a data preprocessing pipeline which prepared data before sending it to our training function to train our deep learning model. The preprocessing was crucial as it separates incoming video into batches of 60 frames, resizes each non-null image to be consistent with each other, normalizes its pixel values to a range between 0 and 1, adds it to a list, and then converts this list into a NumPy array. We also implemented data augmentation to quadruple the amount of video we had to train the model. This processed data was used to train our sequential architecture based convolutional LSTM model for the binary classification of "fall detected" and "no fall detected". For improved accuracy and reliability, we employed time-distributed convolutional layers to extract spatial features from each frame, and an LSTM layer to analyze temporal relationships between frames. Furthermore, we were able to overcome a problem of initial overfitting by introducing dropout layers in our model construction, as well as shuffling the indices of the video frames to ensure the model is trained on a randomized dataset, which helped improve generalization. Deploying the model was necessary to implement it into a usable product interface for future customers and healthcare practitioners. After training, the model and its weights were saved as a ".keras" file allowing the model to be exported. Once a connection to a live video feed was established, we would invoke the pre-processing pipeline for the incoming data and then feed it to the ML model for detection of falls. Finally, the user experience portion covered how our users would interact with and use our model. We interfaced with our laptop camera to simulate security cameras and enable a live video stream that got fed into our pre-processing and detection pipeline. When a fall was detected by our pre-trained model, an automatic SMS message using Sinch gets sent to nurse and doctor pagers/phones alerting them that a patient has fallen along with their fall location. # Next Steps ## 1. Enhanced Detection Detect other behaviors besides falling, such as: - Bed exit - Restless in bed - Room & toilet exit ## 2. Scaling and Expansion Expand our audience to assisted living facilities such as: - Nursing/retirement homes - Rehab centers ## Detected evidence (automated analysis) Indexed codebase: 8 recognized source files, 16 KB. - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - Python (language) — detected in the code - TensorFlow (technology) — detected in the code ## Codebase structure (from repository index) ### Files (11 of 11) ``` api_client.py app_later/api_client.py app_later/mock_api.py app_later/predict.py Fall Detection.ipynb main.py model.keras README.md requirements.txt send_sms.py test_api.py ``` ### Dependencies - requirements.txt: fastapi, flask, numpy@==1.21.2, opencv-python, pip@install fastapi[all], requests, tensorflow ### Recent commits (newest first) - Update README.md - Create README.md - ok - added sms result - added send_sms - Update Fall Detection.ipynb - Added training data - Merge branch 'main' of https://github.com/leuneri/fall_detection - New Model - Merge branch 'main' of https://github.com/leuneri/fall_detection - script for interaction with camera - Merge branch 'main' of https://github.com/leuneri/fall_detection - new model - added sms - added sms - twilio - Merge remote-tracking branch 'origin/main' - new uplaod - deleted redundant frames - updated model ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` opencv-python numpy==1.21.2 requests tensorflow flask fastapi pip install fastapi[all] ``` ### main.py ```python from fastapi import FastAPI, File, UploadFile, HTTPException from keras.models import load_model from fastapi.responses import JSONResponse from typing import List import numpy as np import cv2 import logging import tempfile import shutil import os app = FastAPI() logging.basicConfig(level=logging.INFO) # Load your model as before model = load_model("model.keras") def load_and_preprocess_images_from_files(files: List[UploadFile]): with tempfile.TemporaryDirectory() as temp_dir: images = [] for file in files: # Save temporary file temp_file = os.path.join(temp_dir, file.filename) with open(temp_file, 'wb') as f: shutil.copyfileobj(file.file, f) # Process the image image = cv2.imread(temp_file) if image is None: logging.warning(f"Failed to load image: {temp_file}") continue image = cv2.resize(image, (128, 96)) image = image / 255.0 images.append(image) if len(images) != 60: raise ValueError("Expected 60 images, found {}".format(len(images))) images_array = np.array(images) images_array = np.expand_dims(images_array, axis=0) return images_array @app.post("/predict/") async def predict_from_folder(files: List[UploadFile] = File(...)): try: # Preprocess and load images images_array = await load_and_preprocess_images_from_files(files) # Ensure the input shape is correct if images_array.ndim == 3: images_array = np.expand_dims(images_array, axis=0) # Predict prediction = model.predict(images_array) # Process prediction result = (prediction > 0.5).astype('int').tolist() return JSONResponse(content={"result": result}) except Exception as e: logging.error(f"Error during prediction: {e}") return HTTPException(status_code=500, detail=f"Prediction error: {e}") ``` ### send_sms.py ```python from sinch import Client sinch_client = Client( key_id="db1aa87a-bd62-4bb8-8ba5-56d37b5bf4ee", key_secret="a.EJur9KQ55mD7E.Lw6iQxSeFn", project_id="5fb58ea0-10d4-4227-b71d-c0bbc32986b4" ) def send_message(): send_batch_response = sinch_client.sms.batches.send( body="Fall detection alert: Room 207 at Bed B! \n\n - Automated message sent from CareSentry API", to=["13022526216"], from_="12085810360", delivery_report="none" ) send_message() ``` ### test_api.py ```python import requests import os # The URL of the API endpoint url = 'http://127.0.0.1:8000/predict/' # Path to the folder containing your images folder_path = 'vidframes1' # Collecting files to send files = [('files', (filename, open(os.path.join(folder_path, filename), 'rb'), 'image/jpeg')) for filename in os.listdir(folder_path) if filename.lower().endswith(('.png', '.jpg', '.jpeg'))] # Making the request response = requests.post(url, files=files) # Assuming your endpoint returns JSON with a 'result' key if response.status_code == 200: print("Success:", response.json()) else: print("Error:", response.status_code, response.text) ``` ### api_client.py ```python import numpy as np from fastapi import FastAPI, HTTPException from keras.models import load_model import cv2 import os import logging from pathlib import Path from typing import List from send_sms import send_message # import shutil send_every_n_frames = 60 # Adjust based on your needs # Load your pre-trained model (adjust the path to your saved model correctly) current_directory = Path.cwd() model_path = current_directory / "model.keras" # Use pathlib for OS-agnostic path handling try: model = load_model(model_path) except Exception as e: logging.error(f"Failed to load model: {e}") folder_path = current_directory / "vidframes1" def clear_folder(): # Loop through all items in the folder for item_name in os.listdir(folder_path): # Create the full path to the item item_path = os.path.join(folder_path, item_name) # Deletes item in path if os.path.isfile(item_path): os.remove(item_path) def send_frame_to_folder(frames): frame_number = 10 # Encode frame as JPEG for frame in frames: _, encoded_image = cv2.imencode('.jpg', frame) # image_bytes = encoded_image.tobytes() filename = f"frame_{frame_number}.jpg" image_path = os.path.join(folder_path, filename) frame_number += 1 # Write the encoded image to a file with open(image_path, 'wb') as file: file.write(encoded_image) def load_and_preprocess_images(folder_path): images = [] # Use .iterdir() to iterate over the contents of the folder for path in folder_path.iterdir(): if path.is_file(): # Ensure it's a file image_path = str(path) # Convert Path object to string image = cv2.imread(image_path) if image is None: logging.warning(f"Failed to load image: {image_path}") continue image = cv2.resize(image, (128, 96)) image = image / 255.0 # Normalize pixel values images.append(image) if len(images) != send_every_n_frames: raise ValueError("Expected 60 images, found {}".format(len(images))) images_array = np.array(images) images_array = np.expand_dims(images_array, axis=0) # Add batch dimension return images_array def predict_from_folder(): try: # Load and preprocess images images_array = load_and_preprocess_images(folder_path) # if images_array.ndim == 5: # Ensure the input shape is correct # images_array = np.expand_dims(images_array, axis=0) # Predict prediction = model.predict(images_array) # Assume the model outputs a sigmoid activation and use 0.5 as the threshold for binary classification result = (prediction > 0.5).astype('int').tolist() # send sms send_message() return {"result": result} except Exception as e: logging.error(f"Error during prediction: {e}") raise HTTPException(status_code=500, detail=f"Prediction error: {e}") # Initialize the camera cap = cv2.VideoCapture(0) # Check if the webcam is opened correctly if not cap.isOpened(): raise IOError("Cannot open webcam") frame_count = 0 frames = [] try: while True: ret, frame = cap.read() # # Check if frame is captured # if not ret: # break # Display the captured frame cv2.imshow('Camera Output', frame) frames.append(frame) # Increment the frame count frame_count += 1 # Send every Nth frame to the API if frame_count % send_every_n_frames == 0: clear_folder() send_frame_to_folder(frames) load_and_preprocess_images(folder_path) result = predict_from_folder() frame_count = 0 if result["result"][0][0] == 1: print("this actually worked") break # continue #TODO: add logic here to imform nurse # Break the loop with 'q' if cv2.waitKey(1) & 0xFF == ord('q'): break finally: # Release the camera and close all windows cap.release() cv2.destroyAllWindows() ``` ### app_later/mock_api.py ```python from flask import Flask, request, jsonify app = Flask(__name__) @app.route('/api', methods=['POST']) def detect_fall(): # Simulate fall detection processing here # For now, we just return a mock response return jsonify({"fall_detected": True, "message": "Fall detected in the received frame"}) if __name__ == '__main__': app.run(debug=True, port=5000) ``` ### app_later/predict.py ```python import tensorflow import os import keras from keras.models import load_model import cv2 import numpy as np # Load the model model = load_model('model.keras') def predict(video_path): processed_images = [] for img in sorted(os.listdir(video_path)): img = cv2.imread(os.path.join(video_path, img)) if img is not None: img = cv2.resize(img, (128, 96)) img = img / 255 processed_images.append(img) processed_images = np.array(processed_images) # Convert the list to a numpy array predictions = model.predict(processed_images) # Process predictions to determine if a fall is detected threshold = 0.5 predicted_class = "Fall" if predictions >= threshold else "No Fall" confidence_score = predictions if predicted_class == "Positive" else 1 - predictions print(f"Predicted Class: {predicted_class} with confidence {confidence_score[0]:.2f}") predict("C:/Users/M/OneDrive - softromic/Documents/GitHub/fall_detection/vidframes - 1") ``` ### app_later/api_client.py ```python import cv2 import requests import numpy as np def send_frame_to_api(frame): """ Send a frame to an external API for processing. The frame is encoded as a JPEG before being sent. """ # Encode frame as JPEG _, encoded_image = cv2.imencode('.jpg', frame) image_bytes = encoded_image.tobytes() # Specify your API endpoint api_endpoint = "http://your-api-endpoint.com/api" # Prepare headers for the HTTP request headers = {'Content-Type': 'image/jpeg'} # Send the frame to the API using a POST request response = requests.post(api_endpoint, data=image_bytes, headers=headers) ###TODO: consider how to send result if response: return True # Add true response here for how to handle # offer a response with location included (hard-coded for now) # Initialize the camera cap = cv2.VideoCapture(0) # Check if the webcam is opened correctly if not cap.isOpened(): raise IOError("Cannot open webcam") frame_count = 0 send_every_n_frames = 30 # Adjust based on your needs try: while True: ret, frame = cap.read() # Check if frame is captured if not ret: break # Display the captured frame cv2.imshow('Camera Output', frame) # Increment the frame count frame_count += 1 # Send every Nth frame to the API if frame_count % send_every_n_frames == 0: send_frame_to_api(frame) # Break the loop with 'q' if cv2.waitKey(1) & 0xFF == ord('q'): break finally: # Release the camera and close all windows cap.release() cv2.destroyAllWindows() ```