# Project export: MediLive

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Real-time emergency AI nurse aiding senior citizens and first respondents.
- Devpost: https://devpost.com/software/medilive
- GitHub: https://github.com/avimahesh21/MediLive-frontend
- Demo: https://github.com/avimahesh21/MediLive-backend
- Video: https://www.youtube.com/embed/-oBvSqdik_0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — avi22vt (21 commits), Alexander Peal (15 commits), Akshat Chavan (3 commits), Avi Maheshwari (2 commits)

## Devpost submission (written by the team)

### Inspiration

What happens when you can’t be there for your grandparents during a medical crisis? Our team, all of whom have grandparents living in distant senior homes, face a dilemma. Though the declining health and isolation of these family members necessitates external care during emergencies, this need is not easily met due to the high cost and scarcity of full-time caregivers. Additionally, in the event you can’t be with your loved ones in an emergency, first responders may not have personal medical knowledge that could be the difference between life or death. Because of this issue, we saw the need to build a platform that serves as a “caretaker” for senior citizens in times of medical emergencies. This problem has large implications – in the US, 27% of the population over 60 lives by themselves, which accounts for 29 million annual emergency department visits. We sought to build a solution that could provide senior citizens with medical instructions and communicate life-saving information to first respondents on the scene. By doing so, we hope to give families confidence and peace of mind no matter the distance separating them and their loved ones.

### What it does

MediLife is an integrated platform that 1) monitors vital indicators of senior citizen health including head movement, heart rate, and facial positioning 2) releases an AI agent that provides medical directives immediately after detection of abnormal vital signs (ex. cardiac arrest, faints, falls) and 3) notifies first respondents and provides a summarized report of time of emergency, medical history, and current medications. MediLife aims to significantly improve efficiency of medical treatment within the first 20 minutes of a senior citizen emergency.

### How we built it

We built MediLife using a suite of tools and technologies, split up into 3 components: the AI agent, the computer vision model, and the vitals monitoring. These individual components were then combined together cohesively in a front end providing a simple but effective user interface for both senior citizens and first responders. The AI Agent was developed using OpenAI’s GPT-4 model. We implemented OpenAI’s text-to-speech voice and React’s built in speech-to-text to simulate a real conversation with an AI Nurse. We prompt engineered GPT to understand the data it is receiving from our CV and heart rate monitor to allow the nurse to have impactful medical questions and advice. The computer vision model was built using pre-trained TensorFlow.js models that specialize in facial detection and body language. We calculated different ways to interpret danger like feints or seizures using the model’s data. For vitals monitoring, we incorporated hardware into this aspect of our hack to track the heart rate of patients via a HW-827 pulse sensor, an arduino board, and other electrical components. The HW-827 pulse sensor attaches to the fingertip of the senior citizen, shines a green light onto the finger, measures the amount of reflected light with a photosensor, and finally obtains a heart-beat pulse reading. The live data coming from the sensor is then transmitted into arduino code and then through a serial port parser into the program's Node.JS backend. The front-end of MediLife shows a camera facing the senior citizen as well as a live activity feed and vitals monitoring data. All of the components are connected using React.

### Challenges we ran into

One of the primary challenges we ran into was measuring vitals of senior citizens and transmitting that live data into the back end server. Originally, our team wanted to use an Apple Watch to collect heart rate and blood oxygen data; however, this process required using Swift and a MacOS device which we did not have access to. We then tried to pivot to a myZone activity belt tracker in conjunction with Terra API; this required an Android device which we also did not have access to. We were then left with the final solution to incorporate a hardware sensor using a pulse reader which was not extremely accurate in its heart beat readings. In an ideal world, the senior citizen would have a piece of wearable technology that would transfer vital data. Another challenge we faced was the prompt engineering of the AI agent to best address the medical information based on the medical information collected. We found difficulty in getting the AI agent to ask effective questions and provide accurate responses to the senior citizen indicating the best course of action to take. After a lot of fine-tuning and tweaking, we were able to improve the quality of the nurse’s questions, but there’s certainly more room for improvement. Lastly, our team went through many iterations of an intuitive and effective conversation structure. Simulating a real human conversation with pauses, language, and user interactivity was difficult and took a lot of testing. It was also a challenge to feed the model enough relevant data about the medical conversation so that it could have enough knowledge to be helpful.

### Accomplishments we're proud of

We take particular pride in integrating advanced trigger detectors through the use of TensorFlow and pulse sensing technology. The development of an AI avatar that directly communicates with the patient also stands out as a highlight. Most significantly, our team has successfully developed a working MVP that promises to extend its benefits beyond our immediate families, potentially impacting the lives of millions of elderly individuals and their families across the globe.

### What we learned

How to integrate a HW-287 pulse sensor into a full-stack project Prompt engineering scripts to yield an effective AI pilot Developing genuine conversation structures using OpenAI TensorFlow with Javascript to flag unusual head and face movements Web sockets and serial ports to integrate live arduino data

### What's next

Next steps for MediLive possibly include tracking more vital signs through both camera computer vision and body sensors including blood pressure and oxygen, respiratory rate, and body temperature. It will be important to strike a balance between using an abundance of tracking sensors and monitors and not impairing the movement of the senior citizen. We also believe there are opportunities to refine the AI agent in terms of medical advice it can provide to the senior citizens using historical data, medical information, and other analysis of the emergency. A more fleshed out version of our product could integrate with smart home devices like Google Home and Amazon Alexa, as well as be trained on an even more advanced LLM model which could have more functionality like telling a respondent step-by-step how to do a Heimlich Maneuver or CPR.

## README (from the GitHub repository)

# Getting Started with Create React App

This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

## Available Scripts

In the project directory, you can run:

### `npm start`

Runs the app in the development mode.\
Open [http://localhost:3000](http://localhost:3000) to view it in your browser.

The page will reload when you make changes.\
You may also see any lint errors in the console.

### `npm test`

Launches the test runner in the interactive watch mode.\
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.

### `npm run build`

Builds the app for production to the `build` folder.\
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.\
Your app is ready to be deployed!

See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.

### `npm run eject`

**Note: this is a one-way operation. Once you `eject`, you can't go back!**

If you aren't satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.

Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you're on your own.

You don't have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn't feel obligated to use this feature. However we understand that this tool wouldn't be useful if you couldn't customize it when you are ready for it.

## Learn More

You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).

To learn React, check out the [React documentation](https://reactjs.org/).

### Code Splitting

This section has moved here: [https://facebook.github.io/create-react-app/docs/code-splitting](https://facebook.github.io/create-react-app/docs/code-splitting)

### Analyzing the Bundle Size

This section has moved here: [https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size](https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size)

### Making a Progressive Web App

This section has moved here: [https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app](https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app)

### Advanced Configuration

This section has moved here: [https://facebook.github.io/create-react-app/docs/advanced-configuration](https://facebook.github.io/create-react-app/docs/advanced-configuration)

### Deployment

This section has moved here: [https://facebook.github.io/create-react-app/docs/deployment](https://facebook.github.io/create-react-app/docs/deployment)

### `npm run build` fails to minify

This section has moved here: [https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify](https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify)


## Detected evidence (automated analysis)

Indexed codebase: 20 recognized source files, 29 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- Node.js (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
- TensorFlow (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (32 of 32)

```
.gitignore
package.json
public/index.html
public/manifest.json
public/models/face_expression_model-shard1
public/models/face_expression_model-weights_manifest.json
public/models/face_landmark_68_model-shard1
public/models/face_landmark_68_model-weights_manifest.json
public/models/face_recognition_model-shard1
public/models/face_recognition_model-shard2
public/models/face_recognition_model-weights_manifest.json
public/models/tiny_face_detector_model-shard1
public/models/tiny_face_detector_model-weights_manifest.json
README.md
src/ActionLog.js
src/App.css
src/App.js
src/App.test.js
src/ArduinoData.js
src/EventEmitter.js
src/Face.css
src/Face.js
src/Footer.js
src/HeartRateMonitor.js
src/index.css
src/index.js
src/nurse.css
src/Nurse.js
src/reportWebVitals.js
src/setupTests.js
src/WebcamCap.js
tailwind.config.js
```

### Dependencies

- package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, bootstrap@^5.3.2, face-api.js@^0.22.2, react@^18.2.0, react-dom@^18.2.0, react-player@^2.14.1, react-scripts@^5.0.1, react-speech-recognition@^3.10.0, react-webcam@^7.2.0, tailwindcss@^3.4.1, typewriter-effect@^2.21.0, web-vitals@^2.1.4

### Recent commits (newest first)

- ui changes
- Merge branch 'master' of https://github.com/avimahesh21/app-frontend
- ui changes
- finished first front-end makeover
- added logo and tailwind
- Merge branch 'master' of https://github.com/avimahesh21/app-frontend
- bug fixes
- Merge branch 'master' of https://github.com/avimahesh21/app-frontend
- cleaned console logs
- Merge branch 'master' of https://github.com/avimahesh21/app-frontend
- twilio message
- fixed master
- Merge pull request #4 from avimahesh21/avi-avatar
- package got messed up
- Merge branch 'adding-arduino' of https://github.com/avimahesh21/app-frontend into avi-avatar
- ui changes
- Merge branch 'master' of https://github.com/avimahesh21/app-frontend into avi-avatar
- added ArduinoData component
- updates ui
- final changes to face recognition and computer vision

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "app-frontend",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "bootstrap": "^5.3.2",
    "face-api.js": "^0.22.2",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-player": "^2.14.1",
    "react-scripts": "^5.0.1",
    "react-speech-recognition": "^3.10.0",
    "react-webcam": "^7.2.0",
    "typewriter-effect": "^2.21.0",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "tailwindcss": "^3.4.1"
  }
}

```

### src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

```

### src/App.js

```javascript
import logo from './visioncare-logo.png'; //'./crosslogo.png';
import patientProfilePic from './patientProfilePic.jpg';
import './App.css';
import WebcamCap from './WebcamCap';
import Nurse from './Nurse';
import 'bootstrap/dist/css/bootstrap.min.css';
import Typewriter from "typewriter-effect";
import Footer from './Footer';
import ActionLog from './ActionLog';
import React, { useState, useEffect, useRef } from "react";
import SpeechRecognition, {
  useSpeechRecognition,
} from "react-speech-recognition";
import ArduinoData from './ArduinoData';
import Face from './Face'
import HeartRateMonitor from './HeartRateMonitor';

function App() {
  const [isListening, setIsListening] = useState(true);
  const [trigger, setTrigger] = useState(false);
  const [triggerDetails, setTriggerDetails] = useState(null);
  const [alert, setAlert] = useState(null);
  const [speaking, setSpeaking] = useState(false);
  const [audioUrl, setAudioUrl] = useState(null);
  const [voiceText, setVoiceText] = useState(null);
  const [isUserSpeaking, setIsUserSpeaking] = useState(false);
  const silenceTimer = useRef(null);
  let fetched = false;


  useEffect(() => {
    if (trigger) {
      sendMessage();
      simulateAlert("Sent Info Text to Paramedics");
    }
  }, [trigger]);



  const {
    transcript,
    listening,
    resetTranscript,
    browserSupportsSpeechRecognition,
    finalTranscript,
  } = useSpeechRecognition();
  const startListening = () =>
    SpeechRecognition.startListening({ continuous: true });

  //todo
  let patientName = 'John Doe';
  let patientData = '{  "patient_name": "${patientName}",  "daily_medications": [    {      "name": "Atorvastatin",      "dose": "20mg",      "frequency": "once a day"    },    {      "name": "Lisinopril",      "dose": "10mg",      "frequency": "once a day"    }  ],  "current_medical_issues": [    {      "issue": "Hypertension",      "diagnosis_date": "2023-01-15"    },    {      "issue": "High Cholesterol",      "diagnosis_date": "2023-02-20"    }  ],  "at_risk_data": {    "smoking_status": "Former smoker",    "family_history": [      "Heart Disease",      "Diabetes"    ],    "BMI": 28.5  },  "previous_appointment_data": [    {      "date": "2023-03-10",      "reason": "Routine check-up",      "notes": "Blood pressure slightly elevated. Recommended dietary changes."    },    {      "date": "2023-04-22",      "reason": "Follow-up for hypertension",      "notes": "Blood pressure improved. Continue current medication."    }  ]}'

  const simulateAlert = (alert) => {
    const newAlert = {
      id: Date.now(), // Ensure a unique ID for each alert
      message: alert + ` at ${new Date().toLocaleTimeString()}`,
    };
    setAlert(newAlert);
  };

  useEffect(() => {
    if (!browserSupportsSpeechRecognition) {
      console.error("Browser does not support speech recognition.");
      return;
    }

    if (trigger && !fetched) {
      fetched = true;
      let trigDetails = "Fainted"; //todo
      setTriggerDetails(trigDetails);
      fetchFirstQuestion(trigDetails);
      simulateAlert(trigDetails);
    }
  }, [trigger]);

  const patientInfo = JSON.parse(patientData);

  const sendMessage = async () => {


    // Construct the message
    const message = `

    We have an update regarding your patient, ${patientName}. They have unexpectedly fainted and require immediate assistance!
    ---
    Current Medical Issues:
    ${patientInfo.current_medical_issues.map(issue => `- ${issue.issue} - Diagnosed on ${issue.diagnosis_date}`).join('\n')}
    ---
    Daily Medications:
    ${patientInfo.daily_medications.map(med => `- ${med.name} ${med.dose}, ${med.frequency}`).join('\n')}
    ---
    At Risk Data:
    - Smoking Status: ${patientInfo.at_risk_data.smoking_status}
    - Family History: ${patientInfo.at_risk_data.family_history.join(', ')}
    - BMI: ${patientInfo.at_risk_data.BMI}
    ---
    Previous Appointment Data:
    ${patientInfo.previous_appointment_data.map(appointment => `- Date: ${appointment.date}, Reason: ${appointment.reason}, Notes: ${appointment.notes}`).join('\n')}
    
    Please review this information and advise on any further actions.
    
    `;


    try {
      const response = await fetch('http://localhost:3001/send-message', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message }) // Assuming the backend only needs the message text
      });

      if (response.ok) {
        const data = await response.json();
        console.log('Message sent:', data.sid);
        // Handle success - update UI or show a message to the user
      } else {
        // Handle errors - response.ok is false if status code is not in the range 200-299
        const errorData = await response.json();
        console.error('Failed to send message:', errorData.error);
        // Update UI to show the error message
      }
    } catch (error) {
      // Catch network errors or issues with the fetch call itself
      console.error('Network error or issue with fetch:', error);
      // Update UI to show the network error message
    }
  };


  const fetchFirstQuestion = async (triggerDetails) => {
    if (!speaking) {
      try {
        const response = await fetch('http://localhost:3001/firstQuestion', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
          },
          body: JSON.stringify({ triggerDetails: triggerDetails, patientData: patientData }),
        });
        if (!response.ok) throw new Error('Network response was not ok');
        const data = await response.json();
        setVoiceText(data.question);
        createAudioURL(data.buffer);
        setSpeaking(true);
        resetTranscript();
        simulateAlert("Contacted Medical Help");
      } catch (error) {
        console.error('Failed to fetch question:', error);
      }
    }
  };

  async function createAudioURL(audioBuffer) {
    const blob = new Blob(
[truncated — 4842 more characters]
```

### tailwind.config.js

```javascript
/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
    "./src/*.{js,jsx,ts,tsx}"
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}


```

### src/Face.css

```css


.myapp {
    position: relative;
    width: 708px;
    height: 490px;
  }
  
  video, canvas {
    position: absolute;
    top: 0;
    left: 0;
  }
  
  video {
    width: 100%;
    height: 100%;
  }
  
  canvas {
    z-index: 10;
  }
  
```

### src/setupTests.js

```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';

```

### src/App.test.js

```javascript
import { render, screen } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
  render(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

```

### src/reportWebVitals.js

```javascript
const reportWebVitals = onPerfEntry => {
  if (onPerfEntry && onPerfEntry instanceof Function) {
    import('web-vitals').then(({ getCLS, getFID, getFCP, getLCP, getTTFB }) => {
      getCLS(onPerfEntry);
      getFID(onPerfEntry);
      getFCP(onPerfEntry);
      getLCP(onPerfEntry);
      getTTFB(onPerfEntry);
    });
  }
};
//test commit

export default reportWebVitals;

```

### src/Footer.js

```javascript
import React from 'react';
import HeartRateMonitor from "./HeartRateMonitor";

const Footer = () => {
    return (
        <footer className="row pb-3"> 
            <div className="col-md-3 text-center">
                <HeartRateMonitor />
            </div>
            
            {/*<div className="col-md-6">
            </div>*/}
            
        </footer>
    );
}

export default Footer;

```

### src/index.css

```css
@tailwind base;
@tailwind components;
@tailwind utilities;

body {
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
    'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
    sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

code {
  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
    monospace;
}

```

[10 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]