# Project export: MedInfoBot

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Say goodbye to prescription confusion. MedInfoBot is your trusted AI prescription companion, making healthcare accessible and medication understanding simple.
- Devpost: https://devpost.com/software/medinfobot
- GitHub: https://github.com/AndrewLive/calhacks2023
- Team: 2 GitHub contributor(s) — AndrewLive (6 commits), Nancy Lau (5 commits)

## Devpost submission (written by the team)

### Overview

Are you tired of the confusion and uncertainty that often accompanies medication prescriptions? Introducing MedInfoBot, your trusted companion on your healthcare journey. We understand that understanding medication details can be challenging, and that's why we've designed MedInfoBot to simplify medication understanding and make healthcare more accessible than ever before.

### What it does

With MedInfoBot, you can effortlessly extract important information from your prescriptions, including medication names, dosages, and what it does. Say goodbye to the complexity of medical terms. Our user-friendly interface provides clear and concise explanations of common uses of your medicine and potential side effects, along with audio descriptions, for those visually impaired, empowering anyone to make informed decisions about your own health. Key Features: Prescription Information Processing MedInfoBot employs Optical Character Recognition (OCR) to extract prescription details from images, ensuring accurate and reliable data retrieval. User-Friendly Interface The application boasts an intuitive and user-friendly interface, making it accessible to users of all backgrounds and ages. AI-Powered Explanations MedInfoBot utilizes the OpenAI API to generate clear and concise explanations of medication details, enhancing your understanding of what you're taking. Hardware Integration Powered by the Milk-V Duo board and the CAM-GC2083 camera module, MedInfoBot leverages efficient hardware capabilities for accurate image capture and processing. Expandable Features The Milk-V Duo board's versatility allows for the seamless integration of future additional features, such as human emotion detection and prediction with the camera module. Technologies Hardware Components MedInfoBot operates on a low cost, low power consumption, high performance $5 RISC-V computer, ensuring that our solution is not only accessible but also incredibly efficient. Milk-V Duo Board This ultra-compact embedded development platform is powered by the CV1800B chip, capable of running Linux and RTOS. It provides a reliable, low-cost, and high-performance foundation for MedInfoBot. The board features dual processors, versatile GPIO pins, USB support, and more, making it a perfect match for this healthcare application. CAM-GC2083 Camera Module The GLAXYCORE's GC2083 CMOS Image Sensor, is a camera module thatoffers up to 2MP resolution, ensuring high-quality imaging. It seamlessly integrates with the 16P MIPI CSI interface on the Milk-V Duo board, enabling advanced imaging capabilities. With features like optical excellence, high sensitivity, and impressive dynamic range, this camera module enhances the MedInfoBot's image capture capabilities significantly. Software Components This bot makes liberal use of various machine learning/AI models to perform its desired functions. PyTesseract PyTesseract is a Python wrapper for Google's Tesseract Optical Character Recognition application. This allows the bot to take an image as an input, and output the recognizable text that appears in the image. OpenAI/ChatGPT OpenAI's ChatGPT LLM was used to process the text output from PyTesseract and extract the medication name from the text. It was also then used to return useful information about the specified medication. GTTS GTTS (Google Text To Speech) is a python wrapper for Google's Text to Speech functions. It was used to generate an audio file of the medication name and description outputted by the previous software component. Together, these hardware components and software technologies power MedInfoBot to simplify medication understanding and promote healthcare accessibility.

### What's next

As we move forward, MedInfoBot aims to expand its capabilities and impact in the healthcare space. Here's a glimpse of what's on the horizon: Multi-Language Support Accessibility to healthcare information for a global audience with support for multiple languages. Integration with Hume AI Exploring emotion detection capabilities to provide personalized support alongside medication information. Voice Assistance Making MedInfoBot even more accessible with voice command features for medication inquiries. User Profiles Personalized profiles to track medication history, receive reminders, and get tailored recommendations. Medical Database Integration Currently, ChatGPT is used to search for information about the given medication. In the future, this can be upgraded to searching in a pre-compiled/pre-assembled medical database in order to speed up bot functions and improve medication accuracy. MedInfoBot's journey doesn't end here! We are your gateway to simplified medication understanding and informed healthcare decisions. Join us in making healthcare more accessible and understandable for everyone.

## README (from the GitHub repository)

# calhacks2023
Calhacks 2023 Project


## Detected evidence (automated analysis)

Indexed codebase: 4 recognized source files, 4 KB.
- Python (language) — detected in the code
- OpenAI (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (5 of 5)

```
.gitignore
capture_image.sh
prescription.py
README.md
run_prescription.sh
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- output formatting
- got rid of chatgpt headers in text
- create html to display results and show audio
- captures image and output
- revert prescription.py changes
- updated code for rtsp
- capture from rtsp stream from camera
- adjustments to chatgpt prompt
- python script to analyze image of prescription and use ChatGPT to return info about it
- ignores env files
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### capture_image.sh

```shell
#!/bin/bash

RTSP_URL="rtsp://192.168.42.1:8554/stream0"
OUTPUT_DIR="images"

mkdir -p "$OUTPUT_DIR"

IMAGE_NUM=1
while [ -e "$OUTPUT_DIR/captured_image$IMAGE_NUM.jpg" ]; do
  ((IMAGE_NUM++))
done

ffmpeg -i "$RTSP_URL" -vframes 1 "$OUTPUT_DIR/captured_image$IMAGE_NUM.jpg"

echo "Image captured and saved as captured_image$IMAGE_NUM.jpg"

```

### run_prescription.sh

```shell
#!/bin/bash

./capture_image.sh

latest_image=$(ls -t images/captured_image*.jpg | head -n 1)

output=$(./prescription.py "$latest_image")

cat <<EOF > prescription_output.html
<!DOCTYPE html>
<html>
<head>
    <title>Prescription Output</title>
    <style>
        body {
            font-family: Arial, sans-serif;
            background-color: #f5f5f5;
            margin: 0;
            padding: 0;
        }
        h1 {
            background-color: #007BFF;
            color: #fff;
            padding: 20px;
            text-align: center;
        }
        .container {
            max-width: 800px;
            margin: 20px auto;
            padding: 20px;
            background-color: #fff;
            box-shadow: 0 0 10px rgba(0, 0, 0, 0.2);
        }
        img {
            max-width: 100%;
            height: auto;
            display: block;
            margin: 0 auto;
        }
    </style>
</head>
<body>
    <h1>Prescription Output</h1>
    <div class="container">
        <p>
            $output
        </p>

        <audio controls>
            <source src="prescription_info.mp3" type="audio/mpeg">
            Your browser does not support the audio element.
        </audio>

        <img src="$latest_image" alt="Prescription Image">
        
    </div>
</body>
</html>
EOF

# Open the generated HTML file in the default web browser on macOS
open prescription_output.html

```

### prescription.py

```python
from dotenv import load_dotenv
import openai
import os
from PIL import Image
import pytesseract
import sys
from gtts import gTTS
import subprocess

if __name__ == '__main__':
    '''
    Usage: ./prescription.py <prescription_image>

    This program uses OCR to read the text on a prescription label.
    Afterwards, it uses ChatGPT to extract the medication name, dosage, and form,
    and states what the medication is commonly used for and its effects.
    '''
    load_dotenv()
    openai.api_key = os.getenv('APIKEY')

    # ./prescription <img file>
    if (len(sys.argv) != 2):
        exit(1)

    content = '''
            Please extract ONLY the medication name, dosage, form, and instructions from the following text.
            Your response should consist ONLY of the extracted medication name, dosage, and form, and nothing else.
            If there is no medication name present, please state 'NO MEDICATION DETECTED':


            '''
    
    content += pytesseract.image_to_string(Image.open(sys.argv[1]))

    
    messages = [ {"role": "system", "content": content} ]

    chat = openai.ChatCompletion.create( 
            model="gpt-3.5-turbo", messages=messages 
            ) 
    
    medication_name = chat.choices[0].message.content
    print(f'{medication_name}')

    content = f'''
            Please describe the following medication:
            {medication_name}

            Please be short and concise with your explanation.
            Please describe common uses and side effects of the medication as well.
            Also, DO NOT, under any circumstance, restate the medication name, dosage, or form.
            '''
    
    messages.append({"role": "system", "content": content})
    chat = openai.ChatCompletion.create( 
            model="gpt-3.5-turbo", messages=messages 
            ) 
    medication_info = chat.choices[0].message.content
    print(f'\n{medication_info}')
    
    outfile = open('./prescription_info.txt', 'w')
    outfile.write(f'{medication_name}\n\n\n{medication_info}')
    outfile.close()

    tts = gTTS(text = f'{medication_name}\n\n\n{medication_info}', lang = 'en', slow = False)
    tts_path = f'./prescription_info.mp3'
    tts.save(tts_path)
```