# Project export: care.ai

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: For go-to assistant for everything in primary healthcare
- Devpost: https://devpost.com/software/medbot-u3hzt8
- GitHub: https://github.com/joyendra/TreeHacks2024
- Video: https://www.youtube.com/embed/NcpG03cECDI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Best Use of Monster Generative AI APIs (4x XBox Series S [1st] & 1 million Monster API credits [2nd] & $400 Cash [3rd]))
- Team: 2 GitHub contributor(s) — s-kachroo (5 commits), Joyendra Roy Biswas (4 commits)

## Devpost submission (written by the team)

### Inspiration

Care.ai was inspired by our self-conducted study involving 60 families and 23 smart devices, focusing on elderly healthcare. Over three months, despite various technologies, families preferred the simplicity of voice-activated assistants like Alexa. This preference led us to develop an intuitive, user-friendly AI healthcare chatbot tailored to everyday needs.

### What it does

Care.ai, an AI healthcare chatbot, leverages custom-trained Large Language Models (LLMs) and visual recognition technology hosted on the Intel Cloud for robust processing power. These models, refined and accessible via Hugging Face, underwent further fine-tuning through MonsterAPI, enhancing their accuracy and responsiveness to medical queries. The web application, powered by the Reflex library, provides a seamless and intuitive front-end experience, making it easy for users to interact with and benefit from the chatbot's capabilities. Care.ai supports real-time data analytics and critical care necessary for humans.

### How we built it

We built our AI healthcare chatbot by training LLMs and visual recognition systems on the Intel Cloud, then hosting and fine-tuning these models on Hugging Face with MonsterAPI. The chatbot's user-friendly web interface was developed using the Reflex library, creating a seamless user interaction platform. For data collection, We researched datasets and performed literature review We used the pre-training data for developing and fine-tuning our LLM and visual models We collect live data readings using sensors to test against our trained models We categorized our project into three parts: Interactive Language Models: We developed deep learning models on Intel Developer Cloud and fine-tuned our Hugging Face hosted models using MonsterAPI. We further used Reflex Library to be the face of Care.ai and create a seamless platform. Embedded Sensor Networks: Developed our IoT sensors to track the real-time data and test our LLVMs on the captured data readings. Compliance and Security Components: Intel Developer Cloud to extract emotions and de-identify patient's voice to be HIPAA

### Challenges we ran into

Integrating new technologies posed significant challenges, including optimizing model performance on the Intel Cloud, ensuring seamless model fine-tuning via MonsterAPI and achieving intuitive user interaction through the Reflex library. Balancing technical complexity with user-friendliness and maintaining data privacy and security were among the key hurdles we navigated.

### Accomplishments we're proud of

We're proud of creating a user-centric AI healthcare chatbot that combines advanced LLMs and visual recognition hosted on the cutting-edge Intel Cloud. Successfully fine-tuning these models on Hugging Face and integrating them with a Reflex-powered interface showcases our technical achievement. Our commitment to privacy, security, and intuitive design has set a new standard in accessible home healthcare solutions.

### What we learned

We learned the importance of integrating advanced AI with user-friendly interfaces for healthcare. Balancing technical innovation with accessibility, the intricacies of cloud hosting, model fine-tuning, and ensuring data privacy were key lessons in developing an effective, secure, and intuitive AI healthcare chatbot.

### What's next

Next, Care.ai is expanding its disease recognition capabilities, enhancing user interaction with natural language processing improvements, and exploring partnerships for broader deployment in healthcare systems to revolutionize home healthcare access and efficiency.

## README (from the GitHub repository)

# TreeHacks2024
Stanford TreeHacks 2024 project repo


## Detected evidence (automated analysis)

Indexed codebase: 13 recognized source files, 18 KB.
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- AWS (technology) — claimed on Devpost, not found in the code
- C++ (language) — claimed on Devpost, not found in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- Firebase (technology) — claimed on Devpost, not found in the code
- Java (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (23 of 23)

```
.DS_Store
.gitignore
BLIP2.ipynb
data_cleaner.ipynb
HIPAA_Voice (1).ipynb
README.md
reflex/.DS_Store
reflex/webui/.DS_Store
reflex/webui/.gitignore
reflex/webui/requirements.txt
reflex/webui/rxconfig.py
reflex/webui/webui/__init__.py
reflex/webui/webui/components/__init__.py
reflex/webui/webui/components/chat.py
reflex/webui/webui/components/loading_icon.py
reflex/webui/webui/components/modal.py
reflex/webui/webui/components/navbar.py
reflex/webui/webui/components/sidebar.py
reflex/webui/webui/state.py
reflex/webui/webui/styles.py
reflex/webui/webui/webui.py
router.py
train.csv
```

### Dependencies

- reflex/webui/requirements.txt: openai@==0.28, reflex@>=0.2.0

### Recent commits (newest first)

- Added files
- Merge pull request #1 from joyendra/reflex-monster
- monster code changes
- Merge remote-tracking branch 'origin/main'
- navbars deleted and other minor modifications
- Added gitignore
- Merge branch 'main' of https://github.com/joyendra/TreeHacks2024
- Added files
- GPT keys added and model changed
- reflex init commit
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### reflex/webui/requirements.txt

```
reflex>=0.2.0
openai==0.28

```

### router.py

```python
from fastapi import FastAPI, UploadFile, File
from pydantic import BaseModel
import torch
from transformers import AutoProcessor, Blip2ForConditionalGeneration

# Load your model
processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
device = "cuda" if torch.cuda.is_available() else "cpu"
model = Blip2ForConditionalGeneration.from_pretrained("ybelkada/blip2-opt-2.7b-fp16-sharded", device_map="auto", load_in_8bit=True)
model.load_state_dict(torch.load('model.pt'))

app = FastAPI()

class Item(BaseModel):
    question: str

@app.post("/predict/")
async def predict(item: Item, file: UploadFile = File(...)):
    # Read image file
    image = await file.read()
    inputs = processor(image.convert('RGB'), text=item.question, return_tensors="pt").to(device, torch.float16)

    # Make prediction
    generated_ids = model.generate(**inputs, max_new_tokens=10)

    # Post processing
    answer = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()

    return {"answer": answer}
```

### reflex/webui/rxconfig.py

```python
import reflex as rx


config = rx.Config(
    app_name="webui",
)

```

### reflex/webui/webui/webui.py

```python
"""The main Chat app."""

import reflex as rx

from webui import styles
from webui.components import chat, modal, navbar, sidebar
from webui.state import State


def index() -> rx.Component:
    """The main app."""
    return rx.chakra.vstack(
        navbar(),
        chat.chat(),
        chat.action_bar(),
        sidebar(),
        modal(),
        bg=styles.bg_dark_color,
        color=styles.text_light_color,
        min_h="100vh",
        align_items="stretch",
        spacing="0",
    )


# Add state and page to the app.
app = rx.App(style=styles.base_style)
app.add_page(index)

```

### reflex/webui/webui/styles.py

```python
import reflex as rx

bg_dark_color = "#111"
bg_medium_color = "#222"

border_color = "#fff3"

accennt_light = "#6649D8"
accent_color = "#5535d4"
accent_dark = "#4c2db3"

icon_color = "#fff8"

text_light_color = "#fff"
shadow_light = "rgba(17, 12, 46, 0.15) 0px 48px 100px 0px;"
shadow = "rgba(50, 50, 93, 0.25) 0px 50px 100px -20px, rgba(0, 0, 0, 0.3) 0px 30px 60px -30px, rgba(10, 37, 64, 0.35) 0px -2px 6px 0px inset;"

message_style = dict(display="inline-block", p="4", border_radius="xl", max_w="30em")

input_style = dict(
    bg=bg_medium_color,
    border_color=border_color,
    border_width="1px",
    p="4",
)

icon_style = dict(
    font_size="md",
    color=icon_color,
    _hover=dict(color=text_light_color),
    cursor="pointer",
    w="8",
)

sidebar_style = dict(
    border="double 1px transparent;",
    border_radius="10px;",
    background_image=f"linear-gradient({bg_dark_color}, {bg_dark_color}), radial-gradient(circle at top left, {accent_color},{accent_dark});",
    background_origin="border-box;",
    background_clip="padding-box, border-box;",
    p="2",
    _hover=dict(
        background_image=f"linear-gradient({bg_dark_color}, {bg_dark_color}), radial-gradient(circle at top left, {accent_color},{accennt_light});",
    ),
)

base_style = {
    rx.chakra.Avatar: {
        "shadow": shadow,
        "color": text_light_color,
        "bg": border_color,
    },
    rx.chakra.Button: {
        "shadow": shadow,
        "color": text_light_color,
        "_hover": {
            "bg": accent_dark,
        },
    },
    rx.chakra.Menu: {
        "bg": bg_dark_color,
        "border": f"red",
    },
    rx.chakra.MenuList: {
        "bg": bg_dark_color,
        "border": f"1.5px solid {bg_medium_color}",
    },
    rx.chakra.MenuDivider: {
        "border": f"1px solid {bg_medium_color}",
    },
    rx.chakra.MenuItem: {
        "bg": bg_dark_color,
        "color": text_light_color,
    },
    rx.chakra.DrawerContent: {
        "bg": bg_dark_color,
        "color": text_light_color,
        "opacity": "0.9",
    },
    rx.chakra.Hstack: {
        "align_items": "center",
        "justify_content": "space-between",
    },
    rx.chakra.Vstack: {
        "align_items": "stretch",
        "justify_content": "space-between",
    },
}

```

### reflex/webui/webui/state.py

```python
import os
import requests
import json
import openai
import reflex as rx

# openai.api_key = os.getenv("OPENAI_API_KEY")
# openai.api_base = os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1")
openai.api_key = "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"
openai.api_base = "https://api.openai.com/v1"

MONSTER_API_KEY = os.getenv("MONSTER_API_KEY")
MONSTER_SECRET_KEY = os.getenv("MONSTER_SECRET_KEY")

if not openai.api_key and not MONSTER_API_KEY:
    raise Exception("Please set OPENAI_API_KEY or MONSTER_API_KEY")


def get_access_token():
    """
    :return: access_token
    """
    return "d2b97bee-4f34-40c9-a347-c59b4a195488"


class QA(rx.Base):
    """A question and answer pair."""

    question: str
    answer: str


DEFAULT_CHATS = {
    "Interact": [],
}


class State(rx.State):
    """The app state."""

    # A dict from the chat name to the list of questions and answers.
    chats: dict[str, list[QA]] = DEFAULT_CHATS

    # The current chat name.
    current_chat = "Interact"

    # The current question.
    question: str

    # Whether we are processing the question.
    processing: bool = False

    # The name of the new chat.
    new_chat_name: str = ""

    # Whether the drawer is open.
    drawer_open: bool = False

    # Whether the modal is open.
    modal_open: bool = False

    api_type: str = "monster" if MONSTER_API_KEY else "openai"

    def create_chat(self):
        """Create a new chat."""
        # Add the new chat to the list of chats.
        self.current_chat = self.new_chat_name
        self.chats[self.new_chat_name] = []

        # Toggle the modal.
        self.modal_open = False

    def toggle_modal(self):
        """Toggle the new chat modal."""
        self.modal_open = not self.modal_open

    def toggle_drawer(self):
        """Toggle the drawer."""
        self.drawer_open = not self.drawer_open

    def delete_chat(self):
        """Delete the current chat."""
        del self.chats[self.current_chat]
        if len(self.chats) == 0:
            self.chats = DEFAULT_CHATS
        self.current_chat = list(self.chats.keys())[0]
        self.toggle_drawer()

    def set_chat(self, chat_name: str):
        """Set the name of the current chat.

        Args:
            chat_name: The name of the chat.
        """
        self.current_chat = chat_name
        self.toggle_drawer()

    @rx.var
    def chat_titles(self) -> list[str]:
        """Get the list of chat titles.

        Returns:
            The list of chat names.
        """
        return list(self.chats.keys())

    async def process_question(self, form_data: dict[str, str]):
        # Get the question from the form
        question = form_data["question"]

        # Check if the question is empty
        if question == "":
            return

        if self.api_type == "openai":
            model = self.openai_process_question
        else:
            model = self.monster_process_question

        async for value in model(question):
            yield value

    async def openai_process_question(self, question: str):
        """Get the response from the API.

        Args:
            form_data: A dict with the current question.
        """

        # Add the question to the list of questions.
        qa = QA(question=question, answer="")
        self.chats[self.current_chat].append(qa)

        # Clear the input and start the processing.
        self.processing = True
        yield

        # Build the messages.
        messages = [
            {"role": "system", "content": "You are a friendly chatbot named Reflex."}
        ]
        for qa in self.chats[self.current_chat]:
            messages.append({"role": "user", "content": qa.question})
            messages.append({"role": "assistant", "content": qa.answer})

        # Remove the last mock answer.
        messages = messages[:-1]

        # Start a new session to answer the question.
        session = openai.ChatCompletion.create(
            model="gpt-4",
            messages=messages,
            stream=True,
        )

        # Stream the results, yielding after every word.
        for item in session:
            if hasattr(item.choices[0].delta, "content"):
                answer_text = item.choices[0].delta.content
                self.chats[self.current_chat][-1].answer += answer_text
                self.chats = self.chats
                yield

        # Toggle the processing flag.
        self.processing = False

    import requests

    async def monster_process_question(self, question: str):
        """Get the response from the external API.

        Args:
            question: The question to process.
        """
        # Add the question to the list of questions.
        qa = QA(question=question, answer="")
        self.chats[self.current_chat].append(qa)

        self.processing = True
        yield

        payload = {
            "input_variables": {"prompt": question},
            "prompt": question,
            "stream": False,
            "max_tokens": 256,
            "n": 1,
            "best_of": 1,
            "presence_penalty": 0,
            "frequency_penalty": 0,
            "repetition_penalty": 1,
            "temperature": 1,
            "top_p": 1,
            "top_k": -1,
            "min_p": 0,
            "use_beam_search": False,
            "length_penalty": 1,
            "early_stopping": False
        }

        headers = {
            "accept": "application/json",
            "Authorization": "Bearer d2b97bee-4f34-40c9-a347-c59b4a195488",
            "Content-Type": "application/json"
        }

        try:
            response = requests.post(
                "https://5d7d0c35-402b-4225-97fb-0fb8fd9d0b51.monsterapi.ai/generate",
                json=payload,
                headers=headers
            )
            response.raise_for_status()
            answer_data = response.json()

            answer_text = answer_data.get("answers", [])[0]

            self.chats[sel
[truncated — 318 more characters]
```

### reflex/webui/webui/components/__init__.py

```python
from .loading_icon import loading_icon
from .navbar import navbar
from .modal import modal
from .sidebar import sidebar

```

### reflex/webui/webui/components/loading_icon.py

```python
import reflex as rx


class LoadingIcon(rx.Component):
    """A custom loading icon component."""

    library = "react-loading-icons"
    tag = "SpinningCircles"
    stroke: rx.Var[str]
    stroke_opacity: rx.Var[str]
    fill: rx.Var[str]
    fill_opacity: rx.Var[str]
    stroke_width: rx.Var[str]
    speed: rx.Var[str]
    height: rx.Var[str]

    def get_event_triggers(self) -> dict:
        return {"on_change": lambda status: [status]}


loading_icon = LoadingIcon.create

```

### reflex/webui/webui/components/navbar.py

```python
import reflex as rx

from webui import styles
from webui.state import State


def navbar():
    return rx.chakra.box(
        rx.chakra.hstack(
            rx.chakra.hstack(
                rx.chakra.button(
                    "+ New chat",
                    bg=styles.accent_color,
                    px="4",
                    py="2",
                    h="auto",
                    on_click=State.toggle_modal,
                ),
                spacing="8",
            ),
            justify="space-between",
        ),
        bg=styles.bg_dark_color,
        backdrop_filter="auto",
        backdrop_blur="lg",
        p="4",
        border_bottom=f"1px solid {styles.border_color}",
        position="sticky",
        top="0",
        z_index="100",
    )

```

### reflex/webui/webui/components/sidebar.py

```python
import reflex as rx

from webui import styles
from webui.state import State


def sidebar_chat(chat: str) -> rx.Component:
    """A sidebar chat item.

    Args:
        chat: The chat item.
    """
    return rx.chakra.hstack(
        rx.chakra.box(
            chat,
            on_click=lambda: State.set_chat(chat),
            style=styles.sidebar_style,
            color=styles.icon_color,
            flex="1",
        ),
        rx.chakra.box(
            rx.chakra.icon(
                tag="delete",
                style=styles.icon_style,
                on_click=State.delete_chat,
            ),
            style=styles.sidebar_style,
        ),
        color=styles.text_light_color,
        cursor="pointer",
    )


def sidebar() -> rx.Component:
    """The sidebar component."""
    return rx.chakra.drawer(
        rx.chakra.drawer_overlay(
            rx.chakra.drawer_content(
                rx.chakra.drawer_header(
                    rx.chakra.hstack(
                        rx.chakra.text("Chats"),
                        rx.chakra.icon(
                            tag="close",
                            on_click=State.toggle_drawer,
                            style=styles.icon_style,
                        ),
                    )
                ),
                rx.chakra.drawer_body(
                    rx.chakra.vstack(
                        rx.foreach(State.chat_titles, lambda chat: sidebar_chat(chat)),
                        align_items="stretch",
                    )
                ),
            ),
        ),
        placement="left",
        is_open=State.drawer_open,
    )

```

[2 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]