# Project export: Instruct IQ

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Our project aids elders facing technical hurdles by using image capture and processing to get the next steps. Leveraging Language Models (LLM), we give users with a sequence of troubleshooting steps.
- Devpost: https://devpost.com/software/senior-savvy-solutions
- GitHub: https://github.com/megha19998/TreeHacks2024Frontend
- Demo: https://github.com/codlocker/Treehacks-InferenceAPI
- Video: https://www.youtube.com/embed/vbRP5-bYU2k?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Ipsit Sahoo (2 commits), Megha Sharma (2 commits)

## Devpost submission (written by the team)

### Overview

The inspiration My inspiration stems from my parents, who struggle with understanding various features of Facebook. Explaining the icons or where to click or type is challenging due to her educational background. Our annotated images aim to provide direct guidance to help individuals like her navigate technology with ease. This motivation drives us to build a product that empowers seniors worldwide, ensuring they don't feel disappointed or left behind. Idea Video: https://www.videoscribe.co/app/preview/d8d65832-5b52-4460-be68-f872a3f1eede/ PlayList: https://www.youtube.com/playlist?list=PLSPImV-J0zlmwSoTFdlD0puZTUZWY18cO Our target audience Our project aids elders facing technical hurdles by utilizing image capture and processing to ascertain the next steps. We would also want to target companies willing to make their products more accessible. Leveraging Language Model (LLM) technology, we present users with a sequence of troubleshooting steps. Leveraging Language Model (LLM) technology, we present users with a sequence of troubleshooting steps. Examples include resetting Facebook accounts, recovering passwords, or managing daily deliveries. The domain aims to push the boundaries of automation while maximizing feedback from seniors to optimize usability. Examples include resetting Facebook accounts, recovering passwords, or managing daily deliveries. The domain aims to push the boundaries of automation while maximizing feedback from seniors to optimize usability. Technologies Used We build our backend as worker role using Intel Developer Cloud and Intel Extension for Transformer libraries backed by Chroma DB as a workflow running server requests to process LLM queries and RAG. The front-end is a Reflex app built purely in Python. The app serves as a web role with an image processing backend to annotate images uploaded via camera or as a file.

### Challenges we ran into

This was our first hackathon using LLM, so getting the terminologies right presented a significant challenge. The team at Intel provided us with some amazing sessions, explaining their tech stack and clarifying our doubts along the way. Additionally, we had to use Reflex for our front-end, which was relatively new and required a different learning curve compared to someone with a JavaScript background. The team was incredibly helpful in providing continued support, and we thoroughly enjoyed experimenting with their platform and exploring its boundaries.

### Accomplishments we're proud of

Over the course of 36 hours, we dedicated a significant amount of time to gathering feedback from mentors and sponsors regarding our pitch. This marked our first accomplishment, as it provided validation for our idea. We were successful in crafting an acceptable end-to-end experience that displays a conclusive proof of concept for our idea, of which we are proud. We feel satisfied with what the two of us achieved with this project.

### What we learned

Over the course we learned about iterating our design after discussion with mentors and judges. This gave us the knowledge needed to understand the marketability of this product. Secondly, we also got to work on cool LLM tools which gave us an opening to explore the possibilities.

### What's next

for Senior Savvy Solutions Future work: Going forward, we would like to let users simply upload any product instruction manual, and watch as our cutting-edge AI technology instantly comprehends its content. Then, harnessing the power of your smartphone camera, our platform seamlessly identifies your surroundings and pinpoints the product you're interacting with. While our current prototype focuses on OCR integration for text recognition, we aim to further enhance our platform by implementing real-time video processing capabilities, allowing users to receive guidance while viewing product demonstrations. Empowerment lies at the heart of our platform. Once the user inputs their query, our intelligent model springs into action, deciphering the nuances and intricacies of their request. Not content with mere comprehension, our model goes the extra mile, proactively seeking clarification when needed to ensure crystal-clear guidance. Whether it's directing the user to rotate or manipulate an object, pinpointing the exact button to press among a myriad of options, or simply elucidating the function of a particular button, our AI-driven solution delivers unparalleled assistance tailored to each user's unique needs. Due to severe time constraints, integration of AR was not possible. Our vision extends beyond individual applications to embrace a broader ecosystem of seamless integration and enhanced user experiences. Through the introduction of an API version of our app, coupled with reusable UI components and plugins, we aim to revolutionize the landscape of interactive guidance for physical products. Similar to the familiar walkthroughs encountered in software applications, our solution empowers companies to craft immersive experiences that gamify user interactions, elevating the overall product experience to new heights. Not only does this approach enrich user engagement, but it also delivers substantial cost savings by reducing the reliance on manual assistance and call center resources. With our innovative solution, companies can unlock a new realm of user-centric experiences while optimizing operational efficiency and resource allocation.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 4 recognized source files, 13 KB.
- FastAPI (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- Redis (technology) — detected in the code

## Codebase structure (from repository index)

### Files (6 of 6)

```
.gitignore
requirements.txt
rxconfig.py
virtuas2/__init__.py
virtuas2/easyOCR.py
virtuas2/virtuas2.py
```

### Dependencies

- requirements.txt: alembic@==1.13.1, anyio@==4.2.0, async-timeout@==4.0.3, bidict@==0.23.0, certifi@==2024.2.2, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, cloudpickle@==2.2.1, contourpy@==1.2.0, cryptography@==42.0.3, cycler@==0.12.1, Deprecated@==1.2.14, docopt@==0.6.2, easyocr@==1.7.1, exceptiongroup@==1.2.0, fastapi@==0.96.1, filelock@==3.13.1, fonttools@==4.49.0, fsspec@==2024.2.0, gunicorn@==20.1.0, h11@==0.14.0, httpcore@==1.0.3, httpx@==0.25.2, idna@==3.6, imageio@==2.34.0, importlib-resources@==6.1.1, install@==1.3.5, Jinja2@==3.1.3, kiwisolver@==1.4.5, lazy_loader@==0.3, Mako@==1.3.2, markdown-it-py@==3.0.0, MarkupSafe@==2.1.5, matplotlib@==3.8.2, mdurl@==0.1.2, mpmath@==1.3.0, networkx@==3.2.1, ninja@==1.11.1.1, numpy@==1.26.4, opencv-python@==4.9.0.80, opencv-python-headless@==4.9.0.80, packaging@==23.2, pillow@==10.2.0, pipdeptree@==2.14.0, pipreqs@==0.4.13, platformdirs@==3.11.0, psutil@==5.9.8, PyAudio@==0.2.14, pyclipper@==1.3.0.post5, pycparser@==2.21, pydantic@==1.10.14, pygame@==2.5.2, PyGithub@==2.2.0, Pygments@==2.17.2, PyJWT@==2.8.0, PyNaCl@==1.5.0, pyparsing@==3.1.1, python-bidi@==0.4.2, python-dateutil@==2.8.2, python-dotenv@==1.0.1, python-engineio@==4.9.0, python-multipart@==0.0.5, python-socketio@==5.11.1, PyYAML@==6.0.1, redis@==4.6.0, reflex@==0.4.0, reflex-hosting-cli@==0.1.8, requests@==2.31.0, rich@==13.7.0, scikit-image@==0.22.0, scipy@==1.12.0, shapely@==2.0.3, simple-websocket@==1.0.0, six@==1.16.0, sniffio@==1.3.0, SpeechRecognition@==3.10.1, SQLAlchemy@==2.0.27, sqlmodel@==0.0.14, starlette@==0.27.0, starlette-admin@==0.9.0, sympy@==1.12, tabulate@==0.9.0, tifffile@==2024.2.12, torch@==2.2.0, torchvision@==0.17.0, typer@==0.9.0, typing_extensions@==4.9.0, urllib3@==2.2.0, uvicorn@==0.27.1, watchdog@==2.3.1, watchfiles@==0.19.0, websockets@==12.0, wrapt@==1.16.0, wsproto@==1.2.0, yarg@==0.1.9, zipp@==3.17.0

### Recent commits (newest first)

- Merge pull request #2 from megha19998/develop
- Req update
- Merge pull request #1 from megha19998/develop
- Update
- Update UX
- additional func
- initial commit frontend

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
alembic==1.13.1
anyio==4.2.0
async-timeout==4.0.3
bidict==0.23.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
contourpy==1.2.0
cryptography==42.0.3
cycler==0.12.1
Deprecated==1.2.14
docopt==0.6.2
easyocr==1.7.1
exceptiongroup==1.2.0
fastapi==0.96.1
filelock==3.13.1
fonttools==4.49.0
fsspec==2024.2.0
gunicorn==20.1.0
h11==0.14.0
httpcore==1.0.3
httpx==0.25.2
idna==3.6
imageio==2.34.0
importlib-resources==6.1.1
install==1.3.5
Jinja2==3.1.3
kiwisolver==1.4.5
lazy_loader==0.3
Mako==1.3.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.8.2
mdurl==0.1.2
mpmath==1.3.0
networkx==3.2.1
ninja==1.11.1.1
numpy==1.26.4
opencv-python==4.9.0.80
opencv-python-headless==4.9.0.80
packaging==23.2
pillow==10.2.0
pipdeptree==2.14.0
pipreqs==0.4.13
platformdirs==3.11.0
psutil==5.9.8
PyAudio==0.2.14
pyclipper==1.3.0.post5
pycparser==2.21
pydantic==1.10.14
pygame==2.5.2
PyGithub==2.2.0
Pygments==2.17.2
PyJWT==2.8.0
PyNaCl==1.5.0
pyparsing==3.1.1
python-bidi==0.4.2
python-dateutil==2.8.2
python-dotenv==1.0.1
python-engineio==4.9.0
python-multipart==0.0.5
python-socketio==5.11.1
PyYAML==6.0.1
redis==4.6.0
reflex==0.4.0
reflex-hosting-cli==0.1.8
requests==2.31.0
rich==13.7.0
scikit-image==0.22.0
scipy==1.12.0
shapely==2.0.3
simple-websocket==1.0.0
six==1.16.0
sniffio==1.3.0
SpeechRecognition==3.10.1
SQLAlchemy==2.0.27
sqlmodel==0.0.14
starlette==0.27.0
starlette-admin==0.9.0
sympy==1.12
tabulate==0.9.0
tifffile==2024.2.12
torch==2.2.0
torchvision==0.17.0
typer==0.9.0
typing_extensions==4.9.0
urllib3==2.2.0
uvicorn==0.27.1
watchdog==2.3.1
watchfiles==0.19.0
websockets==12.0
wrapt==1.16.0
wsproto==1.2.0
yarg==0.1.9
zipp==3.17.0

```

### rxconfig.py

```python
import reflex as rx

config = rx.Config(
    app_name="virtuas2",
)
```

### virtuas2/easyOCR.py

```python
import cv2
import easyocr
import matplotlib.pyplot as plt
import numpy as np

image_path = '/Users/meghasharma/TreeHacks2024/facebook.jpeg'
img = cv2.imread(image_path)
reader = easyocr.Reader(['en'], gpu=False)
text_ = reader.readtext(img)
threshold = 0.25
for t_, t in enumerate(text_):
    if(t[1] == "Forgotten password?"):
        print(t[1])
        bbox, text, score = t 
        print(text)
        cv2.rectangle(img, bbox[0], bbox[2], (0, 255, 0), 5)
        cv2.putText(img, "click here", bbox[0], cv2.FONT_HERSHEY_COMPLEX, 0.65, (255, 0, 0), 2)
        plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
        plt.show()
```

### virtuas2/virtuas2.py

```python
"""Welcome to Reflex! This file outlines the steps to create a basic app."""

from rxconfig import config
import reflex as rx
from typing import List
import cv2
import easyocr
import matplotlib.pyplot as plt
import numpy as np
import os
import time
from PIL import Image
from pathlib import Path
import speech_recognition as sr
import pygame
import time
from dotenv import load_dotenv
from github import Github
from github import Auth
import json
import requests

load_dotenv()
docs_url = "https://reflex.dev/docs/getting-started/introduction"
filename = f"{config.app_name}/{config.app_name}.py"
recognizer = sr.Recognizer()

GITHUB_API="https://api.github.com/users/codlocker"
API_TOKEN=os.environ.get('GIT_API_KEY')
gist_id_read=os.environ.get('GIT_GIST_ID_READ')
gist_id_write=os.environ.get('GIT_GIST_ID_WRITE')

def update_gist(token, gist_id_write, json_data):
    headers = {
        'Authorization': f'token {API_TOKEN}',
        'Accept': 'application/vnd.github.v3+json'
    }
    data = {
        'files': {
            'request.json': {
                'content': json.dumps(json_data)
            },
        }
    }
    response = requests.patch(f'https://api.github.com/gists/{gist_id_write}', json=data, headers=headers)
    
    if response.status_code == 200:
        return response.json()['html_url']
    else:
        print(f'Failed to update gist: {response.status_code}')
        return None

def read_gist(gist_id_read):
    headers = {
        'Authorization': f'token {API_TOKEN}',
        'Accept': 'application/vnd.github.v3+json'
    }

    response = requests.get(f'https://api.github.com/gists/{gist_id_read}', headers=headers)
    print()
    print()
    #print(response)
    #responseSteps = [line for line in response['answer'].split('\n') if len(line) > 0][1:]
    if response.status_code == 200:
        gist_data = response.json()
        files = gist_data['files']
        for file_info in files.values():
            if file_info['filename'] == "response.json":
                #responseSteps = [line for line in file_info['content']['answer'].split('\n') if len(line) > 0][1:]
                print()
                print()
                jsonFile = json.loads(file_info['content'])
                print("I am printing json")
                print()
                print()
                print(jsonFile)
                print()
                print(type(jsonFile))
                responseSteps = [line for line in jsonFile['answer'].split('\n') if (len(line) > 0)][1:]
                #print(responseSteps)
                return responseSteps
    else:
        print(f'Failed to read gist: {response.status_code}')
        return ""
    
# # Example JSON data
# json_data = {
#     "answer" : response
# }

# # Example usage
# token = '<fdfdsfdsfdsf>'
# gist_id = 'ac86958f23270c6f346f950b7e418d58'
# # gist_url = update_gist(token, gist_id, json_data)
# #if gist_url:
# #     print(f'Gist updated successfully: {gist_url}')
# read_content = read_gist(gist_id)
# print(read_content)

class State(rx.State):
    """The app state."""
    img: list[str]
    inputBox: str
    step: List[str] = []

    async def handle_upload(self, files: list[rx.UploadFile]):
        """Handle the upload of file(s).

        Args:
            files: The uploaded files.
        """
        for file in files:
            upload_data = await file.read()
            outfile = Path(rx.get_upload_dir()) / file.filename
            # outfile = rx.get_asset_path(file.filename)

            # Save the file.
            with open(outfile, "wb") as file_object:
                file_object.write(upload_data)
            print(file_object.name)
            tempImg = performOCR(str(file_object.name))
            print("here")
            print(tempImg)
            
            # Update the img var.
            self.img.append(tempImg)
    
    def mouse_callback(self,event, x, y, flags, param):
        global captured, frame
        if event == cv2.EVENT_LBUTTONDOWN:
            captured = True
            frame = param

    def handleChange(self, newValue):
        self.inputBox = newValue
    
    def clearImageArray(self): 
        self.img.clear()
        return rx.clear_selected_files()

    def onClick(self):
        #self.step =  ["step 1","step 2","step 3","step 4","step 5","step 6"]
        print()
        print()
        print(self.inputBox)
        response = str(self.inputBox)
        result = {
            "question" : self.inputBox
        }
        resp = update_gist(API_TOKEN, gist_id_write, result)
        print(API_TOKEN)
        print(resp)
        time.sleep(20)
        response = read_gist(gist_id_read)
        self.step = response
    
    def finish_item(self, step: str):
        self.step = [i for i in self.step if i != step]

    def captureImage(self):
        global captured, frame
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open camera.")
            return
        cv2.namedWindow('Camera')
        cv2.setMouseCallback('Camera', self.mouse_callback)

        captured = False
        frame = None
        current_directory = os.getcwd()
        print("Current working directory:", current_directory)
        while not captured:
            ret, frame = cap.read()
            if not ret:
                print("Error: Could not capture frame.")
                break
            cv2.imshow('Camera', frame)
            key = cv2.waitKey(1)
            if key == 27:  
                break
        img_path = Path(rx.get_upload_dir()) / "capturedImage.png"
        if frame is not None:
            print("I am here")
            cv2.imwrite(str(img_path), frame)
        print(frame)
        print("I am here now")
        cap.release()
        cv2.destroyAllWindows()

        #State.handle_upload('captured_image.png')
        time.sleep(5)
        tempImg = performOCR(str(img_path))
        self.img.append(tempImg)

    def listen(self):
        with sr.Micr
[truncated — 7123 more characters]
```