# Project export: MedVision

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: It can take or upload a picture of medicine package label, and provide necessary information using text-to-speech feature.
- Devpost: https://devpost.com/software/medvision
- GitHub: https://github.com/erik-ksth/medVision1.0
- Team: 0 GitHub contributor(s) — 

## Devpost submission (written by the team)

### Inspiration

When our group was brainstorming the ideas, we found out how hard it is for us to understand the medicine based on the package. That leads into thinking about the struggle that people with impaired visions and elder people may have. Envision a world where visually impaired and elderly individuals no longer struggle to read medicine labels or depend on others. With MedVision Web, this vision becomes a reality. It offers a straightforward and efficient software solution that scans medication labels and instantly vocalizes the medication name, purpose, and dosage. Together, we're revolutionizing the healthcare experience, ensuring that people of all ages and visual abilities can independently and confidently manage their well-being.

### What it does

Users can capture or upload an image of a medicine package label, and our software employs Google Vision AI and Together.AI to scan and extract text from the image. It then provides users with necessary information, including a text-to-speech feature, providing an efficient and user-friendly solution.

### How we built it

We built a full-stack application that uses ReactJS for the front end and Python with Flask for the back end. It integrates with various AI services, including Together.AI and Google Cloud and Firebase for database functionality. Additionally, we have attempted to develop a mobile app using Flutter.

### Challenges we ran into

As this is our first time implementing a full-stack app using various AI technologies and tools, we had difficulties when integrating their APIs into our app. Thanks to the workshops and mentors, we were able to quickly overcome these challenges and proceed with our work. Even though we developed the front-end and back-end separately, we struggled to integrate them due to time constraints. Also, testing a complex full-stack application with multiple AI services is definitely a challenging process.

### Accomplishments we're proud of

This project involves various parts being developed separately, and the most significant accomplishment we are proud of is successfully integrating all of them and offering impactful features to users.

### What we learned

We learned a lot about how to integrate APIs, different creative ideas & tools, collaboration with teams, and finally we had fun!

### What's next

We believe that MedVision is more than just a project, it's a transformative tool for enhancing public health. Our vision for the future is to create a chatbot which can make health and medication information accessible at users' fingertips. We're dedicated to enhancing the user experience with a responsive design and user authentication, providing a secure environment for users to access a wealth of health resources. We aspire to elevate the health community by pushing the boundaries of innovation and making a positive impact in healthcare technology.

## README (from the GitHub repository)

# Getting Started with Create React App

This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

## Available Scripts

In the project directory, you can run:

### `npm start`

Runs the app in the development mode.\
Open [http://localhost:3000](http://localhost:3000) to view it in your browser.

The page will reload when you make changes.\
You may also see any lint errors in the console.

### `npm test`

Launches the test runner in the interactive watch mode.\
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.

### `npm run build`

Builds the app for production to the `build` folder.\
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.\
Your app is ready to be deployed!

See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.

### `npm run eject`

**Note: this is a one-way operation. Once you `eject`, you can't go back!**

If you aren't satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.

Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you're on your own.

You don't have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn't feel obligated to use this feature. However we understand that this tool wouldn't be useful if you couldn't customize it when you are ready for it.

## Learn More

You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).

To learn React, check out the [React documentation](https://reactjs.org/).

### Code Splitting

This section has moved here: [https://facebook.github.io/create-react-app/docs/code-splitting](https://facebook.github.io/create-react-app/docs/code-splitting)

### Analyzing the Bundle Size

This section has moved here: [https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size](https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size)

### Making a Progressive Web App

This section has moved here: [https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app](https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app)

### Advanced Configuration

This section has moved here: [https://facebook.github.io/create-react-app/docs/advanced-configuration](https://facebook.github.io/create-react-app/docs/advanced-configuration)

### Deployment

This section has moved here: [https://facebook.github.io/create-react-app/docs/deployment](https://facebook.github.io/create-react-app/docs/deployment)

### `npm run build` fails to minify

This section has moved here: [https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify](https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify)


## Detected evidence (automated analysis)

Indexed codebase: 21 recognized source files, 25 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- Firebase (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (28 of 28)

```
.gitignore
backend/api.py
backend/config/.gitignore
backend/firebase/firestore.py
backend/main.py
backend/my_modules/text_to_speech.py
backend/my_modules/together_ai.py
backend/my_modules/vision_script.py
backend/requirements.txt
handles/handleSubmit.js
package.json
public/index.html
public/manifest.json
public/robots.txt
README.md
server.js
src/App.css
src/App.js
src/App.test.js
src/firebase.js
src/index.css
src/index.js
src/med-vision.json
src/pages/home.js
src/pages/info.js
src/reportWebVitals.js
src/setupTests.js
tailwind.config.js
```

### Dependencies

- backend/requirements.txt: firebase-admin@==5.3.0, google-cloud-texttospeech@==2.5.0, google-cloud-vision@==2.7.0, Playsound@==1.3.0, requests@==2.26.0, together@==0.2.6
- package.json: @emotion/react@^11.11.1, @emotion/styled@^11.11.0, @mui/icons-material@^5.14.15, @mui/material@^5.14.15, @mui/styled-engine-sc@^6.0.0-alpha.3, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.6.0, express@^4.18.2, firebase@^10.5.2, firebase-admin@^11.11.0, react@^18.2.0, react-dom@^18.2.0, react-router-dom@^6.17.0, react-scripts@5.0.1, styled-components@^6.1.0, tailwindcss@^3.3.5, use-sound@^4.0.1, web-vitals@^2.1.4

### Recent commits (newest first)

- fix: play sound
- build: info page
- fix: btn bug
- fix: disabled btn
- feat: image toggle
- fix: error
- build: blah blah
- build: photo uploading 2.0
- fix: camera not working
- upload btn done
- create upload button
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "med-vision",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@emotion/react": "^11.11.1",
    "@emotion/styled": "^11.11.0",
    "@mui/icons-material": "^5.14.15",
    "@mui/material": "^5.14.15",
    "@mui/styled-engine-sc": "^6.0.0-alpha.3",
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "axios": "^1.6.0",
    "express": "^4.18.2",
    "firebase": "^10.5.2",
    "firebase-admin": "^11.11.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-router-dom": "^6.17.0",
    "react-scripts": "5.0.1",
    "styled-components": "^6.1.0",
    "use-sound": "^4.0.1",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "tailwindcss": "^3.3.5"
  }
}

```

### backend/requirements.txt

```
# Google Vision API
google-cloud-vision==2.7.0

# Google Text-to-Speech API
google-cloud-texttospeech==2.5.0

# Requests
requests==2.26.0 

# Together AI API
together==0.2.6

# Firebase libraries
firebase-admin==5.3.0

# Playsound
Playsound==1.3.0

```

### server.js

```javascript
const express = require('express');
const app = express();
const PORT = 3000;

app.get('/', (req, res) => {
  res.send('Hello from Node.js server!');
});

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

```

### src/App.js

```javascript
import "./App.css";
import { BrowserRouter as Router, Routes, Route } from "react-router-dom";
import Home from "./pages/home";
import Info from "./pages/info";

function App() {
  return (
    <>
      <Router>
        <Routes>
          <Route path="/" element={<Home />} />
          <Route path="/info" element={<Info />} />
        </Routes>
      </Router>
    </>
  );
}

export default App;

```

### src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

```

### backend/main.py

```python
from my_modules import text_to_speech
from my_modules import together_ai as ta
from my_modules.vision_script import *
from playsound import playsound
from flask import Flask, request, jsonify
from api import api 

app = Flask(__name__)
    
@app.route('/api/process-together-ai', methods=['POST'])
def main():
    # Get text data from together.AI
    text = "Thank you for using MedVision, AI powered medical assistant for your healthy life. According to the input data, I provide the useful information for you. " + ta.process_text_with_together_ai()

    # Generate speech
    voice_name = "en-US-Standard-J"
    text_to_speech.text_to_wav(voice_name, text)
    audio_file = voice_name + ".wav"
    playsound(audio_file)

    # parsed_strings = parse_data(ta.process_text_with_together_ai())
    # results = {
    #     "Usage": parsed_strings[0],
    #     "Dosage": parsed_strings[1],
    #     "Storage": parsed_strings[2]
    # }

    # return jsonify(results)


# Main function
if __name__ == "__main__":
    main()

    # app = Flask(__name__)
    # app.register_blueprint(api, url_prefix='/api')
    # app.run(debug=True)
```

### tailwind.config.js

```javascript
/** @type {import('tailwindcss').Config} */
module.exports = {
  content: ["./src/**/*.{html,js}"],
  theme: {
    extend: {
      colors: {
        'primaryGreen': '#006d77' ,
        'primaryTeel' : '#83c5be' ,
        'primaryBlue' : '#edf6f9'
      }
    },
  },
  plugins: [],
}
```

### src/setupTests.js

```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';

```

### backend/api.py

```python
from flask import Blueprint, jsonify

api = Blueprint('api', __name__)

@api.route('/api/data', methods=['GET'])
def get_data():
    data = {"message": "Data from the backend"}
    print("get_data() function called")
    return jsonify(data)

```

### src/App.test.js

```javascript
import { render, screen } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
  render(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

```

[12 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]