# Project export: DermaDetect This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: DermaDetect leverages AI for accessible skin health checks, focusing on affordability and privacy for underserved populations. - Devpost: https://devpost.com/software/skin-ai-zlwdsy - GitHub: https://github.com/abccodes/Treehacks2024 - Result: winner (Convex: Best Use of Convex Features ($1.25k Cash + 6mo Pro [1st] & $500 Cash + 3mo Pro [2nd] & $250 Cash [3rd])) - Team: 4 GitHub contributor(s) — drkchn (54 commits), Aidan (40 commits), Tommy Yau (28 commits), SirKentut (13 commits) ## Devpost submission (written by the team) ### Inspiration DermaDetect was born out of a commitment to improve healthcare equity for underrepresented and economically disadvantaged communities, including seniors, other marginalized populations, and those impacted by economic inequality. Recognizing the prohibitive costs and emotional toll of traditional skin cancer screenings, which often result in benign outcomes, we developed an open-source AI-powered application to provide preliminary skin assessments. This innovation aims to reduce financial burdens and emotional stress, offering immediate access to health information and making early detection services more accessible to everyone, regardless of their societal status. ### What it does AI-powered analysis: Fine-tuned Resnet50 Convolutional Neural Network classifier that predicts skin lesions as benign versus cancerous by leveraging the open-source HAM10000 dataset. AI-powered analysis: Fine-tuned Resnet50 Convolutional Neural Network classifier that predicts skin lesions as benign versus cancerous by leveraging the open-source HAM10000 dataset. Protecting patient data confidentiality: Our application uses OAuth technology (Clerk and Convex) to authenticate and verify users logging into our application, protecting patient data when users upload images and enter protected health information (PHI). Protecting patient data confidentiality: Our application uses OAuth technology (Clerk and Convex) to authenticate and verify users logging into our application, protecting patient data when users upload images and enter protected health information (PHI). Understandable and age-appropriate information: Prediction Guard LLM technology offers clear explanations of results, fostering informed decision-making for users while respecting patient data privacy. Understandable and age-appropriate information: Prediction Guard LLM technology offers clear explanations of results, fostering informed decision-making for users while respecting patient data privacy. Journal entry logging: Using the Convex backend database schema allows users to make multiple journal entries, monitor their skin, and track moles over long periods. Journal entry logging: Using the Convex backend database schema allows users to make multiple journal entries, monitor their skin, and track moles over long periods. Seamless triaging: Direct connection to qualified healthcare providers eliminates unnecessary user anxiety and wait times for concerning cases. Seamless triaging: Direct connection to qualified healthcare providers eliminates unnecessary user anxiety and wait times for concerning cases. ### How we built it Machine learning model TensorFlow, Keras: Facilitated our model training and model architecture, Python, OpenCV, Prediction Guard LLM, Intel Developer Cloud, Pandas, NumPy, Sklearn, Matplotlib Frontend TypeScript, Convex, React.js, Shadcn (Components), FramerMotion (Animated components), TailwindCSS Backend TypeScript, Convex Database & File storage, Clerk (OAuth User login authentication), Python, Flask, Vite, InfoBip (Twillio-like service) ### Challenges we ran into We had a lot of trouble cleaning and applying the HAM10000 skin images dataset. Due to long run times, we found it very challenging to make any progress on tuning our model and sorting the data. We eventually started splitting our dataset into smaller batches and training our model on a small amount of data before scaling up which worked around our problem. We also had a lot of trouble normalizing our data, and figuring out how to deal with a large Melanocytic nevi class imbalance. After much trial and error, we were able to correctly apply data augmentation and oversampling methods to address the class imbalance issue. We had a lot of trouble cleaning and applying the HAM10000 skin images dataset. Due to long run times, we found it very challenging to make any progress on tuning our model and sorting the data. We eventually started splitting our dataset into smaller batches and training our model on a small amount of data before scaling up which worked around our problem. We also had a lot of trouble normalizing our data, and figuring out how to deal with a large Melanocytic nevi class imbalance. After much trial and error, we were able to correctly apply data augmentation and oversampling methods to address the class imbalance issue. One of our biggest challenges was setting up our backend Flask server. We encountered so many environment errors, and for a large portion of the time, the server was only able to run on one computer. After many Google searches, we persevered and resolved the errors. One of our biggest challenges was setting up our backend Flask server. We encountered so many environment errors, and for a large portion of the time, the server was only able to run on one computer. After many Google searches, we persevered and resolved the errors. ### Accomplishments we're proud of We are incredibly proud of developing a working open-source, AI-powered application that democratizes access to skin cancer assessments. We are incredibly proud of developing a working open-source, AI-powered application that democratizes access to skin cancer assessments. Tackling the technical challenges of cleaning and applying the HAM10000 skin images dataset, dealing with class imbalances, and normalizing data has been a journey of persistence and innovation. Tackling the technical challenges of cleaning and applying the HAM10000 skin images dataset, dealing with class imbalances, and normalizing data has been a journey of persistence and innovation. Setting up a secure and reliable backend server was another significant hurdle we overcame. The process taught us the importance of resilience and resourcefulness, as we navigated through numerous environmental errors to achieve a stable and scalable solution that protects patient data confidentiality. Setting up a secure and reliable backend server was another significant hurdle we overcame. The process taught us the importance of resilience and resourcefulness, as we navigated through numerous environmental errors to achieve a stable and scalable solution that protects patient data confidentiality. Integrating many technologies that were new to a lot of the team such as Clerk for authentication, Convex for user data management, Prediction Guard LLM, and Intel Developer Cloud. Integrating many technologies that were new to a lot of the team such as Clerk for authentication, Convex for user data management, Prediction Guard LLM, and Intel Developer Cloud. Extending beyond the technical domain, reflecting a deep dedication to inclusivity, education, and empowerment in healthcare. Extending beyond the technical domain, reflecting a deep dedication to inclusivity, education, and empowerment in healthcare. ### What we learned Critical importance of data quality and management in AI-driven applications. The challenges we faced in cleaning and applying the HAM10000 skin images dataset underscored the need for meticulous data preprocessing to ensure AI model accuracy, reliability, and equality. Critical importance of data quality and management in AI-driven applications. The challenges we faced in cleaning and applying the HAM10000 skin images dataset underscored the need for meticulous data preprocessing to ensure AI model accuracy, reliability, and equality. How to Integrate many different new technologies such as Convex, Clerk, Flask, Intel Cloud Development, Prediction Guard LLM, and Infobip to create a seamless and secure user experience. How to Integrate many different new technologies such as Convex, Clerk, Flask, Intel Cloud Development, Prediction Guard LLM, and Infobip to create a seamless and secure user experience. ### What's next Finding users to foster future development and feedback. Finding users to foster future development and feedback. Partnering with healthcare organizations and senior communities for wider adoption. Partnering with healthcare organizations and senior communities for wider adoption. Continuously improving upon data curation, model training, and user experience through ongoing research and development. Continuously improving upon data curation, model training, and user experience through ongoing research and development. ## README (from the GitHub repository) # Overview This project focuses on the development of a full-stack machine learning project for the classification of skin cancer, leveraging the comprehensive Skin Cancer MNIST: HAM10000 dataset. The dataset, curated by Maderna in 2018, provides a diverse collection of dermatoscopic images crucial for training and testing our algorithms. It encompasses over 10,000 labeled images of skin lesions, categorized into seven different diagnostic categories, making it an invaluable resource for advancing research and development in the field of dermatology and automated diagnostic systems. ------------------ ![Tech Diagram ()](https://github.com/abccodes/Treehacks2024/assets/79234681/7425daf8-feb2-48ab-a621-b7f16aa495fe) ------------------ ![Overall Diagram](https://github.com/abccodes/Treehacks2024/assets/79234681/8a3d4574-b091-4c2d-b23a-35fa0c20d968) ------------------ ============================================================================ # Convex + TypeScript + ESLint + Vite + React + Clerk + Tailwind + shadcn/ui This template provides a minimal setup to get Convex working, with TypeScript, ESLint and React using [Vite](https://vitejs.dev/). It uses [Clerk](https://clerk.dev/) for user authentication. Start by editing `convex/myFunctions.ts` and interact with your React app. See Convex docs at https://docs.convex.dev/home ## Setting up ``` npm create convex@latest -t react-vite-clerk-shadcn ``` Then: 1. Follow steps 1 to 3 in the [Clerk onboarding guide](https://docs.convex.dev/auth/clerk#get-started) 2. Paste the Issuer URL as `CLERK_JWT_ISSUER_DOMAIN` to your dev deployment environment variable settings on the Convex dashboard (see [docs](https://docs.convex.dev/auth/clerk#configuring-dev-and-prod-instances)) 3. Paste your publishable key as `VITE_CLERK_PUBLISHABLE_KEY=""` to the `.env.local` file in this directory. If you want to sync Clerk user data via webhooks, check out this [example repo](https://github.com/thomasballinger/convex-clerk-users-table/). ============================================================================ # Setup Server: Classifier and Prediction Guard Intel LLM ``` cd treehacks2024 python3 -m venv venv source venv/bin/activate pip install -r requirements.txt ``` # run the server ``` cd treehacks2024 source venv/bin/activate python server/app.py ``` # exit venv ``` exit ``` ============================================================================ # Inspiration DermaDetect was born out of a commitment to improve healthcare equity for underrepresented and economically disadvantaged communities, including seniors, other marginalized populations, and those impacted by economic inequality. Recognizing the prohibitive costs and emotional toll of traditional skin cancer screenings, which often result in benign outcomes, we developed an open-source AI-powered application to provide preliminary skin assessments. This innovation aims to reduce financial burdens and emotional stress, offering immediate access to health information and making early detection services more accessible to everyone, regardless of their societal status. # What it does AI-powered analysis: Fine-tuned Resnet50 Convolutional Neural Network classifier that predicts skin lesions as benign versus cancerous by leveraging the open-source HAM10000 dataset. Protecting patient data confidentiality: Our application uses OAuth technology (Clerk and Convex) to authenticate and verify users logging into our application, protecting patient data when users upload images and enter protected health information (PHI). Understandable and age-appropriate information: Prediction Guard LLM technology offers clear explanations of results, fostering informed decision-making for users while respecting patient data privacy. Journal entry logging: Using the Convex backend database schema allows users to make multiple journal entries, monitor their skin, and track moles over long periods. Seamless triaging: Direct connection to qualified healthcare providers eliminates unnecessary user anxiety and wait times for concerning cases. # How we built it Machine learning model TensorFlow, Keras: Facilitated our model training and model architecture, Python, OpenCV, Prediction Guard LLM, Intel Developer Cloud, Pandas, NumPy, Sklearn, Matplotlib Frontend TypeScript, Convex, React.js, Shadcn (Components), FramerMotion (Animated components), TailwindCSS Backend TypeScript, Convex Database & File storage, Clerk (OAuth User login authentication), Python, Flask, Vite, InfoBip (Twillio-like service) # Challenges we ran into We had a lot of trouble cleaning and applying the HAM10000 skin images dataset. Due to long run times, we found it very challenging to make any progress on tuning our model and sorting the data. We eventually started splitting our dataset into smaller batches and training our model on a small amount of data before scaling up which worked around our problem. We also had a lot of trouble normalizing our data, and figuring out how to deal with a large Melanocytic nevi class imbalance. After much trial and error, we were able to correctly apply data augmentation and oversampling methods to address the class imbalance issue. One of our biggest challenges was setting up our backend Flask server. We encountered so many environment errors, and for a large portion of the time, the server was only able to run on one computer. After many Google searches, we persevered and resolved the errors. # Accomplishments that we're proud of We are incredibly proud of developing a working open-source, AI-powered application that democratizes access to skin cancer assessments. Tackling the technical challenges of cleaning and applying the HAM10000 skin images dataset, dealing with class imbalances, and normalizing data has been a journey of persistence and innovation. Setting up a secure and reliable backend server was another significant hurdle we overcame. The process taught us the importance of resilience and resourcefulness, as we navigated through numerous environmental errors to achieve a stable and scalable solution that protects patient data confidentiality. Integrating many technologies that were new to a lot of the team such as Clerk for authentication, Convex for user data management, Prediction Guard LLM, and Intel Developer Cloud. Extending beyond the technical domain, reflecting a deep dedication to inclusivity, education, and empowerment in healthcare. # What we learned Critical importance of data quality and management in AI-driven applications. The challenges we faced in cleaning and applying the HAM10000 skin images dataset underscored the need for meticulous data preprocessing to ensure AI model accuracy, reliability, and equality. How to Integrate many different new technologies such as Convex, Clerk, Flask, Intel Cloud Development, Prediction Guard LLM, and Infobip to create a seamless and secure user experience. # What's next for DermaDetect Finding users to foster future development and feedback. Partnering with healthcare organizations and senior communities for wider adoption. Continuously improving upon data curation, model training, and user experience through ongoing research and development. ## Detected evidence (automated analysis) Indexed codebase: 74 recognized source files, 161 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TensorFlow (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (82 of 82) ``` README.md treehacks2024/.eslintrc.cjs treehacks2024/.gitignore treehacks2024/components.json treehacks2024/convex/_generated/api.d.ts treehacks2024/convex/_generated/api.js treehacks2024/convex/_generated/dataModel.d.ts treehacks2024/convex/_generated/server.d.ts treehacks2024/convex/_generated/server.js treehacks2024/convex/auth.config.js treehacks2024/convex/files.ts treehacks2024/convex/myFunctions.ts treehacks2024/convex/README.md treehacks2024/convex/schema.ts treehacks2024/convex/tsconfig.json treehacks2024/index.html treehacks2024/package.json treehacks2024/postcss.config.js treehacks2024/requirements.txt treehacks2024/server/app.py treehacks2024/src/App.tsx treehacks2024/src/components/test.jsx treehacks2024/src/components/typography/code.tsx treehacks2024/src/components/typography/link.tsx treehacks2024/src/components/ui/accordion.tsx treehacks2024/src/components/ui/alert-dialog.tsx treehacks2024/src/components/ui/alert.tsx treehacks2024/src/components/ui/aspect-ratio.tsx treehacks2024/src/components/ui/avatar.tsx treehacks2024/src/components/ui/badge.tsx treehacks2024/src/components/ui/button.tsx treehacks2024/src/components/ui/calendar.tsx treehacks2024/src/components/ui/card.tsx treehacks2024/src/components/ui/carousel.tsx treehacks2024/src/components/ui/checkbox.tsx treehacks2024/src/components/ui/collapsible.tsx treehacks2024/src/components/ui/command.tsx treehacks2024/src/components/ui/context-menu.tsx treehacks2024/src/components/ui/dialog.tsx treehacks2024/src/components/ui/dropdown-menu.tsx treehacks2024/src/components/ui/form.tsx treehacks2024/src/components/ui/hover-card.tsx treehacks2024/src/components/ui/input.tsx treehacks2024/src/components/ui/label.tsx treehacks2024/src/components/ui/menubar.tsx treehacks2024/src/components/ui/navigation-menu.tsx treehacks2024/src/components/ui/popover.tsx treehacks2024/src/components/ui/progress.tsx treehacks2024/src/components/ui/radio-group.tsx treehacks2024/src/components/ui/scroll-area.tsx treehacks2024/src/components/ui/select.tsx treehacks2024/src/components/ui/separator.tsx treehacks2024/src/components/ui/sheet.tsx treehacks2024/src/components/ui/skeleton.tsx treehacks2024/src/components/ui/slider.tsx treehacks2024/src/components/ui/switch.tsx treehacks2024/src/components/ui/table.tsx treehacks2024/src/components/ui/tabs.tsx treehacks2024/src/components/ui/textarea.tsx treehacks2024/src/components/ui/toast.tsx treehacks2024/src/components/ui/toaster.tsx treehacks2024/src/components/ui/toggle.tsx treehacks2024/src/components/ui/tooltip.tsx treehacks2024/src/components/ui/use-toast.ts treehacks2024/src/ErrorBoundary.tsx treehacks2024/src/index.css treehacks2024/src/lib/utils.tsx treehacks2024/src/main.tsx treehacks2024/src/main/components/DescriptionCard.tsx treehacks2024/src/main/components/ImageCard.tsx treehacks2024/src/main/components/LoadingScreen.tsx treehacks2024/src/main/components/Navbar.tsx treehacks2024/src/main/components/Upload.module.css treehacks2024/src/main/components/Upload.tsx treehacks2024/src/main/pages/HomePage.tsx treehacks2024/src/main/pages/InputPage.tsx treehacks2024/src/main/pages/OutputPage.tsx treehacks2024/src/Shufflehero.jsx treehacks2024/tailwind.config.js treehacks2024/tsconfig.json treehacks2024/tsconfig.node.json treehacks2024/vite.config.ts ``` ### Dependencies - treehacks2024/package.json: @clerk/clerk-react@^4.30.5, @hookform/resolvers@^3.3.2, @radix-ui/react-accordion@^1.1.2, @radix-ui/react-alert-dialog@^1.0.5, @radix-ui/react-aspect-ratio@^1.0.3, @radix-ui/react-avatar@^1.0.4, @radix-ui/react-checkbox@^1.0.4, @radix-ui/react-collapsible@^1.0.3, @radix-ui/react-context-menu@^2.1.5, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-hover-card@^1.0.7, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.0.2, @radix-ui/react-menubar@^1.0.4, @radix-ui/react-navigation-menu@^1.1.4, @radix-ui/react-popover@^1.0.7, @radix-ui/react-progress@^1.0.3, @radix-ui/react-radio-group@^1.1.3, @radix-ui/react-scroll-area@^1.0.5, @radix-ui/react-select@^2.0.0, @radix-ui/react-separator@^1.0.3, @radix-ui/react-slider@^1.1.2, @radix-ui/react-slot@^1.0.2, @radix-ui/react-switch@^1.0.3, @radix-ui/react-tabs@^1.0.4, @radix-ui/react-toast@^1.1.5, @radix-ui/react-toggle@^1.0.3, @radix-ui/react-tooltip@^1.0.7, @types/node@^20.7.0, @types/react@^18.2.21, @types/react-dom@^18.2.7, @typescript-eslint/eslint-plugin@^6.7.0, @typescript-eslint/parser@^6.7.0, @vitejs/plugin-react@^4.0.4, @xixixao/uploadstuff@^0.0.5, autoprefixer@^10.4.16, class-variance-authority@^0.7.0, clsx@^2.0.0, cmdk@^0.2.0, convex@^1.8.0, date-fns@^2.30.0, embla-carousel-autoplay@^8.0.0-rc22, embla-carousel-react@^8.0.0-rc22, eslint@^8.49.0, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.3, framer-motion@^11.0.5, npm@^10.4.0, npm-run-all@^4.1.5, postcss@^8.4.30, react@^18.2.0, react-day-picker@^8.9.1, react-dom@^18.2.0, react-hook-form@^7.47.0, react-router-dom@^6.22.1, tailwind-merge@^1.14.0, tailwindcss@^3.3.3, tailwindcss-animate@^1.0.7, typescript@^5.2.2, vite@^4.4.9, zod@^3.22.4 - treehacks2024/requirements.txt: Flask@==2.0.3, Flask_Cors@==3.0.3, infobip_api_python_client@==3.0.3, keras@==2.15.0, numpy@==1.26.4, opencv_python@==4.9.0.80, Pillow@==10.2.0, predictionguard@==1.9.1, python-dotenv@==1.0.1, Requests@==2.31.0, setuptools@==58.0.4, tensorflow@==2.15.0, typing_extensions@==4.9.0 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Delete treehacks2024/README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Merge pull request #28 from abccodes/small-fix - small fix: - Merge pull request #27 from abccodes/derek-landing-home-page - Updated project name - Updated images and increased text size - Added images for landing page - Update README.md - Update README.md - Update README.md - Update README.md ## Key source files (fetched from GitHub, selected and truncated for size) ### treehacks2024/requirements.txt ``` Flask==2.0.3 Flask_Cors==3.0.3 infobip_api_python_client==3.0.3 keras==2.15.0 numpy==1.26.4 opencv_python==4.9.0.80 Pillow==10.2.0 predictionguard==1.9.1 python-dotenv==1.0.1 Requests==2.31.0 setuptools==58.0.4 tensorflow==2.15.0 typing_extensions==4.9.0 ``` ### treehacks2024/package.json ``` { "name": "treehacks2024", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "npm-run-all --parallel dev:frontend dev:backend", "dev:frontend": "vite --open", "dev:backend": "convex dev", "predev": "convex dev --until-success && convex dashboard", "build": "tsc && vite build", "lint": "tsc && eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0", "preview": "vite preview" }, "dependencies": { "@clerk/clerk-react": "^4.30.5", "@hookform/resolvers": "^3.3.2", "@radix-ui/react-accordion": "^1.1.2", "@radix-ui/react-alert-dialog": "^1.0.5", "@radix-ui/react-aspect-ratio": "^1.0.3", "@radix-ui/react-avatar": "^1.0.4", "@radix-ui/react-checkbox": "^1.0.4", "@radix-ui/react-collapsible": "^1.0.3", "@radix-ui/react-context-menu": "^2.1.5", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-hover-card": "^1.0.7", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-menubar": "^1.0.4", "@radix-ui/react-navigation-menu": "^1.1.4", "@radix-ui/react-popover": "^1.0.7", "@radix-ui/react-progress": "^1.0.3", "@radix-ui/react-radio-group": "^1.1.3", "@radix-ui/react-scroll-area": "^1.0.5", "@radix-ui/react-select": "^2.0.0", "@radix-ui/react-separator": "^1.0.3", "@radix-ui/react-slider": "^1.1.2", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-switch": "^1.0.3", "@radix-ui/react-tabs": "^1.0.4", "@radix-ui/react-toast": "^1.1.5", "@radix-ui/react-toggle": "^1.0.3", "@radix-ui/react-tooltip": "^1.0.7", "@xixixao/uploadstuff": "^0.0.5", "class-variance-authority": "^0.7.0", "clsx": "^2.0.0", "cmdk": "^0.2.0", "convex": "^1.8.0", "date-fns": "^2.30.0", "embla-carousel-autoplay": "^8.0.0-rc22", "embla-carousel-react": "^8.0.0-rc22", "framer-motion": "^11.0.5", "npm": "^10.4.0", "react": "^18.2.0", "react-day-picker": "^8.9.1", "react-dom": "^18.2.0", "react-hook-form": "^7.47.0", "react-router-dom": "^6.22.1", "tailwind-merge": "^1.14.0", "tailwindcss-animate": "^1.0.7", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "^20.7.0", "@types/react": "^18.2.21", "@types/react-dom": "^18.2.7", "@typescript-eslint/eslint-plugin": "^6.7.0", "@typescript-eslint/parser": "^6.7.0", "@vitejs/plugin-react": "^4.0.4", "autoprefixer": "^10.4.16", "eslint": "^8.49.0", "eslint-plugin-react-hooks": "^4.6.0", "eslint-plugin-react-refresh": "^0.4.3", "npm-run-all": "^4.1.5", "postcss": "^8.4.30", "tailwindcss": "^3.3.3", "typescript": "^5.2.2", "vite": "^4.4.9" } } ``` ### treehacks2024/src/App.tsx ```typescript import { BrowserRouter, Routes, Route } from "react-router-dom"; import HomePage from "./main/pages/HomePage"; import OutputPage from "./main/pages/OutputPage.tsx"; export default function App() { return ( } /> } /> ); } ``` ### treehacks2024/src/main.tsx ```typescript import { ClerkProvider, useAuth } from "@clerk/clerk-react"; import { ConvexReactClient } from "convex/react"; import { ConvexProviderWithClerk } from "convex/react-clerk"; import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; import { ErrorBoundary } from "./ErrorBoundary"; import "./index.css"; const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string); ReactDOM.createRoot(document.getElementById("root")!).render( ); ``` ### treehacks2024/server/app.py ```python from flask import Flask, request, jsonify from PIL import Image from io import BytesIO import requests from flask_cors import CORS import cv2 from tensorflow.keras.models import load_model from tensorflow.keras.utils import to_categorical from keras.preprocessing.image import img_to_array import numpy as np import predictionguard as pg import os import random from dotenv import load_dotenv from infobip_api_client.api_client import ApiClient, Configuration import http.client import json load_dotenv('.env.local') app = Flask(__name__) CORS(app) os.environ['PREDICTIONGUARD_TOKEN'] = "q1VuOjnffJ3NO2oFN8Q9m8vghYc84ld13jaqdF7E" model = load_model('server/models/skin_v2.h5') infobip_client_config = Configuration( host=os.getenv("INFOBIP_BASE_URL"), api_key={"APIKeyHeader": os.getenv("INFOBIP_API_KEY")}, api_key_prefix={"APIKeyHeader": os.getenv("INFOBIP_API_PREFIX")}, ) infobip_api_client = ApiClient(infobip_client_config) lesion_type_dict = { 'nv': 'Melanocytic nevi', 'mel': 'Melanoma', 'bkl': 'Benign keratosis-like lesions ', 'bcc': 'Basal cell carcinoma', 'akiec': 'Actinic keratoses', 'vasc': 'Vascular lesions', 'df': 'Dermatofibroma' } lesion_ID = [ 'nv', 'mel', 'bkl', 'bcc', 'akiec', 'vasc', 'df' ] @app.route('/api/predict', methods=['POST']) def predict(): # data = request.get_json() # image_url = data.get('url') images = [ 'https://i.ibb.co/VpG4mfT/ISIC-0029075.jpg', 'https://i.ibb.co/RcqFMJj/ISIC-0029079.jpg', 'https://i.ibb.co/wpJyRYr/ISIC-0029082.jpg', 'https://i.ibb.co/h7jhJpj/ISIC-0029086.jpg', 'https://i.ibb.co/ZWg0n7r/ISIC-0029089.jpg', 'https://i.ibb.co/5FBtsH3/ISIC-0029094.jpg', 'https://i.ibb.co/wC0NR8R/ISIC-0029100.jpg', 'https://i.ibb.co/vHWtMg5/ISIC-0029107.jpg', 'https://i.ibb.co/wJyt7TS/ISIC-0029112.jpg', 'https://i.ibb.co/yfQ5tcK/ISIC-0029118.jpg'] image_url = random.choice(images) img_array = download_and_preprocess_image(image_url) prediction = model.predict(img_array) predicted_class = np.argmax(prediction, axis=1)[0] predicted_label = lesion_type_dict[lesion_ID[predicted_class]] print(prediction, predicted_class) prompt = """### System: You are a profession Dermatologist who has been in the industry for 40 years. You will receive a input of a skin disease name of a patient. State it if the disease is cancerous or not. If it's cancerous, show your care and concern to the user and ask them to ask for a doctor's advise as soon as possible. Explain what that is to a patient, in very simple term, so people that without a biology background can still understand. Don't mention the user has no biology background. Tell them it's still important to get advise from a professional Doctor. No number list or bullet points, 3 complete sentenses only. ### User: I was using a skin disease classifier, and it says I may have {disease}. Explain it within 3 complete sentenses only. ### Respond: """.format(disease=predicted_label) prompt_response = pg.Completion.create( model="Neural-Chat-7B", prompt=prompt )['choices'][0]['text'] is_healthy = predicted_class in [0, 2, 5, 6] if not is_healthy: sms_prompt = """### System: You are a SMS agent. Write an SMS message to inform a physician whose patient got identified a skin symptom with potential cancerous characteristics on our app DermaDetect. You will receive a input about the skin disease name of the patient. Share that information to the physician Keep the message clean and concise. No number list or bullet points, 3 complete sentenses only. Start with this and complete the paragraph: Hi, your patient {name} might have... ### User: A person was using a skin disease classifier, and it says he or she may have {disease}. Tell this to his or her physician. ### Respond: """.format(name="John Doe", disease=predicted_label) sms_message_prompt_response = pg.Completion.create( model="Neural-Chat-7B", prompt=sms_prompt )['choices'][0]['text'] conn = http.client.HTTPSConnection("l3vqgj.api.infobip.com") payload = json.dumps({ "messages": [ { "destinations": [{"to":"19174368930"}], "from": "Skin.ai", "text": sms_message_prompt_response } ] }) headers = { 'Authorization': f"{os.getenv('INFOBIP_API_PREFIX')} {os.getenv('INFOBIP_API_KEY')}", 'Content-Type': 'application/json', 'Accept': 'application/json' } conn.request("POST", "/sms/2/text/advanced", payload, headers) res = conn.getresponse() data = res.read() print(data.decode("utf-8")) return jsonify({'predicted_label': predicted_label, 'prompt_response': prompt_response, 'image_url': image_url, 'is_healthy': is_healthy}) def download_and_preprocess_image(image_url, image_size=(224, 224)): response = requests.get(image_url) img = cv2.imdecode(np.frombuffer(response.content, np.uint8), -1) if img is not None: img = cv2.resize(img, image_size) img_array = img_to_array(img) img_array = np.expand_dims(img_array, axis=0) return img_array else: print("Error downloading image") return None if __name__ == '__main__': app.run(debug=True) ``` ### treehacks2024/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.9.0. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### treehacks2024/postcss.config.js ```javascript export default { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` ### treehacks2024/vite.config.ts ```typescript import path from "path"; import react from "@vitejs/plugin-react"; import { defineConfig } from "vite"; // https://vitejs.dev/config/ export default defineConfig({ plugins: [react()], resolve: { alias: { "@": path.resolve(__dirname, "./src"), }, }, }); ``` ### treehacks2024/index.html ```html Convex + React (Vite)
``` ### treehacks2024/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { content: ["./index.html", "./src/**/*.{js,ts,jsx,tsx}"], theme: { container: { center: true, padding: "2rem", screens: { "2xl": "1400px", }, }, extend: { colors: { border: "hsl(var(--border))", input: "hsl(var(--input))", ring: "hsl(var(--ring))", background: "hsl(var(--background))", foreground: "hsl(var(--foreground))", primary: { DEFAULT: "hsl(var(--primary))", foreground: "hsl(var(--primary-foreground))", }, secondary: { DEFAULT: "hsl(var(--secondary))", foreground: "hsl(var(--secondary-foreground))", }, destructive: { DEFAULT: "hsl(var(--destructive))", foreground: "hsl(var(--destructive-foreground))", }, muted: { DEFAULT: "hsl(var(--muted))", foreground: "hsl(var(--muted-foreground))", }, accent: { DEFAULT: "hsl(var(--accent))", foreground: "hsl(var(--accent-foreground))", }, popover: { DEFAULT: "hsl(var(--popover))", foreground: "hsl(var(--popover-foreground))", }, card: { DEFAULT: "hsl(var(--card))", foreground: "hsl(var(--card-foreground))", }, }, borderRadius: { lg: "var(--radius)", md: "calc(var(--radius) - 2px)", sm: "calc(var(--radius) - 4px)", }, keyframes: { "accordion-down": { from: { height: 0 }, to: { height: "var(--radix-accordion-content-height)" }, }, "accordion-up": { from: { height: "var(--radix-accordion-content-height)" }, to: { height: 0 }, }, }, animation: { "accordion-down": "accordion-down 0.2s ease-out", "accordion-up": "accordion-up 0.2s ease-out", }, }, }, plugins: [require("tailwindcss-animate")], }; ``` [64 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]