# Project export: StyleAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: StyleAI is a cutting-edge fashion assistant designed to revolutionize how you approach your wardrobe leveraging advanced computer vision and natural language processing. - Devpost: https://devpost.com/software/styleai - GitHub: https://github.com/ArvindVivek/style-ai - Team: 1 GitHub contributor(s) — Arvind Vivekanandan (10 commits) ## Devpost submission (written by the team) ### Inspiration The inspiration behind StyleAI stemmed from the desire to revolutionize the way we interact with fashion in the digital realm. We envisioned a platform that seamlessly integrates cutting-edge AI models for image recognition, language processing, and image generation to provide users with personalized and dynamic fashion recommendations. ### What it does StyleAI is an innovative platform that allows users to effortlessly manage their wardrobe and receive intelligent fashion suggestions. By leveraging Together.ai for natural language capabilities and Together.ai for image generation, users can interact conversationally to get outfit recommendations for various events. Additionally, StyleAI analyzes clothing items in images, identifies their color, and assists in creating versatile and stylish outfits. ### How we built it We employed a multi-faceted approach to build StyleAI. We integrated the Roboflow API for image segmentation, extracting Regions of Interest (ROIs) from clothing images. This data was then processed through a K-Means clustering algorithm to identify dominant colors. Utilizing Together.ai, we implemented natural language processing for seamless interaction. Together.ai's image generation capabilities were harnessed to dynamically visualize outfit suggestions. ### Challenges we ran into One of the primary challenges we faced was efficiently integrating the various components of StyleAI, ensuring smooth communication between image processing, language understanding, and image generation modules. Additionally, optimizing the workflow to handle a diverse range of fashion items and styles posed a significant technical hurdle. ### Accomplishments we're proud of We're immensely proud of creating a cohesive and intelligent platform that empowers users to effortlessly curate their wardrobe and receive personalized fashion advice. Achieving a seamless integration of image recognition, natural language processing, and image generation showcases the potential of AI-driven solutions in the fashion domain. ### What we learned Through the development of StyleAI, we gained valuable insights into the intricacies of combining image processing, language understanding, and image generation technologies. This project reinforced the potential of AI to revolutionize the fashion industry and enhance user experiences. ### What's next In the future, we envision expanding StyleAI's capabilities to include advanced features such as augmented reality try-ons, personalized style recommendations based on user preferences, and integration with e-commerce platforms for seamless shopping experiences. Additionally, we aim to refine the user interface for a more intuitive and engaging interaction. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 22 recognized source files, 53 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (31 of 31) ``` .DS_Store .gitattributes .gitignore client/.DS_Store client/README.md client/style-ai-client/.DS_Store client/style-ai-client/convex/_generated/api.d.ts client/style-ai-client/convex/_generated/api.js client/style-ai-client/convex/_generated/dataModel.d.ts client/style-ai-client/convex/_generated/server.d.ts client/style-ai-client/convex/_generated/server.js client/style-ai-client/convex/init.ts client/style-ai-client/convex/messages.ts client/style-ai-client/convex/README.md client/style-ai-client/convex/schema.ts client/style-ai-client/convex/tsconfig.json client/style-ai-client/index.html client/style-ai-client/LICENSE client/style-ai-client/package.json client/style-ai-client/README.md client/style-ai-client/src/App.tsx client/style-ai-client/src/index.css client/style-ai-client/src/main.tsx client/style-ai-client/src/vite-env.d.ts client/style-ai-client/tsconfig.json client/style-ai-client/vite.config.ts README.md server/cv_model.py server/llm.py server/playground.py server/README.md ``` ### Dependencies - client/style-ai-client/package.json: @faker-js/faker@^8.0.2, @types/babel__core@^7.20.0, @types/node@^16.11.12, @types/react@^18.0.0, @types/react-dom@^18.0.0, @vitejs/plugin-react@3.1.0, convex@1.5.1, npm-run-all@^4.1.5, react@^17.0.2, react-dom@^17.0.2, typescript@~5.0.3, vite@^4.4.2 ### Recent commits (newest first) - restructure - client init - removed package locks - Merge pull request #1 from jcamel2/main - just the beginning - llm init - class and color - init color classification - cv model init - init - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### client/style-ai-client/package.json ``` { "name": "convex-tour-chat-0", "version": "0.0.0", "scripts": { "dev": "npm-run-all dev:init --parallel dev:server dev:client", "build": "tsc && vite build", "dev:server": "convex dev", "dev:client": "vite --open", "dev:init": "convex dev --run init --until-success" }, "dependencies": { "@faker-js/faker": "^8.0.2", "convex": "1.5.1", "react": "^17.0.2", "react-dom": "^17.0.2" }, "devDependencies": { "@types/babel__core": "^7.20.0", "@types/node": "^16.11.12", "@types/react": "^18.0.0", "@types/react-dom": "^18.0.0", "@vitejs/plugin-react": "3.1.0", "npm-run-all": "^4.1.5", "typescript": "~5.0.3", "vite": "^4.4.2" } } ``` ### client/style-ai-client/src/main.tsx ```typescript import { StrictMode } from "react"; import ReactDOM from "react-dom"; import "./index.css"; import App from "./App"; import { ConvexProvider, ConvexReactClient } from "convex/react"; const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL); ReactDOM.render( , document.getElementById("root") ); ``` ### client/style-ai-client/src/App.tsx ```typescript import { useQuery, useMutation } from "convex/react"; import { api } from "../convex/_generated/api"; import { useEffect, useState } from "react"; import { faker } from "@faker-js/faker"; // For demo purposes. In a real app, you'd have real user data. const NAME = faker.person.firstName(); export default function App() { const messages = useQuery(api.messages.list); const sendMessage = useMutation(api.messages.send); const [newMessageText, setNewMessageText] = useState(""); useEffect(() => { // Make sure scrollTo works on button click in Chrome setTimeout(() => { window.scrollTo({ top: document.body.scrollHeight, behavior: "smooth" }); }, 0); }, [messages]); return (

Style AI: Fashion Finder

Connected as {NAME}

{messages?.map((message) => (
{message.author}

{message.body}

))}
{ e.preventDefault(); await sendMessage({ body: newMessageText, author: NAME }); setNewMessageText(""); }} > { const text = e.target.value; setNewMessageText(text); }} placeholder="Write a message…" />
); } ``` ### client/style-ai-client/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.5.1. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### server/llm.py ```python import together import os from dotenv import load_dotenv load_dotenv() together.api_key = os.getenv("TOGETHER_API_KEY") model_list = together.Models.list() model_name = "togethercomputer/RedPajama-INCITE-7B-Instruct" output = together.Complete.create( prompt=": What are Isaac Asimov's Three Laws of Robotics?\n:", model=model_name, max_tokens=256, temperature=0.8, top_k=60, top_p=0.6, repetition_penalty=1.1, stop=["", "\n\n"], ) # print generated text print(output["prompt"][0] + output["output"]["choices"][0]["text"]) ``` ### server/cv_model.py ```python from roboflow import Roboflow import cv2 import numpy as np from PIL import Image, ImageDraw import colorgram from io import BytesIO import os from sklearn.cluster import KMeans from dotenv import load_dotenv import supervision as sv from typing import Tuple from lavis.models import load_model_and_preprocess import torch # Load environment variables load_dotenv() ROBOFLOW_API_KEY = os.getenv("ROBOFLOW_API_KEY") # Initialize Roboflow # Model Credits: https://universe.roboflow.com/roboflow-jvuqo/fashion-assistant-segmentation rf = Roboflow(api_key=ROBOFLOW_API_KEY) project = rf.workspace().project("fashion-assistant-segmentation") model = project.version(5).model DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") PROMPT = "What is the style of this clothing?" model2, vis_processors, txt_processors = load_model_and_preprocess( name="blip_vqa", model_type="vqav2" ) prompt_processed = txt_processors["eval"](PROMPT) images_numpy = [] predictions = [] def load_image(image_path: str) -> Tuple[PIL.Image.Image, np.ndarray]: image_pil = Image.open(image_path).convert("RGB") image_numpy = np.asarray(image_pil) image_numpy = cv2.cvtColor(image_numpy, cv2.COLOR_RGB2BGR) return image_pil, image_numpy # Function to extract ROI def extract_roi(data, img_path): # Get the points for the ROI points = data["predictions"][0]["points"] # Create an empty mask mask = Image.new( "L", (int(data["predictions"][0]["width"]), int(data["predictions"][0]["height"])), 0, ) draw = ImageDraw.Draw(mask) # Convert points to tuples points = [(int(point["x"]), int(point["y"])) for point in points] # Draw polygon on the mask draw.polygon(points, fill=255) # Get the bounding box of the mask bbox = mask.getbbox() # Crop the original image using the bounding box roi = Image.open(img_path) # Replace with the path to your original image roi = roi.crop(bbox) return roi # Function to get dominant color def get_dominant_color(image): colors = colorgram.extract(image, 1) dominant_color = colors[0].rgb return dominant_color # Infer on local images and iterate through data folder for image in os.listdir("data/subset"): image_path = f"data/subset/{image}" prediction = model.predict(f"data/subset/{image}").json() class_value = prediction["predictions"][0]["class"] roi = extract_roi(prediction, f"data/subset/{image}") # Get the dominant color dominant_color = get_dominant_color(roi) # Combine class and RGB color result = f"Class: {class_value}, Color: RGB({dominant_color[0]}, {dominant_color[1]}, {dominant_color[2]})" image_pil, image_numpy = load_image(image_path=str(image_path)) image_processed = vis_processors["eval"](image_pil).unsqueeze(0).to(DEVICE) prediction2 = model2.predict_answers( samples={"image": image_processed, "text_input": prompt_processed}, inference_method="generate", )[0] images_numpy.append(image_numpy) predictions.append(prediction2) print(result, prediction2) ``` ### client/style-ai-client/vite.config.ts ```typescript import { defineConfig } from "vite"; import react from "@vitejs/plugin-react"; // https://vitejs.dev/config/ export default defineConfig({ plugins: [react()], build: { chunkSizeWarningLimit: 10000, }, }); ``` ### client/style-ai-client/index.html ```html Convex Chat
``` ### server/playground.py ```python import cv2 import numpy as np from PIL import Image, ImageDraw import colorgram from io import BytesIO import os from sklearn.cluster import KMeans img_path = f"data/subset/photo_2023-05-29-19-51-04_jpeg.rf.da18962278da1f32caa5cf1d237f5309.jpg" # Function to extract ROI def extract_roi(data): # Get the points for the ROI points = data["predictions"][0]["points"] # Create an empty mask mask = Image.new( "L", (int(data["predictions"][0]["width"]), int(data["predictions"][0]["height"])), 0, ) draw = ImageDraw.Draw(mask) # Convert points to tuples points = [(int(point["x"]), int(point["y"])) for point in points] # Draw polygon on the mask draw.polygon(points, fill=255) # Get the bounding box of the mask bbox = mask.getbbox() # Crop the original image using the bounding box roi = Image.open(img_path) # Replace with the path to your original image roi = roi.crop(bbox) return roi # Function to get dominant color def get_dominant_color(image): colors = colorgram.extract(image, 1) dominant_color = colors[0].rgb return dominant_color # Usage segmentation_data = { "predictions": [ { "x": 741.0, "y": 675.0, "width": 572.0, "height": 684.0, "confidence": 0.988563060760498, "class": "sneaker", "points": [ {"x": 958.0, "y": 334.0}, {"x": 952.0, "y": 340.0}, {"x": 952.0, "y": 348.0}, {"x": 948.0, "y": 352.0}, {"x": 946.0, "y": 352.0}, {"x": 944.0, "y": 354.0}, {"x": 944.0, "y": 356.0}, {"x": 938.0, "y": 362.0}, {"x": 936.0, "y": 360.0}, {"x": 926.0, "y": 360.0}, {"x": 924.0, "y": 358.0}, {"x": 922.0, "y": 358.0}, {"x": 920.0, "y": 356.0}, {"x": 920.0, "y": 354.0}, {"x": 918.0, "y": 354.0}, {"x": 916.0, "y": 352.0}, {"x": 914.0, "y": 352.0}, {"x": 910.0, "y": 348.0}, {"x": 910.0, "y": 346.0}, {"x": 908.0, "y": 344.0}, {"x": 906.0, "y": 344.0}, {"x": 902.0, "y": 340.0}, {"x": 902.0, "y": 338.0}, {"x": 900.0, "y": 336.0}, {"x": 884.0, "y": 336.0}, {"x": 880.0, "y": 340.0}, {"x": 880.0, "y": 356.0}, {"x": 876.0, "y": 360.0}, {"x": 866.0, "y": 360.0}, {"x": 860.0, "y": 354.0}, {"x": 856.0, "y": 354.0}, {"x": 854.0, "y": 352.0}, {"x": 810.0, "y": 352.0}, {"x": 808.0, "y": 354.0}, {"x": 804.0, "y": 354.0}, {"x": 800.0, "y": 358.0}, {"x": 798.0, "y": 358.0}, {"x": 796.0, "y": 360.0}, {"x": 790.0, "y": 360.0}, {"x": 788.0, "y": 362.0}, {"x": 786.0, "y": 362.0}, {"x": 780.0, "y": 368.0}, {"x": 778.0, "y": 368.0}, {"x": 770.0, "y": 376.0}, {"x": 766.0, "y": 376.0}, {"x": 764.0, "y": 378.0}, {"x": 762.0, "y": 378.0}, {"x": 756.0, "y": 384.0}, {"x": 754.0, "y": 384.0}, {"x": 748.0, "y": 390.0}, {"x": 746.0, "y": 390.0}, {"x": 742.0, "y": 394.0}, {"x": 740.0, "y": 394.0}, {"x": 732.0, "y": 402.0}, {"x": 730.0, "y": 402.0}, {"x": 724.0, "y": 408.0}, {"x": 722.0, "y": 408.0}, {"x": 720.0, "y": 410.0}, {"x": 720.0, "y": 412.0}, {"x": 716.0, "y": 416.0}, {"x": 714.0, "y": 416.0}, {"x": 710.0, "y": 420.0}, {"x": 708.0, "y": 420.0}, {"x": 696.0, "y": 432.0}, {"x": 694.0, "y": 432.0}, {"x": 692.0, "y": 434.0}, {"x": 690.0, "y": 434.0}, {"x": 690.0, "y": 436.0}, {"x": 672.0, "y": 454.0}, {"x": 672.0, "y": 456.0}, {"x": 664.0, "y": 464.0}, {"x": 664.0, "y": 466.0}, {"x": 656.0, "y": 474.0}, {"x": 656.0, "y": 480.0}, {"x": 654.0, "y": 482.0}, {"x": 654.0, "y": 486.0}, {"x": 652.0, "y": 488.0}, {"x": 652.0, "y": 490.0}, {"x": 650.0, "y": 492.0}, {"x": 650.0, "y": 502.0}, {"x": 654.0, "y": 506.0}, {"x": 654.0, "y": 508.0}, {"x": 656.0, "y": 510.0}, {"x": 656.0, "y": 514.0}, {"x": 658.0, "y": 516.0}, {"x": 658.0, "y": 518.0}, {"x": 660.0, "y": 518.0}, {"x": 664.0, "y": 522.0}, {"x": 664.0, "y": 524.0}, {"x": 666.0, "y": 526.0}, {"x": 666.0, "y": 528.0}, {"x": 668.0, "y": 530.0}, {"x": 670.0, "y": 530.0}, {"x": 672.0, "y": 532.0}, {"x": 674.0, "y": 532.0}, {"x": 682.0, "y": 540.0}, {"x": 682.0, "y": 550.0}, {"x": 680.0, "y": 552.0}, {"x": 680.0, "y": 556.0}, {"x": 678.0, "y": 558.0}, {"x": 678.0, "y": 560.0}, {"x": 674.0, "y": 564.0}, {"x": 674.0, "y": 568.0}, {"x": 672.0, "y": 570.0}, {"x": 672.0, "y": 574.0}, {"x": 670.0, "y": 576.0}, {"x": 670.0, "y": 580.0}, {"x": 668.0, "y": 582.0}, {"x": 668.0, "y": 598.0}, [truncated — 16082 more characters] ``` ### client/style-ai-client/src/vite-env.d.ts ```typescript /// ``` [8 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]