# Project export: Style AI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 11.0
- Tagline: In recent years, online shopping has become quite popular. However, it's hard to tell how clothes look and fit on you. Style AI recommends a list of clothes and lets you try them on virtually.
- Devpost: https://devpost.com/software/style-ai
- GitHub: https://github.com/ericyucb/CalHacks-11.0
- Video: https://www.youtube.com/embed/2QxLqM3wMR8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — ethankou (6 commits), AJPL88 (5 commits)
## Devpost submission (written by the team)
### Inspiration
In recent years, especially post-COVID, online shopping has become extremely common. One big issue when shopping online is that users are unable to try on clothes before ordering them. This results in people getting clothes that end up not fitting or not looking great, which is something nobody wants. In addition, many people face constant difficulties in their life that limit their This gave us the inspiration to create Style AI as a way to let people try on clothes virtually before ordering them online.
### What it does
Style AI takes a photo of you and analyzes the clothes you are currently wearing and gives detailed clothing recommendations of specific brands, shirt types, and colors. Then, the user has the option to try on each of the recommendations virtually.
### How we built it
We used OpenCV to capture a photo of the user. Then the image is inputted to Gemini API to generate a list of clothing recommendations. These recommendations are then passed into google shopping API, which uses google search to find where the user can buy the recommended clothes. Then, we filter through the results to find clothes that have the correct image format. The image of the shirt is superimposed onto a live OpenCV video stream of the user. To overlay the shirt on the user, we segmented the shirt image into 3 sections: left sleeve, center, and right sleeve. We also perform segmentation on the user using MediaPipe. Then, we warp each segment of the shirt onto the user's body in the video stream. We made the website using Reflex.
### Challenges we ran into
The shirt overlay aspect was much more challenging than expected. At first, we planned to use a semantic segmentation model for the shirt of the user because then we could warp and transform the shape of the real shirt to the shirt mask on the user. The issue was that semantic segmentation was very slow so the shirt wasn't able to overlay on the user in real-time. We solved this by using a combination of various OpenCV functions so the shirt could be overlaid in real-time.
### Accomplishments we're proud of
We are proud of every part of our project, since each required lots of research, and we are all proud of the individual contributions to the project. We are also proud that we were able to overcome many challenges and adapt to things that went wrong. Specifically, we were proud that we were able to use a completely new framework, reflex, which allowed us to work in python natively across both the frontend and the backend.
### What we learned
We learned how to use Reflex to create websites. We also learned how to use APIs. Also, we learned about more functionalities of MediaPipe and OpenCV when writing the shirt overlay code.
### What's next
Expand Style AI for all types of clothing such as pants and shoes. Implementation of a "bulk order" functionality allowing users to order across online retailers. Add more personalized recommendations. Enable real-time voice assisted chat bot conversations to simulate talking to a fashion expert in-person.
## README (from the GitHub repository)
# CalHacks-11.0
## Detected evidence (automated analysis)
Indexed codebase: 14 recognized source files, 37 KB.
- Python (language) — detected in the code
## Codebase structure (from repository index)
### Files (22 of 22)
```
.DS_Store
.gitignore
.idea/.gitignore
.idea/CalHacks-11.0.iml
.idea/inspectionProfiles/profiles_settings.xml
.idea/inspectionProfiles/Project_Default.xml
.idea/misc.xml
.idea/modules.xml
.idea/vcs.xml
README.md
reflex_app/__init__.py
reflex_app/.gitignore
reflex_app/reflex_app/__init__.py
reflex_app/reflex_app/reflex_app.py
reflex_app/requirements.txt
reflex_app/rxconfig.py
reflex_app/shirt_fitter/__init__.py
reflex_app/shirt_fitter/.DS_Store
reflex_app/shirt_fitter/.env
reflex_app/shirt_fitter/gemini.py
reflex_app/shirt_fitter/mastercamera.py
segment_test.py
```
### Dependencies
- reflex_app/requirements.txt: reflex@==0.6.3
### Recent commits (newest first)
- it works
- added mastercamera.py
- master sword
- aura
- made gemini_run function
- try catch for gemini
- Merge branch 'main' of https://github.com/ericyucb/CalHacks-11.0
- commit
- shirt mesh fitting completed
- added reflex_app website
- updated gemini
- gemini working
- fixed shirt fitter
- add shirt
- hip and shoulder thing
- detects shoulder and arm segments
- first commit
## Key source files (fetched from GitHub, selected and truncated for size)
### reflex_app/requirements.txt
```
reflex==0.6.3
```
### segment_test.py
```python
# import torch
# from torchvision import models
# import cv2
# import numpy as np
# import torchvision.transforms as T
# # Load the pre-trained DeepLabV3 model for semantic segmentation
# model = models.segmentation.deeplabv3_resnet101(pretrained=True).eval()
# # Define the transformation to prepare the input image
# preprocess = T.Compose([
# T.ToPILImage(),
# T.Resize(256),
# T.ToTensor(),
# T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
# ])
# def decode_segmap(image, source_image, nc=21):
# # Define a color for each class (21 classes in the PASCAL VOC dataset)
# label_colors = np.array([(0, 0, 0), # 0=background
# (128, 0, 0), # 1=aeroplane
# (0, 128, 0), # 2=bicycle
# (128, 128, 0), # 3=bird
# (0, 0, 128), # 4=boat
# (128, 0, 128), # 5=bottle
# (0, 128, 128), # 6=bus
# (128, 128, 128), # 7=car
# (64, 0, 0), # 8=cat
# (192, 0, 0), # 9=chair
# (64, 128, 0), # 10=cow
# (192, 128, 0), # 11=dining table
# (64, 0, 128), # 12=dog
# (192, 0, 128), # 13=horse
# (64, 128, 128), # 14=motorbike
# (192, 128, 128), # 15=person (this is the torso class we are interested in)
# (0, 64, 0), # 16=potted plant
# (128, 64, 0), # 17=sheep
# (0, 192, 0), # 18=sofa
# (128, 192, 0), # 19=train
# (0, 64, 128)]) # 20=tv/monitor
# r = np.zeros_like(image).astype(np.uint8)
# g = np.zeros_like(image).astype(np.uint8)
# b = np.zeros_like(image).astype(np.uint8)
# for l in range(0, nc):
# idx = image == l
# if l < len(label_colors): # Avoid index out of bounds error
# r[idx] = label_colors[l, 0]
# g[idx] = label_colors[l, 1]
# b[idx] = label_colors[l, 2]
# rgb = np.stack([r, g, b], axis=2)
# return cv2.addWeighted(source_image, 0.6, rgb, 0.4, 0)
# cap = cv2.VideoCapture(0)
# while True:
# ret, frame = cap.read()
# if not ret:
# break
# # Preprocess the frame for the DeepLabV3 model
# input_image = preprocess(frame).unsqueeze(0)
# # Perform inference
# with torch.no_grad():
# output = model(input_image)['out'][0]
# output_predictions = output.argmax(0).byte().cpu().numpy()
# # Decode the output and overlay it on the original frame
# segmented_image = decode_segmap(output_predictions, frame)
# # Display the original frame and the segmented image
# cv2.imshow('Segmentation', segmented_image)
# if cv2.waitKey(1) & 0xFF == ord('q'):
# break
# cap.release()
# cv2.destroyAllWindows()
```
### reflex_app/rxconfig.py
```python
import reflex as rx
config = rx.Config(
app_name="reflex_app",
)
```
### .idea/vcs.xml
```xml
```
### .idea/misc.xml
```xml
```
### .idea/modules.xml
```xml
```
### .idea/inspectionProfiles/profiles_settings.xml
```xml
```
### reflex_app/shirt_fitter/gemini.py
```python
import os
import google.generativeai as genai
import cv2
import json
import time
import math
genai.configure(api_key='AIzaSyBDhYAAOLh8HNWXOsXZRXEomgH_jlZbZt4')
def upload_to_gemini(path, mime_type=None):
"""Uploads the given file to Gemini."""
file = genai.upload_file(path, mime_type=mime_type)
print(f"Uploaded file '{file.display_name}' as: {file.uri}")
return file
def gemini_run(model, frame):
img_path = 'shirt_fitter/gemini_images/test0.png'
cv2.imwrite(img_path, frame)
prompt_parts = [
upload_to_gemini(img_path, mime_type="image/png"),
"You are a fashion expert who gives fashion advice to people. Your job is to analyze a photo of me and give me 3 specific recommendations of t-shirts from various brands. Output the 3 clothes as a JSON dictionary that includes the brand, shirt type, and color. Please suggest darker colored clothing. Also, provide a concise and simple description of what shirt I am wearing. Reference me in 2nd person."
]
try:
response = model.generate_content(prompt_parts)
deez = json.loads(response.text)
description = deez['description']
recs_dict = deez['recommendations']
recs = [f'{n["brand"]} {n["color"]} {n["shirt_type"]}' for n in recs_dict]
return description, recs
except:
return None
def aura():
generation_config = {
"temperature": 0,
"top_p": 0.95,
"top_k": 40,
"max_output_tokens": 8192,
"response_mime_type": "application/json",
}
model = genai.GenerativeModel(
model_name="gemini-1.5-flash-8b",
generation_config=generation_config,
)
# Video capture loop
path = 'shirt_fitter/gemini_images'
cam = cv2.VideoCapture(0)
timer = 6.5
prev_time = time.perf_counter()
while True:
current_time = time.perf_counter()
dt = current_time - prev_time
timer -= dt
ret, frame = cam.read()
frame = cv2.resize(frame, (640, 360))
# Perform action after timer
if timer < 0:
output = gemini_run(model, frame)
if output:
return output
description, recs = output
print(description)
print(recs)
break
else:
return False
print('Try again')
# Display the countdown or "Capturing" message
if math.floor(timer) == 0:
text = 'Capturing!'
text_position = (50, 50)
font_size = 1.2
else:
frame = cv2.putText(frame, 'Get Ready...', (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1.2, (255, 255, 255), 3, cv2.LINE_AA)
text_position = (280, 200)
text = str(math.floor(timer))
font_size = 3
frame = cv2.putText(frame, text, text_position, cv2.FONT_HERSHEY_SIMPLEX, font_size,
(255, 255, 255), 3, cv2.LINE_AA)
cv2.imshow('frame', frame)
cv2.waitKey(1)
prev_time = current_time
cv2.destroyAllWindows()
```
### reflex_app/reflex_app/reflex_app.py
```python
# app = rx.App()
# app.add_page(index)
import reflex as rx
from typing import List, Dict
import requests
import os
import urllib.parse
import aiohttp
import asyncio
import cv2
from PIL import Image
from io import BytesIO
from shirt_fitter.mastercamera import generateSegmentation, master
from shirt_fitter.gemini import aura
search_url = "https://www.searchapi.io/api/v1/search"
# Define the state to hold form data and other UI state
class SearchState(rx.State):
merch: List[Dict[str,str]] = [{}]
loading:bool = False
error:bool = False
recommendations: str = []
description: str = ""
params = {
"engine": "google_shopping",
"q": "Uniqlo Red Airism Tee",
"gl": "us",
"location": "California,United States",
"num": "1",
"api_key": "hVgDDfo4YQNdBaet1KDpNq8G",
}
def try_on(self):
master("shirt_fitter/test_clothes/shirt0.png","shirt_fitter/test_clothes/shirt1.png","shirt_fitter/test_clothes/shirt2.png")
def create_aura(self):
output = aura()
if output:
description, recommendation = output
self.description = description
for i in range(3):
shirt_data = self.fetch_shirt_data(recommendation[i])
print("testing shirt_data returns ", shirt_data["shopping_results"])
if shirt_data:
individual_shirt = shirt_data["shopping_results"][0]
url = individual_shirt["thumbnail"]
img = requests.get(url)
if img.status_code == 200:
# Open a file in write-binary mode and save the image
img = Image.open(BytesIO(img.content))
img.save("shirt_fitter/test_clothes/shirt" + str(i)+".png")
generateSegmentation("shirt_fitter/test_clothes/shirt" + str(i)+".png")
print(f"Image saved as shirt"+str(i))
else:
print(f"Failed to retrieve image. Status code: {img.status_code}")
self.merch.append(individual_shirt)
else:
self.create_aura()
def fetch_shirt_data(self, shirt_name):
try:
self.params["q"] = shirt_name + " white background, shirt only"
response = requests.get(f"{search_url}?{urllib.parse.urlencode(self.params)}")
response.raise_for_status()
return response.json()
except requests.exceptions.RequestException as err:
print(err)
self.error = True
finally:
self.loading = False
def create_product_image(alt_text, image_src):
"""Create a product image with specific dimensions and styling."""
return rx.image(
src=image_src,
alt=alt_text,
height="16rem",
object_fit="cover",
width="100%",
)
def create_heading(text):
"""Create a styled heading with specific font properties."""
return rx.heading(
text,
font_weight="600",
color="#1F2937",
font_size="1.25rem",
line_height="1.75rem",
as_="h3",
)
def create_description_text(text):
"""Create a styled description text for products."""
return rx.text(
text, margin_top="0.5rem", color="#4B5563"
)
def create_price_and_cart_container(price):
"""Create a container with price and 'Add to Cart' button."""
return rx.flex(
create_price_text(price=price),
display="flex",
align_items="center",
justify_content="space-between",
margin_top="1rem",
)
def create_price_text(price):
"""Create a styled price text for products."""
return rx.text.span(
"$"+price,
font_weight="700",
color="#1F2937",
font_size="1.25rem",
line_height="1.75rem",
)
def create_product_details_box(title, description, price):
"""Create a box containing product details including title, description, and price."""
return rx.box(
create_heading(text=title),
create_description_text(text=description),
create_price_and_cart_container(price=price),
padding="1rem",
)
def get_product(product):
return rx.box(
create_product_image(
alt_text="No Shirts Yet",
image_src=product['thumbnail'],
),
create_product_details_box(
title=product['title'],
description="",
price=product['extracted_price'],
),
rx.el.button(
"Add to Cart",
background_color="#3B82F6",
_hover={"background-color": "#2563EB"},
padding_left="1rem",
padding_right="1rem",
padding_top="0.5rem",
padding_bottom="0.5rem",
border_radius="0.25rem",
color="#ffffff", on_click=rx.redirect(product["product_link"])
),
background_color="#ffffff",
overflow="hidden",
border_radius="0.5rem",
box_shadow="0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06)",
)
# return rx.vstack(
# rx.heading(
# product['title']
# ),
# rx.image(src=product['thumbnail'], alt=product['title'], width=350, height=350),
# rx.link("Buy ", product['product_link'])
# )
def create_upload_section():
"""Create the main upload section with heading, description, and form."""
return rx.box(
rx.heading(
"Get Personalized Fashion Recommendations",
font_weight="700",
margin_bottom="1.5rem",
font_size="1.875rem",
line_height="2.25rem",
color="#1E40A
[truncated — 1670 more characters]
```
### .idea/inspectionProfiles/Project_Default.xml
```xml