# Project export: AfroVision

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2025
- Tagline: Zoom's virtual backgrounds often distort or cut off curly textured and afrohair due to biased segmentationmodels. AfroVision improves segmentation to ensure better representation across all hairtypes.
- Devpost: https://devpost.com/software/afrovision
- GitHub: https://github.com/Pamelachristina/AfroVision
- Team: 1 GitHub contributor(s) — Pamelachristina (2 commits)

## Devpost submission (written by the team)

### Inspiration

This project is deeply personal for me. As a software engineer apprentice at Sony Interactive Entertainment, I often join professional meetings on Zoom and Microsoft Teams. However, I frequently leave my camera off—not out of preference, but because Zoom’s virtual backgrounds fail to represent me properly. My curly, textured hair is often distorted, cut off, or blends unnaturally, making me feel misrepresented and self-conscious on screen. This isn't just my experience; many Black professionals and individuals with curly or afro-textured hair face the same issue. Existing segmentation models are primarily trained on straight and wavy hair, leaving our textures as an afterthought. AfroVision aims to fix this by improving hair segmentation for virtual backgrounds, ensuring people with all hair types are represented accurately and equitably.

### What it does

AfroVision enhances virtual background rendering by improving the segmentation of curly, coily, and afro-textured hair. We implemented DeepLabV3 with ResNet101 to segment hair while testing how different models handle diverse textures. Our system processes images, generates segmentation masks, and seamlessly blends the subject with a virtual background.

### How we built it

Data Collection: We sourced datasets focusing on curly, wavy, and afro-textured hair, scraping additional images to ensure diversity. Segmentation Model: Used DeepLabV3-ResNet101 for semantic segmentation. Image Processing: Preprocessed images, normalized data, and resized masks for improved accuracy. Testing & Evaluation: Applied segmentation to real-world images to assess effectiveness on different hair textures.

### Challenges we ran into

Dataset Limitations: Many publicly available datasets lack representation of diverse hair textures. We had to scrape and manually verify images to supplement the dataset. Google Drive Issues: Encountered technical difficulties accessing and managing data stored in Google Drive, slowing down our workflow. Processing Errors: Debugging file path errors and refining image preprocessing took significant time. Time Constraints: With a limited window, fine-tuning on afro-textured hair remains a next step beyond this hackathon.

### Accomplishments we're proud of

Successfully segmented multiple curly hair textures and demonstrated virtual background blending. Created a working pipeline for future fine-tuning and improvement. Identified key areas where segmentation models fall short for Black and curly-haired users, setting a foundation for addressing this bias.

### What we learned

Importance of Representation in AI: Bias in training data leads to biased models. We saw firsthand how standard segmentation models struggle with diverse hair types. Fine-Tuning is Essential: Pre-trained models work to an extent but must be customized to better capture curls and coils. Efficient Debugging Matters: Managing large datasets and troubleshooting Google Drive path issues is a skill in itself.

### What's next

Integrate with Zoom SDK: Implement AfroVision into Zoom's virtual background system to enable real-time segmentation improvements. Fine-Tune on Afro Hair Data: We need to train DeepLabV3 further with more labeled images of afro-textured hair. Expand Dataset & Annotation: Manually annotate and generate higher-quality segmentation masks for kinky, coily, and tightly curled hair. Improve Real-Time Performance: Optimize processing speed for live applications like Zoom and Google Meet. Open Source Contribution: Publish the dataset and model improvements to benefit other developers tackling bias in computer vision. Fine-Tune on Afro Hair Data: We need to train DeepLabV3 further with more labeled images of afro-textured hair. Expand Dataset & Annotation: Manually annotate and generate higher-quality segmentation masks for kinky, coily, and tightly curled hair. Improve Real-Time Performance: Optimize processing speed for live applications like Zoom and Google Meet. Open Source Contribution: Publish the dataset and model improvements to benefit other developers tackling bias in computer vision.

## README (from the GitHub repository)

## AfroVision
Improving Hair Segmentation for Curly and Afro-Textured Hair

## Inspiration
Virtual backgrounds often distort or cut off curly, textured, and afro hair due to biased segmentation models trained mostly on straight and wavy hair. AfroVision enhances segmentation accuracy to ensure better representation across all hair types.

## What It Does
AfroVision improves virtual background rendering by enhancing segmentation for curly, coily, and afro-textured hair. We implemented DeepLabV3 with ResNet101 to refine hair detection and blending in virtual settings.

## How It Was Built
Data Collection: Curated datasets featuring diverse hair textures, supplemented by scraping additional images.
Segmentation Model: Used DeepLabV3-ResNet101 for precise hair segmentation.
Preprocessing & Augmentation: Normalized images and resized masks for consistency.
Testing & Evaluation: Applied segmentation to real-world images, analyzing effectiveness across different hair textures.

## Challenges
Dataset Gaps: Most datasets lack adequate representation of Black and curly hair.
File Management Issues: Google Drive access problems slowed workflow.
Processing Errors: Debugging segmentation outputs and image masks took time.
Time Constraints: While functional, further fine-tuning is necessary for real-time applications.

## Accomplishments
Built a working segmentation model tailored for diverse hair textures.
Demonstrated real-world improvements in virtual background blending.
Identified biases in existing models and laid the foundation for future fine-tuning.
What We Learned
AI Bias Affects Real-World Applications: Standard models struggle with afro-textured hair due to limited training data.
Fine-Tuning is Key: Pre-trained models help, but domain-specific data is essential.
Efficient Debugging Saves Time: Managing datasets, handling file paths, and optimizing preprocessing are crucial.

## Next Steps
🚀 Integrate with Zoom SDK for real-time segmentation
📊 Fine-tune on afro-textured hair datasets
📂 Expand dataset & annotation for better model generalization
⚡ Optimize performance for real-time applications
🛠 Open-source AfroVision to help address bias in AI



## Detected evidence (automated analysis)

Indexed codebase: 2 recognized source files, 5 KB.
- Python (language) — detected in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (9 of 9)

```
.DS_Store
.gitignore
initial_segmentation_model.ipynb
notebooks/.ipynb_checkpoints/Untitled-checkpoint.ipynb
notebooks/Untitled.ipynb
README.md
requirements.txt
segmentation.ipynb
test.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Removed MODNet from direct tracking
- Created using Colab
- Created using Colab
- Removed large files and ignored venv
- Final setup before TreeHacks
- Initial commit - project setup
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### test.py

```python
import torch
from torchvision import models, transforms
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt

# Load the pre-trained DeepLabV3 model
model = models.segmentation.deeplabv3_resnet101(pretrained=True)
model.eval()  # Set to evaluation mode

# Image Preprocessing
def preprocess_image(image_path):
    image = Image.open(image_path).convert("RGB")
    
    preprocess = transforms.Compose([
        transforms.ToTensor(),  # Convert image to tensor
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),  # Normalize with ImageNet stats
    ])
    
    image_tensor = preprocess(image).unsqueeze(0)  # Add batch dimension
    return image_tensor

# Run the model on an image
def segment_image(image_path):
    input_image = preprocess_image(image_path)
    
    with torch.no_grad():
        output = model(input_image)
    
    # Extract the segmentation mask
    segmentation_mask = output['out'][0]  # First item in the batch
    segmentation_mask = torch.argmax(segmentation_mask, dim=0)  # Find the most likely class for each pixel
    
    return segmentation_mask

# Segment the image and replace the background
def replace_background(image_path, segmentation_mask, background_path):
    # Convert segmentation mask to binary mask for "person" class (class 15)
    binary_mask = np.where(segmentation_mask == 15, 1, 0)  # 1 for "person", 0 for background

    # Replace the background with a virtual background (ensure background is the same size)
    background_image = Image.open(background_path).resize((segmentation_mask.shape[1], segmentation_mask.shape[0]))

    # Convert image and background to numpy arrays
    foreground = np.array(Image.open(image_path).convert("RGBA"))
    background = np.array(background_image.convert("RGBA"))

    # Combine the foreground (person) with the background using the binary mask
    final_image = foreground * binary_mask[..., None] + background * (1 - binary_mask[..., None])

    # Convert back to an image
    final_image = Image.fromarray(final_image.astype(np.uint8))

    return final_image

# Example usage
image_path = "/Users/pamelasanchezhernandez/AfroVision/sample.jpg"  # Update this path
 
background_path = "/Users/pamelasanchezhernandez/AfroVision/price_is_right.jpg" # Update to the actual background path

# Step 1: Segment the image
segmentation_mask = segment_image(image_path)

# Step 2: Replace background using the segmentation mask
final_image = replace_background(image_path, segmentation_mask, background_path)

# Show the final result
final_image.show()







```