# Project export: AI UI/UX Design Assistant This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Accessibility is often overlooked in design. This AI Assistant guides UI designers/developers to fix issues like missing alt text, inconsistent fonts, and poor color contrast for better UX - Devpost: https://devpost.com/software/ai-ui-ux-design-assistant - GitHub: https://github.com/sjswee31/floating-design-assistant - Team: 1 GitHub contributor(s) โ€” Sarah Swee (3 commits) ## Devpost submission (written by the team) ### Inspiration Experience working as a Accessibility/Search Engine Optimization Specialist for Web Design. I was tasked to fix alternative text and text hierarchy and realized that old websites did not have accessibility top of mind. I knew they were issues for people who were low vision/blind, deaf, or had a hard time navigating websites that are not made with accessibility in mind. ### What it does The goal of this AI Design Assistant intends on giving advice to UI/UX designers on what to consider during their website building phase. ### How we built it I built a chrome extension that is connected to a Gemini API for a LLM to take screenshots of parts of the website to give advice on what needs improvement, following WCAG Requirements, UX Critique with the POV of a senior designer, and branding auditing. ### Challenges we ran into Had difficulty figuring out how I could get the AI agent to parse through the html. I would have needed more experience in optimization and parallel computing and API credits. I tried parsing through html, but the queries ran very long and did not load. The goal of parshing would have identified images that did not have associated alt or aria-labels needed for screen readers to identify an image on a webpage. However, for future direction, I believe it could be possible to expand upon this. ### Accomplishments we're proud of Building my first AI Agent ever with the goal of accessibility in mind. ### What we learned The importance of UI, thinking from the perspective of both the user and the designer. ### What's next Maybe expanding by adding screen reader features to help assist blind users! ## README (from the GitHub repository) # UI/Web Accessibility Agent - Chrome Extension An AI-powered Chrome extension that provides real-time design and accessibility feedback on any webpage using Google's Gemini API. ## Features - ๐ŸŽจ **Floating Assistant**: Draggable, resizable UI that stays on top of any webpage - ๐Ÿ” **Three Analysis Modes**: - **Accessibility (WCAG Focus)**: Detailed accessibility audit with WCAG 2.1 guidelines - **UX Critique**: User experience evaluation and recommendations - **Branding Audit**: Brand identity and visual consistency analysis - ๐Ÿ’ฌ **Custom Queries**: Ask specific questions about the webpage design - ๐Ÿ“ฑ **Responsive Design**: Works on any screen size - โšก **Real-time Analysis**: Instant feedback using Gemini Vision API ## Setup Instructions ### 1. Prerequisites - Python 3.8 or higher - Google Chrome browser - Gemini API key from [Google AI Studio](https://makersuite.google.com/app/apikey) ### 2. Install Dependencies ```bash cd server pip install -r requirements.txt ``` ### 3. Set Up API Key ```bash export GEMINI_API_KEY="your-gemini-api-key-here" ``` Or create a `.env` file in the server directory: ``` GEMINI_API_KEY=your-gemini-api-key-here ``` ### 4. Start the Server ```bash cd server python server.py ``` The server will start on `http://localhost:5001` ### 5. Load the Chrome Extension 1. Open Chrome and go to `chrome://extensions/` 2. Enable "Developer mode" (toggle in top right) 3. Click "Load unpacked" 4. Select the `chrome-extension` folder from this project 5. The extension should now appear in your extensions list ### 6. Test the Setup Run the test script to verify everything is working: ```bash python test_extension.py ``` ## Usage 1. **Load the Extension**: After loading the extension, visit any website 2. **Floating Assistant Appears**: A floating assistant will appear in the top-right corner 3. **Choose Analysis Mode**: Select from the dropdown: - **Accessibility**: Get WCAG 2.1 compliance feedback - **UX Critique**: User experience recommendations - **Branding Audit**: Brand identity analysis 4. **Get Feedback**: Click "Get Design Suggestion" to analyze the current page 5. **Custom Questions**: Use the text area to ask specific questions about the design 6. **Drag & Resize**: Move the assistant around and resize it as needed ## Analysis Modes ### Accessibility (WCAG Focus) Provides detailed accessibility audit including: - Color contrast issues - Keyboard navigation problems - Screen reader compatibility - ARIA attribute recommendations - HTML/CSS implementation details - WCAG 2.1 guideline references ### UX Critique Evaluates user experience aspects: - Layout consistency - Navigation flow - Interactive elements - Visual appeal - Mobile responsiveness ### Branding Audit Analyzes brand identity: - Color scheme consistency - Typography hierarchy - Visual elements - Brand voice and tone - Recognition and memorability ## File Structure ``` slide_accessibility_checker/ โ”œโ”€โ”€ chrome-extension/ โ”‚ โ”œโ”€โ”€ manifest.json # Extension configuration โ”‚ โ”œโ”€โ”€ background.js # Background service worker โ”‚ โ”œโ”€โ”€ assistant.js # Content script for floating UI โ”‚ โ”œโ”€โ”€ assistant.css # Styles (if separate) โ”‚ โ”œโ”€โ”€ assistant.html # HTML template (if separate) โ”‚ โ””โ”€โ”€ icon.png # Extension icon โ”œโ”€โ”€ server/ โ”‚ โ”œโ”€โ”€ server.py # Flask server with Gemini API โ”‚ โ”œโ”€โ”€ prompts.py # AI prompts for different modes โ”‚ โ”œโ”€โ”€ requirements.txt # Python dependencies โ”‚ โ””โ”€โ”€ logging_config.py # Logging configuration โ”œโ”€โ”€ test_extension.py # Setup verification script โ””โ”€โ”€ README.md # This file ``` ## Troubleshooting ### Extension Not Loading - Check that all files are in the `chrome-extension` folder - Verify `manifest.json` is valid JSON - Check Chrome's extension page for error messages ### Server Connection Issues - Ensure the server is running on port 5001 - Check firewall settings - Verify the API key is set correctly ### API Errors - Check your Gemini API key is valid - Verify you have sufficient API credits - Check the server logs for detailed error messages ### Floating Assistant Not Appearing - Refresh the webpage after loading the extension - Check browser console for JavaScript errors - Ensure the extension is enabled ## API Usage and Credits The extension uses Google's Gemini Pro Vision API. To check your API usage: 1. Go to [Google AI Studio](https://makersuite.google.com/app/apikey) 2. Check your usage dashboard 3. Monitor your quota and billing ## Development ### Adding New Analysis Modes 1. Add the mode to `server/prompts.py` 2. Update the dropdown in `chrome-extension/assistant.js` 3. Test the new mode ### Customizing Prompts Edit `server/prompts.py` to modify the AI prompts for each analysis mode. ### Styling Changes Modify the CSS in `chrome-extension/assistant.js` to change the appearance of the floating assistant. ## Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes 4. Test thoroughly 5. Submit a pull request ## License This project is open source and available under the MIT License. ## Support If you encounter issues: 1. Check the troubleshooting section above 2. Run `python test_extension.py` to diagnose problems 3. Check the server logs for detailed error messages 4. Open an issue on the repository with detailed information ## Option 1: Chrome Web Store (Recommended for Public Sharing) ### 1. Prepare Your Extension: ```bash # Create a ZIP file of your chrome-extension folder cd chrome-extension zip -r ../design-assistant-extension.zip * ``` ### 2. Chrome Web Store Process: 1. **Go to**: [Chrome Web Store Developer Dashboard](https://chrome.google.com/webstore/devconsole/) 2. **Sign in** with your Google account 3. **Pay one-time fee**: $5 USD registration fee 4. **Upload your ZIP file** 5. **Fill out store listing**: - Extension name: "Floating Design Assistant" - Description: "AI-powered design and accessibility feedback tool" - Screenshots/videos of your extension in action - Privacy policy (required) ### 3. Store Listing Requirements: - **High-quality screenshots** (1280x800px) - **Detailed description** of features - **Privacy policy** (since you're using AI APIs) - **Clear usage instructions** ## Option 2: Direct Sharing (For Developers/Technical Users) ### 1. Create a GitHub Repository: ```bash # Initialize git repository git init git add . git commit -m "Initial commit: Floating Design Assistant Chrome Extension" # Create GitHub repo and push # (You'll need to create the repo on GitHub first) git remote add origin https://github.com/yourusername/design-assistant-extension.git git push -u origin main ``` ### 2. Create Installation Instructions: Create a `README.md` with: ```markdown # Floating Design Assistant Chrome Extension ## Installation (Developer Mode): 1. Download this repository 2. Open Chrome โ†’ chrome://extensions/ 3. Enable "Developer mode" 4. Click "Load unpacked" 5. Select the `chrome-extension` folder 6. Set up the server (see server/README.md) ``` ## Option 3: Self-Hosted Distribution ### 1. Package for Distribution: ```bash # Create a distribution package mkdir distribution cp -r chrome-extension distribution/ cp -r server distribution/ cp README.md distribution/ cp test_extension.py distribution/ cp check_api_credits.py distribution/ # Create installation script echo '#!/bin/bash echo "Installing Floating Design Assistant..." cd server pip install -r requirements.txt echo "Installation complete! See README.md for setup instructions." ' > distribution/install.sh chmod +x distribution/install.sh ``` ### 2. Share the Package: - Upload to Google Drive, Dropbox, or similar - Share the download link - Include setup instructions ## ๐Ÿ“‹ Important Considerations: ### **Privacy & Security:** - **API Key Management**: Users need their own Gemini API key - **Privacy Policy**: Required for Chrome Web Store - **Data Handlin [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 10 recognized source files, 46 KB. - CSS (language) โ€” detected in the code - Flask (technology) โ€” detected in the code - HTML (language) โ€” detected in the code - JavaScript (language) โ€” detected in the code - Python (language) โ€” detected in the code ## Codebase structure (from repository index) ### Files (12 of 12) ``` assistant.css assistant.html assistant.js background.js check_api_credits.py logging_config.py manifest.json prompts.py README.md requirements.txt server.py test_extension.py ``` ### Dependencies - requirements.txt: flask@==3.0.0, flask-cors@==4.0.0, google-generativeai@==0.3.2, Pillow@==10.0.0, python-dotenv@==1.0.0 ### Recent commits (newest first) - README.md - Rename floating-design-assistant.zip to floating-zip/floating-design-assistant.zip - Add files via upload - Initial commit: Design Assistant Chrome Extension ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` flask==3.0.0 flask-cors==4.0.0 google-generativeai==0.3.2 Pillow==10.0.0 python-dotenv==1.0.0 ``` ### server.py ```python # === server.py === from flask import Flask, request, jsonify from flask_cors import CORS import base64 from PIL import Image import io import time import os import logging from logging_config import logger, log_request, log_response, log_error from prompts import PROMPTS # Configuration SECTION_WIDTH = 800 # Width of each section in pixels SECTION_HEIGHT = 600 # Height of each section in pixels MAX_SECTIONS_PER_REQUEST = 5 # Number of sections to process at once REQUEST_TIMEOUT = 300 # 5 minutes MAX_IMAGE_SIZE = 20 * 1024 * 1024 # 20MB max image size # Caching setup cache = {} # Simple in-memory cache CACHE_TTL = 300 # Cache entries expire after 5 minutes app = Flask(__name__) CORS(app) # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Initialize Google Gemini API try: # Try loading from .env file first try: from dotenv import load_dotenv load_dotenv() except ImportError: logger.warning("python-dotenv not installed, falling back to environment variables") api_key = os.environ.get("GEMINI_API_KEY") if not api_key: raise ValueError("GEMINI_API_KEY not found. Please set it in your environment or .env file") import google.generativeai as genai genai.configure(api_key=api_key) model = genai.GenerativeModel('gemini-1.5-pro') logger.info("Successfully initialized Gemini API") except Exception as e: log_error(f"Failed to initialize Gemini API: {str(e)}") raise def validate_image_size(image_data): """Validate that an image is within allowed size limits.""" if len(image_data) > MAX_IMAGE_SIZE: raise ValueError(f"Image too large: {len(image_data)} bytes > {MAX_IMAGE_SIZE} bytes") return image_data def compress_image_section(image_data, section_id): """Optimize image compression for a specific section.""" try: if not image_data.startswith('data:image/'): raise ValueError("Invalid image format") # Use cache if available cache_key = f"section_{section_id}" if cache_key in cache: cached = cache[cache_key] if time.time() - cached['timestamp'] < CACHE_TTL: return cached['data'] # Decode and process image image = Image.open(io.BytesIO(base64.b64decode(image_data.split(",")[1]))) # Resize to optimal dimensions image = image.resize((SECTION_WIDTH, SECTION_HEIGHT), Image.Resampling.LANCZOS) # Convert to RGB if not already if image.mode != 'RGB': image = image.convert('RGB') # Save with optimized settings buffer = io.BytesIO() image.save( buffer, format="JPEG", quality=85, optimize=True, progressive=True ) compressed_data = buffer.getvalue() # Store in cache cache[cache_key] = { 'data': compressed_data, 'timestamp': time.time() } return compressed_data except Exception as e: logger.error(f"Image processing failed for section {section_id}: {e}") return None @app.route("/api/check", methods=["POST"]) def check(): start_time = time.time() try: data = request.get_json() log_request(f"Mode: {data.get('mode')}, Image received: {bool(data.get('image'))}") if not data: log_error("No data received") return jsonify({"error": "No data received"}), 400 mode = data.get("mode", "accessibility") prompt = data.get("prompt") or PROMPTS.get(mode, "Please critique this website's UI.") image_data = data.get("image") if not image_data: log_error("No image data received") return jsonify({"error": "No image data received"}), 400 # Validate and compress the image try: validate_image_size(image_data) compressed_image = compress_image_section(image_data, "main") if not compressed_image: return jsonify({"error": "Failed to process image"}), 400 except Exception as e: log_error(f"Image validation failed: {str(e)}") return jsonify({"error": f"Image processing error: {str(e)}"}), 400 try: # Create image part for Gemini using the correct format image_part = { "mime_type": "image/jpeg", "data": compressed_image } # Generate response using Gemini with proper content format response = model.generate_content([prompt, image_part]) if response.text: feedback = response.text log_response(f"Successfully generated feedback using Gemini") else: feedback = "No feedback generated from Gemini API" log_error("Empty response from Gemini API") processing_time = time.time() - start_time return jsonify({ "success": True, "feedback": feedback, "processing_time": processing_time, "api_used": "gemini-1.5-pro" }) except Exception as e: log_error(f"Gemini API error: {str(e)}") return jsonify({ "error": True, "message": f"API Error: {str(e)}", "api_used": "gemini-1.5-pro" }), 500 except Exception as e: logger.error(f"Unhandled error: {str(e)}") return jsonify({"error": str(e)}), 500 @app.route("/api/health", methods=["GET"]) def health_check(): """Health check endpoint to verify API connectivity""" try: # Test Gemini API with a simple prompt response = model.generate_content("Hello") return jsonify({ "status": "healthy", "api": "gemini-1.5-pro", "timestamp": tim [truncated โ€” 290 more characters] ``` ### assistant.css ```css #floating-assistant .header { cursor: move; background-color: #f8f9fa; padding: 12px; border-radius: 8px; font-weight: 600; color: #333; display: flex; align-items: center; gap: 8px; margin-bottom: 16px; } ``` ### assistant.html ```html
โ‡ฒ Design Assistant
Query Time: 0s
``` ### logging_config.py ```python import logging import os from datetime import datetime # Create logs directory if it doesn't exist if not os.path.exists('logs'): os.makedirs('logs') # Get today's date for log file name today = datetime.now().strftime('%Y-%m-%d') log_file = f'logs/accessibility_checker_{today}.log' # Configure logging logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[ logging.FileHandler(log_file), logging.StreamHandler() ] ) # Create a logger logger = logging.getLogger(__name__) # Add custom logging levels def log_request(request_data): """Log incoming request details""" logger.info(f"Received request: {request_data}") def log_response(response_data): """Log response details""" logger.info(f"Sent response: {response_data}") def log_error(error_message, request_data=None): """Log error with request context""" if request_data: logger.error(f"Error processing request {request_data}: {error_message}") else: logger.error(f"Error: {error_message}") ``` ### background.js ```javascript // === background.js === chrome.runtime.onMessage.addListener((request, sender, sendResponse) => { if (request.action === "capture") { console.log('Starting capture process...'); chrome.tabs.captureVisibleTab(null, { format: "png" }, function (dataUrl) { if (chrome.runtime.lastError) { console.error('Error capturing tab:', chrome.runtime.lastError.message); sendResponse({ error: true, message: "Failed to capture tab" }); return; } console.log('Captured image data URL'); console.log('Sending request to server...'); fetch("http://localhost:5001/api/check", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ image: dataUrl, mode: request.mode, prompt: request.prompt }) }) .then(res => { if (!res.ok) { throw new Error(`HTTP error! status: ${res.status}`); } return res.json(); }) .then(data => { console.log('Sending feedback to content script'); sendResponse({ success: true, feedback: data.feedback, processing_time: data.processing_time || 0 }); }) .catch(err => { console.error('Error:', err); sendResponse({ error: true, message: `Error: ${err.message}` }); }); }); return true; // Keep the channel open for async response } }); ``` ### check_api_credits.py ```python #!/usr/bin/env python3 """ Script to check Gemini API credits and test connectivity """ import os import requests import json from dotenv import load_dotenv def check_api_key(): """Check if Gemini API key is set and valid""" load_dotenv() api_key = os.environ.get("GEMINI_API_KEY") if not api_key: print("โŒ GEMINI_API_KEY is not set") print("\n๐Ÿ’ก To get a Gemini API key:") print(" 1. Go to https://makersuite.google.com/app/apikey") print(" 2. Create a new API key") print(" 3. Set it as environment variable: export GEMINI_API_KEY='your-key'") return None print("โœ… GEMINI_API_KEY is set") return api_key def test_api_connectivity(api_key): """Test if the API key works with a simple request""" try: import google.generativeai as genai genai.configure(api_key=api_key) # Try vision models first for visual analysis model_names = [ 'gemini-1.5-pro-vision', 'gemini-pro-vision', 'gemini-1.5-pro', 'gemini-1.5-flash', 'gemini-pro' ] for model_name in model_names: try: print(f" Trying model: {model_name}") model = genai.GenerativeModel(model_name) # Simple test request response = model.generate_content("Hello, this is a test. Please respond with 'API is working' if you can see this message.") if response.text: print(f"โœ… API connectivity test successful with model: {model_name}") print(f" Response: {response.text}") return True, model_name else: print(f" Model {model_name} returned empty response") except Exception as e: print(f" Model {model_name} failed: {str(e)[:100]}...") continue print("โŒ All model attempts failed") return False, None except Exception as e: print(f"โŒ API connectivity test failed: {e}") return False, None def check_usage_dashboard(): """Provide information about checking usage""" print("\n๐Ÿ“Š To check your API usage and credits:") print(" 1. Go to https://makersuite.google.com/app/apikey") print(" 2. Click on your API key") print(" 3. Check the 'Usage' tab") print(" 4. Monitor your quota and billing") def main(): print("๐Ÿ” Checking Gemini API Setup") print("=" * 40) # Check API key api_key = check_api_key() if not api_key: return # Test connectivity print("\n๐Ÿงช Testing API connectivity...") success, working_model = test_api_connectivity(api_key) if success: print(f"\nโœ… Your Gemini API is working correctly with model: {working_model}!") print(" You can now use the Chrome extension.") # Update the server to use the working model print(f"\n๐Ÿ’ก Note: The server will use model: {working_model}") else: print("\nโŒ API connectivity issues detected.") print(" Please check your API key and try again.") # Usage information check_usage_dashboard() if __name__ == "__main__": main() ``` ### prompts.py ```python PROMPTS = { "accessibility": ( "Act as a senior UI/UX designer and WCAG 2.1 accessibility expert. " "You are analyzing a full-page website screenshot to identify accessibility issues. " "Provide a structured accessibility audit with actionable frontend recommendations. " "Your response must be formatted as a **hierarchical bullet list**, organized by section (e.g., Header, Main Content, Footer).\n\n" "For each section, include:\n" "1. **Issue Category** (e.g., Color Contrast, Alt Text, Keyboard Navigation)\n" " - **Specific Issue**: Describe the exact problem.\n" " - **Affected User Group(s)**: (e.g., low vision, colorblind, screen reader users, motor impairment)\n" " - **WCAG Reference**: (e.g., WCAG 2.1 SC 1.4.3)\n" " - **Impact Summary**: What user experience or interaction is blocked or impaired?\n" " - **Frontend Recommendation**:\n" " โ€ข HTML Elements to Adjust (e.g.,