# Project export: MailPilot - A Professional Email Writer This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: A pure javascript chrome extension using Google Gemini that takes multimodal input to write emails in multiple tones; professional to diplomatic, along with detailed email summaries and translation. - Devpost: https://devpost.com/software/a-professional-email-writer - GitHub: https://github.com/jhuynh06/email-overlay-ext - Video: https://www.youtube.com/embed/0Bz3MOY5VIs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — jhuynh06 (3 commits), Claude (2 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration of this project runs from going through multiple emails causing production friction. We wanted, as developers, to be able to focus on coding as much as possible. After learning about multi-modal inputs, we decided this would be the best project. ### What it does This is a pure javascript Chrome extension. It allows the user to input a Google Gemini key. From there, a popup appears on either two supported platforms, Gmail or Outlook. It is multi-modal, such that it takes all available attachments to create the best responding email. It can also compose new emails. It is optimized specifically for writing emails, with no prompting needed by the user. It also has a summary/analysis tool specifically tailored for professional use. Lastly, there is a translation tool to make communications between individuals even smoother. ### How we built it The app was built through Javascript, HTML, and CSS. Using Google's various Gemini models, the user can test which model might improve their use case. It uses Gemini 1.5 ability to parse through files, audio, and images to increase the amount of information used in the context window. ### Challenges we ran into We ran into several UI/UX issues. We started with something that directly embedded into Gmail's compose bar, but eventually iterated over to make an overlay type of extension. This made clarity much better for the user. Not only that, we also had to iterate over our prompts for best optimization. ### Accomplishments we're proud of We're extremely proud of the multi-modal functionality of the extension. It allows for the user to get the best results with more information, leading to more accurate and professional emails. ### What we learned Gemini integration Chrome extension building Multimodal LLM usage ### What's next for - A Professional Email Writer There are still several improvements that we could still make. We believe that our prompts could still be improved to give even more accuracy. Some more features that could help is a sort of suggestion feature that works on the user's previous emails with another recipient. By building on their past tone/way of diction, this could create more human/life-like interactions. ## README (from the GitHub repository) # AI Email Assistant Chrome Extension A powerful Chrome extension that provides AI-powered email assistance using Google's Gemini API. Features a Grammarly-like overlay interface that appears when composing emails in Gmail and Microsoft Outlook, offering intelligent response generation, email analysis, and multimodal attachment processing. ## 🚀 Features ### Core Functionality - **Smart Email Response Generation**: Create contextual email replies based on original email content and chosen tone - **Email Analysis & Summarization**: Intelligent analysis with formatted summaries, key points, and action items - **Universal Text Detection**: Focus-based overlay injection that works seamlessly across email platforms - **Multimodal Attachment Analysis**: Download and analyze email attachments (PDFs, images, documents) for comprehensive context ### Advanced Capabilities - **Multiple Response Tones**: Formal, casual, brief, detailed, friendly, and professional options - **Attachment Processing**: Full content analysis of PDFs, documents, images, and spreadsheets - **Translation Framework**: Ready-to-implement translation system with language selection - **Settings Persistence**: All user preferences saved across browser sessions - **Rate Limit Handling**: Robust error handling with exponential backoff retry logic ## 🎯 Supported Platforms - **Gmail**: `mail.google.com` - **Outlook Web**: `outlook.live.com`, `outlook.office.com` ## 📋 Prerequisites - Chrome browser (Manifest V3 compatible) - Google Gemini API key ([Get one here](https://makersuite.google.com/app/apikey)) ## 🛠️ Installation ### Method 1: Load Unpacked (Development) 1. **Download the Extension** ```bash git clone cd overlay-extension ``` 2. **Open Chrome Extensions** - Navigate to `chrome://extensions/` - Enable "Developer mode" in the top right corner 3. **Load the Extension** - Click "Load unpacked" - Select the `overlay-extension` folder - The extension icon will appear in your browser toolbar ### Method 2: Install from Chrome Web Store *Coming soon - extension pending review* ## ⚙️ Configuration ### 1. Get Google Gemini API Key 1. Visit [Google AI Studio](https://makersuite.google.com/app/apikey) 2. Create a new API key 3. Copy the API key for configuration ### 2. Configure Extension Settings 1. Click the extension icon in your browser toolbar 2. Enter your Google Gemini API key 3. Configure your preferences: - **Model Selection**: Choose between Gemini 1.5 Flash (fast) or Pro (best quality) - **Default Tone**: Set your preferred response style - **Response Length**: Short, medium, or long responses - **Creativity Level**: Conservative, balanced, or creative - **Advanced Features**: Enable attachment analysis and multimodal processing 4. Click "Test API" to verify your connection 5. Click "Save Settings" to store your configuration ## 🎮 How to Use ### Basic Email Response Generation 1. **Open Gmail or Outlook** in your browser 2. **Reply to an email** or compose a new message 3. **Click in the compose area** - the AI overlay will appear automatically 4. **Click "Generate"** to create an AI-powered response 5. **Customize settings** by clicking the menu button (⋯) to select different actions ### Advanced Features #### Multimodal Attachment Analysis 1. Enable "Analyze Email Attachments" in extension settings 2. Enable "Full Multimodal Analysis (Beta)" for complete attachment processing 3. When replying to emails with attachments, the AI will: - Download and analyze PDF documents, images, spreadsheets - Reference specific content from attachments in responses - Provide insights based on actual file contents #### Email Analysis & Summarization 1. Click the menu button (⋯) in the overlay 2. Select "Analyze & Summarize Email" 3. Click "Generate" to receive: - Brief summary of main topics - Key points and action items - Important dates and deadlines - Insights from attachments (if enabled) #### Response Tone Customization Choose from multiple response tones: - **Formal**: Professional and respectful - **Casual**: Friendly and conversational - **Brief**: Concise and direct - **Detailed**: Comprehensive but focused - **Friendly**: Warm and personable - **Professional**: Business-appropriate ## 🏗️ Technical Architecture ### Extension Structure ``` overlay-extension/ ├── manifest.json # Chrome Extension Manifest V3 ├── popup.html/js # Settings UI with gradient design ├── background.js # Service worker for API calls ├── overlay.js # Core overlay logic & Gemini service ├── overlay.css # Grammarly-style positioning & styling ├── content-gmail.js # Gmail-specific integration ├── content-outlook.js # Outlook-specific integration └── icons/ # Extension icons ``` ### Key Components #### 1. Universal Email Assistant (`content-gmail.js`) - Focus-based text detection similar to Grammarly - Intelligent email context filtering - Dynamic overlay creation and positioning - Email context extraction and processing #### 2. Gemini API Service (`overlay.js`) - **GeminiService**: Core AI interaction with retry logic - **MultimodalAttachmentService**: Advanced attachment processing - Rate limit handling with exponential backoff - Support for both base64 and File API uploads #### 3. Settings Management (`popup.js`) - Chrome storage sync integration - Dependency validation (multimodal requires attachments) - Real-time API testing and validation ## 🔧 Development ### Prerequisites - Node.js (for syntax checking) - Git (for version control) ### Development Commands ```bash # Check JavaScript syntax node -c *.js # Load extension in Chrome # Navigate to chrome://extensions/ → Developer mode → Load unpacked # Debug console logging # Check browser console for detailed operation logs ``` ### Project Scripts ```bash # Syntax validation npm run lint # If package.json exists # Manual syntax check node -c overlay.js node -c content-gmail.js node -c popup.js ``` ## 🔒 Privacy & Security ### Data Handling - **API Key Storage**: Securely stored using Chrome's sync storage - **Email Content**: Only sent to Google's Gemini API for processing - **No Third-Party Storage**: No data stored or transmitted to other services - **HTTPS Encryption**: All communication uses secure HTTPS ### Permissions Required - `storage`: Save user settings and API key - `activeTab`: Access current tab for content injection - `scripting`: Execute content scripts - `downloads`: Handle email attachment processing - Host permissions for Gmail, Outlook, and Gemini API ### Security Best Practices - API keys stored locally only - Content scripts isolated per domain - No external data transmission beyond Gemini API - Secure attachment processing with size limits ## 🐛 Troubleshooting ### Extension Not Working - ✅ Verify extension is enabled in `chrome://extensions/` - ✅ Check API key is correctly entered in popup settings - ✅ Try refreshing the Gmail/Outlook page - ✅ Check browser console for error messages ### Overlay Not Appearing - ✅ Ensure you're on a supported platform (Gmail/Outlook) - ✅ Try clicking "Reply" or "Compose" to trigger injection - ✅ Check if other extensions are interfering - ✅ Verify you're clicking in the message compose area ### API Errors - ✅ Test API key using "Test API" button in settings - ✅ Check internet connection stability - ✅ Verify available API quota in Google Cloud Console - ✅ Try regenerating API key if persistent issues ### Attachment Processing Issues - ✅ Ensure "Analyze Email Attachments" is enabled - ✅ Check file size limits (20MB maximum) - ✅ Verify supported file types (PDF, DOC, images, etc.) - ✅ Check browser console for attachment processing logs ## 📈 Performance ### Optimization Features - **Gemini Flash Model**: Default for faster responses and better rate limits - **Intelligent Caching**: Reduces redundant API ca [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 8 recognized source files, 147 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (9 of 9) ``` background.js content-gmail.js content-outlook.js manifest.json overlay.css overlay.js popup.html popup.js README.md ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Merge pull request #5 from jhuynh06/improve-multimodal-response-prompts - Improve multimodal response prompts for natural conversation - Added READNE - Merge pull request #4 from jhuynh06/feature/multimodal-attachments - Complete multimodal attachment analysis functionality - revert fake - brawl stars - moon - sun - hi daniel - hello! - Merge pull request #3 from jhuynh06/radio-fix - radio button - ...i think the sun is almost coming out - woahh is that the milky way? there's so many stars! - is that the big dipper? - no roof no problem - wait... i have a roof??? - stargazing in my livin' room - oh boy, here we go again! ## Key source files (fetched from GitHub, selected and truncated for size) ### popup.js ```javascript document.addEventListener('DOMContentLoaded', function() { const form = document.getElementById('settingsForm'); const apiKeyInput = document.getElementById('apiKey'); const geminiModelSelect = document.getElementById('geminiModel'); const defaultToneSelect = document.getElementById('defaultTone'); const maxTokensSelect = document.getElementById('maxTokens'); const temperatureSelect = document.getElementById('temperature'); const analyzeAttachmentsCheckbox = document.getElementById('analyzeAttachments'); const multimodalAnalysisCheckbox = document.getElementById('multimodalAnalysis'); const translationLanguageSelect = document.getElementById('translationLanguage'); const saveBtn = document.getElementById('saveBtn'); const testBtn = document.getElementById('testBtn'); const statusDiv = document.getElementById('status'); const toggleAdvanced = document.getElementById('toggleAdvanced'); const advancedContent = document.getElementById('advancedContent'); // Load saved settings loadSettings(); // Advanced settings toggle toggleAdvanced.addEventListener('click', function() { const isVisible = advancedContent.classList.contains('show'); advancedContent.classList.toggle('show'); toggleAdvanced.textContent = isVisible ? 'Advanced Settings ▼' : 'Advanced Settings ▲'; }); // Form submission form.addEventListener('submit', function(e) { e.preventDefault(); saveSettings(); }); // Test API button testBtn.addEventListener('click', function() { testApiConnection(); }); // Handle multimodal analysis dependency analyzeAttachmentsCheckbox.addEventListener('change', function() { if (!this.checked) { multimodalAnalysisCheckbox.checked = false; multimodalAnalysisCheckbox.disabled = true; } else { multimodalAnalysisCheckbox.disabled = false; } }); // Initialize dependency state if (!analyzeAttachmentsCheckbox.checked) { multimodalAnalysisCheckbox.disabled = true; } async function loadSettings() { try { const result = await chrome.storage.sync.get([ 'geminiApiKey', 'geminiModel', 'defaultTone', 'maxTokens', 'temperature', 'analyzeAttachments', 'multimodalAnalysis', 'translationLanguage' ]); if (result.geminiApiKey) { apiKeyInput.value = result.geminiApiKey; } if (result.geminiModel) { geminiModelSelect.value = result.geminiModel; } if (result.defaultTone) { defaultToneSelect.value = result.defaultTone; } if (result.maxTokens) { maxTokensSelect.value = result.maxTokens; } if (result.temperature) { temperatureSelect.value = result.temperature; } if (result.analyzeAttachments !== undefined) { analyzeAttachmentsCheckbox.checked = result.analyzeAttachments; } if (result.multimodalAnalysis !== undefined) { multimodalAnalysisCheckbox.checked = result.multimodalAnalysis; } if (result.translationLanguage) { translationLanguageSelect.value = result.translationLanguage; } } catch (error) { showStatus('Error loading settings', 'error'); } } async function saveSettings() { const apiKey = apiKeyInput.value.trim(); if (!apiKey) { showStatus('Please enter your API key', 'error'); return; } try { saveBtn.textContent = 'Saving...'; saveBtn.disabled = true; await chrome.storage.sync.set({ geminiApiKey: apiKey, geminiModel: geminiModelSelect.value, defaultTone: defaultToneSelect.value, maxTokens: parseInt(maxTokensSelect.value), temperature: parseFloat(temperatureSelect.value), analyzeAttachments: analyzeAttachmentsCheckbox.checked, multimodalAnalysis: multimodalAnalysisCheckbox.checked, translationLanguage: translationLanguageSelect.value }); showStatus('Settings saved successfully!', 'success'); // Notify content scripts about updated settings chrome.tabs.query({active: true, currentWindow: true}, function(tabs) { if (tabs[0]) { chrome.tabs.sendMessage(tabs[0].id, { action: 'settingsUpdated' }); } }); } catch (error) { showStatus('Error saving settings: ' + error.message, 'error'); } finally { saveBtn.textContent = 'Save Settings'; saveBtn.disabled = false; } } async function testApiConnection() { const apiKey = apiKeyInput.value.trim(); if (!apiKey) { showStatus('Please enter your API key first', 'error'); return; } try { testBtn.textContent = 'Testing...'; testBtn.disabled = true; const selectedModel = geminiModelSelect.value || 'gemini-1.5-pro'; const response = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${selectedModel}:generateContent?key=${apiKey}`, { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ contents: [{ parts: [{ text: "Hello, this is a test message." [truncated — 1088 more characters] ``` ### background.js ```javascript // Background Service Worker for AI Email Assistant class BackgroundService { constructor() { this.init(); } init() { // Listen for extension installation chrome.runtime.onInstalled.addListener((details) => { this.handleInstall(details); }); // Listen for messages from content scripts chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { this.handleMessage(message, sender, sendResponse); return true; // Keep message channel open for async responses }); // Tab updates are handled by manifest content scripts } handleInstall(details) { if (details.reason === 'install') { // Set default settings chrome.storage.sync.set({ geminiModel: 'gemini-1.5-flash', defaultTone: 'formal', maxTokens: 300, temperature: 0.7 }); // Extension icon will show popup for API key setup } } async handleMessage(message, sender, sendResponse) { try { switch (message.action) { case 'getSettings': const settings = await this.getSettings(); sendResponse({ success: true, data: settings }); break; case 'saveSettings': await this.saveSettings(message.data); sendResponse({ success: true }); break; case 'testApiKey': const isValid = await this.testApiKey(message.apiKey); sendResponse({ success: true, valid: isValid }); break; case 'generateResponse': const response = await this.generateResponse(message.data); sendResponse({ success: true, data: response }); break; case 'generateSummary': const summary = await this.generateSummary(message.data); sendResponse({ success: true, data: summary }); break; default: sendResponse({ success: false, error: 'Unknown action' }); } } catch (error) { console.error('Background script error:', error); sendResponse({ success: false, error: error.message }); } } async getSettings() { return new Promise((resolve) => { chrome.storage.sync.get([ 'geminiApiKey', 'geminiModel', 'defaultTone', 'maxTokens', 'temperature' ], (result) => { resolve(result); }); }); } async saveSettings(settings) { return new Promise((resolve) => { chrome.storage.sync.set(settings, () => { resolve(); }); }); } async testApiKey(apiKey) { try { // Use Flash model for testing - better rate limits const response = await fetch( `https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=${apiKey}`, { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ contents: [{ parts: [{ text: 'Hello' }] }], generationConfig: { maxOutputTokens: 10 } }) } ); if (response.status === 429) { // Rate limited but API key works return true; } return response.ok; } catch (error) { console.error('API key test error:', error); return false; } } async generateResponse(data) { const settings = await this.getSettings(); if (!settings.geminiApiKey) { throw new Error('API key not configured'); } try { const selectedModel = settings.geminiModel || 'gemini-1.5-pro'; const response = await fetch( `https://generativelanguage.googleapis.com/v1beta/models/${selectedModel}:generateContent?key=${settings.geminiApiKey}`, { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ contents: [{ parts: [{ text: data.prompt }] }], generationConfig: { temperature: data.temperature || settings.temperature || 0.7, maxOutputTokens: Math.min((data.maxTokens || settings.maxTokens || 300) * 4, 2048) } }) } ); if (!response.ok) { throw new Error(`API request failed: ${response.status}`); } const result = await response.json(); if (result.candidates && result.candidates[0] && result.candidates[0].content) { return result.candidates[0].content.parts[0].text.trim(); } else { throw new Error('No response content received'); } } catch (error) { console.error('Generate response error:', error); throw error; } } async generateSummary(data) { const settings = await this.getSettin [truncated — 1644 more characters] ``` ### popup.html ```html

AI Email Assistant

Powered by Google Gemini