# Project export: D.E.V.I.S This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Voice controlled software engineering team - Devpost: https://devpost.com/software/d-e-v-i-s - GitHub: https://github.com/jessechoe10/DEVIS - Team: 2 GitHub contributor(s) — Jesse Choe (3 commits), kkaura251 (1 commits) ## Devpost submission (written by the team) ### Inspiration Devin is really annoying to use, but imagine you could just deploy full-stack apps with just your voice - pretty much J.A.R.V.I.S for software engineering. ### What it does Generates apps based on voice input and even uses computer use agents to mimic UI designs. ### How we built it Scrapybara for computer use to screenshot similar designs, Gemini's VLM to take the screenshot and describe it in natural language, ElevenLabs / Whisper for voice communication, and o1 for all of our software agents. ### Challenges we ran into Figuring out how the APIs worked and stringing everything together. Also a lot of latency issues. ### Accomplishments we're proud of We somehow managed to get something working in this short timeframe. ### What we learned You just gotta have fun with it tbh. ### What's next A lot more integrations and getting a fully working web app. This could be a decent startup idea too, especially if it controls an army of software agents. ## README (from the GitHub repository) # D.E.V.I.S ## Design Evolution Via Intelligent Systems DEVIS is a multi-agent system that helps create and deploy UI/UX designs through voice interaction and AI-powered code generation. ### Components 1. **Voice Agent**: Captures and processes verbal feedback from engineers 2. **Frontend Agent**: Generates React components and styles based on requirements 3. **Screenshot Agent**: Captures and analyzes reference designs using Anthropic's Claude 4. **Deployment Agent**: Automatically deploys generated code to GitHub and Vercel ### Setup 1. Install dependencies: ```bash pip install -r requirements.txt ``` 2. Create a `.env` file with the following keys: ``` OPENAI_API_KEY=your_openai_key ANTHROPIC_API_KEY=your_anthropic_key GITHUB_TOKEN=your_github_token VERCEL_TOKEN=your_vercel_token ``` 3. Install system dependencies for PyAudio: ```bash brew install portaudio ``` ### Usage 1. Run the main script: ```bash python main.py ``` 2. Speak your UI/UX requirements when prompted 3. Optionally provide a reference URL for design inspiration 4. The system will: - Process your voice input - Generate React components and styles - Create a GitHub repository - Deploy to Vercel automatically ### Example ```python from devis import DEVIS devis = DEVIS() devis.run_design_cycle(reference_url="https://example.com") ``` ### Notes - Make sure your microphone is properly configured - Ensure you have sufficient permissions for GitHub and Vercel APIs - The generated code uses React and Tailwind CSS by default ## Detected evidence (automated analysis) Indexed codebase: 6 recognized source files, 25 KB. - Anthropic (technology) — detected in the code - FastAPI (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Vercel (technology) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (8 of 8) ``` .cursorrules deployment_agent.py frontend_agent.py main.py README.md requirements.txt screenshot_agent.py voice_agent.py ``` ### Dependencies - requirements.txt: anthropic@==0.8.1, elevenlabs@==0.3.0, fastapi@==0.109.2, google-ai-generative-language@==0.2.0, google-generativeai@==0.3.2, openai@>=1.0.0, Pillow@==10.2.0, pyaudio@==0.2.14, PyGithub@==2.1.1, python-dotenv@==1.0.0, python-multipart@==0.0.6, scrapybara@==0.1.0, sounddevice@==0.4.6, soundfile@==0.12.1, speechrecognition@==3.10.1, uvicorn@==0.27.1, vercel@==0.2.1 ### Recent commits (newest first) - removed stuff - added gitignore - added proj - Add files via upload - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` openai>=1.0.0 anthropic==0.8.1 python-dotenv==1.0.0 fastapi==0.109.2 uvicorn==0.27.1 python-multipart==0.0.6 Pillow==10.2.0 PyGithub==2.1.1 vercel==0.2.1 speechrecognition==3.10.1 pyaudio==0.2.14 soundfile==0.12.1 sounddevice==0.4.6 scrapybara==0.1.0 google-ai-generative-language==0.2.0 google-generativeai==0.3.2 elevenlabs==0.3.0 ``` ### main.py ```python from voice_agent import VoiceAgent from frontend_agent import FrontendAgent from deployment_agent import DeploymentAgent import os from dotenv import load_dotenv import subprocess import webbrowser from pathlib import Path load_dotenv() class DEVIS: def __init__(self): """Initialize DEVIS with voice, frontend and deployment capabilities""" self.voice_agent = VoiceAgent() self.frontend_agent = FrontendAgent() self.deployment_agent = DeploymentAgent() self.project_dir = None def setup_local_project(self): """Set up local React project""" self.voice_agent.speak("Setting up your React project...") self.project_dir = Path(os.path.expanduser("~/Documents/devis-ui")) if not self.project_dir.exists(): subprocess.run(["npx", "create-react-app", str(self.project_dir)]) self.voice_agent.speak("Project setup complete!") def run_local_dev_server(self): """Run local development server""" if self.project_dir: self.voice_agent.speak("Starting development server...") subprocess.Popen(["npm", "start"], cwd=self.project_dir) webbrowser.open("http://localhost:3000") self.voice_agent.speak("Development server is running. You can see your app in the browser.") def update_local_code(self, generated_code, generated_styles): """Update local project with new code""" if self.project_dir: self.voice_agent.speak("Updating your code...") app_file = self.project_dir / "src" / "App.js" styles_file = self.project_dir / "src" / "App.css" with open(app_file, "w") as f: f.write(generated_code) with open(styles_file, "w") as f: f.write(generated_styles) self.voice_agent.speak("Code updated! Check the browser to see your changes.") def run(self): """Main loop for voice-controlled software development""" welcome_msg = """ Welcome to DEVIS - Your Voice Controlled Software Development Assistant! I'll help you build and deploy a web application through voice commands. Let's start getting everything set up. """ self.voice_agent.speak(welcome_msg) try: # Initial setup self.setup_local_project() # Get initial requirements initial_requirements = self.voice_agent.listen_for_input("What kind of web app would you like to create?") if initial_requirements: self.voice_agent.speak("Great! I'll create a baseline app based on your requirements.") baseline_code = self.frontend_agent.generate_code(initial_requirements, self.project_dir) baseline_styles = self.frontend_agent.generate_styles(initial_requirements, self.project_dir) self.update_local_code(baseline_code, baseline_styles) # Start development server self.run_local_dev_server() while True: # Listen for command command = self.voice_agent.listen_for_input("What changes would you like to make?") if not command: break # Check if user is satisfied if "looks good" in command.lower(): self.voice_agent.speak("Great! Would you like to deploy your application now?") deploy_command = self.voice_agent.listen_for_input("Say 'yes' to deploy or 'no' to continue development.") if deploy_command and "yes" in deploy_command.lower(): # Deployment phase self.voice_agent.speak("Starting deployment process...") # Create GitHub repository self.voice_agent.speak("Creating GitHub repository...") repo_name = "devis-generated-ui" repo = self.deployment_agent.create_github_repo(repo_name) if repo: self.voice_agent.speak(f"Created GitHub repository: {repo['html_url']}") # Push code to GitHub if self.project_dir: self.voice_agent.speak("Pushing your code to GitHub...") files = { "src/App.js": (self.project_dir / "src" / "App.js").read_text(), "src/App.css": (self.project_dir / "src" / "App.css").read_text(), "package.json": (self.project_dir / "package.json").read_text() } self.deployment_agent.push_to_github(repo["full_name"], files) self.voice_agent.speak("Code pushed to GitHub successfully!") # Deploy to Vercel self.voice_agent.speak("Deploying to Vercel...") deployment = self.deployment_agent.deploy_to_vercel(self.project_dir) if deployment: self.voice_agent.speak(f"Your web app is now live at: {deployment['url']}") break self.voice_agent.speak("Deployment complete! Thank you for using DEVIS.") break else: self.voice_agent.speak("Okay, let's continue development. What changes would you like to make?" [truncated — 1392 more characters] ``` ### deployment_agent.py ```python import os import requests from dotenv import load_dotenv import json load_dotenv() class DeploymentAgent: def __init__(self): self.github_token = os.getenv("GITHUB_TOKEN") self.vercel_token = os.getenv("VERCEL_TOKEN") self.github_headers = { "Authorization": f"token {self.github_token}", "Accept": "application/vnd.github.v3+json" } self.vercel_headers = { "Authorization": f"Bearer {self.vercel_token}" } def create_github_repo(self, repo_name, description=""): """Create a new GitHub repository""" url = "https://api.github.com/user/repos" data = { "name": repo_name, "description": description, "private": False } response = requests.post(url, headers=self.github_headers, json=data) return response.json() if response.status_code == 201 else None def push_to_github(self, repo_name, files): """Push generated files to GitHub""" base_url = f"https://api.github.com/repos/{repo_name}/contents" for file_path, content in files.items(): data = { "message": f"Add {file_path}", "content": content.encode('utf-8').hex(), "branch": "main" } response = requests.put(f"{base_url}/{file_path}", headers=self.github_headers, json=data) if response.status_code not in [201, 200]: print(f"Error pushing {file_path}: {response.json()}") def deploy_to_vercel(self, github_repo_url): """Deploy the GitHub repository to Vercel""" url = "https://api.vercel.com/v9/projects" # Create project project_data = { "name": github_repo_url.split("/")[-1], "gitRepository": { "type": "github", "repo": github_repo_url } } response = requests.post(url, headers=self.vercel_headers, json=project_data) if response.status_code != 201: print(f"Error creating Vercel project: {response.json()}") return None project_id = response.json()["id"] # Trigger deployment deploy_url = f"https://api.vercel.com/v13/deployments" deploy_data = { "projectId": project_id, "target": "production" } response = requests.post(deploy_url, headers=self.vercel_headers, json=deploy_data) return response.json() if response.status_code == 201 else None if __name__ == "__main__": agent = DeploymentAgent() # Example usage repo = agent.create_github_repo("test-ui-project") if repo: files = { "index.js": "console.log('Hello World')", "styles.css": "body { margin: 0; }" } agent.push_to_github(repo["full_name"], files) deployment = agent.deploy_to_vercel(repo["html_url"]) print("Deployment status:", deployment) ``` ### frontend_agent.py ```python import openai import os from pathlib import Path class FrontendAgent: def __init__(self): self.client = openai.Client() def _read_current_code(self, project_dir): """Read current App.js and App.css content""" try: app_js = (Path(project_dir) / "src" / "App.js").read_text() app_css = (Path(project_dir) / "src" / "App.css").read_text() return app_js, app_css except: return None, None def generate_code(self, requirements, project_dir=None): """Generate or update App.js based on requirements""" current_code = None if project_dir: current_code, _ = self._read_current_code(project_dir) prompt = f"""You are generating a complete React App.js file. Output ONLY the code, no markdown, no explanations. Requirements: {requirements} Rules: 1. Include all necessary imports 2. Use modern React practices 3. Output a complete, working App.js file 4. Use normal CSS for styling 5. DO NOT include any markdown or code fences 6. DO NOT include any explanations 7. Output ONLY the code that should be in App.js 8. Just use pure CSS. Don't use any frameworks. Import relevant css files like App.css/index.css. Current code to iterate on: {current_code if current_code else 'No existing code'} """ response = self.client.chat.completions.create( model="o1-mini", messages=[ {"role": "user", "content": prompt} ] ) return response.choices[0].message.content.strip() def generate_styles(self, component_description, project_dir=None): """Generate or update App.css based on requirements""" _, current_css = None, None if project_dir: _, current_css = self._read_current_code(project_dir) prompt = f"""You are generating a complete App.css file. Output ONLY the CSS code, no markdown, no explanations. Component description: {component_description} Rules: 1. Include all necessary styles 2. Use modern CSS practices 3. Output a complete, working App.css file 4. DO NOT USE ANY CSS FRAMEWORKS. 5. DO NOT include any markdown or code fences 6. DO NOT include any explanations 7. Output ONLY the code that should be in App.css 8. Just use pure CSS. Don't use any frameworks like Tailwind. Current CSS to iterate on: {current_css if current_css else 'No existing CSS'} """ response = self.client.chat.completions.create( model="o1-mini", messages=[ {"role": "user", "content": prompt} ] ) return response.choices[0].message.content.strip() def cleanup(self): """Clean up resources""" pass if __name__ == "__main__": agent = FrontendAgent() sample_req = "Create a modern navigation bar with a logo, links, and a search bar" code = agent.generate_code(sample_req) styles = agent.generate_styles(sample_req) print("Generated Code:", code) print("Generated Styles:", styles) ``` ### screenshot_agent.py ```python from scrapybara import Scrapybara from scrapybara.core.api_error import ApiError from scrapybara.tools import BrowserTool from dotenv import load_dotenv import os from google import genai from PIL import Image import base64 import io load_dotenv() class ScreenshotAgent: def __init__(self): self.client = Scrapybara(api_key=os.getenv("SCRAPYBARA_API_KEY")) self.instance = None self.genai_client = genai.Client(api_key=os.getenv("GEMINI_API_KEY")) def capture_screenshot(self, url): """Capture screenshot of a webpage""" try: # Start Ubuntu instance self.instance = self.client.start_ubuntu() # Start browser and get CDP URL cdp_url = self.instance.browser.start().cdp_url # Create browser tool browser_tool = BrowserTool(self.instance) # Navigate to URL browser_tool(command="go_to", url=url) # Take screenshot screenshot_result = browser_tool(command="screenshot") # Save to screenshot.png in the current directory output_path = os.path.join(os.path.dirname(__file__), "screenshot.png") with open(output_path, "wb") as f: f.write(base64.b64decode(screenshot_result.base_64_image)) # Cleanup self.instance.browser.stop() self.instance.stop() self.instance = None return output_path except ApiError as e: print(f"Scrapybara API Error {e.status_code}: {e.body}") if self.instance: self.instance.stop() self.instance = None return None except Exception as e: print(f"Error capturing screenshot: {e}") if self.instance: self.instance.stop() self.instance = None return None def analyze_screenshot(self, screenshot_path): """Analyze screenshot using Google Gemini""" try: if not screenshot_path: return None with Image.open(screenshot_path) as img: response = self.genai_client.models.generate_content( model="gemini-2.0-flash", contents=[{ "role": "user", "parts": [{ "text": "Analyze this UI screenshot and describe the layout, colors, components, and design patterns used. Focus on actionable details that could be used to recreate a similar design.", "inline_data": { "mime_type": "image/png", "data": img } }] }] ) return response.text except Exception as e: print(f"Error analyzing screenshot: {e}") return None def cleanup(self): """Clean up resources""" if self.instance: try: self.instance.browser.stop() self.instance.stop() except: pass finally: self.instance = None if __name__ == "__main__": # Test the screenshot agent agent = ScreenshotAgent() test_urls = [ "https://www.google.com", "https://github.com", "https://news.ycombinator.com" ] for url in test_urls: print(f"\nTesting with {url}") screenshot_path = agent.capture_screenshot(url) if screenshot_path: print(f"Screenshot saved to: {screenshot_path}") if os.path.exists(screenshot_path): size = os.path.getsize(screenshot_path) print(f"Screenshot size: {size} bytes") if size > 1000: # Basic validation that file has content print(" Success!") else: print(" Screenshot seems too small") else: print(" Screenshot file not found") else: print(" Failed to capture screenshot") # Cleanup between tests agent.cleanup() ``` ### voice_agent.py ```python import io import sounddevice as sd import soundfile as sf from openai import OpenAI from dotenv import load_dotenv from elevenlabs.client import ElevenLabs from elevenlabs import play import os import re import time import tempfile import numpy as np load_dotenv() class VoiceAgent: def __init__(self): self.client = OpenAI() self.voice_client = ElevenLabs(api_key=os.getenv('ELEVENLABS_API_KEY')) self.voice_id = "JBFqnCBsd6RMkjVDRZzb" # Rachel voice self.sample_rate = 44100 def speak(self, text): """Convert text to speech using ElevenLabs""" try: audio = self.voice_client.text_to_speech.convert( text=text, voice_id=self.voice_id, model_id="eleven_multilingual_v2", output_format="mp3_44100_128", ) play(audio) # Add a small delay to ensure the message is heard time.sleep(len(text.split()) * 0.3) except Exception as e: print(f"Speech synthesis failed: {e}") print(f"Fallback to text: {text}") # Fallback to printing def record_audio(self, duration=5): """Record audio from microphone""" self.speak("Recording...") # Record audio recording = sd.rec( int(duration * self.sample_rate), samplerate=self.sample_rate, channels=1, dtype=np.float32 ) sd.wait() # Save to temporary file temp_file = tempfile.NamedTemporaryFile(suffix='.wav', delete=False) sf.write(temp_file.name, recording, self.sample_rate) return temp_file.name def transcribe_audio(self, audio_file): """Transcribe audio using Whisper API""" try: with open(audio_file, "rb") as f: transcript = self.client.audio.transcriptions.create( model="whisper-1", file=f ) return transcript.text except Exception as e: print(f"Transcription failed: {e}") return None def listen_for_input(self, prompt): """Listen for voice input with a specific prompt""" self.speak(prompt) time.sleep(3) # Record audio audio_file = self.record_audio() # Transcribe text = self.transcribe_audio(audio_file) # Cleanup os.unlink(audio_file) if text: self.speak(f"I heard: {text}") return text else: self.speak("I could not understand what you said. Please try again.") return None def extract_url(self, text): """Extract URL from text using regex and validation""" # Basic URL pattern url_pattern = r'https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+' urls = re.findall(url_pattern, text) if urls: return urls[0] # If no URL found, use OpenAI to try to understand the URL response = self.client.chat.completions.create( model="o1-mini", messages=[ {"role": "user", "content": f"Extract a website URL from this {text}. If no valid URL is found, try to construct one from the domain name. Return just the URL with https:// prefix or 'invalid' if no URL can be constructed."} ] ) extracted_url = response.choices[0].message.content.strip() return None if extracted_url == "invalid" else extracted_url def get_reference_url(self): """Get reference URL from voice input with validation""" url = None while not url: text = self.listen_for_input( "Please provide a URL for the reference design you'd like to use. " "You can say the full URL or just the website name." ) if not text: return None url = self.extract_url(text) if not url: self.speak("I couldn't understand the URL. Please try again.") else: self.speak(f"Great! I'll use {url} as the reference design.") return url def process_design_feedback(self, screenshot_analysis): """Process feedback about the design based on screenshot analysis""" self.speak("I've analyzed the design. What changes would you like to make?") feedback = self.listen_for_input("Please provide your feedback.") if not feedback: return None response = self.client.chat.completions.create( model="o1-mini", messages=[ {"role": "user", "content": f"You are a UI/UX expert. Based on the screenshot analysis and user feedback, provide specific UI component requirements.\nScreenshot Analysis: {screenshot_analysis}\nUser Feedback: {feedback}"} ] ) return response.choices[0].message.content if __name__ == "__main__": agent = VoiceAgent() url = agent.get_reference_url() print("Reference URL:", url) ```