# Project export: Laila This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: automating the web, so you can focus on what matters - Devpost: https://devpost.com/software/lailya - GitHub: https://github.com/timobraz/Calhacks11 - Video: https://www.youtube.com/embed/JhrLWurl5k4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Timofey Obraztsov (26 commits), Andrew Hwang (19 commits), jason136 (6 commits), caosl (4 commits) ## Devpost submission (written by the team) ### Inspiration We all know how important it is to have time for meaningful work and to maintain a balanced life. But when we discovered that 40% of workers spend a quarter of their work week on manual, repetitive tasks—like logging emails, updating spreadsheets, and managing schedules—it really caught our attention. These tasks, while small, add up over time, leading to increased burnout and reduced job satisfaction. We felt there had to be a better way—a way to automate these mundane processes so that people can focus on the work that truly matters to them. This insight drove us to create a solution that not only enhances productivity but also helps individuals reclaim their time and maintain a healthier work-life balance. ### What it does Introducing LAILA, an intelligent digital assistant designed to make your workday easier. LAILA automates repetitive web tasks in real-time, allowing you to focus on what truly matters and boosting your productivity effortlessly. By combining advanced action models, voice technology, and generative AI, LAILA takes your voice commands or input, interprets them, and transforms them into clear, actionable steps. These tasks are then carried out instantly and displayed on your screen, so you can track the progress as it happens—all without lifting a finger. In essence, LAILA turns tedious tasks into simple, automated actions, helping you reclaim your time and work smarter, not harder. ### How we built it We developed a comprehensive system architecture diagram to illustrate the communication flow and interaction between various components of our solution. Here's the tech stack: Front End Next.js and React: For building a responsive, dynamic user interface. Tailwind CSS and Shadcn: For efficient and customizable styling. TypeScript: Ensures type safety and scalability. Back End Twilio: Enables seamless calling capabilities. Deepgram: Utilized for speech-to-text processing. SingleStore: A real-time data management system for fast analytics. Apache Kafka: Manages real-time event streaming. Perplexity and Gemini: For interpreting tasks and generating instructions using AI. Phoenix: Evaluates and observes LLM (Large Language Model) performance. Selenium: Automates browser tasks for efficient web interactions. ### Challenges we ran into Integrating multiple APIs together cohesively Getting the Gemini AI to generate clear, accurate, and actionable instructions Connecting Kafka and SingleStore Ensuring real-time performance ### Accomplishments we're proud of Developed a working MVP that uses an advanced action model to automate web tasks. Integrated a diverse range of technologies—like AI, voice recognition, and real-time data handling—into a cohesive, functioning system. ### What we learned How critical it is to thoroughly read and understand API documentation. The potential of action models for automating more complex web flows and pipelines. Working with a diverse tech stack and multiple APIs taught us the importance of teamwork and adaptability. ### What's next for LAILA Handle more complex web flows Expanding to more complex web flows Integrating more AI models ## README (from the GitHub repository) # Calhacks11 ## Detected evidence (automated analysis) Indexed codebase: 41 recognized source files, 228 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (53 of 53) ``` .gitignore fullstack/.eslintrc.json fullstack/.gitignore fullstack/components.json fullstack/next.config.js fullstack/next.config.mjs fullstack/package.json fullstack/postcss.config.mjs fullstack/README.md fullstack/src/app/api/gemini/route.tsx fullstack/src/app/api/perplexity/route.ts fullstack/src/app/api/request/[taskid]/route.ts fullstack/src/app/api/request/route.ts fullstack/src/app/api/transcribe/route.ts fullstack/src/app/api/twilio/route.ts fullstack/src/app/api/twilioRecordingComplete/route.ts fullstack/src/app/dashboard/action/page.tsx fullstack/src/app/dashboard/analytics/page.tsx fullstack/src/app/dashboard/config/page.tsx fullstack/src/app/dashboard/history/page.tsx fullstack/src/app/dashboard/layout.tsx fullstack/src/app/dashboard/page.tsx fullstack/src/app/dashboard/task/page.tsx fullstack/src/app/globals.css fullstack/src/app/layout.tsx fullstack/src/app/page.tsx fullstack/src/app/start/deepgram/page.tsx fullstack/src/app/start/page.tsx fullstack/src/components/demo.tsx fullstack/src/components/drawer.tsx fullstack/src/components/TranscribeButton.tsx fullstack/src/components/ui/background-beams.tsx fullstack/src/components/ui/button.tsx fullstack/src/components/ui/drawer.tsx fullstack/src/components/ui/input.tsx fullstack/src/components/ui/sidebar.tsx fullstack/src/components/ui/text-hover-effect.tsx fullstack/src/contexts/LoadingContext.tsx fullstack/src/lib/kafka.ts fullstack/src/lib/utils.ts fullstack/tailwind.config.ts fullstack/tsconfig.json package.json prettier.config.cjs python/.gitignore python/main.py python/pipeline.py python/prompts.py python/requirements.txt python/schema.sql python/selenium_driver.py python/spider_pipeline.html README.md ``` ### Dependencies - fullstack/package.json: @deepgram/sdk@^3.8.1, @radix-ui/react-dialog@^1.1.2, @radix-ui/react-slot@^1.1.0, @tabler/icons-react@^3.19.0, @types/fluent-ffmpeg@^2.1.21, @types/mysql@^2.15.26, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/recharts@^1.8.29, @types/uuid@^10.0.0, aceternity-ui@^0.2.2, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.15, ffmpeg-static@^5.1.0, fluent-ffmpeg@^2.1.2, formidable@^3.5.2, framer-motion@^11.11.9, kafkajs@^2.2.4, lucide-react@^0.453.0, mysql2@^3.11.3, net@^1.0.2, next@14.2.15, postcss@^8, react@^18, react-dom@^18, react-icons@^5.3.0, recharts@^2.13.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, tls@^0.0.1, twilio@^5.3.4, typescript@^5, uuid@^10.0.0, vaul@^1.1.0, xmlbuilder@^15.1.1 - package.json: @types/formidable@^3.4.5, concurrently@^8.2.2, lint-staged@^15.2.0, prettier@^3.1.0 - python/requirements.txt: arize-phoenix@==5.2.2, arize-phoenix-otel@==0.6.1, google-generativeai@<0.6.0, kafka-python, matplotlib, numpy, openinference-instrumentation-vertexai, pymysql, python-dotenv@==1.0.1, selenium, SQLAlchemy, urllib3, vertexai ### Recent commits (newest first) - fix - commit - asd - S - commited something - history - config for question - question + answer from backend - dash - transcibe - asdf - g - new navigate - mw - Remove sensitive information and use environment variables - updates to broker - updates to broker - proompting - proompting - broker ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "sfhacks", "version": "0.1.0", "scripts": { "dev": "concurrently -n fullstack,python -c blue,green \"cd fullstack && yarn dev\" \"cd python && python3 main.py\"", "postinstall": "cd fullstack && yarn install && cd ../python && pip3 install -r requirements.txt", "format": "prettier --write fullstack/src/**/*.{css,ts,tsx}" }, "dependencies": {}, "engines": { "node": "^18 || ^20" }, "devDependencies": { "@types/formidable": "^3.4.5", "concurrently": "^8.2.2", "lint-staged": "^15.2.0", "prettier": "^3.1.0" } } ``` ### python/requirements.txt ``` selenium numpy kafka-python pymysql SQLAlchemy urllib3 python-dotenv==1.0.1 matplotlib google-generativeai<0.6.0 vertexai openinference-instrumentation-vertexai arize-phoenix==5.2.2 arize-phoenix-otel==0.6.1 ``` ### fullstack/package.json ``` { "name": "fullstack", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@deepgram/sdk": "^3.8.1", "@radix-ui/react-dialog": "^1.1.2", "@radix-ui/react-slot": "^1.1.0", "@tabler/icons-react": "^3.19.0", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "ffmpeg-static": "^5.1.0", "fluent-ffmpeg": "^2.1.2", "formidable": "^3.5.2", "framer-motion": "^11.11.9", "kafkajs": "^2.2.4", "lucide-react": "^0.453.0", "mysql2": "^3.11.3", "net": "^1.0.2", "next": "14.2.15", "react": "^18", "react-dom": "^18", "react-icons": "^5.3.0", "recharts": "^2.13.0", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7", "tls": "^0.0.1", "twilio": "^5.3.4", "uuid": "^10.0.0", "vaul": "^1.1.0", "xmlbuilder": "^15.1.1" }, "devDependencies": { "@types/fluent-ffmpeg": "^2.1.21", "@types/mysql": "^2.15.26", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "@types/recharts": "^1.8.29", "@types/uuid": "^10.0.0", "aceternity-ui": "^0.2.2", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### python/main.py ```python import asyncio import json from kafka import KafkaConsumer, KafkaProducer from pipeline import SpiderPipeline KAFKA_BROKER = "8.tcp.us-cal-1.ngrok.io:16714" # Use your ngrok address here async def process_message(message, pipeline: SpiderPipeline, producer: KafkaProducer, consumer: KafkaConsumer): message_string = message.value.decode("utf-8") message_json = json.loads(message_string) tasks = [] if message_json.get("action") == "navigate": url = message_json["message"] uuid = message_json.get("uuid") tasks.append(pipeline.navigate(url, producer, uuid)) if message_json.get("action") == "create_browser": print("MESSAGE", message_json) uuid = message_json["uuid"] query = message_json["message"] print(uuid, query, "creating browser") producer.send( uuid, json.dumps( { "message": f"Created Spidey browser", "preview": None, "display": True, } ).encode("utf-8"), ) tasks.append(pipeline.run(query, producer, consumer, uuid)) tasks.append(pipeline.take_screenshots(producer, uuid)) if tasks: asyncio.gather(*tasks) async def consume_messages(consumer, pipeline, producer): while True: messages = consumer.poll(timeout_ms=1000) tasks = [] for topic_partition, msgs in messages.items(): for message in msgs: tasks.append(process_message(message, pipeline, producer, consumer)) if tasks: await asyncio.gather(*tasks) await asyncio.sleep(0.1) # Small delay to prevent CPU overuse async def main(): consumer = KafkaConsumer( "requests_topic", bootstrap_servers=[KAFKA_BROKER], ) producer = KafkaProducer( bootstrap_servers=[KAFKA_BROKER], ) pipeline = SpiderPipeline() print("READY FOR REQUESTS") await consume_messages(consumer, pipeline, producer) if __name__ == "__main__": asyncio.run(main()) ``` ### fullstack/src/app/page.tsx ```typescript import { SidebarDemo } from '@/components/demo'; export default function Home() { return (
); } ``` ### fullstack/src/app/layout.tsx ```typescript import type { Metadata } from 'next'; import localFont from 'next/font/local'; import './globals.css'; const geistSans = localFont({ src: './fonts/GeistVF.woff', variable: '--font-geist-sans', weight: '100 900', }); const geistMono = localFont({ src: './fonts/GeistMonoVF.woff', variable: '--font-geist-mono', weight: '100 900', }); export const metadata: Metadata = { title: 'Create Next App', description: 'Generated by create next app', }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### fullstack/src/app/dashboard/page.tsx ```typescript 'use client'; import { useState, useEffect } from 'react'; import { useRouter } from 'next/navigation'; function Page() { const [activeTab, setActiveTab] = useState('overview'); const router = useRouter(); useEffect(() => { console.log('Setting up EventSource'); const eventSource = new EventSource('/api/twilioRecordingComplete'); eventSource.onopen = () => { console.log('EventSource connection opened'); }; eventSource.onmessage = (event) => { console.log('Received message:', event.data); try { const transcription = JSON.parse(event.data).value; console.log('Parsed transcription:', transcription); // Navigate to the task page with the transcription as a query parameter router.push(`/dashboard/task?transcription=${encodeURIComponent(transcription)}`); // Close the event source after receiving the transcription eventSource.close(); } catch (error) { console.error('Error parsing message:', error); } }; eventSource.onerror = (error) => { console.error('EventSource error:', error); }; return () => { console.log('Closing EventSource'); eventSource.close(); }; }, [router]); const renderContent = () => { switch (activeTab) { case 'overview': return
Overview Content
; case 'tasks': return
Tasks Content
; case 'analytics': return
Analytics Content
; default: return
Select a tab
; } }; return (
{/* Sidebar */}

Dashboard

{/* Main content */}

{activeTab.charAt(0).toUpperCase() + activeTab.slice(1)}

{renderContent()}
); } export default Page; ``` ### fullstack/src/app/dashboard/layout.tsx ```typescript 'use client'; import React, { useState } from 'react'; import { Sidebar, SidebarBody, SidebarLink } from '../../components/ui/sidebar'; import { IconArrowLeft, IconSettings, IconChartBar, IconHistory, IconSparkles } from '@tabler/icons-react'; import Link from 'next/link'; import { motion } from 'framer-motion'; import Image from 'next/image'; import { cn } from '@/lib/utils'; import jason from '../../public/jason.jpeg'; function Layout({ children }: { children: React.ReactNode }) { const links = [ { label: 'Task', href: '/dashboard/task', icon: , }, { label: 'Previous Actions', href: '/dashboard/history', icon: , }, { label: 'Analytics', href: '/dashboard/analytics', icon: , }, { label: 'Configurations', href: '/dashboard/config', icon: , }, { label: 'Logout', href: '/start', icon: , }, ]; const [open, setOpen] = useState(false); return (
{open ? : }
{links.map((link, idx) => ( ))}
), }} />
{children}
); } export const Logo = () => { return (
Laila ); }; export const LogoIcon = () => { return (
); }; export default Layout; ``` ### fullstack/src/app/start/page.tsx ```typescript 'use client'; import React, { useState, useRef, useEffect, useCallback } from 'react'; import { BackgroundBeams } from '../../components/ui/background-beams'; import { TextHoverEffect } from '@/components/ui/text-hover-effect'; import { useRouter } from 'next/navigation'; import { motion } from 'framer-motion'; import Recorder from '@/components/TranscribeButton'; // Separate component for the textarea const TextArea = React.memo(({ value, onChange }: { value: string; onChange: (value: string) => void }) => { const textareaRef = useRef(null); const handleChange = (e: React.ChangeEvent) => { onChange(e.target.value); }; useEffect(() => { if (textareaRef.current) { textareaRef.current.style.height = 'auto'; textareaRef.current.style.height = `${textareaRef.current.scrollHeight}px`; } }, [value]); const router = useRouter(); useEffect(() => { async function fetchMessage() { const response = await fetch(`/api/twilioRecordingComplete`, { method: 'GET', headers: { 'Content-Type': 'application/json', }, }); const reader = response.body?.getReader(); if (!reader) return; let currentLine = ''; while (true) { const { done, value } = await reader.read(); if (done) break; currentLine += new TextDecoder().decode(value); if (currentLine.includes('\n')) { const messages = currentLine.split('\n').filter((item) => Boolean(item)); router.push(messages[0]); } } } fetchMessage(); }, []); return (