# Project export: BrowseBlind This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: World's First AI Browser. Allowing blind people to interact for the first time with the internet. Also helping visually impaired. Also, automated processes help average people save time. - Devpost: https://devpost.com/software/browseblind - GitHub: https://github.com/RoyceAroc/browseblind - Demo: https://browseblind.co/ - Video: https://www.youtube.com/embed/39JkocXmKnA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Fetch.ai: Agentic Track Prize) - Team: 1 GitHub contributor(s) — Royce Arockiasamy (16 commits) ## Devpost submission (written by the team) ### Inspiration I was doing an application last week when one of the questions about diversity prompted me into thinking about how we take things for more than just granted. It struck me that blind people aren't able to interact with the internet (shop on e-commerce websites, research to learn, play video games, social media - the list is quite endless). I then asked my dad, who works in the computer-science industry, about whether the visually impaired and blind are able to contribute to their likeness in the workforce when I realized how far off the scales were tipped. Assil Eye Institute If I were in their shoes, then I would immediately build such a software to be connected like and with everyone else, but in their case I would be blind so I wouldn't be able to, luckily I'm not so I know the right thing to do would be to build a software for them. Also, I realized that for people like me who are quite lazy or would like parts of their browser processes being automated (like telling it to reserve my tennis court from 5-6 PM rather than doing it myself), an AI browser might also sound quite valuable. Rather more of a business market "want" than a "need", I think regardless it's novelty in the market and it's usefulness for a customer segment (did a bit of customer discovery as well reaching out to centers and found out directly from people that they are very much look forward to it) makes this idea worth venturing about this hackathon in CalHacks. ### What it does It's the same as your regular web browser application (Chrome, Safari, Brave, etc) in the way you surf the web, but in addition, you can do so freely while being blind-folded. Well, how does that work? Also, controlled by text and/or speech input (pressing the space bar for over a second starts a recording and leaving it ends the recording), any user's instructions are followed by the browser. When a user lands on a page, a quick summary of the page is read including the nav bar components, etc so the user can navigate and explore the web just like any other person. A user can ask the browser to click on any element/part of the page and also fill in information on the page via true NLP. They can also open/close tabs, search using the search engine, ask questions (variant of RAG approach taken) over the current page, save the page locally/print the page, and so many other tasks that the current set of agents have to offer. Every time something new is displayed on the browser (ex. going to a different page) they are notified so they can take judgements. ### How we built it Using the PyQt5 Browser Development framework, I started my code by building a browser. I organized my prompt engineering agentic framework using Fetch AI's agentic system. For prompting questions with images/text I used Gemini's models. I also used Gemini's bounding box model for detecting where to proceed next on the page. This was not too accurate, so I coupled it with my algorithm I wrote where I take the html code of a website and parsed it down to the important segments (removing PII and unnecessary contents) in order to save tokens and decide based on html where to move next as a backup. After much testing, I settled on Groq for the decision-making segments of the LLM chain for its speed. I also used it for the STT part where the user has the option to speak in our application. The TTS part was handled by DeepGram and other voice agent integrations. Building requires testing and I tested this by imitating a blind person by being blind. One such successful testing included making accounts on websites I have never visited before. ### Challenges we ran into Parsing the HTML took one of the longest parts due to its implicit complexity. Event listeners attached to elements across the DOM tree. Event listeners on images like a hamburger icon which has no text so a mapping is needed. These multiple edge cases had to be considered before reaching the threshold where it was doing perfect on every website as it is doing now. Another challenge I ran into was fine-tuning. The accuracy of this model was at around 60% and it was a hard and enduring work to get it to around 95% where it is currently at right now. ### Accomplishments we're proud of Controlling the browser by "having a 2-street conversation" with it simply blows my mind and really changes the way one surfs the web. It's really fun and really useful at the same time. ### What we learned How to build your own browser. How to fine tune the boundary box model with Gemini through prompt engineering to extract high accurate insights. Learned documentation for multiple voice agent companies that were integrated. ### What's next Perfecting the software model and finding ways to even further reduce token consumption in order to push this into the market as soon as possible. Also, building "background tabs" a feature I didn't have time to finish but essentially you can give a tab a task (ex. find the part on the wikipedia page that talks about fourier transform or find the contact page for X, Y, Z company) and the tab does the task in the background and comes back up when finished. ## README (from the GitHub repository) # Browse Blind World's First AI Browser for the blind https://github.com/user-attachments/assets/8da2bcda-75d1-4843-b967-2b974fdcdbd8 ## HomePage URL Website at [browseblind.co](https://browseblind.co) ## Code Base - Frontend code on root level - Backend folder has desktop application source code ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 55 KB. - HTML (language) — detected in the code - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (37 of 37) ``` .DS_Store .gitignore backend/main.spec backend/poetry.lock backend/pyproject.toml backend/README.md backend/src/agents/main.py backend/src/agents/models/__init__.py backend/src/agents/models/browser_agents/close_tab.py backend/src/agents/models/browser_agents/go_back.py backend/src/agents/models/browser_agents/go_forward.py backend/src/agents/models/browser_agents/print_page.py backend/src/agents/models/browser_agents/reload_page.py backend/src/agents/models/browser_agents/search_engine.py backend/src/agents/models/browser_agents/url_engine.py backend/src/agents/models/browser.py backend/src/agents/models/decision_agents/browser_agent.py backend/src/agents/models/decision_agents/inner_agent.py backend/src/agents/models/decision_agents/onload_agent.py backend/src/agents/models/decision_agents/processing_agent.py backend/src/agents/models/decision_agents/question_agent.py backend/src/agents/models/decision_agents/read_page_agent.py backend/src/agents/models/inner_agents/click/click_agent.py backend/src/agents/models/inner_agents/input/text_agent.py backend/src/agents/schemas/message.py backend/src/llms/deepgram.py backend/src/llms/gemini.py backend/src/llms/groq.py backend/src/main.py backend/src/main.spec backend/src/utils/html_parser.py backend/src/utils/llm_parsing.py backend/src/voice_model.py CNAME index.html packaging/README.md README.md ``` ### Dependencies - backend/pyproject.toml: beautifulsoup4@^4.12.3, google-generativeai@^0.8.3, groq@^0.11.0, keyboard@^0.13.5, pillow@^11.0.0, pygame@^2.6.1, pyinstaller@^6.10.0, pyqt5@^5.15.11, pyqtwebengine@^5.15.7, uagents@^0.16.2 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - change - update - update - we done?? - lots of changes - first victory of the night - first fetch ai agent baby - added html parser - browser - basic browser init - Create CNAME - Delete CNAME - Create CNAME - Delete CNAME - init - Create CNAME ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/pyproject.toml ``` [tool.poetry] name = "browseblind-package" version = "0.1.0" description = "" authors = ["RoyceAroc "] readme = "README.md" [tool.poetry.dependencies] python = ">=3.9,<3.13" pyqt5 = "^5.15.11" pyqtwebengine = "^5.15.7" pyinstaller = "^6.10.0" uagents = "^0.16.2" groq = "^0.11.0" google-generativeai = "^0.8.3" pillow = "^11.0.0" beautifulsoup4 = "^4.12.3" keyboard = "^0.13.5" pygame = "^2.6.1" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" [virtualenvs] virtualenvs.in-project = "false" ``` ### backend/src/agents/main.py ```python from agents.models.browser import create_browser_agents from agents.models.decision_agents.processing_agent import init_processing_agent from agents.models.decision_agents.onload_agent import init_onload_agent from agents.models.decision_agents.browser_agent import init_browser_agent from agents.models.decision_agents.read_page_agent import init_read_page_agent from agents.models.decision_agents.question_agent import init_question_agent from agents.models.decision_agents.inner_agent import init_inner_agent from agents.models.inner_agents.click.click_agent import init_click_agent from agents.models.inner_agents.input.text_agent import init_text_agent def create_agents(browser): browser_agents = create_browser_agents(browser) click_agent = init_click_agent(browser) text_agent = init_text_agent(browser) processing_agent = init_processing_agent(browser) onload_agent = init_onload_agent(browser) browser_agent = init_browser_agent(browser) read_page_agent = init_read_page_agent(browser) question_agent = init_question_agent(browser) inner_agent = init_inner_agent(browser) decision_agents = [ processing_agent, onload_agent, browser_agent, read_page_agent, question_agent, inner_agent, ] return decision_agents + browser_agents + [click_agent, text_agent] ``` ### backend/src/main.py ```python import sys import os from PyQt5.QtWidgets import ( QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout, QLineEdit, QTabWidget, QSizePolicy, QDesktopWidget, QTextEdit, ) from PyQt5.QtWebEngineWidgets import QWebEngineView, QWebEnginePage from PyQt5.QtCore import QUrl, QTimer, pyqtSignal, Qt from PyQt5.QtGui import QFont from uagents import Agent, Bureau, Context from datetime import datetime as dt from agents.main import create_agents from agents.schemas.message import Message import threading import asyncio import json from llms.groq import run_groq, run_groq_stt from llms.gemini import run_gemini from utils.html_parser import get_new_html import PIL.Image import PIL.ImageDraw import re import string import random import sys import threading import pyaudio import wave from PyQt5.QtWidgets import QApplication, QWidget from PyQt5.QtCore import pyqtSignal, QObject import time import keyboard from voice_model import play_sound global main_agent_ctx main_agent_ctx = None global agents_list agents_list = [] global current_msg_id current_msg_id = "" main_agent = Agent(name="main_agent", seed="sigmar recovery phrase") def generate_random_string(length): letters_and_digits = string.ascii_letters + string.digits return "".join(random.choice(letters_and_digits) for _ in range(length)) class ConsoleLoggingPage(QWebEnginePage): def javaScriptConsoleMessage(self, level, message, line, source): level_str = {0: "INFO", 1: "WARNING", 2: "ERROR"}.get(level, "UNKNOWN") print(f"JS [{level_str}] {message} (line {line}) in {source}") class BrowserTab(QWidget): screenshot_completed = pyqtSignal() def __init__(self, parent=None, url="https://browseblind.co"): super().__init__() self.parent_widget = parent self.layout = QVBoxLayout() self.url_bar = QLineEdit() self.url_bar.setStyleSheet( """ QLineEdit { padding: 8px 12px; border: 2px solid #506159; border-radius: 5px; background-color: #a1928a; font-size: 14px; color: #333333; margin: 5px; } QLineEdit:focus { border-color: #a1928a; background-color: white; } QLineEdit:hover { border-color: #dbb6a2; } """ ) self.web_view = QWebEngineView() self.html = None self.fields = None self.summary = None self.page = ConsoleLoggingPage(self.web_view) self.web_view.setPage(self.page) self.url_bar.returnPressed.connect(self.navigate_to_url) self.layout.addWidget(self.url_bar) self.web_view.setUrl(QUrl(url)) self.web_view.urlChanged.connect(self.update_url_bar) self.web_view.titleChanged.connect(self.update_title) self.web_view.loadFinished.connect(self.run_onload_script) self.layout.addWidget(self.web_view) self.setLayout(self.layout) def navigate_to_url(self): url = self.url_bar.text() if not url.startswith(("http://", "https://")): url = "https://" + url self.web_view.setUrl(QUrl(url)) def update_url_bar(self, q): self.url_bar.setText(q.toString()) def update_title(self, title): if self.parent_widget: index = self.parent_widget.indexOf(self) self.parent_widget.setTabText(index, title if title else "New Tab") def capture_screenshot(self): screenshot = self.web_view.grab() screenshot.save("tab.png", "png") self.screenshot_completed.emit() def run_onload_script(self): QTimer.singleShot(100, lambda: self.execute_onload_script(True)) def v2_onload_script(self): self.execute_onload_script() def execute_onload_script(self, init=False): js = """ (function() { let elements = document.querySelectorAll('*'); let results = []; elements.forEach(function(element) { let elementInfo = { tag: element.tagName, events: [], xpath: getXPath(element) }; let eventAttributes = Array.from(element.attributes).filter(attr => attr.name.startsWith('on') ); eventAttributes.forEach(attr => { elementInfo.events.push({ type: attr.name.slice(2), source: 'inline' }); }); if (elementInfo.events.length > 0) { results.push(elementInfo); } }); let final = { "results": results, "innerHTML": document.documentElement.outerHTML } return JSON.stringify(final, null, 2); })(); function getXPath(element) { let parts = []; let currentElement = element; while (currentElement && currentElement.nodeType === Node.ELEMENT_NODE) { let siblings = Array.from(currentElement.parentNode ? currentElement.parentNode.children : []); let sameTagSiblings = siblings.filter(sibling => sibling.nodeName === currentElement.nodeName); if (sameTagSiblings.length > 1) { let index = sameTagSiblings.indexOf(currentElement) + 1; parts.push(currentElement.nodeName.toLowerCase() + `[${index}]`); } else { parts.push(currentElement.nodeName.toLowerCase()); } currentElement = currentElemen [truncated — 21807 more characters] ``` ### index.html ```html Browse Blind | AI Browser

BrowseBlind.co

⠃⠗⠕⠺⠎⠑⠃⠇⠊⠝⠙

BrowseBlind Demo
  • World's First Interactive AI Browser
  • For the first time, visually impaired can interact with the internet with more than a screenreader
  • Ask to click/fill anything, open/close tabs, ask questions, summarize, RAG, automate a process, etc
  • Optimized Agentic Framework costing < $1 for 8 hours of internet usage
  • Powered by Text/STT Input and TTS Output

Powered by

Google Logo Fetch Logo Groq Logo Deepgram Logo
``` ### backend/src/voice_model.py ```python import llms.deepgram import pygame import os chosen_model = "deepgram" def play_sound(data): if chosen_model == "deepgram": llms.deepgram.make_audio(data) pygame.mixer.init() sound = pygame.mixer.Sound(os.path.dirname(__file__) + "/audio.mp3") sound.play() ``` ### backend/src/llms/gemini.py ```python import random import google.generativeai as genai from configs import GEMINI_API_KEY genai.configure(api_key=random.choice(GEMINI_API_KEY)) model = genai.GenerativeModel("gemini-1.5-flash") def run_gemini(image, prompt): response = model.generate_content([prompt, image]) s = response.text return s ``` ### backend/src/utils/llm_parsing.py ```python import json def extract_and_parse_json(input_string): try: input_string = input_string.replace("“", '"').replace("”", '"') start_idx = input_string.find("{") end_idx = input_string.find("}", start_idx) json_str = input_string[start_idx : end_idx + 1] json_obj = json.loads(json_str) return json_obj except Exception as e: return None ``` ### backend/src/llms/deepgram.py ```python import requests import random from configs import DEEPGRAM_TOKEN url = "https://api.deepgram.com/v1/speak?model=aura-asteria-en" headers = { "Authorization": f"Token {random.choice(DEEPGRAM_TOKEN)}", "Content-Type": "text/plain", } def make_audio(data): response = requests.post(url, headers=headers, data=data) with open("audio.mp3", "wb") as f: f.write(response.content) return "audio.mp3" ``` ### backend/src/llms/groq.py ```python import os import random from groq import Groq from configs import GROQ_API_KEY client = Groq( api_key=random.choice(GROQ_API_KEY), ) def run_groq(prompt): chat_completion = client.chat.completions.create( messages=[ { "role": "user", "content": prompt, }, ], model="llama3-8b-8192", ) return chat_completion.choices[0].message.content def run_groq_stt(filename): with open(filename, "rb") as file: transcription = client.audio.transcriptions.create( file=(filename, file.read()), model="whisper-large-v3-turbo", prompt="User giving instructions", response_format="json", language="en", temperature=0.0, ) return transcription.text ``` ### backend/src/utils/html_parser.py ```python from bs4 import BeautifulSoup, Comment, NavigableString def get_xpath(element): parts = [] for parent in element.parents: siblings = parent.find_all(element.name, recursive=False) if len(siblings) > 1: index = siblings.index(element) + 1 parts.append(f"{element.name}[{index}]") else: parts.append(element.name) element = parent parts.reverse() return "/" + "/".join(parts) def get_new_html(old_html, event_listeners): global html_code html_code = "" soup = BeautifulSoup(old_html, "html.parser") global elements elements = [] for element in soup.find_all(True): def list_append(clickable, elem, xpath=None, input_status=False): global html_code global elements if input_status: elements.append({"type": "input", "xpath": get_xpath(elem)}) input_attributes = elem.attrs input_attributes_str = " ".join( f"data='{value}'" for key, value in input_attributes.items() ) html_code += ( "" ) else: if clickable: if xpath is not None: elements.append({"type": "click", "xpath": xpath}) else: elem_inner_text = elem.get_text(strip=True).strip() elements.append({"type": "click", "xpath": get_xpath(elem)}) if len(list(element.children)) == 1: html_code += ( "" + elem_inner_text + "" ) else: text = elem.strip() if len(text) != 0: html_code += " " + elem.strip() + " " tag_name = element.name.lower() if tag_name == "a" or tag_name == "button": list_append(True, element) elif tag_name == "script" or tag_name == "style" or tag_name == "noscript": continue elif tag_name == "input" or tag_name == "textarea" or tag_name == "select": list_append(False, element, None, True) else: roy = list(element.children) for idx in range(len(roy)): if isinstance(roy[idx], NavigableString) and not isinstance( roy[idx], Comment ): curr_elem = list(element.children)[idx] list_append(False, curr_elem) curr_elem_xpath = get_xpath(element) for event in event_listeners: if event["xpath"] == curr_elem_xpath: list_append(True, element, curr_elem_xpath) return html_code, elements ``` [17 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]