# Project export: PULSE.AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: An AI code agent tool that integrates live browser session context for smarter coding assistance. It analyzes tabs, runs extra searches, and uses RAG to provide Codeium’s Cascade with extra context. - Devpost: https://devpost.com/software/pulse-ai-0oafm5 - GitHub: https://github.com/ekang7/PulseAI - Video: https://www.youtube.com/embed/wFDW0y9zVGU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Codeium: Best Use of Windsurf ($1k Cash + 1-Year Windsurf + Codeium Subscription)) - Team: 4 GitHub contributor(s) — LANCE BAE (25 commits), Edward Kang (16 commits), Yuen Ler Chow (13 commits), paula1284 (12 commits) ## Devpost submission (written by the team) ### Inspiration Developers constantly search the internet for solutions while coding—whether it's project planning documents, API documentation, recent research papers, or Stack Overflow discussions. We wanted to seamlessly integrate browser tab context into an AI code editor (Windsurf), making coding more efficient by surfacing relevant information automatically. ### What it does PulseAI captures browser context (screenshots, URLs, and titles), performs extra searches for adjacent topics via Perplexity, and stores & retrieves semantically relevant data using ChromaDB and Retrieval Augmented Generation (RAG) to bolster AI-assisted coding on Codeium's Windsurf-Cascade editor. We have two main workflows, a Passive Workflow for building up the browser context vector embedding ChromaDB and an Active Workflow for handling real-time user queries to the AI Code Editor. Passive Workflow Steps: The Chrome extension extracts raw browser context (a screenshot of the webpage along with its title and url) when the user decides to click “Add this”. Text is extracted using OCR and Pixtral (a multimodal Mistral model) describes any images in the raw data. Processed browser data is stored in ChromaDB for retrieval. Perplexity's Sonar API runs searches based on aforementioned browser data and user queries to preemptively retrieve other relevant information. Namely, we get the topic of the summarized context, generate similar topics, and perform searches for each of them to retrieve additional relevant contexts. Mistral's ministral-8b is used for quick summarization of the previous output. We utilize structured outputs for both APIs to ensure information is presented in a consistent fashion. ChromaDB embeds and stores search results and browser data for quick and effective retrieval. We get semantic vector embeddings here obtained from using the SentenceTransformers all-MiniLM-L6-v2 model. Active Workflow Steps: The user enters a query for the AI Code Editor. For context for Perplexity, RAG retrieves the top-k most relevant results from ChromaDB with semantic KNN with respect to the user query. Mistral's ministral-8b is used for quick summarization of the top-k results. We then use Perplexity’s Sonar API to retrieve additional similar topics. These results are parsed using Mistral and embedded in the ChromaDB. With these new results in the table, we use RAG to again retrieve the top-k most relevant results from ChromaDB with semantic KNN with respect to the user query. Mistral's ministral-8b is used for quick summarization of the top-k results resulting in a nice summarized context We created a custom MCP server/tool that is called by Cascade to inject a summary into its context. Cascade uses the summarized context to generate more relevant tips and code completions. Frontend In addition to the main tool, we built a frontend where hackers can view and delete database entries, for added flexibility and control. ### How we built it We built a Chrome extension to extract browser data, an agentic backend to process our information and store it in ChromaDB, a callable tool (MCP) that is callable by Cascade, and a visualization tool that lets the user view and edit the context generation process in real-time. ### Challenges we ran into Creating MCP tool: Learning about MCP tools, building one, and integrating it into Cascade's tool kit was something completely new to us. We had to research different potential methods of injecting dynamic context into coding assistants' contexts e.g., we were unable to find a method for Cursor. Debugging the MCP workflow for Codeium and making it fit smoothly into our pipeline took a significant amount of effort. Obtaining real-time browser context: We also struggled with getting real-time browser context in a usable format. We overcame this by taking advantage of Mistral’s new multimodal model Pixtral to form a pipeline to process the information and Mistral’s smaller ministral-8b model to summarize it into a more efficient form. ### Accomplishments we're proud of When we first decided on our project, we acknowledged that there were going to be several moving parts, and it's great to see that we were able to integrate them perfectly. We're also quite proud of our idea, as it expands a coding agent's ability to learn about its project outside of its IDE and at the same time minimizes context switches for the user. Furthermore, our product works almost exactly as intended and is a great proof of concept; with more work, we're confident it could be a move towards the next level of AI coding assistance. ### What we learned We learned about... How to make a browser extension that interacts with external servers. The MCP protocol, which was an introduction to how LLM tools are defined and used. Codeium's Windsurf/Cascade workspace and how to introduce personalized functionality. Perplexity's and Mistral's APIs and how to take advantage of structured outputs. RAG and how to use ChromaDB to retrieve semantically relevant information. ### What's next for PulseAI We aim to improve overall latency and implement a more automated browser + AI code editor experience that is even more hands-off while still respecting users' privacy. ## README (from the GitHub repository) # PulseAI: Browser-Based Coding Assistant Tool (TreeHacks 2025 Codeium Prize Winner for Best Use of Windsurf) PulseAI is a tool that enhances your AI coding assistant experience by providing your assistant with information about your (voluntarily captured) browser activity. By providing your assistant with a better understanding of your work, you'll be able to work with an agent that provides targeted suggestions and aid. ## Features - **Screenshot Capture**: A Chrome extension captures your browser screen and sends it to your PulseAI backend for processing. - **Autonomous Search**: Perplexity and Mistral are used to browse the web for related topics, fetching real-time information that may be relevant. - **Vector Search**: All of the retrieved search information is stored using ChromaDB, enabling semantic searches that retrieve relevant content quickly. - **MCP Integration**: Seamless integration with Codeium's Windsurf/Cascade MCP framework. ## Components ### Backend (`/backend`) The backend of PulseAI is built using FastAPI. It handles the processing of screenshots, the autonomous searches, and vector database / RAG. ### Chrome Extension (`/chrome_extension`) The Chrome extension used to capture screenshots when allowed. It sends these to the backend. ### MCP Tool (`/mcp_tool`) We created a new MCP tool that is used by Cascade to retrieve relevant context in real-time. ## Detected evidence (automated analysis) Indexed codebase: 35 recognized source files, 78 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Mistral AI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code ## Codebase structure (from repository index) ### Files (45 of 45) ``` .gitignore backend/.env.sample backend/clients/__init__.py backend/clients/mistral.py backend/clients/perplexity.py backend/db/__init__.py backend/db/vector_store.py backend/examples/vector_store_example.py backend/main.py backend/requirements.txt backend/setup.sh backend/test/setup_data.py backend/test/test_db.py backend/utils.py backend/view_screenshots.py chrome_extension/background.js chrome_extension/contentScript.js chrome_extension/manifest.json chrome_extension/package.json chrome_extension/popup.html chrome_extension/popup.js mcp_tool/client_util.py mcp_tool/requirements.txt mcp_tool/server.py mcp_tool/setup.sh mcp_tool/test_mcp.py README.md visualizer/.gitignore visualizer/package.json visualizer/public/index.html visualizer/public/manifest.json visualizer/public/robots.txt visualizer/README.md visualizer/src/App.css visualizer/src/App.js visualizer/src/App.test.js visualizer/src/components/DatabaseView.js visualizer/src/components/DocumentEditor.js visualizer/src/components/LogsView.js visualizer/src/components/NavBar.js visualizer/src/index.css visualizer/src/index.js visualizer/src/reportWebVitals.js visualizer/src/services/dbService.js visualizer/src/setupTests.js ``` ### Dependencies - backend/requirements.txt: chromadb, fastapi, mistralai, Pillow, pydantic, pytesseract, python-dotenv, requests, simplejson, uuid, uvicorn - chrome_extension/package.json: express@^4.21.2 - mcp_tool/requirements.txt: chromadb, fastmcp, httpx, mcp[cli], requests, uv - visualizer/package.json: @emotion/react@^11.14.0, @emotion/styled@^11.14.0, @mui/icons-material@^6.4.4, @mui/material@^6.4.4, @testing-library/dom@^10.4.0, @testing-library/jest-dom@^6.6.3, @testing-library/react@^16.2.0, @testing-library/user-event@^13.5.0, axios@^1.7.9, react@^19.0.0, react-dom@^19.0.0, react-router-dom@^7.1.5, react-scripts@5.0.1, web-vitals@^2.1.4 ### Recent commits (newest first) - Update README.md - Update README.md - Upgraded perplexity model - comment out - better button - change - changes - Merge branch 'main' of https://github.com/ekang7/PulseAI - ui changes - Merge branch 'main' of https://github.com/ekang7/PulseAI - updated readme - fix bugs - Merge branch 'main' of https://github.com/ekang7/PulseAI - small comment - colored logs - Merge remote-tracking branch 'refs/remotes/origin/main' - added passive and active perplexity things - Merge branch 'main' of https://github.com/ekang7/PulseAI - database viewer UI updates - Merge branch 'main' of https://github.com/ekang7/PulseAI ## Key source files (fetched from GitHub, selected and truncated for size) ### mcp_tool/requirements.txt ``` mcp[cli] httpx fastmcp uv chromadb requests ``` ### chrome_extension/package.json ``` { "dependencies": { "express": "^4.21.2" } } ``` ### backend/requirements.txt ``` requests simplejson mistralai chromadb pydantic python-dotenv pytesseract Pillow uvicorn uuid fastapi ``` ### visualizer/package.json ``` { "name": "visualizer", "version": "0.1.0", "private": true, "dependencies": { "@emotion/react": "^11.14.0", "@emotion/styled": "^11.14.0", "@mui/icons-material": "^6.4.4", "@mui/material": "^6.4.4", "@testing-library/dom": "^10.4.0", "@testing-library/jest-dom": "^6.6.3", "@testing-library/react": "^16.2.0", "@testing-library/user-event": "^13.5.0", "axios": "^1.7.9", "react": "^19.0.0", "react-dom": "^19.0.0", "react-router-dom": "^7.1.5", "react-scripts": "5.0.1", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### mcp_tool/server.py ```python from typing import Any import httpx from mcp.server.fastmcp import FastMCP from client_util import query_documents, summarize_results_with_mistral, call_active_perplexity # Initialize FastMCP server mcp = FastMCP("browser_context_fetcher") @mcp.tool() async def get_context_information(question: str = "") -> str: """ ALWAYS CALL THIS TOOL BEFORE DOING ANYTHING ELSE, ESPECIALLY SEARCHING THE WEB. Retrieves important information about what is relevant to the user's current request. `question` is what the user is asking about. """ # Get context data which includes the user's question try: # Use test question if provided, otherwise get from context if question: user_question = question elif context_data and "question" in context_data: user_question = context_data["question"] else: return "No question found in context" # add some more documents to the vector store related to the user's question call_active_perplexity(user_question) # Query the vector store for relevant documents rag_results = query_documents( query_text=user_question, n_results=3, collection_name="screenshots_collection" ) response = "Here is some potentially relevant information:" + "\n\n".join(rag_results["documents"]) return response except Exception as e: return "Something went wrong. Please let the user that the following error occurred:\n" + str(e) if __name__ == "__main__": # Initialize and run the server mcp.run(transport='stdio') ``` ### backend/main.py ```python from fastapi import FastAPI from pydantic import BaseModel import base64 from datetime import datetime from fastapi.middleware.cors import CORSMiddleware import os import io from PIL import Image import pytesseract from mistralai import Mistral from db.vector_store import add_documents, query_documents import logging from dotenv import load_dotenv from clients import mistral from typing import List, Any import queue from fastapi import Request from fastapi.responses import StreamingResponse import json import asyncio from utils import call_active_perplexity, call_passive_perplexity load_dotenv() MISTRAL_API_KEY = os.getenv("MISTRAL_API_KEY") MODEL = "ministral-8b-latest" client = Mistral(api_key=MISTRAL_API_KEY) # Set up logging logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[ logging.StreamHandler() ] ) logger = logging.getLogger(__name__) app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) class ActivePerplexityPayload(BaseModel): question : str class DocumentQueryPayload(BaseModel): query_text : str n_results : int collection_name : str class CollectiveSummaryPayload(BaseModel): sources : List[Any] class ScreenshotPayload(BaseModel): screenshot: str # data URL (e.g., "data:image/png;base64,iVBORw0KGgoAAAANS...") pageUrl: str pageTitle: str def extract_text_from_image(image_bytes): """Extract text from image using OCR""" image = Image.open(io.BytesIO(image_bytes)) text = pytesseract.image_to_string(image) return text.strip() def resize_image(image_bytes, max_size=(500, 500)): """Resize image while maintaining aspect ratio""" image = Image.open(io.BytesIO(image_bytes)) image.thumbnail(max_size, Image.Resampling.LANCZOS) # Convert back to bytes output_buffer = io.BytesIO() # Keep as PNG to maintain quality image.save(output_buffer, format='PNG') output_buffer.seek(0) return output_buffer.getvalue() def describe_image_with_pixtral(image_bytes): """Get image description using Pixtral model""" # Debug original image original_image = Image.open(io.BytesIO(image_bytes)) # Resize image if needed resized_image = resize_image(image_bytes) # Convert image to base64 for Pixtral base64_image = base64.b64encode(resized_image).decode('utf-8') return mistral.get_image_description(base64_image) @app.post("/api/query_documents") async def query_documents_endpoint(payload: DocumentQueryPayload): results = query_documents(payload.query_text, payload.n_results, payload.collection_name) return results @app.get("/api/list_all_documents") async def list_all_documents_endpoint(): from db.vector_store import list_all_documents return list_all_documents("screenshots_collection") class UpdateDocumentPayload(BaseModel): id: str content: str metadata: dict = {} @app.post("/api/update_document") async def update_document_endpoint(payload: UpdateDocumentPayload): """ Receives an ID, new content, and metadata to update a document in the 'screenshots_collection' (or any other collection). """ from db.vector_store import update_document try: update_document( id=payload.id, new_content=payload.content, new_metadata=payload.metadata, collection_name="screenshots_collection" ) return { "status": "success", "message": f"Document {payload.id} updated." } except Exception as e: logger.error(f"Error updating document {payload.id}: {e}") return { "status": "error", "message": str(e) } class DeleteDocumentPayload(BaseModel): id: str @app.post("/api/delete_document") async def delete_document_endpoint(payload: DeleteDocumentPayload): """ Endpoint to delete a document by ID. """ from db.vector_store import delete_document try: delete_document(payload.id, collection_name="screenshots_collection") return {"status": "success", "message": f"Document {payload.id} deleted."} except Exception as e: logger.error(f"Error deleting document {payload.id}: {str(e)}", exc_info=True) return {"status": "error", "message": str(e)} @app.post("/api/collective_summary") async def collective_summary_endpoint(payload: CollectiveSummaryPayload): summary = mistral.get_collective_summary(payload.sources) return {"summary": summary} # api endpoint for active perplexity @app.post("/api/call_active_perplexity") async def call_active_perplexity_endpoint(payload : ActivePerplexityPayload): question = payload.question try: call_active_perplexity(question) except Exception as e: logger.error(f"Error calling active perplexity: {str(e)}", exc_info=True) return {"status": "error", "message": str(e)} @app.post("/api/upload") async def upload_screenshot(payload: ScreenshotPayload): """ Receives a base64-encoded PNG from the Chrome Extension, plus the page title and URL. Process uploaded screenshots: 1. Save the image 2. Extract text using OCR 3. Get image description using Pixtral 4. Store in vector DB """ try: logger.info("Processing new screenshot upload") # The 'screenshot' is a data URL: "data:image/png;base64," # We only want the base64 data after the comma # So, split the data URL to get base64 data header, encoded_image = payload.screenshot.split(",", 1) image_bytes = base64.b64decode(encoded_image) logger.info(f"Decoded image bytes length: {len(image_bytes)}") # Generate timestamp and filenames timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") filename = f [truncated — 3863 more characters] ``` ### visualizer/src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### visualizer/src/App.js ```javascript import React from "react"; import { BrowserRouter as Router, Routes, Route } from "react-router-dom"; import Navbar from "./components/NavBar"; import DatabaseView from "./components/DatabaseView"; import LogsView from "./components/LogsView"; import DocumentEditor from "./components/DocumentEditor"; import { Container } from "@mui/material"; function App() { return ( } /> } /> } /> ); } export default App; ``` ### mcp_tool/setup.sh ```shell export PYTHONPATH=$PYTHONPATH:$(pwd) ``` ### backend/setup.sh ```shell # A script to just add the current directory to your Python path export PYTHONPATH=$PYTHONPATH:$(pwd) ``` [26 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]