# Project export: Project Jarvis

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Making an AI Agent that interacts with the web how humans do.
- Devpost: https://devpost.com/software/project-jarvis
- GitHub: https://github.com/DavidUlloa6310/jarvis
- Video: https://www.youtube.com/embed/sOsbbBz1jNI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — David Ulloa (3 commits), Jose Pujol (1 commits)

## Devpost submission (written by the team)

### Inspiration

We have been recently inspired by the advent of what some are calling "Large Action Models" which can use natural language to perform a given task, like play your favorite song on Spotify. Rabbit R1 has been a recent proponent of this new paradigm.

### What it does

It is an AI agent that can look at the user's web browser and perform task specified by the user through natural language such as "create a new Tweet."

### How we built it

We built it using Meta's Segment Anything model to get the model to understand the web browser. We also leverage OpenCLIP which is an open source multimodal model that can bridge images and text. We use the different components of the web to allow the model to decide the best actionable steps based on the user's query.

### Challenges we ran into

We ran into numerous challenges. We first wanted to use decision transformers but did not have the data, compute power, or large dataset to accomplish it. We then began to figure out how to use the current open source models to piece together a working proof of concept. As an added issue, it is difficult to control inputs and output devices due to security and privacy concerns from web browsers.

### Accomplishments we're proud of

We built a model that can understand the different components of a web site and with relatively substantial accuracy determine what action to take based on a user's prompt. We had no idea how we were going to do this when we first talked about it, but we are proud of the progress we made and how feasible the solution became.

### What we learned

We learned many valuable skills such as running open source models locally and gained deeper understanding for how these models work cohesively to accomplish a task.

### What's next

We will be continuing this project by using a more modern architecture with decision transformers so that we can chain actions together to give the model power to perform more complex tasks. We can also integrate speech-to-text for a seamless user interface.

## README (from the GitHub repository)

# Jarvis


## Detected evidence (automated analysis)

Indexed codebase: 8 recognized source files, 3 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- LangChain (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (10 of 10)

```
.gitignore
backend/embeddings/embed.py
backend/main.py
extension/background.js
extension/content-script.js
extension/manifest.json
extension/popup.html
extension/popup.js
extension/styles.css
README.md
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Made initial embedding file
- Initial Commit
- Initial Commit - README

## Key source files (fetched from GitHub, selected and truncated for size)

### backend/main.py

```python
import json

from flask import Flask, request

app = Flask(__name__)

@app.route("/action", methods = ["POST", "GET"])
def index():
    print(request.data)
    return json.dumps({"message": "test"}), 200, {'Content-Type': 'application/json'}

if __name__ == '__main__':
    app.run(port = 3001, debug = True)

```

### extension/styles.css

```css
.ext-container {
    min-width: 250px;
    min-height: 400px;
}

.sub-container {
    display: flex;
    flex-direction: column;
    gap: 5px;
}

```

### extension/content-script.js

```javascript
const script = document.createElement("script");

// Add scripts you want to add into runtime here:
script.src = chrome.runtime.getURL("popup.js");

(document.head || document.documentElement).appendChild(script);
script.onload = function() {
    script.remove();
};


```

### extension/popup.js

```javascript
document.addEventListener("DOMContentLoaded", function() {
    const button = document.getElementById("queryButton");
    if (button) {
        button.addEventListener("click", function() {
            alert("Button clicked!");
            chrome.runtime.sendMessage({ action: "captureScreenshot" });
        });
    }
});


```

### extension/popup.html

```html
<html>
    <head>
        <title>Jarvis</title>
        <link rel="stylesheet" href = "styles.css">
    </head>
    <body class = "ext-container">
        <h1>Hey, Jarvis here!</h1>
        <p>
        Jarvis is here to help you solve any and all problems on the internet - like a real personal assistant.
        </p>
        <div class = "sub-container">
            <textarea placeholder = "Enter your query..." id = "queryText"></textarea>
            <button id = "queryButton">
                Submit
            </button>
        </div>
        <script src="popup.js"></script>
    </body>
</html>

```

### extension/background.js

```javascript
chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
  if (request.action === "captureScreenshot") {
    chrome.tabs.captureVisibleTab(null, { format: 'png' }, function(imageUri) {

        const headers = {
            "Content-Type": "application/json"
        }

        const image = imageUri.replace(/^data:image\/(png|jpeg);base64,/, ''); // base64 encoded
        const body = {
            image,
        }

        fetch("http://localhost:3001/action", { method : "POST", headers, body}).then(response => {
            if (!response.ok) {
                return new Error("Error");
            }
            return response.json();
        }).then(data => {
            console.log(data);
        }).catch(err => {
            console.log(err);
        })

      sendResponse({ action: "click", point: [1.00, 2.00]});
    });
  }
  return true; // Indicate that sendResponse will be called asynchronously
});


// Encode the segment, hashmap the encoding with the position, pass encoding into model, 

```

### backend/embeddings/embed.py

```python
from langchain_experimental.open_clip import OpenCLIPEmbeddings


MODEL = "ViT-B-32"
CHECKPOINT = "laion2b_s34b_b79k"
clip_embd = OpenCLIPEmbeddings(MODEL, CHECKPOINT)


def split_image(masks, image):
    sub_images = []
    for pred in masks:
        boundary = pred['bbox']
        x, y, w, h = [int(v) for v in boundary]
        sub_images.append(image[y:y+h, x:x+w])
    return sub_images


def embed_image(image):
    return clip_embd.embed_image([image]) # Should be URI


def embed_text(text):
    return clip_embd.embed_documents([text])
```