# Project export: Polaris

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Reimagining an internet made for everyone.
- Devpost: https://devpost.com/software/polaris-9shwca
- GitHub: https://github.com/LenaWang03/TreeHacks-2024
- Demo: https://tree-hacks-2024.vercel.app/
- Team: 5 GitHub contributor(s) — Eric Zhou (16 commits), Gordon Cheung (14 commits), LenaWang03 (13 commits), dylan (13 commits), Gordon Cheung (7 commits)

## Devpost submission (written by the team)

### Inspiration

The inspiration for Polaris came from witnessing the struggles that the elderly face in navigating the rapidly evolving digital landscape. In an age where the internet is integral to everyday life, it became clear that a significant portion of the population is left behind due to interfaces that are not designed with their needs in mind. Our goal was to create a solution that not only bridges this gap but also empowers the elderly to navigate the web with confidence and independence, ensuring that age is not a barrier to digital literacy and access. What It Does Polaris is a Chrome extension that revolutionizes web accessibility for the elderly. By allowing users to describe in natural language what they wish to accomplish on a website, Polaris guides them through the necessary steps. It captures and analyzes web page components, interprets the user’s intent through advanced language models, and provides visual cues and simple instructions via an overlay. This process simplifies web navigation, making digital spaces more inclusive and user-friendly. How We Built It We built Polaris using a combination of different frontend frameworks, APIs and machine learning technologies. The front end, developed as a Chrome extension and accompanying web app, captures user input and webpage elements processed using Beautiful Soup. The backend, powered by multi-modal language models, processes this data to understand the context and intent behind user commands. We leveraged open-source models like Llama2 and Mistral8x7B provided by together.ai's inference APIs and maximized their potential with prompt engineering. Our user-focused interface built with a combination of React, MaterialUI and Vite, allows us to simplify navigating the web for people of all age. Challenges We Ran Into One of the main challenges was ensuring accurate segmentation and interpretation of web page components in real time, which required optimizing our models for speed without sacrificing accuracy. Another challenge was designing an intuitive user interface that could be easily navigated by elderly users, necessitating several iterations based on user feedback. Accomplishments That We're Proud Of We are particularly proud of developing a solution that significantly improves web accessibility for the elderly, a demographic often overlooked in technology design. Successfully integrating complex AI technologies into a user-friendly application that can run efficiently as a Chrome extension stands as a testament to our team’s dedication and technical prowess. What We Learned Throughout this project, we gained deeper insights into the challenges of web accessibility and the potential of AI to address these issues. We learned the importance of user-centered design, especially when creating technology for populations with specific needs. Additionally, we honed our skills in AI development, particularly in prompt engineering and data processing, and learned valuable lessons in teamwork, project management, and iterative design. What's Next for Polaris Looking ahead, we aim to expand Polaris's capabilities to cover more complex web interactions and support additional browsers beyond Chrome. Our vision is a digital world whose benefits are universally accessible to everyone, irrespective of age. We see what is now just a simple but powerful chrome extension one day turning into a suite of tools aimed at improving equitable access to the Internet, providing seamless integrations with other technologies like browsers and OSes."

## README (from the GitHub repository)

# TreeHacks 2024

## Detected evidence (automated analysis)

Indexed codebase: 36 recognized source files, 9135 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
- Vercel (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (46 of 46)

```
.gitignore
client/.eslintrc.cjs
client/.gitignore
client/index.html
client/package.json
client/README.md
client/src/App.tsx
client/src/components/Card/Card.css
client/src/components/Card/Card.tsx
client/src/components/Chat/Chat.css
client/src/components/Chat/Chat.tsx
client/src/components/ChatInput/ChatInput.css
client/src/components/ChatInput/ChatInput.tsx
client/src/components/DefaultPrompt/DefaultPrompt.css
client/src/components/DefaultPrompt/DefaultPrompt.tsx
client/src/components/DefaultPrompts/DefaultPrompt/DefaultPrompts.css
client/src/components/DefaultPrompts/DefaultPrompt/DefaultPrompts.tsx
client/src/index.css
client/src/main.tsx
client/src/pages/Home.tsx
client/src/pages/Search.tsx
client/src/vite-env.d.ts
client/tsconfig.json
client/tsconfig.node.json
client/vite.config.ts
extension/background.js
extension/contentScript.js
extension/index.html
extension/manifest.json
extension/popup.js
extension/style.css
README.md
server/example_data/encoded_img.txt
server/example_data/facebook/facebook.html
server/example_data/facebook/facebook.json
server/example_data/gmail/gmail.html
server/example_data/google/google.html
server/example_data/google/google.json
server/get_url.py
server/load_ai_clients.py
server/models.py
server/next_step.py
server/server.py
server/soup.py
server/task_complete.py
server/utils.py
```

### Dependencies

- client/package.json: @emotion/react@^11.11.3, @emotion/styled@^11.11.0, @mui/material@^5.15.10, @types/react@^18.2.55, @types/react-dom@^18.2.19, @typescript-eslint/eslint-plugin@^6.21.0, @typescript-eslint/parser@^6.21.0, @vitejs/plugin-react@^4.2.1, eslint@^8.56.0, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.5, react@^18.2.0, react-dom@^18.2.0, react-icons@^5.0.1, react-router-dom@^6.22.1, router-dom@^2.2.11, typescript@^5.2.2, vite@^5.1.3

### Recent commits (newest first)

- changed name
- changed tagline
- add search hardcode functionality
- Merge branch 'main' of https://github.com/LenaWang03/TreeHacks-2024
- Gordon/tooltip overlay (#15)
- fix typo
- remove unused import
- changed website title
- Merge pull request #14 from LenaWang03/backend
- Delete describe_screenshot.py
- Merge pull request #13 from LenaWang03/add_popup
- added popup
- Merge pull request #12 from LenaWang03/pycache
- Stop tracking __pycache__ directories
- Merge pull request #11 from LenaWang03/backend
- Finish backend
- Merge branch 'main' of https://github.com/LenaWang03/TreeHacks-2024
- Merge pull request #10 from LenaWang03/frontend-rehaul
- rehaul
- Merge pull request #9 from LenaWang03/server-endpoints

## Key source files (fetched from GitHub, selected and truncated for size)

### client/package.json

```
{
  "name": "treehacks-2024",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc && vite build",
    "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0",
    "preview": "vite preview"
  },
  "dependencies": {
    "@emotion/react": "^11.11.3",
    "@emotion/styled": "^11.11.0",
    "@mui/material": "^5.15.10",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-icons": "^5.0.1",
    "router-dom": "^2.2.11"
  },
  "devDependencies": {
    "@types/react": "^18.2.55",
    "@types/react-dom": "^18.2.19",
    "@typescript-eslint/eslint-plugin": "^6.21.0",
    "@typescript-eslint/parser": "^6.21.0",
    "@vitejs/plugin-react": "^4.2.1",
    "eslint": "^8.56.0",
    "eslint-plugin-react-hooks": "^4.6.0",
    "eslint-plugin-react-refresh": "^0.4.5",
    "react-router-dom": "^6.22.1",
    "typescript": "^5.2.2",
    "vite": "^5.1.3"
  }
}

```

### server/server.py

```python
import uvicorn
from fastapi import Body, FastAPI
from fastapi.middleware.cors import CORSMiddleware
from get_url import get_url
from models import (GenerateNextStepRequest, GenerateNextStepResponse,
                    RedirectRequest, RedirectResponse)
from next_step import get_next_step, get_relevant_tag_ids
from soup import get_tag_details
from task_complete import is_complete

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],  # Allows all methods
    allow_headers=["*"],  # Allows all headers
)


@app.get("/")
async def health_check():
    """
    Health check route to ensure the server is running and responsive.
    """

    return {"message": "Server running!"}


@app.post("/redirect", response_model=RedirectResponse)
async def redirect(request: RedirectRequest):
    """
    Receives a natural language description of what the user wants to do on a website,
    and returns the website to visit and a refined prompt. If the user clicks the buttons,
    simply return the response. Otherwise, get the URL and refine the prompt.
    """
    if request.url is None:
        url = get_url(request.prompt)
        response = {"url": url, "prompt": RedirectRequest(request.prompt, url)}
    return response


@app.post("/generate-next-step/", response_model=GenerateNextStepResponse)
async def generate_next_step(request: GenerateNextStepRequest = Body(...)):
    """
    Receives details about the user's current state on the webpage and returns 
    directions for the next step they should take and the related DOM elements.
    task_complete returns True if the task has been completed.
    """
    # LLM call to determine if task is complete. It is complete if the prompt can be answered by looking on the current page (HTML)
    task_complete = is_complete(request.prompt, request.html)

    if task_complete:
        return {
            "directions": None,
            "relevant_tag_ids": None,
            "task_complete": True,
        }

    # get_next_step uses no knowledge of the HTML of the page and just returns the logical next step
    next_step = get_next_step(request.previous_steps, request.prompt)
    # get_relevant_tag_ids finds the relevant tag ids based on the generated next step and the HTML of the page
    tag_details = get_tag_details(request.html)
    relevant_tag_ids = get_relevant_tag_ids(next_step, tag_details)
    
    return {
        "directions": next_step,
        "relevant_tag_ids": relevant_tag_ids,
        "task_complete": False,
    }

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

```

### client/src/main.tsx

```typescript
import React from 'react'
import ReactDOM from 'react-dom/client'
import App from './App.tsx'
import './index.css'


ReactDOM.createRoot(document.getElementById('root')!).render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
)

```

### client/src/App.tsx

```typescript
import Chat from "./components/Chat/Chat";
import Home from "./pages/Home";
import Search from "./pages/Search";
import { BrowserRouter, Routes, Route } from "react-router-dom";


function App() {
  return (
    <BrowserRouter>
    <Routes>
      <Route path="/" element={<Home />} />
      <Route path="/search" element={<Search />} />
      <Route path="/home" element={<Chat />} />
    </Routes>
    </BrowserRouter>
  );
}

export default App;

```

### client/vite.config.ts

```typescript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react()],
})

```

### client/index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/logo.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Polaris</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.tsx"></script>
  </body>
</html>

```

### server/load_ai_clients.py

```python
import os

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

TOGETHER_API_KEY = os.environ.get("TOGETHER_API_KEY")
OPEN_AI_API_KEY = os.environ.get("OPEN_AI_API_KEY")

def load_together_client():
    client = OpenAI(
        api_key=TOGETHER_API_KEY,
        base_url="https://api.together.xyz",
    )
    return client

def load_openai_client():
    client = OpenAI(api_key=OPEN_AI_API_KEY)

    return client
```

### extension/background.js

```javascript
chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.action === "captureTab") {
    setTimeout(() => {
      chrome.tabs.captureVisibleTab(null, { format: "png" }, (dataUrl) => {
        // Handle the captured image here
        console.log({ dataUrl });
        sendResponse({ status: "Screenshot taken", image_url: dataUrl });
      });
    }, 2000); // Indicates you wish to send a response asynchronously
    return true;
  }
});

```

### extension/style.css

```css
h1 {
  color: red;
}

img {
    width: 120px;
}

body {
  border-radius: 20px;
  padding: 10px;
  width: 250px;
}

.topBar {
  color: #0057ff;
  font-size: 18px;
  display: flex;
  align-items: center;
  justify-content: space-between;
}

.line {
  border-bottom: 1px solid #cacaca;
  padding-top: 10px;
}

.subheading {
    font-size: 16px;
    
}

.bottomBar {
  color: #0057ff;
  font-size: 18px;
  display: flex;
  align-items: center;
  gap:10px;
  font-weight: bold;
  /* flex-direction: row; */
}

```

### server/task_complete.py

```python
from load_ai_clients import load_together_client


def is_complete(prompt: str, html: str):
    client = load_together_client()
    response = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": f"Can this task: '{prompt}' be completed on this page? Use the following HTML to answer: '{html}'. Answer just 'Yes' or 'No', one word.",
            }
        ],
        model="meta-llama/Llama-2-70b-chat-hf",
        max_tokens=1,
    )

    return response == "Yes"
```

[20 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]