# Project export: Filtra

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: An application that helps you focus by filtering any unnecessary distractions from your web browser.
- Devpost: https://devpost.com/software/filtra
- GitHub: https://github.com/theivanyeung/Filtra
- Team: 3 GitHub contributor(s) — Jack Luo (7 commits), Ivan Yeung (5 commits), zacharyzhang04 (4 commits)

## Devpost submission (written by the team)

### Inspiration

You get online to do your real analysis coursework, opening a new tab in Chrome to reach an online Latex compiler. But there's an ad! New AF1's for only 75 bucks? You click the ad and get redirected to Amazon, and before you know it, you fail real analysis, fail to get your degree, gamble away your money, and are very, very sad. Filtra helps you blur those distractions on your browser. You finish your work on time, which leads to more learning, more productivity, and better grades.

### What it does

There is a focus mode and a relaxing mode. On relax mode, nothing is filtered. On focus mode, everything is filtered depending on whether or not it helps you become more productive or not.

### How we built it

We use proprietary LLM technology to decide what elements will help you become more productive or not. We have a NextJS Frontend and Flask backend that integrate seamlessly with OpenAI API in order to decide which HTML elements in the website DOM are necessary.

### Challenges we ran into

Querying all the elements of the DOM is relatively slow, and even slower is OpenAI API. It takes much time for OpenAI to decide which elements can be unfiltered. Thus, for each loaded website, one challenge is that it takes many seconds for the screen to load and the HTML elements to show up. One way to speed this up is to use faster ML technologies than LLMs in order to determine if an element is "productive" or not.

### Accomplishments we're proud of

The technology works to filter elements out in the website, and the UI is clean and easy to use. It is very self-explanatory (see video link below). Once you have the application running, you can easily toggle between two modes, "focus" and "relax."

### What we learned

Transformer technology can be used for a wide variety of use cases, and LLMs can be used for things beyond NLP. In this case, the OpenAI API is able to process web elements in the form of HTML elements in our DOM, and filter out elements and divs that are deemed unproductive.

### What's next

Speed, speed, speed...! The only issue that we are facing is speed of API responses, which causes some of the web pages to load and be processed very slowly. In the future, we plan to test out other ML models to see if we can achieve more speed and greater accuracy in determining which web elements are useless and can be filtered out.

## README (from the GitHub repository)

# Filtra

## Detected evidence (automated analysis)

Indexed codebase: 23 recognized source files, 54 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (32 of 32)

```
app/.eslintrc.json
app/.gitignore
app/app/api/engine/route.ts
app/app/api/synthesize/route.ts
app/app/api/transcribe/route.ts
app/app/layout.tsx
app/app/page.tsx
app/components/modules/useToastManager.ts
app/next.config.mjs
app/package.json
app/README.md
app/styles/customTheme.ts
app/styles/fonts.css
app/styles/global.css
app/tsconfig.json
app/Vesper/OpticalCircle.js
app/Vesper/rings/CenterRing.js
app/Vesper/rings/InnerRing.js
app/Vesper/rings/OuterRing.js
app/Vesper/rings/UpperRing.js
chrome-extension/background.js
chrome-extension/content.js
chrome-extension/manifest.json
engine/.env
engine/api/app.py
engine/components/prompt.py
engine/README.md
LICENSE
platform/index.html
platform/main.js
platform/package.json
README.md
```

### Dependencies

- app/package.json: @chakra-ui/icons@^2.1.1, @chakra-ui/react@^2.8.2, @emotion/react@^11.11.3, @emotion/styled@^11.11.0, @react-three/drei@^9.97.6, @react-three/fiber@^8.15.16, @types/node@^20, @types/react@^18, @types/react-dom@^18, axios@^1.6.7, electron@^28.2.3, eslint@^8, eslint-config-next@14.1.0, framer-motion@^11.0.5, next@14.1.0, react@^18, react-dom@^18, three@^0.161.0, typescript@^5

### Recent commits (newest first)

- Merge branch 'master' of https://github.com/theivanyeung/Filtra
- general: updated description for project
- finished electron platform
- completed frontend
- Merge branch 'master' of https://github.com/theivanyeung/Filtra
- prompt engineering: evaluating distractions
- add the background
- Merge pull request #1 from theivanyeung/main
- merging
- content for the chrome extension
- backend: update image classifer and fix bugs
- baackend: analyze text for distraction
- backend: edge case text
- initialized electron app
- intiialized frameworks
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### platform/package.json

```
{
  "scripts": {
    "start": "electron ."
  }
}

```

### app/package.json

```
{
  "name": "app",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@chakra-ui/icons": "^2.1.1",
    "@chakra-ui/react": "^2.8.2",
    "@emotion/react": "^11.11.3",
    "@emotion/styled": "^11.11.0",
    "@react-three/drei": "^9.97.6",
    "@react-three/fiber": "^8.15.16",
    "axios": "^1.6.7",
    "electron": "^28.2.3",
    "framer-motion": "^11.0.5",
    "next": "14.1.0",
    "react": "^18",
    "react-dom": "^18",
    "three": "^0.161.0"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "eslint": "^8",
    "eslint-config-next": "14.1.0",
    "typescript": "^5"
  }
}

```

### platform/main.js

```javascript
const {
  app,
  BrowserWindow,
  screen,
  ipcMain,
  shell,
  desktopCapturer,
} = require("electron");
const path = require("path");

let win;

function createWindow() {
  const { width, height } = screen.getPrimaryDisplay().workAreaSize;

  const size = 500;

  win = new BrowserWindow({
    width: Math.round(size),
    height: Math.round(size),
    icon: path.join(__dirname, "/build/icon.ico"),
    frame: false,
    transparent: true,
    webPreferences: {
      nodeIntegration: true,
      contextIsolation: true,
      enableRemoteModule: true,
      preload: path.join(__dirname, "electron_modules/preload.js"),
    },
  });

  win.setPosition(width / 2 - size / 2, height / 2 - size / 2);

  // Dev Window

  // win.webContents.openDevTools();

  win.loadURL("http://localhost:3000/");
}

app.disableHardwareAcceleration();

app.enableSandbox();

app.on("ready", createWindow);

{
  /**
   * DEVICE CONNECTION MODULE
   */
}

// let cachedProcesses = null;
// let cachedActiveWindow = null;

// processWindows.getProcesses((err, processes) => {
//   if (err) {
//     console.error("Error", err);
//     return;
//   }
//   cachedProcesses = processes;
// });

// processWindows.getActiveWindow((err, processInfo) => {
//   if (err) {
//     console.error("Error", err);
//     return;
//   }
//   cachedActiveWindow = processInfo;
// });

// ipcMain.on("request-processes", (event) => {
//   event.reply("provide-processes", cachedProcesses);
// });

// ipcMain.on("request-active-window", (event) => {
//   event.reply("provide-active-window", cachedActiveWindow);
// });

```

### app/app/layout.tsx

```typescript
"use client";

import "../styles/global.css";
import "../styles/fonts.css";

import { ChakraProvider } from "@chakra-ui/react";
import customTheme from "@/styles/customTheme";

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body>
        <ChakraProvider theme={customTheme}>{children} </ChakraProvider>
      </body>
    </html>
  );
}


```

### engine/api/app.py

```python
import os
import openai
from flask import Flask, request, jsonify
from flask_cors import CORS
import traceback
import json
import requests
from transformers import ViltProcessor, ViltForQuestionAnswering
from PIL import Image
import io
import base64
import sys

# Prompts
sys.path.append("..")
from components.prompts import text_analyze_prompt

app = Flask(__name__)
CORS(app)

# Tranformer Models
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")

# Flask


@app.errorhandler(500)
def handle_internal_server_error(error):
    return (
        jsonify(
            {
                "error": "Internal Server Error",
                "details": str(error),
                "trace": traceback.format_exc(),
            }
        ),
        500,
    )


@app.route("/")
def home():
    return "If humanity successfully builds AGI, how the hell are we gonna control it? In the meantime, I'm building this with the end goal of eventually helping me do shit while I'm operating my digital world. Starting out with an AI specializing in distraction management."


# Engine


def prompt_refinery(prompt):
    pass


def response_refinery(response):
    pass


@app.route("/engine", methods=["POST"])
def engine():
    content = request.json.get("content")

    if not content:
        return jsonify({"error": "No text uploaded"}), 400

    print(json.dumps(content, indent=3))

    return jsonify({"response": "SHITTTT"})


@app.route("/engine/text", methods=["POST"])
def text_classifier():
    prompt = request.json.get("content")

    if not prompt:
        return jsonify({"error": "No text uploaded"}), 400

    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {
                "role": "system",
                "content": "You are Vesper, an assistant focused on minimizing digital distractions by filtering out distractions within apps/websites instead of blocking them entirely, providing a personalized, adaptive, and productive digital experience and helping users maintain their focus",
            },
            {"role": "user", "content": text_analyze_prompt + prompt},
        ],
    )

    return jsonify({"response": response.choices[0].message.content})


@app.route("/engine/image", methods=["POST"])
def image_classifier():
    image_url = request.json.get("content")
    isbase = request.json.get("isbase")

    if not image_url:
        return jsonify({"error": "No image URL uploaded"}), 400

    if isbase:
        # Decode the base64 data to get the image data
        decoded_data = base64.b64decode(image_url)

        # Convert the image data to an Image object
        image = Image.open(io.BytesIO(decoded_data))
    else:
        image = Image.open(requests.get(image_url, stream=True).raw)

    image = image.convert("RGB")

    text = "Is this image so extremely distracting that the viewer will become addicted to it?"

    encoding = processor(image, text, return_tensors="pt")

    outputs = model(**encoding)
    logits = outputs.logits
    idx = logits.argmax(-1).item()

    return jsonify({"response": model.config.id2label[idx]})


@app.route("/engine/videos", methods=["POST"])
def video_classifier():
    video = request.json.get("video")

    pass


# Agent

openai.api_key = os.getenv("OPENAI_API_KEY")


@app.route("/entity", methods=["POST"])
def entity():
    data = request.json.get("data")

    if not data:
        return jsonify({"error": "Data not provided"}), 400

    # response = openai.ChatCompletion.create(
    #     model="gpt-3.5-turbo",
    #     messages=[
    #         {"role": "system", "content": "You are Vesper, an assistant focused on minimizing digital distractions by filtering out distractions within apps/websites instead of blocking them entirely, providing a personalized, adaptive, and productive digital experience and helping users maintain their focus"},
    #         {"role": "user", "content": prompt},
    #     ]
    # )

    # return jsonify({"response": response.choices[0].message.content})

    return jsonify({"data": data})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

```

### app/app/page.tsx

```typescript
"use client";

import { useEffect, useState, useRef } from "react";

import { Button, Flex, Heading } from "@chakra-ui/react";
import { MoonIcon, SunIcon } from "@chakra-ui/icons";

import { IpcRendererEvent } from "electron";

import useToastManager from "@/components/modules/useToastManager";

import OpticalCircle from "@/Vesper/OpticalCircle";

const Home = () => {
  const [focus, setFocus] = useState(false);
  const [prompt, setPrompt] = useState<string>("");
  const [activeWindow, setActiveWindow] = useState<string>("");

  const { errorToast2, responseToast } = useToastManager();

  {
    /**
     * SPEECH TO TEXT MODULE
     */
  }

  const mediaRecorderRef = useRef<MediaRecorder | null>(null);
  const audioContextRef = useRef<AudioContext | null>(null);
  const processorRef = useRef<ScriptProcessorNode | null>(null);

  // Function to start recording
  const startRecording = () => {
    navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => {
      audioContextRef.current = new AudioContext();
      const audioStream =
        audioContextRef.current.createMediaStreamSource(stream);

      processorRef.current = audioContextRef.current.createScriptProcessor(
        2048,
        1,
        1
      );

      audioStream.connect(processorRef.current);
      processorRef.current.connect(audioContextRef.current.destination);

      mediaRecorderRef.current = new MediaRecorder(stream);
      mediaRecorderRef.current?.addEventListener(
        "dataavailable",
        handleDataAvailable
      );
      mediaRecorderRef.current?.start();
    });
  };

  // Function to handle available audio data
  const handleDataAvailable = async (e: BlobEvent) => {
    const audioData = new Blob([e.data], { type: "audio/webm; codecs=opus" });
    const arrayBuffer = await new Response(audioData).arrayBuffer();
    const uint8Array = new Uint8Array(arrayBuffer);
    const base64AudioData = Buffer.from(uint8Array).toString("base64");

    const response = await fetch("/api/transcribe", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ audioData: base64AudioData }),
    });

    if (response.ok) {
      const { transcription } = await response.json();
      if (transcription) {
        console.log(transcription);
      } else {
        console.log("");
      }
    } else {
      const errorMessage = await response.text();
      console.error("Failed to transcribe audio:", errorMessage);
      throw new Error(`Failed to transcribe audio: ${errorMessage}`);
    }
  };

  // Function to stop recording
  const stopRecording = () => {
    mediaRecorderRef.current?.stop();
    processorRef.current?.disconnect();
    audioContextRef.current?.close();
  };

  {
    /**
     * TEXT TO SPEECH MODULE
     */
  }

  const speakText = async (text: string) => {
    const response = await fetch("/api/synthesize", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text }),
    });

    const data = await response.json();

    const audioContent = data.audioContent;

    const buffer = new Uint8Array(audioContent.data).buffer;
    const blob = new Blob([buffer], { type: "audio/mpeg" });
    const url = URL.createObjectURL(blob);
    const audio = new Audio(url);
    audio.play();
  };

  {
    /**
     * ENGINE
     */
  }

  const submitMessage = async (prompt: string) => {
    const command = prompt;
    setPrompt("");

    try {
      const response = await fetch("/api/engine", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ prompt: command }),
      });

      console.log(response);
    } catch (error) {
      errorToast2(error);
    }
  };

  const handleFormSubmit = (event?: React.FormEvent<HTMLFormElement>) => {
    event?.preventDefault();
    submitMessage(prompt);
  };

  {
    /**
     * AUDIO
     */
  }

  const modeAudio = useRef(new Audio());

  const playModeAudio = (mode: boolean) => {
    modeAudio.current.src = mode
      ? "/sound-effects/focus.mp3"
      : "/sound-effects/leisure.mp3";
    modeAudio.current.play();
  };

  {
    /**
     * DEVICE CONNECTION MODULE
     */
  }

  {
    /**
     * MODE MODULE
     */
  }

  // useEffect(() => {
  //   if (window.electron && window.electron.ipcRenderer) {
  //     const fetchProcessesAndActiveWindow = () => {
  //       window.electron.ipcRenderer.send("request-active-window");
  //     };

  //     const activeWindowListener = (event: IpcRendererEvent, data: any) => {
  //       console.log(data);
  //     };

  //     window.electron.ipcRenderer.on(
  //       "provide-active-window",
  //       activeWindowListener
  //     );

  //     const pollingInterval = setInterval(fetchProcessesAndActiveWindow, 1000);

  //     return () => {
  //       clearInterval(pollingInterval);
  //       window.electron.ipcRenderer.removeListener(
  //         "provide-active-window",
  //         activeWindowListener
  //       );
  //     };
  //   }
  // }, []);

  const toggleMode = () => {
    setFocus(!focus);
  };

  const postAPI = async () => {
    const str = "";

    const response = await fetch("/api/test", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ str }),
    });

    if (response.ok) {
      const { message } = await response.json();
      console.log(message);
    } else {
      const errorMessage = await response.text();
      console.error("Failed to receive message:", errorMessage);
      throw new Error(`Failed to receive message: ${errorMessage}`);
    }
  };

  return (
    <Flex
      justifyContent={"center"}
      alignItems={"center"}
      w={"100vw"}
      h={"100vh"}
    >
      <Flex
        flexDirection={"column"}
        justifyContent={"center"}
        alignItems={"center"}
        w={"90%"}
        h={"90%"}
        borderRadius={"1000px"}
        border={"
[truncated — 1541 more characters]
```

### app/app/api/engine/route.ts

```typescript
import { NextRequest, NextResponse } from "next/server";
import axios from "axios";

const API_URL = "http://127.0.0.1:5000";

export async function POST(req: NextRequest) {
  const { prompt } = await req.json();

  try {
    // API CALL

    const response = await axios.post(`${API_URL}/engine`, {
      prompt,
    });
    return NextResponse.json({
      response: response.data.response,
    });
  } catch (err) {
    // ERROR HANDLING

    if (err instanceof Error) {
      console.error("Error: ", err.message);
      return NextResponse.json(
        { error: `Failed to response with task: ${err.message}` },
        { status: 500 }
      );
    } else {
      console.error("Unknown error: ", err);
      return NextResponse.json(
        { error: `Failed to response with task: Unknown error occurred.` },
        { status: 500 }
      );
    }
  }
}

```

### app/app/api/synthesize/route.ts

```typescript
import { NextRequest, NextResponse } from "next/server";
import textToSpeech, { protos } from "@google-cloud/text-to-speech";

// credential retrieval

const credentials = process.env.GOOGLE_APPLICATION_CREDENTIALS;

if (typeof credentials !== "string") {
  throw new Error("GOOGLE_APPLICATION_CREDENTIALS is not a string");
}

const clientConfig = {
  projectId: "vigama",
  credentials: JSON.parse(Buffer.from(credentials, "base64").toString()),
};

const client = new textToSpeech.TextToSpeechClient(clientConfig);

export async function POST(req: NextRequest) {
  const { text } = await req.json();

  const request: protos.google.cloud.texttospeech.v1.ISynthesizeSpeechRequest =
    {
      input: { text: text },
      voice: {
        languageCode: "en-US",
        name: "en-US-Studio-O",
        ssmlGender: "FEMALE",
      },
      audioConfig: { audioEncoding: "MP3" },
    };

  try {
    // API CALL

    const [response] = await client.synthesizeSpeech(request);

    return NextResponse.json({
      audioContent: response.audioContent,
    });
  } catch (err) {
    // ERROR HANDLING

    if (err instanceof Error) {
      console.error("Error: ", err.message);
      return NextResponse.json(
        { error: `Failed to synthesize text: ${err.message}` },
        { status: 500 }
      );
    } else {
      console.error("Unknown error: ", err);
      return NextResponse.json(
        {
          error: `Failed to transcribe audio: Unknown error occurred.`,
        },
        { status: 500 }
      );
    }
  }
}

```

### app/app/api/transcribe/route.ts

```typescript
import { NextRequest, NextResponse } from "next/server";

import { SpeechClient, protos } from "@google-cloud/speech";

const { RecognitionConfig } = protos.google.cloud.speech.v1;
const { AudioEncoding } = RecognitionConfig;

const credentials = process.env.GOOGLE_APPLICATION_CREDENTIALS;

if (typeof credentials !== "string") {
  throw new Error("GOOGLE_APPLICATION_CREDENTIALS is not a string");
}

const clientConfig = {
  projectId: "vigama",
  credentials: JSON.parse(Buffer.from(credentials, "base64").toString()),
};

const client = new SpeechClient(clientConfig);

export async function POST(req: NextRequest) {
  const { audioData: base64AudioData } = await req.json();

  const audioBuffer = Buffer.from(base64AudioData, "base64");

  const audioContent = {
    content: audioBuffer.toString("base64"),
  };
  const config: protos.google.cloud.speech.v1.IRecognitionConfig = {
    encoding: AudioEncoding.WEBM_OPUS,
    sampleRateHertz: 48000,
    languageCode: "en-US",
  };
  const request: protos.google.cloud.speech.v1.IRecognizeRequest = {
    audio: audioContent,
    config: config,
  };

  try {
    // API CALL

    const [response] = await client.recognize(request);
    const transcription =
      response &&
      response.results &&
      response.results
        .map((result) => {
          return (
            result && result.alternatives && result.alternatives[0].transcript
          );
        })
        .join("\n");

    return NextResponse.json(
      {
        transcription,
      },
      { status: 200 }
    );
  } catch (err) {
    // ERROR HANDLING

    if (err instanceof Error) {
      console.error("Error: ", err.message);
      return NextResponse.json(
        { error: `Failed to transcribe audio: ${err.message}` },
        { status: 500 }
      );
    } else {
      console.error("Unknown error: ", err);
      return NextResponse.json(
        { error: `Failed to transcribe audio: Unknown error occurred.` },
        { status: 500 }
      );
    }
  }
}

```

### chrome-extension/background.js

```javascript
let logs = [];

chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
  if (message.log) {
    logs.push(message.log);
  }
});

chrome.runtime.onConnect.addListener((port) => {
  if (port.name === "debug-panel") {
    port.onMessage.addListener((msg) => {
      if (msg.request === "getLogs") {
        port.postMessage({ logs: logs });
      }
    });
  }
});
```

[11 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]