# Project export: HumeTube

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Have you ever been intrigued by the feelings of those who watched the same YouTube video as you? With Hume and YouTube, we can display an analysis of human emotions expressed during key video moments.
- Devpost: https://devpost.com/software/humetube
- GitHub: https://github.com/EthanThatOneKid/humetube
- Video: https://www.youtube.com/embed/fU0F_mixcl4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Ethan Davidson (20 commits), Karnikaa Velumani (19 commits)

## Devpost submission (written by the team)

### Inspiration

Our inspiration for this project stemmed primarily from Hume AI. Upon seeing Hume AI's capabilities of capturing human emotions, we came up with the idea to analyze human expressions towards YouTube videos. Similarly to YouTube's "most replayed" analytics, we wanted to have the ability to see the most common emotion towards certain scenes in videos. For example, if someone was watching a scary movie on YouTube, they could see when the average person had evoked an emotion of fear during the video.

### What it does

Our product will offer users a comprehensive timeline during which human emotions were evoked throughout the video. This feature allows viewers to gain insights into the collective emotional responses experienced by the average individual while watching specific moments in the video.

### How we built it

HumeTube works as a chrome browser extension that captures data from the users by utilizing the front-facing camera. We then extract the timestamp of the YouTube video from the DOM and simultaneously collect the emotions of the client to then interpret using the Hume API for emotional classification. The user can then load a YouTube video in which the browser extension will query the data layer to retrieve the emotional history playback list that indicates strong correlations between the average emotions observed throughout the video's timeline. This information will be accessible via a collapsible HTML table, directly injected next to the YouTube player.

### Challenges we ran into

A couple challenges we ran into was navigating outdated documentation of creating a chrome browser extension and furthermore, providing access to the laptop to capture recordings of the user's emotions.

### Accomplishments we're proud of

The biggest accomplishments we've achieved come from overcoming our challenges. The initial milestone was developing the browser extension to record every timestamp of the video. The second achievement involved configuring the camera to capture a single frame, encapsulating our reactions attached to a specific timestamp.

### What we learned

We learned a lot from this experience. We learned how to create browser extensions, manipulate data using Hume's batching API, and store data using CockroachDB.

### What's next

We have a lot of features in mind to continue implementing further including aggregating emotions based on regions, providing a trend line based on the intensity of emotions evoked during specific scenes, and potentially creating a larger frontend to collect and analyze all data that retrieves the bigger picture (such as "video that had the highest average emotion of happiness", etc.).

## README (from the GitHub repository)

# HumeTube

Browser extension crowd-sourcing [**@YouTube**](https://github.com/youtube)
video reactions with [**@HumeAI**](https://github.com/HumeAI).

Project created for [Cal Hacks 10.0](https://cal-hacks-10.devpost.com/)!

## Inspiration

Our inspiration for this project stemmed primarily from Hume AI. Upon seeing
Hume AI's capabilities of capturing human emotions, we came up with the idea to
analyze human expressions towards YouTube videos. Similarly to YouTube's "most
replayed" analytics, we wanted to have the ability to see the most common
emotion towards certain scenes in videos. For example, if someone was watching a
scary movie on YouTube, they could see when the average person had evoked an
emotion of fear during the video.

## What it does

Our product will offer users a comprehensive timeline during which human
emotions were evoked throughout the video. This feature allows viewers to gain
insights into the collective emotional responses experienced by the average
individual while watching specific moments in the video.

Demo: <https://www.youtube.com/watch?v=fU0F_mixcl4>

## How we built it

HumeTube works as a chrome browser extension that captures data from the users
by utilizing the front-facing camera. We then extract the timestamp of the
YouTube video from the DOM and simultaneously collect the emotions of the client
to then interpret using the Hume API for emotional classification.

The user can then load a YouTube video in which the browser extension will query
the data layer to retrieve the emotional history playback list that indicates
strong correlations between the average emotions observed throughout the video's
timeline. This information will be accessible via a collapsible HTML table,
directly injected next to the YouTube player.

## Challenges we ran into

A couple challenges we ran into was navigating outdated documentation of
creating a chrome browser extension and furthermore, providing access to the
laptop to capture recordings of the user's emotions.

## Accomplishments that we're proud of

The biggest accomplishments we've achieved come from overcoming our challenges.
The initial milestone was developing the browser extension to record every
timestamp of the video. The second achievement involved configuring the camera
to capture a single frame, encapsulating our reactions attached to a specific
timestamp.

## What we learned

We learned a lot from this experience. We learned how to create browser
extensions, manipulate data using Hume's batching API, and store data using
CockroachDB.

## What's next for HumeTube

We have a lot of features in mind to continue implementing further including
aggregating emotions based on regions, providing a trend line based on the
intensity of emotions evoked during specific scenes, and potentially creating a
larger frontend to collect and analyze all data that retrieves the bigger
picture (such as "video that had the highest average emotion of happiness",
etc.).

## Built with

- Deno
- Hume AI
- JavaScript
- TypeScript

## Data flow

![image](https://github.com/EthanThatOneKid/humetube/assets/31261035/76ad08a6-1129-48cb-a034-fc3a42afde8e)

> **NOTE**
>
> PII is not present in this data pipeline as the Hume API returns only the
> calculated emotions discarding the video frames.

## Development

### Extension

1. Open `chrome://extensions` in Chrome.
1. Enable **Developer mode**.
1. Click **Load unpacked**.
1. Select the `extension` directory.

### HTTP server

```sh
cd server
```

```sh
deno task start
```

<!-- TODO: Set up HTTP server for development. -->

### Database

The HumeTube API server uses Deno Kv to store data. The database is set up
automatically on [Deno Deploy](https://deno.com/deploy).

## Deployment

[Deno Deploy](https://deno.com/deploy).

## License

[`LICENSE`](LICENSE)

## Devpost submission

Devpost submission: <https://devpost.com/software/humetube>.

---

Created with 🐻 by [**@EthanThatOneKid**](https://etok.codes/) and
[**@karnikaavelumani**](https://karni.codes/)


## Detected evidence (automated analysis)

Indexed codebase: 20 recognized source files, 34 KB.
- CSS (language) — detected in the code
- JavaScript (language) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (26 of 26)

```
.gitignore
.vscode/settings.json
extension/background.js
extension/content.js
extension/lib/humetube/create_interval.js
extension/lib/humetube/index.js
extension/lib/humetube/ingest.js
extension/lib/humetube/state.js
extension/lib/youtube/index.js
extension/lib/youtube/parse_video_id.js
extension/manifest.json
extension/scrollbars.css
LICENSE
README.md
server/deno.jsonc
server/deno.lock
server/deps.ts
server/lib/hume/api.ts
server/lib/hume/mod.ts
server/lib/humetube/kv/mod.ts
server/lib/humetube/kv/system.ts
server/lib/humetube/mod.ts
server/lib/humetube/system_interface.ts
server/main.ts
server/tasks/callback_server.ts
server/tasks/generate_fake_analysis.ts
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Update README.md
- run `deno task all`
- Update README.md
- wip
- Table style
- Update content.js
- include number of snapshots analyzed
- wip
- Update content.js
- Merge branch 'main' of https://github.com/EthanThatOneKid/humetube
- fix bugs
- Update content.js
- WIP
- Update content.js
- update extension with prod url
- wip
- WIP
- wip
- wip
- wip

## Key source files (fetched from GitHub, selected and truncated for size)

### server/main.ts

```typescript
import { load } from "humetube/deps.ts";
import { API } from "humetube/lib/hume/mod.ts";
import type { SystemInterface } from "humetube/lib/humetube/mod.ts";
import { KvSystem } from "humetube/lib/humetube/kv/mod.ts";

const MINUTE = 60_000;

await load({ export: true });

const kv = await Deno.openKv();

if (import.meta.main) {
  main();
}

function main() {
  kv.listenQueue(handleQueueEvent);

  Deno.serve(
    {
      onListen({ port }) {
        console.log(`Listening on http://127.0.0.1:${port}`);
      },
    },
    handleRequest,
  );
}

async function handleQueueEvent(event: unknown) {
  const { channel, videoID } = event as { channel: string; videoID: string };
  if (channel === "analyze-predictions") {
    const system = makeSystem();
    const recentAnalysis = await system.getAnalysis({ videoID })
      .catch(() => null);
    if (
      !recentAnalysis || recentAnalysis.lastUpdatedAt > Date.now() - 10 * MINUTE
    ) {
      return;
    }

    system.analyze({ videoID });
    return;
  }
}

async function handleRequest(request: Request): Promise<Response> {
  const url = new URL(request.url);
  if (
    request.method === "POST" &&
    (url.pathname === "/ingest-snapshots" ||
      url.pathname === "/ingest-snapshots/")
  ) {
    return await handleIngestSnapshots(request);
  }

  if (
    request.method === "POST" &&
    (url.pathname === "/ingest-predictions" ||
      url.pathname === "/ingest-predictions/")
  ) {
    return await handleIngestPredictions(request);
  }

  const videoID = new URLPattern({ pathname: "/emotions/:videoID" })
    .exec(url)
    ?.pathname.groups?.videoID ?? "";
  if (request.method === "GET" && videoID) {
    return await handleGetAnalysis(videoID);
  }

  return new Response("Not found", { status: 404 });
}

async function handleIngestSnapshots(request: Request): Promise<Response> {
  const snapshots = await request.json();
  const system = makeSystem();
  const result = await system.ingestSnapshots(snapshots);
  return new Response(JSON.stringify(result), { status: 200 });
}

async function handleIngestPredictions(request: Request): Promise<Response> {
  const predictions = await request.json();
  const system = makeSystem();
  const result = await system.ingestPredictions(predictions);
  for (const videoID of result.videoIDs) {
    // Enqueue analysis of this video in 10 minutes.
    kv.enqueue(
      { channel: "analyze-predictions", videoID },
      { delay: 10 * MINUTE },
    );
  }

  return new Response(JSON.stringify(result), { status: 200 });
}

async function handleGetAnalysis(videoID: string): Promise<Response> {
  const system = makeSystem();
  const result = await system.getAnalysis({ videoID });
  return new Response(JSON.stringify(result), { status: 200 });
}

function makeSystem(
  jobCompleteCallbackURL?: string,
  api?: API,
): SystemInterface {
  const apiURL = Deno.env.get("HUMETUBE_API_URL");
  if (!apiURL) {
    throw new Error("HUMETUBE_API_URL not set.");
  }

  if (!jobCompleteCallbackURL) {
    jobCompleteCallbackURL = `${apiURL}/ingest-predictions`;
  }

  if (!api) {
    const apiKey = Deno.env.get("HUME_API_KEY");
    if (!apiKey) {
      throw new Error("HUME_API_KEY not set.");
    }

    api = new API(
      apiKey,
      jobCompleteCallbackURL,
    );
  }

  return new KvSystem(api, kv);
}

```

### extension/lib/youtube/index.js

```javascript
export * from "./parse_video_id.js";

```

### extension/lib/humetube/index.js

```javascript
export * from "./create_interval.js";
export * from "./state.js";

```

### server/deps.ts

```typescript
export { load } from "https://deno.land/std@0.205.0/dotenv/mod.ts";
export { ulid } from "https://deno.land/x/ulid@v0.3.0/mod.ts";

```

### extension/scrollbars.css

```css
.humetube-emotions-table::-webkit-scrollbar { 
    width: 10px; 
    background-color: #333;
}

.humetube-emotions-table:hover::-webkit-scrollbar-thumb {
    background-color: #333;
    border-radius: 5px;
}

.humetube-emotions-table::-webkit-scrollbar-track {
    background-color: #888;
    border-radius: 5px;
}

details > summary {
    cursor: pointer;
    padding: .5rem 1rem;
}

.humetube-emotions-timeline > summary > * {
    display: inline;
}
```

### extension/background.js

```javascript
import { parseVideoID } from "./lib/youtube/index.js";
import { createInterval, getState, updateState } from "./lib/humetube/index.js";

const recordingEnabled = true;

// https://stackoverflow.com/a/11598753
chrome.runtime.onInstalled.addListener(async () => {
  for (const contentScript of chrome.runtime.getManifest().content_scripts) {
    for (const tab of await chrome.tabs.query({ url: contentScript.matches })) {
      chrome.scripting.executeScript({
        target: { tabId: tab.id },
        files: contentScript.js,
      });
    }
  }
});

chrome.tabs.onActivated.addListener(syncWithContentScript);
chrome.tabs.onCreated.addListener(syncWithContentScript);
chrome.tabs.onUpdated.addListener(syncWithContentScript);
chrome.webNavigation.onCompleted.addListener(syncWithContentScript);

async function syncWithContentScript(info) {
  // When the user navigates to a non-YouTube page, clear the interval.
  const tabID = info.tabId;
  const tab = await chrome.tabs.get(tabID);
  const videoID = parseVideoID(tab.url);
  if (!videoID) {
    return;
  }

  // Try to render the emotions timeline.
  fetch(`https://humetube.deno.dev/emotions/${videoID}`)
    .then(async (response) => {
      if (!response.ok) {
        return;
      }

      // Parse the JSON data.
      const data = await response.json();
      if (data.emotions.length === 0) {
        return;
      }

      sendDataToContentScript(tabID, data);
    });

  if (recordingEnabled) {
    const state = getState();
    if (!videoID) {
      clearInterval(state.intervalID);
      state.activeTabID = null;
      state.intervalID = null;
      updateState(state);
      // TODO: Ingest stashed data via ingest().
      return;
    }

    // When the user navigates to a YouTube page, update the tab ID and video ID.
    state.videoID = videoID;
    state.activeTabID = info.tabId;

    // If the interval is not already running, start it.
    if (!state.intervalID) {
      state.intervalID = createInterval();
    }

    // Update the state.
    updateState(state);
  }
}

function sendDataToContentScript(tabID, data) {
  chrome.tabs.sendMessage(tabID, {
    action: "renderEmotionsTimeline",
    data,
  });
}

```

### extension/content.js

```javascript
const width = 320; // We will scale the photo width to this.
let height; // This will be computed based on the input stream.
let contentTimestamp = null;
let streaming = false;

setup();

async function setup() {
  // Set up expression recording.
  const mediaCaptureElements = await createMediaCaptureElements();
  if (mediaCaptureElements !== undefined) {
    for (const el of mediaCaptureElements) {
      document.body.appendChild(el);
    }
  }

  // Set up timestamp recording.
  document.querySelector(".video-stream").addEventListener(
    "timeupdate",
    (event) => {
      contentTimestamp = event.currentTarget.currentTime;
    },
  );

  // Establish connection with background script.
  chrome.runtime.onMessage.addListener(handleBackgroundMessage);
}

function handleBackgroundMessage(request, sender, sendResponse) {
  switch (request.action) {
    case "getPageData": {
      const isAdPresent =
        document.querySelector(".ytp-ad-preview-container") !== null;
      if (isAdPresent) {
        return;
      }

      const pageData = getPageData();
      sendResponse(pageData);
      break;
    }

    case "renderEmotionsTimeline": {
      renderEmotionsTimeline(request.data);
      break;
    }
  }
}

function getPageData() {
  let dataURI = null;
  const canvas = document.querySelector(".humetube-camera-canvas");
  const video = document.querySelector(".humetube-camera-video");
  height ??= video.videoHeight / (video.videoWidth / width);
  if (width && height && canvas && video) {
    canvas.width = width;
    canvas.height = height;

    const ctx = canvas.getContext("2d");
    ctx.clearRect(0, 0, width, height);
    ctx.drawImage(video, 0, 0, width, height);

    dataURI = canvas.toDataURL("image/jpeg");
  }

  return { contentTimestamp, dataURI };
}

async function createMediaCaptureElements() {
  if (document.querySelector(".humetube-camera-video")) {
    return;
  }

  const video = document.createElement("video");
  video.classList.add("humetube-camera-video");
  video.style.display = "none";

  if (document.querySelector(".humetube-camera-canvas")) {
    return;
  }

  const canvas = document.createElement("canvas");
  canvas.classList.add("humetube-camera-canvas");
  canvas.style.display = "none";

  await navigator.mediaDevices
    .getUserMedia({ video: true, audio: false })
    .then((stream) => {
      video.srcObject = stream;
      video.play();
      console.log({ video, stream });
    })
    .catch((err) => {
      console.error(`An error occurred: ${err}`);
    });

  video.addEventListener(
    "canplay",
    () => {
      if (streaming) {
        return;
      }

      height = video.videoHeight / (video.videoWidth / width);

      // Firefox currently has a bug where the height can't be read from
      // the video, so we will make assumptions if this happens.

      if (isNaN(height)) {
        height = width / (4 / 3);
      }

      video.setAttribute("width", width);
      video.setAttribute("height", height);
      canvas.setAttribute("width", width);
      canvas.setAttribute("height", height);
      streaming = true;
    },
    false,
  );

  return [video, canvas];
}

function renderEmotionsTimeline(data) {
  if (document.querySelector(".humetube-emotions-timeline")) {
    return;
  }

  // Render the emotions timeline.
  const container = document.createElement("details");
  container.open = true;
  container.classList.add("humetube-emotions-timeline");
  container.style.fontFamily = "Roboto, sans-serif";
  container.style.maxHeight = "400px";
  container.style.width = "400px";
  container.style.border = "1px solid rgba(255,255,255,0.2)";
  container.style.borderRadius = "12px";
  container.style.backgroundColor = "#212121";
  container.style.color = "#fff";
  container.style.overflowX = "hidden";
  container.style.position = "relative";
  container.style.backgroundImage = "linear-gradient(#212121, #000000)";
  container.style.padding = "6px 0px";

  const summary = document.createElement("summary");
  summary.style.width = "100%";
  summary.style.backgroundColor = "#212121";
  const title = document.createElement("h2");
  title.textContent = "Hume emotional timeline";
  title.style.position = "sticky";
  title.style.top = "0";
  title.style.margin = "0";
  title.style.padding = "15px";
  title.style.fontSize = "20px";
  summary.appendChild(title);

  const table = document.createElement("table");
  table.classList.add("humetube-emotions-table");
  table.style.overflow = "auto";
  table.style.width = "400px";
  table.style.height = "100%";
  table.style.padding = "0 15px 0 15px";
  table.style.fontSize = "12px";
  table.style.borderRadius = "8px";
  table.style.overflowY = "scroll";
  table.style.scrollbarGutter = "stable";
  table.style.scrollbarColor = "#3ea6ff #212121";

  for (const emotion of data.emotions) {
    const row = table.insertRow();
    const cell1 = row.insertCell();
    const cell2 = row.insertCell();
    const cell3 = row.insertCell();
    const cell4 = row.insertCell();

    const anchor = document.createElement("a");
    const url = new URL(window.location.href);
    url.searchParams.set("t", emotion.timestamp);
    anchor.href = url.href;
    anchor.textContent = formatTime(emotion.timestamp);
    cell1.appendChild(anchor);
    cell2.textContent = emotion.name;

    // Create a colored bar to represent emotion intensity
    const intensityBar = document.createElement("div");
    intensityBar.style.width = `${emotion.amplitude * 100}%`;
    intensityBar.style.height = "10px";
    intensityBar.style.backgroundColor = "#263850";
    intensityBar.style.borderRadius = "4px";
    cell3.appendChild(intensityBar);
    cell3.style.width = "100%";

    cell4.style.fontSize = `${emotion.amplitude * 22 + 2}px`;
    cell4.style.textAlign = "center";
    cell4.textContent = emotion.emoji;

    anchor.style.textDecoration = "none";
    anchor.style.color = "#3ea6ff";
    anchor.style.backgroundColor = "#263850";
    anchor.style.borderRadius =
[truncated — 1332 more characters]
```

### server/tasks/callback_server.ts

```typescript
Deno.serve(async (request) => {
  console.log("Incoming request!");
  const body = await request.json();
  console.log(JSON.stringify(body, null, 2));
  return new Response("OK");
});

```

### server/tasks/generate_fake_analysis.ts

```typescript
import type { GetAnalysisResult } from "humetube/lib/humetube/mod.ts";
import { EMOTIONS } from "humetube/lib/hume/mod.ts";

const EMOTIONS_ARRAY = Object.keys(EMOTIONS);

if (import.meta.main) {
  const amount = Deno.args[0] ? parseInt(Deno.args[0]) : 1;
  const result = makeFakeGetAnalysisResult(amount);
  console.log(JSON.stringify(result));
}

function makeFakeGetAnalysisResult(amount: number): GetAnalysisResult {
  const emotions: GetAnalysisResult["emotions"] = [];
  let timestamp = 0;
  for (let i = 0; i < amount; i++) {
    timestamp += Math.ceil(Math.random() * 3);
    const emotion = makeFakeEmotion(timestamp);
    emotions.push(emotion);
  }

  return { emotions };
}

function makeFakeEmotion(
  timestamp: number,
): GetAnalysisResult["emotions"][number] {
  const randomEmotionIndex = Math.floor(Math.random() * EMOTIONS_ARRAY.length);
  const emotionName =
    EMOTIONS_ARRAY[randomEmotionIndex] as keyof typeof EMOTIONS;
  const emotionEmoji = EMOTIONS[emotionName];
  return {
    timestamp,
    amplitude: Math.random(),
    name: emotionName,
    emoji: emotionEmoji,
  };
}

```

### server/lib/hume/mod.ts

```typescript
export * from "./api.ts";

```

[9 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]