# Project export: Vibe Tracker

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Making personal health manageable.
- Devpost: https://devpost.com/software/vibe-tracker
- GitHub: https://github.com/EthanThatOneKid/vibetracker.git
- Video: https://www.youtube.com/embed/veP4vTQXDdM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Ethan Davidson (22 commits)

## Devpost submission (written by the team)

### Inspiration

I have pain in my right shoulder, it has been clearly because of my poor posture sitting in front of my laptop. I wish there would have been an assistant who would have reminded me every time I slouch for more than 15-20 seconds to correct my posture. We realized this could be a good utility application for most of the office workers. MediaPipe offers pose detection and landmarking of the body. And, A study found that the average office employee spends 1,700 hours a year in front of the computer. i.e. 6 hours a workday. There is good potential to track the emotional wellbeing by measuring the expressions and accordingly

### What it does

With the help of live video input, Feature 1: Measures the slouch intensity and notifies if its beyond the threshold. (using Mediapipe) Feature 2: Measures the expressions elicited by browser-window/applications. (using Hume)

### How we built it

1st Iteration: We used vanilla HTML/CSS/JS to run the application, soon realising the shortcomings, i.e. only restricted to the browser. Only Posture correction feature was added using Mediapipe. 2nd Iteration: We used Electron to build a native app, which was able integrate both the features that is expression measurements and slouch detection. We used Hume API for expression measurements.

### Challenges we ran into

We teamed up 24 hours before the deadline. : shortage of time Deciding which platform to go with, considering the permissions and other aspects.

### Accomplishments we're proud of

Ethan a full-stack developer tagging with beginner- Akshay. Using Gemini API to calculate the slouch intensity. (This was surly not the intended use case) Breadth of various technology stacks used on the project.

### What we learned

Learning a novel approach to detect the slouch using Gemini-Mediapipe API. Learned about contributing and working on open-source projects. Team work Working towards deadline.

### What's next

Gait analysis for detection of early onset of orthopedic conditions. Better UI, be able deploy for end user. Monitor emotional health and produce better insights using Hume's measuring expressions.

## README (from the GitHub repository)

# Vibetracker

Making personal health manageable.

## Development

Install Node.js.

Install the project dependencies:

```bash
npm i
```

Run the development server:

```bash
npm start
```

P.S. Create a copy of `vibetracker.example.json` and rename it to
`vibetracker.json` and fill in the necessary
[Hume credentials](https://dev.hume.ai/docs/introduction/api-key).

---

Developed with <3 at Calhacks 11.0


## Detected evidence (automated analysis)

Indexed codebase: 9 recognized source files, 16 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
- TypeScript (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (13 of 13)

```
.gitignore
.vscode/settings.json
emotion-history.js
hume.js
index.html
LICENSE.md
main.js
mediapipe.js
package.json
preload.js
README.md
styles.css
vibetracker.example.json
```

### Dependencies

- package.json: @mediapipe/tasks-vision@^0.10.17, @paymoapp/active-window@^2.1.1, axios@^1.7.7, electron@^33.0.1, hume@^0.9.1

### Recent commits (newest first)

- wip
- edit readme
- add slouch result container element
- update with Vibe Tracker branding
- get top emotion per frame
- wip
- fix stuff
- shit
- reset to commit
- idk
- wippppppppppppppppppp
- wip
- poll for hume job
- captures webcam every interval
- communicate between renderer and electron main
- initialize hume api client
- add axios request
- add active-window
- overlay mediapipe
- show web cam in app

## Key source files (fetched from GitHub, selected and truncated for size)

### LICENSE.md

```markdown
    DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
            Version 2, December 2004

Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>

Everyone is permitted to copy and distribute verbatim or modified copies of this
license document, and changing it is allowed as long as the name is changed.

    DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

0. You just DO WHAT THE FUCK YOU WANT TO.

```

### package.json

```
{
  "name": "electron-quick-start",
  "version": "1.0.0",
  "description": "A minimal Electron application",
  "main": "main.js",
  "scripts": {
    "start": "electron ."
  },
  "repository": "https://github.com/electron/electron-quick-start",
  "keywords": [
    "Electron",
    "quick",
    "start",
    "tutorial",
    "demo"
  ],
  "author": "GitHub",
  "license": "CC0-1.0",
  "devDependencies": {
    "electron": "^33.0.1"
  },
  "dependencies": {
    "@mediapipe/tasks-vision": "^0.10.17",
    "@paymoapp/active-window": "^2.1.1",
    "axios": "^1.7.7",
    "hume": "^0.9.1"
  }
}

```

### main.js

```javascript
// Modules to control application life and create native browser window
const { app, BrowserWindow, ipcMain } = require("electron");
const path = require("node:path");
const fs = require("node:fs");
const { default: ActiveWindow } = require("@paymoapp/active-window");
const { createJob, pollJob, base64UriToBlob } = require("./hume.js");
const { storeEmotion } = require("./emotion-history.js");

let batch = [];

function createWindow() {
  // Create the browser window.
  const mainWindow = new BrowserWindow({
    width: 800,
    height: 600,
    webPreferences: {
      preload: path.join(__dirname, "preload.js"),
    },
  });

  // https://www.electronjs.org/docs/latest/tutorial/ipc#pattern-1-renderer-to-main-one-way
  ipcMain.on("incoming-capture", async (_event, capture) => {
    await addCaptureToQueue(capture);
  });

  // and load the index.html of the app.
  mainWindow.loadFile("index.html");

  // Open the DevTools.
  // mainWindow.webContents.openDevTools()
}

// This method will be called when Electron has finished
// initialization and is ready to create browser windows.
// Some APIs can only be used after this event occurs.
app.whenReady().then(() => {
  createWindow();

  app.on("activate", function () {
    // On macOS it's common to re-create a window in the app when the
    // dock icon is clicked and there are no other windows open.
    if (BrowserWindow.getAllWindows().length === 0) {
      createWindow();
    }
  });
});

// Quit when all windows are closed, except on macOS. There, it's common
// for applications and their menu bar to stay active until the user quits
// explicitly with Cmd + Q.
app.on("window-all-closed", function () {
  if (process.platform !== "darwin") {
    app.quit();
  }
});

// In this file you can include the rest of your app's specific main process
// code. You can also put them in separate files and require them here.

ActiveWindow.initialize();

if (!ActiveWindow.requestPermissions()) {
  console.log(
    "Error: You need to grant screen recording permission in System Preferences > Security & Privacy > Privacy > Screen Recording"
  );
  process.exit(0);
}

async function addCaptureToQueue(capture, threshold = 10) {
  batch.push(capture);

  if (batch.length >= threshold) {
    console.log("Batch is full, sending to Hume");
    await sendBatchToHume();
  }
}

async function sendBatchToHume() {
  const data = batch.map((capture) => base64UriToBlob(capture));
  batch = [];

  console.log(`Sending ${data.length} captures to Hume!`);
  const humeJob = await createJob(data);

  // stuck on poll
  const jobResult = await pollJob(humeJob.jobID);

  // console.log({ jobResult, jrLen: jobResult.length });
  const emotionsPerFrame = jobResult
    .map((frameResults) => {
      return frameResults.flatMap((face) => {
        return face.emotions.map((emotion) => {
          const activeWindow = ActiveWindow.getActiveWindow();
          return {
            emotion: emotion.name,
            score: emotion.score,
            appID: activeWindow.application,
          };
        });
      });
    })
    .flat();

  // TODO: Select emotion IDs to include and exclude, or alias.
  for (const emotion of emotionsPerFrame) {
    console.log({ emotion });
    // storeEmotion(emotion);
  }
}

```

### styles.css

```css
/* styles.css */

/* Add styles here to customize the appearance of your app */

```

### emotion-history.js

```javascript
let emotionHistory = [];

// emotion is appID, timestamp, emotionType, and score.
function storeEmotion(emotion) {
  emotionHistory.push(emotion);
}

function getEmotionByAppID(appID) {
  return emotionHistory.filter((emotion) => emotion.appID === appID);
}

function clearEmotionHistory() {
  emotionHistory = [];
}

module.exports = {
  storeEmotion,
  getEmotionByAppID,
  clearEmotionHistory,
};

```

### preload.js

```javascript
const { contextBridge, ipcRenderer } = require("electron/renderer");

/**
 * The preload script runs before `index.html` is loaded
 * in the renderer. It has access to web APIs as well as
 * Electron's renderer process modules and some polyfilled
 * Node.js functions.
 *
 * https://www.electronjs.org/docs/latest/tutorial/sandbox
 */
window.addEventListener("DOMContentLoaded", () => {
  const replaceText = (selector, text) => {
    const element = document.getElementById(selector);
    if (element) {
      element.innerText = text;
    }
  };

  for (const type of ["chrome", "node", "electron"]) {
    replaceText(`${type}-version`, process.versions[type]);
  }
});

contextBridge.exposeInMainWorld("electronAPI", {
  incomingCapture: (capture) => ipcRenderer.send("incoming-capture", capture),
});

```

### index.html

```html
<!-- Copyright 2023 The MediaPipe Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->
<!DOCTYPE html>
<head>
  <meta charset="UTF-8" />
  <link href="./styles.css" rel="stylesheet" />
  <script type="module" src="./mediapipe.js"></script>
</head>
<body>
  <h1>Vibetracker</h1>

  <section id="demos" class="invisible">
    <p>Vibe with us for a while...</p>

    <div id="liveView" class="videoView">
      <button id="webcamButton" class="mdc-button mdc-button--raised">
        <span class="mdc-button__ripple"></span>
        <span class="mdc-button__label">Start vibing</span>
      </button>
      <div style="position: relative">
        <video
          id="webcam"
          style="width: 1280px; height: 720px; position: abso"
          autoplay
          playsinline
        ></video>
        <canvas
          class="output_canvas"
          id="output_canvas"
          width="1280"
          height="720"
          style="position: absolute; left: 0px; top: 0px"
        ></canvas>
        <div class="result"></div>
      </div>
    </div>
  </section>
</body>

```

### hume.js

```javascript
const fs = require("node:fs");
const { HumeClient } = require("hume");

const configString = fs.readFileSync("vibetracker.json", "utf8");
const config = JSON.parse(configString);

// https://doc.deno.land/https://esm.sh/hume
const humeClient = new HumeClient({
  apiKey: config.humeApiKey,
  secretKey: config.humeApiSecret,
});

/**
 * createJob creates a new Hume job with the given blobs.
 */
async function createJob(
  blobs,
  apiKey = config.humeApiKey,
  apiURL = HUME_API_URL
) {
  // Create a FormData object.
  const formData = new FormData();

  // Add the JSON data.
  formData.append("json", JSON.stringify({ models: { face: {} } }));

  // Append blobs to the FormData object.
  for (const blob of blobs) {
    formData.append("file", blob);
  }

  // Define the headers.
  const headers = new Headers({
    "X-Hume-Api-Key": apiKey,
    accept: "application/json; charset=utf-8",
  });

  // Make the fetch request.
  const url = makeCreateJobURL(apiURL);
  const response = await fetch(url, {
    method: "POST",
    headers,
    body: formData,
  });
  if (response.ok) {
    const data = await response.json();
    return { jobID: data.job_id };
  }

  throw new Error("Failed to create Hume job: " + (await response.text()));
}

/**
 * base64UriToBlob converts a Base64-encoded data URI to a Blob.
 */
function base64UriToBlob(base64Uri) {
  // Remove the data URI scheme prefix if present
  const base64String = base64Uri.replace(/^data:image\/png;base64,/, "");

  // Decode the Base64 string into a Uint8Array
  const binaryString = atob(base64String);
  const bytes = new Uint8Array(binaryString.length);
  for (let i = 0; i < binaryString.length; i++) {
    bytes[i] = binaryString.charCodeAt(i);
  }

  // Create a Blob from the Uint8Array
  const blob = new Blob([bytes], { type: "image/png" }); // Adjust the type as needed
  return blob;
}

/**
 * pollJob polls the Hume API for the predictions of a job.
 */
async function pollJob(jobID, sleep = 5e3) {
  while (true) {
    await new Promise((resolve) => setTimeout(resolve, sleep));

    try {
      const allPredictions =
        await humeClient.expressionMeasurement.batch.getJobPredictions(jobID);
      return allPredictions.map((p) => extractEmotions(p));
    } catch (error) {
      console.error("Failed to get predictions:", error);
    }
  }
}

function extractEmotions(data) {
  const emotions = [];

  data.results.predictions.forEach((prediction) => {
    prediction.models.face.groupedPredictions.forEach((group) => {
      group.predictions.forEach((frame) => {
        emotions.push({
          timestamp: frame.time,
          emotions: frame.emotions.map((emotion) => ({
            name: emotion.name,
            score: emotion.score,
          })),
        });
      });
    });
  });

  return emotions;
}

function makeCreateJobURL(apiURL = HUME_API_URL) {
  return `${apiURL}/batch/jobs`;
}

const HUME_API_URL = "https://api.hume.ai/v0";

module.exports = {
  createJob,
  base64UriToBlob,
  pollJob,
  makeCreateJobURL,
  HUME_API_URL,
};

```

### mediapipe.js

```javascript
// Copyright 2023 The MediaPipe Authors.

// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import {
  DrawingUtils,
  FilesetResolver,
  PoseLandmarker,
} from "./node_modules/@mediapipe/tasks-vision/vision_bundle.mjs";

const demosSection = document.getElementById("demos");

// Global variables.
let slouchNotification;
let webcamStartedAt;
let slouchingMilliseconds = 0;
let slouching = false;
let poseLandmarker;
let runningMode = "IMAGE";
let enableWebcamButton;
let webcamRunning = false;
const videoHeight = "360px";
const videoWidth = "480px";

let intervalID;

function startCaptureIntervalIfNotStarted() {
  if (intervalID !== undefined) {
    return;
  }

  webcamStartedAt = Date.now();
  intervalID = setInterval(async () => {
    const capture = await captureVideoFrameAsDataURI(video);
    window.electronAPI.incomingCapture(capture);
  }, 1000);
}

function captureVideoFrameAsDataURI(
  videoElement,
  format = "image/png",
  quality = 1.0
) {
  const canvas = document.createElement("canvas");
  canvas.width = videoElement.videoWidth;
  canvas.height = videoElement.videoHeight;
  const context = canvas.getContext("2d");
  return new Promise((resolve, reject) => {
    context.drawImage(videoElement, 0, 0);
    const dataURI = canvas.toDataURL(format, quality);
    resolve(dataURI);
  });
}

// Before we can use PoseLandmarker class we must wait for it to finish
// loading. Machine Learning models can be large and take a moment to
// get everything needed to run.
const createPoseLandmarker = async () => {
  const vision = await FilesetResolver.forVisionTasks(
    "./node_modules/@mediapipe/tasks-vision/wasm"
  );
  poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath: `https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task`,
      delegate: "GPU",
    },
    runningMode: runningMode,
    numPoses: 2,
  });
  demosSection.classList.remove("invisible");
};
createPoseLandmarker();

/********************************************************************
  // Demo 2: Continuously grab image from webcam stream and detect it.
  ********************************************************************/

const video = document.getElementById("webcam");
const canvasElement = document.getElementById("output_canvas");
const canvasCtx = canvasElement.getContext("2d");
const drawingUtils = new DrawingUtils(canvasCtx);

// Check if webcam access is supported.
const hasGetUserMedia = () => !!navigator.mediaDevices?.getUserMedia;

// If webcam supported, add event listener to button for when user
// wants to activate it.
if (hasGetUserMedia()) {
  enableWebcamButton = document.getElementById("webcamButton");
  enableWebcamButton.addEventListener("click", enableCam);
} else {
  console.warn("getUserMedia() is not supported by your browser");
}

// Enable the live webcam view and start detection.
function enableCam(_event) {
  if (!poseLandmarker) {
    console.log("Wait! poseLandmaker not loaded yet.");
    return;
  }

  if (webcamRunning === true) {
    webcamRunning = false;
    enableWebcamButton.innerText = "Start vibing";
  } else {
    webcamRunning = true;
    enableWebcamButton.innerText = "Stop vibing";
  }

  // Activate the webcam stream.
  navigator.mediaDevices
    .getUserMedia({ video: true, audio: false })
    .then((stream) => {
      video.srcObject = stream;
      video.addEventListener("loadeddata", predictWebcam);
      startCaptureIntervalIfNotStarted();
    });
}

let lastVideoTime = -1;
async function predictWebcam() {
  canvasElement.style.height = videoHeight;
  video.style.height = videoHeight;
  canvasElement.style.width = videoWidth;
  video.style.width = videoWidth;
  // Now let's start detecting the stream.
  if (runningMode === "IMAGE") {
    runningMode = "VIDEO";
    await poseLandmarker.setOptions({ runningMode: "VIDEO" });
  }

  let startTimeMs = performance.now();
  if (lastVideoTime !== video.currentTime) {
    lastVideoTime = video.currentTime;
    poseLandmarker.detectForVideo(video, startTimeMs, (result) => {
      if (result.landmarks.length === 0) {
        return;
      }

      const landmark = result.landmarks[0];
      if (landmark === undefined) {
        return;
      }

      // https://www.linkedin.com/pulse/daily-life-example-human-pose-estimation-serkan-erdonmez/
      const leftShoulder = result.landmarks[0][11];
      const rightShoulder = result.landmarks[0][12];
      const shoulderHeight = (leftShoulder.y + rightShoulder.y) * 0.5;

      const nose = result.landmarks[0][0];
      const noseToShoulderDistance = Math.abs(nose.y - shoulderHeight);

      updateSlouching(noseToShoulderDistance, 0.2, 1e3);
      console.log(noseToShoulderDistance);
      const resultContainer = document.querySelector(".result");
      if (slouching) {
        resultContainer.innerText = "Not straight!";
        slouchingMilliseconds;
        if (slouchNotification === undefined) {
          slouchNotification = new Notification("Unslouch", {
            body: "Sloucher!!!",
          });
        }
      } else {
        resultContainer.innerText = "Good!";
        if (slouchNotification !== undefined) {
          slouchNotification.close();
          slouchNotification = undefined;
        }
      }

      canvasCtx.save();
      canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
      for (const landmark of result.landmarks) {
        dra
[truncated — 794 more characters]
```