# Project export: Locked.AI

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Have you ever fallen asleep in class? Found yourself reaching for your phone every time you tried to get work done? Locked.AI redefines the way you stay focused, using AI software to keep you on task.
- Devpost: https://devpost.com/software/locked-ai
- GitHub: https://github.com/kesavanramakrishnan/lockedai
- Video: https://www.youtube.com/embed/dBTVkrHbIdE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — kesavanramakrishnan (1 commits)

## Devpost submission (written by the team)

### Inspiration

We, and many of our peers are constantly afflicted by our generation's diminishing attention span and growing reliance on social media. We came up with Locked.AI to remedy the poor focus that we may face when it comes to long periods of lectures and studying.

### What it does

Locked.AI utilizes an artificial intelligence software that tracks eye and head movements to distinguish focused moments from unfocused ones, promptly alerting the user through their mobile device to get them back on track. After a studying session or lecture, Locked will give users a feedback report of how focused they were.

### How we built it

We built Locked.AI using Python's OpenCV, mediapipe and Flask libraries for the backend and HTML/CSS for the frontend.

### Challenges we ran into

One challenge we faced was being able to properly align the AI models with our goals. For instance, it was difficult perceiving depth with tracking sleepiness. We also ran into trouble detecting focus states as the thresholds required were extremely precise. Another difficulty was communicating between our Flask backend and HTML/CSS frontend. It took some time to figure out how to display our video recording with Javascript.

### Accomplishments we're proud of

We are proud of making a finished product with technologies we are new to.

### What we learned

We learned new Python technology and how to efficiently collaborate in a team.

### What's next

Locked.AI plans to improve their face and eye detection accuracy and host its service on a server.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 7 recognized source files, 270 KB.
- CSS (language) — detected in the code
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (11 of 11)

```
app.py
EyeTrack.py
requirements.txt
sleep.py
static/scripts.js
templates/base.html
templates/index.html
website/.DS_Store
website/assets/.DS_Store
website/package.json
website/styles.css
```

### Dependencies

- requirements.txt: flask, flask_cors, mediapipe, numpy, opencv-python, pushover_complete
- website/package.json: bootstrap@^5.3.3

### Recent commits (newest first)

- Cleaning up app.py and updating dependencies
- Final code

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
opencv-python
mediapipe
numpy
flask
flask_cors
pushover_complete
```

### website/package.json

```
{
  "dependencies": {
    "bootstrap": "^5.3.3"
  }
}

```

### app.py

```python
from flask import Flask, request, render_template, Response
import cv2
import mediapipe as mp
import numpy as np
from flask_cors import CORS
from flask_cors import cross_origin
from pushover_complete import PushoverAPI
import smtplib
from email.mime.text import MIMEText

app = Flask(__name__)
CORS(app)

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        # Handle the form submission
        message = request.form.get('message')
        # Here you would trigger the notification to your iPhone
        send_notification_to_iphone(message)
        return 'Notification sent!'
    return render_template('index.html')

global user_key

def send_notification_to_iphone(message):
    device = 'email'
    api_token = 'axup77m6g6snv69v5n9vv2r68c8q16'
    pushover = PushoverAPI(api_token)
    pushover.send_message(user_key,message,title="Notification from Flask")

# Initialize Mediapipe Face Mesh detector


global stream_active 
stream_active = True

global sleep_data
sleep_data = []

global sleepTrack
sleepTrack = True

global focusTrack
focusTrack = True


def euc_dist(p1, p2, w, h):
        return np.sqrt((int(p1.x * w) - int(p2.x * w))**2 + (int(p1.y * h) - int(p2.y * h))**2)        

def generate_frames():
    global baseline_eye_distance, baseline_face_width, calibrate_eye_distance_total, calibrate_face_width_total, frame_count

    cap = cv2.VideoCapture(0)
    fps = cap.get(cv2.CAP_PROP_FPS)

    mp_face_mesh = mp.solutions.face_mesh
    face_mesh = mp_face_mesh.FaceMesh(refine_landmarks=True)

    # Calibration variables
    baseline_eye_distance = None  # Distance between eyelids during calibration
    baseline_face_width = None    # Face width during calibration

    calibrate_eye_distance_total = 0
    calibrate_face_width_total = 0
    calibration_frames = fps * 10
    frame_count = 0

    closed_eye_counter = 0
    sleep_threshold = 5 * fps
    sleeping = 0
    
    vert_dists = []
    hori_dists = []
    hori = None
    vert = None

    while True:
        
        ret, frame = cap.read()
        if not ret:
            break

        frame = cv2.flip(frame, 1)
        rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        # Process the frame to get landmarks
        results = face_mesh.process(rgb_frame)
      
        h, w, _ = frame.shape
        global focusTrack
        global sleepTrack
        if results.multi_face_landmarks:
           if stream_active:
                landmarks = results.multi_face_landmarks[0].landmark

                # Get key landmarks for the eye
                top_right = landmarks[159]  # Top of right eye
                bottom_right = landmarks[145]  # Bottom of right eye
                
                # Get key landmarks for focus
                forehead = landmarks[151]
                chin = landmarks[175]
                lc = landmarks[411]
                rc = landmarks[187]
                # Convert normalized coordinates to pixel coordinates
                top_right_px = (int(top_right.x * w), int(top_right.y * h))
                bottom_right_px = (int(bottom_right.x * w), int(bottom_right.y * h))

                # Calculate the pixel distance between the top and bottom eyelid
                eye_distance = np.linalg.norm(np.array(top_right_px) - np.array(bottom_right_px))

                # Calculate face width using landmarks on the left and right side of the face
                left_cheek = landmarks[234]  # Left edge of the face
                right_cheek = landmarks[454]  # Right edge of the face
                left_cheek_px = (int(left_cheek.x * w), int(left_cheek.y * h))
                right_cheek_px = (int(right_cheek.x * w), int(right_cheek.y * h))

                # Calculate face width in pixels
                face_width = np.linalg.norm(np.array(left_cheek_px) - np.array(right_cheek_px))

                # Calibration step: Set baseline eye distance and face width
                if frame_count < calibration_frames:
                    cv2.putText(frame, "Calibrating... Keep your eyes open and don't move your head!", (50, 50),
                                cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
                    calibrate_eye_distance_total += eye_distance
                    calibrate_face_width_total += face_width
                    vert_dists.append(euc_dist(forehead, chin, w, h))
                    hori_dists.append(euc_dist(lc, rc, w, h))
                    frame_count += 1
                    
                if sleepTrack:        
                    baseline_eye_distance = calibrate_eye_distance_total / calibration_frames
                    baseline_face_width = calibrate_face_width_total / calibration_frames

                    # Calculate the scaling factor based on face width
                    scaling_factor = baseline_face_width / face_width

                    # Normalize the current eye distance
                    normalized_eye_distance = eye_distance * scaling_factor
                    # print(f"Normalized Eye Distance: {normalized_eye_distance}")

                    # Display circles on the landmarks
                    cv2.circle(frame, top_right_px, 3, (0, 255, 0), -1)
                    cv2.circle(frame, bottom_right_px, 3, (0, 255, 0), -1)

                    # Determine if eyes are open or closed using a threshold
                    if normalized_eye_distance < baseline_eye_distance * 0.70:
                        closed_eye_counter += 1
                        sleep_data.append(0)
                        cv2.putText(frame, "Eyes Closed", (50, 100), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2, cv2.LINE_AA)
                    else:
                        print("Eyes Open")
                        sleep_data.append(1)
                        cv2.putText(frame, "Eyes Open", (50, 100), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
                    if closed_eye_counter >= sleep_threshold:
               
[truncated — 2262 more characters]
```

### EyeTrack.py

```python
import cv2
import mediapipe as mp
import numpy as np

cap = cv2.VideoCapture(0)
face_mesh = mp.solutions.face_mesh.FaceMesh(refine_landmarks=True)

minX = float('inf')
maxX = float('-inf')

while True:
    ret, frame = cap.read()
    frame = cv2.flip(frame, 1)
    if not ret:
        print('Unable to retrieve from webcam')
        break
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    h, w, _ = frame.shape

    mesh = face_mesh.process(rgb)
    landmark_pts = mesh.multi_face_landmarks
    if landmark_pts:    

        # Creating ROI based on center nose point
        nose = landmark_pts[0].landmark[168]
        cx = int(nose.x * w)
        cy = int(nose.y * h)
        
        size = 600
        x1 = max(0, cx - size//2)
        y1 = max(0, cy - size//2)
        x2 = min(w, cx + size//2)
        y2 = min(h, cy + size//2)

        cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0))
        roi = frame[y1:y2, x1:x2]

        # Tracking center eye coordinate
        rh, rw, _ = roi.shape
        eye_r = landmark_pts[0].landmark[473]
        rx = int(eye_r.x * rw)
        ry = int(eye_r.y * rh)
        x = int(eye_r.x * w)
        y = int(eye_r.y * h)
        cv2.circle(frame, (x, y), 2, (0, 255, 0))
        print(f"cx: {rx}, cy: {ry}")

        if rx > maxX:
            maxX = rx
        elif rx < minX:
            minX = rx 
    
    cv2.imshow("Face Mesh", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()
print(f"min: {minX}, max: {maxX}")
```

### sleep.py

```python
import cv2
import mediapipe as mp
import numpy as np

# Initialize Mediapipe Face Mesh detector
mp_face_mesh = mp.solutions.face_mesh
face_mesh = mp_face_mesh.FaceMesh(refine_landmarks=True)

cap = cv2.VideoCapture(0)

# Calibration variables
baseline_eye_distance = None  # Distance between eyelids during calibration
baseline_face_width = None    # Face width during calibration

calibrate_eye_distance_total = 0
calibrate_face_width_total = 0
calibration_frames = 120
frame_count = 0

while True:
    ret, frame = cap.read()
    if not ret:
        break

    frame = cv2.flip(frame, 1)
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

    # Process the frame to get landmarks
    results = face_mesh.process(rgb_frame)

    h, w, _ = frame.shape

    if results.multi_face_landmarks:
        landmarks = results.multi_face_landmarks[0].landmark

        # Get key landmarks for the eye
        top_right = landmarks[159]  # Top of right eye
        bottom_right = landmarks[145]  # Bottom of right eye

        # Convert normalized coordinates to pixel coordinates
        top_right_px = (int(top_right.x * w), int(top_right.y * h))
        bottom_right_px = (int(bottom_right.x * w), int(bottom_right.y * h))

        # Calculate the pixel distance between the top and bottom eyelid
        eye_distance = np.linalg.norm(np.array(top_right_px) - np.array(bottom_right_px))

        # Calculate face width using landmarks on the left and right side of the face
        left_cheek = landmarks[234]  # Left edge of the face
        right_cheek = landmarks[454]  # Right edge of the face
        left_cheek_px = (int(left_cheek.x * w), int(left_cheek.y * h))
        right_cheek_px = (int(right_cheek.x * w), int(right_cheek.y * h))

        # Calculate face width in pixels
        face_width = np.linalg.norm(np.array(left_cheek_px) - np.array(right_cheek_px))

        # Calibration step: Set baseline eye distance and face width
        if frame_count < calibration_frames:
            cv2.putText(frame, "Calibrating... Keep your eyes open!", (50, 50),
                        cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
            calibrate_eye_distance_total += eye_distance
            calibrate_face_width_total += face_width
            frame_count += 1
            
        baseline_eye_distance = calibrate_eye_distance_total / calibration_frames
        baseline_face_width = calibrate_face_width_total / calibration_frames


        # Calculate the scaling factor based on face width
        scaling_factor = baseline_face_width / face_width

        # Normalize the current eye distance
        normalized_eye_distance = eye_distance * scaling_factor
        # print(f"Normalized Eye Distance: {normalized_eye_distance}")

        # Display circles on the landmarks
        cv2.circle(frame, top_right_px, 3, (0, 255, 0), -1)
        cv2.circle(frame, bottom_right_px, 3, (0, 255, 0), -1)

        # Determine if eyes are open or closed using a threshold
        if normalized_eye_distance < baseline_eye_distance * 0.71:
            print("Eyes Closed")
        else:
            print("Eyes Open")

    # Show the video feed
    cv2.imshow("Eye Tracker", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

```

### templates/base.html

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Send Notification</title>
</head>
<body>
    <h1>Send a Notification to Your iPhone</h1>
    <form method="post">
        <label for="message">Message:</label>
        <input type="text" id="message" name="message" required>
        <button type="submit">Send</button>
    </form>
</body>
</html>
```

### static/scripts.js

```javascript
/*!
 * Start Bootstrap - Grayscale v7.0.6 (https://startbootstrap.com/theme/grayscale)
 * Copyright 2013-2023 Start Bootstrap
 * Licensed under MIT (https://github.com/StartBootstrap/startbootstrap-grayscale/blob/master/LICENSE)
 */
//
// Scripts
//


let sessionButton = document.getElementById("sessionButton");
let videoStream = document.getElementById("videoStream");
let trackFocus = document.getElementById("trackFocus");
let trackSleep = document.getElementById("trackSleep");
let emailInput = document.getElementById("emailInput");
let streamStarted = false;

sessionButton.addEventListener("click", toggleSession);

window.addEventListener("DOMContentLoaded", (event) => {
  // Navbar shrink function
  var navbarShrink = function () {
    const navbarCollapsible = document.body.querySelector("#mainNav");
    if (!navbarCollapsible) {
      return;
    }
    if (window.scrollY === 0) {
      navbarCollapsible.classList.remove("navbar-shrink");
    } else {
      navbarCollapsible.classList.add("navbar-shrink");
    }
  };

  // Shrink the navbar
  navbarShrink();

  // Shrink the navbar when page is scrolled
  document.addEventListener("scroll", navbarShrink);

  // Activate Bootstrap scrollspy on the main nav element
  const mainNav = document.body.querySelector("#mainNav");
  if (mainNav) {
    new bootstrap.ScrollSpy(document.body, {
      target: "#mainNav",
      rootMargin: "0px 0px -40%",
    });
  }

  // Collapse responsive navbar when toggler is visible
  const navbarToggler = document.body.querySelector(".navbar-toggler");
  const responsiveNavItems = [].slice.call(
    document.querySelectorAll("#navbarResponsive .nav-link")
  );
  responsiveNavItems.map(function (responsiveNavItem) {
    responsiveNavItem.addEventListener("click", () => {
      if (window.getComputedStyle(navbarToggler).display !== "none") {
        navbarToggler.click();
      }
    });
  });
});

function toggleSession() {
  if(emailInput.value.length > 10){
    if (!streamStarted) {
      const data = {
        "user_email": emailInput.value,
        "focus": trackFocus.checked,
        "sleep": trackSleep.checked,
      };

      fetch("http://127.0.0.1:5000/receive_data", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify(data),
      })
        .then((response) => response.text())
        .then((data) => console.log(data))
        .catch((error) => console.error("Error:", error));

      videoStream.src = "http://127.0.0.1:5000/video_feed";
      sessionButton.textContent = "End Session";
      streamStarted = true;

      fetch("http://127.0.0.1:5000/resume_stream", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
      })
        .then((response) => response.text())
        .then((data) => console.log(data))
        .catch((error) => console.error("Error:", error));
    } else {
      videoStream.src = "";
      sessionButton.textContent = "Start Session";
      streamStarted = false;
      fetch("http://127.0.0.1:5000/pause_stream", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
      })
        .then((response) => response.text())
        .then((data) => showModal(data))
        .catch((error) => console.error("Error:", error));
    }
  }
}



  function showModal(data) {
    document.getElementById("chartModal").style.display = "flex";
    renderChart(data);
  }

  function closeModal() {
    efficiencyChart.destroy();
    document.getElementById("chartModal").style.display = "none";
  }

  let efficiencyChart

  function renderChart(data) {
    const parsed = JSON.parse(data);
    const combined = [parsed.zero, parsed.one];
    console.log(combined);
    const labels = ["Unfocused Time", "Focused Time"];
    const ctx = document.getElementById("efficiencyChart").getContext("2d");
    efficiencyChart = new Chart(ctx, {
      type: "doughnut",
      data: {
        labels: labels,
        datasets: [
          {
            label: "Efficiency",
            data: combined,
            backgroundColor: [
              'rgb(255, 99, 132)',
              'rgb(54, 162, 235)',
            ],
            hoverOffset: 4,
          },
        ],
      },
      options: {
        plugins:{
          title:{
            display: true,
            text: "How Focused Were You?"
          }
        },
        responsive: true,
      },
    });
  }

```

### templates/index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, shrink-to-fit=no"
    />
    <meta name="description" content="" />
    <meta name="author" content="" />
    <title>Locked.AI</title>
    <link rel="icon" type="image/x-icon" href="../website/assets/favicon.ico" />
    <!-- Font Awesome icons (free version)-->
    <script
      src="https://use.fontawesome.com/releases/v6.3.0/js/all.js"
      crossorigin="anonymous"
    ></script>
    <!-- Google fonts-->
    <link
      href="https://fonts.googleapis.com/css?family=Varela+Round"
      rel="stylesheet"
    />
    <link
      href="https://fonts.googleapis.com/css?family=Nunito:200,200i,300,300i,400,400i,600,600i,700,700i,800,800i,900,900i"
      rel="stylesheet"
    />
    <!-- Core theme CSS (includes Bootstrap)-->
    <link href="../website/styles.css" rel="stylesheet" />
  </head>
  <body id="page-top">
    <!-- Navigation-->
    <nav class="navbar navbar-expand-lg navbar-light fixed-top" id="mainNav">
      <div class="container px-4 px-lg-5">
        <a class="navbar-brand" href="#page-top">Locked.AI</a>
        <button
          class="navbar-toggler navbar-toggler-right"
          type="button"
          data-bs-toggle="collapse"
          data-bs-target="#navbarResponsive"
          aria-controls="navbarResponsive"
          aria-expanded="false"
          aria-label="Toggle navigation"
        >
          Menu
          <i class="fas fa-bars"></i>
        </button>
        <div class="collapse navbar-collapse" id="navbarResponsive">
          <ul class="navbar-nav ms-auto">
            <li class="nav-item">
              <a class="nav-link" href="#about">About</a>
            </li>
            <li class="nav-item">
              <a class="nav-link" href="#targets">Functionality</a>
            </li>
            <li class="nav-item">
              <a class="nav-link" href="#focus">Get Focused</a>
            </li>
          </ul>
        </div>
      </div>
    </nav>
    <!-- Masthead-->
    <header class="masthead">
      <div
        class="container px-4 px-lg-5 d-flex h-100 align-items-center justify-content-center"
      >
        <div class="d-flex justify-content-center">
          <div class="text-center">
            <h1 class="mx-auto my-0 text-uppercase">REINVENT FOCUS</h1>
            <h2 class="text-white-50 mx-auto mt-2 mb-5">
              An eye-tracking application designed to keep YOU focused, designed at Calhacks 11.0.

            </h2>
            <a class="btn btn-primary" href="#about">Learn More</a>
          </div>
        </div>
      </div>
    </header>
    <!-- About-->
    <section class="about-section text-center" id="about">
      <div class="container px-4 px-lg-5">
        <div class="row gx-4 gx-lg-5 justify-content-center">
          <div class="col-lg-8">
            <h2 class="text-white mb-4">Built with Artificial Intelligence</h2>
            <p class="text-white-50">
              Using Mediapipe and OpenCV, Locked.AI utilizes well-trained AI models to better differentiate between
              focused and unfocused moments in order to provide an accurate and more reflective experience for users. 
            </p>
          </div>
        </div>
        <img class="img-fluid" src="../website/assets/img/aipic.png" alt="..." />
      </div>
    </section>
    <!-- Projects-->
    <section class="projects-section bg-light" id="targets">
      <div class="container px-4 px-lg-5">
        <!-- Featured Project Row-->
        <div class="row gx-0 mb-4 mb-lg-5 align-items-center">
          <div class="col-xl-8 col-lg-7">
            <img
              class="img-fluid mb-3 mb-lg-0"
              src="../website/assets/img/booksgray.png"
              alt="..."
            />
          </div>
          <div class="col-xl-4 col-lg-5">
            <div class="featured-text text-center text-lg-left">
              <h4>Studying</h4>
              <p class="text-black-50 mb-0">
                Locked.AI tracks eye movements during designated study sessions and keeps you on track with a phone notification once your eyes drift off.
              </p>
            </div>
          </div>
        </div>
        <!-- Project One Row-->
        <div class="row gx-0 mb-5 mb-lg-0 justify-content-center">
          <div class="col-lg-6">
            <img
              class="img-fluid"
              src="../website/assets/img/adhdhigh.jpg"
              alt="..."
            />
          </div>
          <div class="col-lg-6">
            <div class="bg-black text-center h-100 project">
              <div class="d-flex h-100">
                <div
                  class="project-text w-100 my-auto text-center text-lg-left"
                >
                  <h4 class="text-white">ADHD</h4>
                  <p class="mb-0 text-white-50">
                    Locked.AI seeks to assist those with attention issues 
                    to stay on task for extended periods of time.
                  </p>
                </div>
              </div>
            </div>
          </div>
        </div>
        <!-- Project Two Row-->
        <div class="row gx-0 justify-content-center">
          <div class="col-lg-6">
            <img
              class="img-fluid"
              src="../website/assets/img/tireddone.jpg"
              alt="..."
            />
          </div>
          <div class="col-lg-6 order-lg-first">
            <div class="bg-black text-center h-100 project">
              <div class="d-flex h-100">
                <div
                  class="project-text w-100 my-auto text-center text-lg-right"
                >
                  <h4 class="text-white">Fatigue</h4>
                  <p class="mb-0 text-white-50">
                    Whether you’re driving or studying, Locked.AI detects 
                    eye closure that suggests you are dozing off, and promptly 
                    rin
[truncated — 5122 more characters]
```