# Project export: DontCrashOutAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Solution to rage quitting from your computer. - Devpost: https://devpost.com/software/dontclashoutai - GitHub: https://github.com/miizadi/DontCrashOutAI - Team: 2 GitHub contributor(s) — Matthew Kim (11 commits), miizadi (8 commits) ## Devpost submission (written by the team) ### Inspiration Our team sometimes gets tempered when using a computer, and we set out to minimize that. Our initial idea was related to video games, as they can often be causes of frustration, especially online ones. But we realized this idea could be scaled to cover emotions throughout all computer usage, not just video games. Whether that be working on homework, playing online games, or programming a project, there are a variety of actions you do on a computer that can get frustrating. This application seeks to alleviate. ### What it does Once the app is open, it starts tracking your emotions via webcam, and when anger is detected, a creative calming notification will be sent to your desktop. This calming notification utilizes Google's Gemini API to generate an infinite amount of creative messages. These messages help remind the user that despite their frustration, everything is going okay and to take a break from the computer. ### How we built it We used Python to build both the front and back end of the application. Using a model that could detect faces within a webcam, we developed a program in the backend that can predict a person's facial expression and emotion. This prediction is quite accurate with facial tracking. We then implemented Google's Gemini API as the creative source for the calming notification messages. ### Challenges we ran into The biggest challenge we came across was finding and fine-tuning a model that catered to both our facial recognition needs and our calming message needs. Initially, we tried to use Google's MediaPipe facial detection software to track and detect faces being captured in the webcam. Unfortunately, it was incompatible with the model that we were using. Instead, we used openCV to perform facial recognition within our program. We also ran into issues with the webcam capturing weird and low-resolution angles, making it hard to capture a face within the frame. Another issue we came across was complications with Google Gemini simply not working with the API key we generated as well as text not generating correctly. ### Accomplishments we're proud of Getting the facial recognition working was our first hurdle that we accomplished and are extremely proud of. The fact that we were able to get our program to somewhat accurately predict facial expressions is something that we are extremely proud of. As we are both novice programmers, getting any of the above items to work was an accomplishment in and of itself. ### What we learned We got a glimpse into the world of facial recognition and AI fine-tuning. We also learned how to utilize the Google Gemini API and integrate it into our project. ### What's next for DontClashOutAI In the future, we hope to implement Google's Mediapipe for more advanced facial and emotion recognition. We would also like to retrain the model for more accurate and consistent results when predicting facial expressions. We're also working hard to clean up the GUI of the application allowing for a more streamlined and efficient experience. This is where we sourced the model from: link ## README (from the GitHub repository) # DontCrashOutAI Cal Hacks 11.0 solution to rage quitting from your computer. A desktop app that will notify you with clever calming messages with anger is detected via facial and emotion recognition. ## Detected evidence (automated analysis) Indexed codebase: 4 recognized source files, 917 KB. - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (6 of 6) ``` .DS_Store backend.py face_model.h5 frontend.py haarcascade_frontalface_default.xml README.md ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Final commit - Created frontend and backend - Functional emotion tracker w gemini notis - Fixed the video scaling issues. Work on Gemini API - Centered webcam feed - Big changes and some breaking changes - gemini - Deleted main2 - Empty commit - Almost finished emotion tracker using webcam - black and white issue or low res issue - Either Black and white issue or too low res - Dunno whats up with .DS_Store pls fix - Testing Facial recognition - Test Commit - test commit - Update README.md ## Key source files (fetched from GitHub, selected and truncated for size) ### backend.py ```python # backend.py import cv2 import numpy as np from tensorflow.keras.models import load_model import google.generativeai as genai import os class Backend: def __init__(self): # Load the emotion detection model self.model = load_model('face_model.h5') # Load the face cascade classifier self.face_cascade = cv2.CascadeClassifier( cv2.data.haarcascades + 'haarcascade_frontalface_default.xml' ) # Emotion class names self.class_names = ['Angry', 'Disgusted', 'Fear', 'Happy', 'Sad', 'Surprise', 'Neutral'] # Configure the generative AI model genai.configure(api_key=os.environ["API_KEY"]) # Create the generation configuration generation_config = { "temperature": 1, "top_p": 0.95, "top_k": 64, "max_output_tokens": 50, "response_mime_type": "text/plain", } # Initialize the generative AI model self.genai_model = genai.GenerativeModel( model_name="gemini-1.5-flash", generation_config=generation_config, ) # Start a chat session with predefined history self.chat_session = self.genai_model.start_chat( history=[ { "role": "user", "parts": [ "Give a bunch of one-liners that will calm the reader down to prevent anger escalation.", ], }, { "role": "model", "parts": [ "Sure, here are some calming one-liners:\n\n- Take a deep breath; this moment will pass.\n- Stay calm; you've handled worse before.\n- Focus on solutions, not problems.\n- Keep your cool; it's not worth the stress.\n- Pause and reset; you control your response.\n- Let go of what's beyond your control.\n- Stay centered; peace begins with you.", ], }, ] ) def process_frame(self, frame): """ Detects faces in the frame and predicts the emotion for each face. Returns a list of emotions with their corresponding face coordinates. """ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) faces = self.face_cascade.detectMultiScale( gray, scaleFactor=1.3, minNeighbors=5, minSize=(30, 30) ) emotions = [] for (x, y, w, h) in faces: face_roi = frame[y:y + h, x:x + w] face_image = cv2.resize(face_roi, (48, 48)) face_image = cv2.cvtColor(face_image, cv2.COLOR_BGR2GRAY) face_image = np.expand_dims(face_image, axis=0) face_image = np.expand_dims(face_image, axis=-1) predictions = self.model.predict(face_image) emotion_label = self.class_names[np.argmax(predictions)] emotions.append((x, y, w, h, emotion_label)) return emotions def get_calming_message(self): """ Retrieves a calming message from the generative AI model. """ response = self.chat_session.send_message( "Give a quick tip on how to calm down when using a computer. No special characters. Each response should be different from the last." ) return response.text.strip() ``` ### frontend.py ```python # frontend.py import tkinter as tk from tkinter import ttk from PIL import Image, ImageTk import cv2 import subprocess import time from backend import Backend class EmotionTrackerApp: def __init__(self, window, window_title): self.window = window self.window.title(window_title) self.window.geometry("800x750") self.window.configure(bg="#f0f0f0") # Video capture source self.video_source = 0 self.vid = cv2.VideoCapture(self.video_source) self.vid.set(cv2.CAP_PROP_FRAME_WIDTH, 640) self.vid.set(cv2.CAP_PROP_FRAME_HEIGHT, 480) # Canvas for video frames self.canvas = tk.Canvas(window, width=640, height=480) self.canvas.pack(pady=20) # Frame for control buttons self.control_frame = ttk.Frame(window) self.control_frame.pack(pady=10) # Start button self.btn_start = ttk.Button( self.control_frame, text="Start", command=self.start_tracking ) self.btn_start.grid(row=0, column=0, padx=5) # Stop button self.btn_stop = ttk.Button( self.control_frame, text="Stop", command=self.stop_tracking ) self.btn_stop.grid(row=0, column=1, padx=5) # Quit button self.btn_quit = ttk.Button( self.control_frame, text="Quit", command=self.quit ) self.btn_quit.grid(row=0, column=2, padx=5) # Label to display detected emotion self.emotion_label = ttk.Label( window, text="Detected Emotion: ", font=("Helvetica", 14) ) self.emotion_label.pack(pady=10) # Initialize the backend self.backend = Backend() self.delay = 15 # Delay between frame updates (milliseconds) # Initialize last notification time to 0 self.last_notification_time = 0 # Tracking state self.is_tracking = False # Start the update loop self.update() self.window.mainloop() def start_tracking(self): """ Starts the facial tracking. """ self.is_tracking = True self.emotion_label.config(text="Detected Emotion: Tracking started.") def stop_tracking(self): """ Stops the facial tracking. """ self.is_tracking = False self.emotion_label.config(text="Detected Emotion: Tracking stopped.") def update(self): ret, frame = self.vid.read() if ret: # Mirror the frame horizontally frame = cv2.flip(frame, 1) if self.is_tracking: # Process the frame using the backend emotions = self.backend.process_frame(frame) # Draw rectangles and labels on the frame for (x, y, w, h, emotion_label) in emotions: cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2) cv2.putText( frame, emotion_label, (x, y - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 2, ) # Update the emotion label in the GUI self.emotion_label.config( text=f"Detected Emotion: {emotion_label}" ) if emotion_label == "Angry": self.send_notification() else: # If not tracking, just display the frame without processing self.emotion_label.config(text="Detected Emotion: Not tracking.") # Convert the frame to RGB and display it in the GUI self.photo = ImageTk.PhotoImage( image=Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)) ) self.canvas.create_image(0, 0, image=self.photo, anchor=tk.NW) # Schedule the next frame update self.window.after(self.delay, self.update) def send_notification(self): """ Retrieves a calming message from the backend and displays a notification. Only sends a notification if 10 seconds have passed since the last one. """ current_time = time.time() if current_time - self.last_notification_time >= 10: calming_message = self.backend.get_calming_message() # For macOS subprocess.run([ "osascript", "-e", f'display notification "{calming_message}" with title "Emotion Alert"' ]) # For Windows (uncomment if using Windows and comment out the macOS command) # from win10toast import ToastNotifier # toaster = ToastNotifier() # toaster.show_toast("Emotion Alert", calming_message, duration=5) # Update the last notification time self.last_notification_time = current_time def quit(self): """ Quits the application. """ self.window.quit() self.vid.release() cv2.destroyAllWindows() # Run the application if __name__ == "__main__": EmotionTrackerApp(tk.Tk(), "Emotion Tracker") ```