# Project export: SpeakEasy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Attention is all you need. Speakeasy helps you keep it. - Devpost: https://devpost.com/software/speakeasy-5ktydn - GitHub: https://github.com/alonr619/SpeakEasy - Video: https://www.youtube.com/embed/C3BJl4_AoKY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Alon Ragoler (43 commits), Alex2262 (19 commits), Derek Yao (17 commits), nicetea17 (15 commits) ## Devpost submission (written by the team) ### Overview As college students, we’ve all been to lectures where the professor doesn’t realize their audience is completely lost, where we leave the building with more questions than answers and feel confused and frustrated. Similarly in professional settings, the brightest ideas can be undermined by subpar pitches and unclear explanations. The result of such scenarios are missed opportunities to close deals and students who gain little to nothing from lectures. Attention drives impact; commanding it is a difficult skill. Whether you’re doing a simple project presentation for a class or hosting a Ted Talk, you need to read visual cues from the audience and respond in real time to effectively engage your listeners. Dynamically adjusting tone, phrasing, and content on the fly is a crucial yet formidable skill to obtain, and even the most skilled speakers often struggle to address the audience in its entirety. Such struggles motivated us to build SpeakEasy, an AI agent that gives you detailed audience analytics and live feedback based on reactions while you speak, so you can give more engaging and responsive presentations. ### What it does SpeakEasy is an AI agent that gives you detailed analytics and actionable insights into how engaged your audience is. Using computer vision, the webcam feature continuously analyzes each audience member’s emotional state and attention level through sentiment analysis and gaze detection. Simultaneously, our platform is capable of live transcription of the speaker’s words through Fish Audio, providing valuable context when giving speaker insights. Live sentiment analysis and transcription data is then integrated through a Claude-based agent that informs you which people seem confused, which parts of your speech were poorly received, and what you can do to regain your audience’s attention. A multitude of analytics present further fruitful information regarding the presenter’s performance and the crowd’s reaction over time. ### How we built it SpeakEasy is built on a FastAPI backend and React frontend that processes real-time video and audio streams through a dual-pipeline architecture. The core of our speech processing relies on Fish Audio's advanced ASR system, which provides high-accuracy and low-latency speech-to-text transcription with overlapping audio chunks to ensure no words are missed. Fish Audio handles the complex task of converting continuous speech into text, which is then integrated into our custom post-processing AI pipeline that refines the transcripts for accuracy and coherence while removing overlapping words. The visual analysis pipeline uses OpenFace integrated with RetinaFace for face detection and a multitask learning model for emotion and gaze analysis. We implemented a dual-speed processing system that provides fast face detection every 200ms and detailed facial emotion analysis every 7 seconds. Gaze analysis and emotion analysis are implemented by augmenting the model with linear heads to extract gaze and classify a probability distribution over 8 emotions. But what does it mean for people to be attending to something else? Since we have a head in our model that outputs gaze, defined by a yaw and a pitch, we can compute a unit vector that represents the direction of a person’s gaze. Then, we can define a region of attention that represents what people should be attending to. In the case of SpeakEasy, the region of attention would be defined as the speaker, or a slideshow that he would be utilizing. Then, we can compute a unit vector that represents the direction from the person’s eyes to this region of attention. We can then take the cosine similarity between their gaze vector and the “true” vector, and scale this to [0, 1] to acquire a probability metric for attention. At the end, Claude AI analyzes the combined audio transcripts and visual data to generate real-time presenter recommendations, all streamed directly to the dashboard via SSE. ### Challenges we ran into At first, Fish Audio’s transcription was missing spoken words because of small delays between the audio chunks we sent to it. In addition, the output was muddled by external noises that were causing the transcription to become inaccurate and incoherent. We solved this by including overlap between audio chunks that were passed into Fish Audio. The output would then go through a Claude-based post-processing layer to clean and assess coherence before displaying to the user, removing words that were recorded in the previous chunk as well as filler words and noise. One challenge was being able to accurately and efficiently capture the sentiment and attention scores for each individual person detected in frame. Sentiment and attention are inherently complex attributes, making accurate detection quite difficult. However, we came up with a two step approach to handling both the efficiency bottlenecks and the accuracy issues: In the first stage, we would rapidly run a large scale one-shot bounding box detection model for all the faces in frame. By running this algorithm independently of sentiment analysis and attention, we would be able to report bounding box detection much more seamlessly to the front end. Then, once we had acquired the bounding boxes, we would crop the image to each individual face based on the predicted bounding boxes, and then feed that image into the model once again utilizing linear layers after a convolutional net to project sentiments and gaze, thereby allowing the model to analyze only one face at a time to produce more accurate results. Another challenge was with the overall architecture of the project. We initially had a single large API route in the backend that would call Fish Audio, the computer vision model, and Claude all in the same place. However, the latency this resulted in made SpeakEasy effectively useless, and we wanted different components of the dashboard to be able to update independently. To fix this, we restructured the platform to have most of the logic on the frontend and split up the backend into several smaller API routes. When called on, each route replies with raw data, which is then filtered and displayed appropriately by each React component. ### Accomplishments we're proud of Overall, we are incredibly proud of our final project, as we built a working multimodal, real-time system that unites computer vision, audio transcription, and language modeling into one cohesive pipeline. Throughout the process of creating SpeakEasy, architecting the flow was one of the most difficult yet rewarding parts. With multiple moving parts, designing the system to integrate everything coherently was essential in realizing our vision for the platform. We also take pride in the level of polish we achieved on both the frontend and backend. From designing an intuitive dashboard that visualizes real-time audience engagement to fine-tuning Fish Audio’s transcription pipeline and integrating Claude’s contextual reasoning, every piece was built with user experience and performance in mind. Of course, none of this would’ve been possible without our teamwork and adaptability. We applaud ourselves for actively communicating our ideas, progress, and needs, as doing so has been crucial to our efficacy. ### What we learned We learned a lot about processing multimodal inputs through agentic workflows. LLMs are extremely powerful, but they still have core limitations that prevent them from being able to fully one-shot platforms with features as complex and intricate as SpeakEasy. Throughout the hackathon, we operated with a philosophy that treats LLMs as a tool to be used rather than a black box to be thrown at everything. We integrated them in various parts of Speakeasy but maintained a strong non-LLM fundamental structure throughout. ### What's next We’d like to refine our algorithms and analytics to be more accurate by integrating data like intonation and other vocal cues to give more comprehensive feedback for the presenter. With dual-facing cameras, we could also give feedback on body language and hand gestures, significantly enhancing speakers’ stage presence. Moreover, we currently have analytics algorithms running in our CV end, which allows us to visualize heat maps and Voronoi diagrams over emotion and attention distributions in the audience which we plan to integrate with the front end to provide speakers with more comprehensible data. Currently, the speaker receives feedback from observing the screen’s streamed advice, which may distract them and decrease the quality of their presentation. With Fish Audio’s text-to-speech capabilities, we can connect our current platform to earbuds so that the speaker can privately listen to recommendations. Additionally, Fish Audio’s distinct emotion control in voice generation inspires us. We envision SpeakEasy to include a feature that directs your emotions by showing first-hand what you would and should sound like in order to hype up the crowd or command the audience, which is decided in response to overall attention and confusion in the audience. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 109 recognized source files, 369 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 122) ``` .gitignore backend/__init__.py backend/algo.py backend/cv.py backend/fish/api.py backend/fish/route.py backend/libs/openface/__init__.py backend/libs/openface/cli.py backend/libs/openface/demo.py backend/libs/openface/face_detection.py backend/libs/openface/landmark_detection.py backend/libs/openface/model/__init__.py backend/libs/openface/model/AU_model.py backend/libs/openface/model/AutomaticWeightedLoss.py backend/libs/openface/model/MTL.py backend/libs/openface/multitask_model.py backend/libs/openface/Pytorch_Retinaface/convert_to_onnx.py backend/libs/openface/Pytorch_Retinaface/data/__init__.py backend/libs/openface/Pytorch_Retinaface/data/config.py backend/libs/openface/Pytorch_Retinaface/data/data_augment.py backend/libs/openface/Pytorch_Retinaface/data/FDDB/img_list.txt backend/libs/openface/Pytorch_Retinaface/data/wider_face.py backend/libs/openface/Pytorch_Retinaface/detect.py backend/libs/openface/Pytorch_Retinaface/layers/__init__.py backend/libs/openface/Pytorch_Retinaface/layers/functions/prior_box.py backend/libs/openface/Pytorch_Retinaface/layers/modules/__init__.py backend/libs/openface/Pytorch_Retinaface/layers/modules/multibox_loss.py backend/libs/openface/Pytorch_Retinaface/LICENSE.MIT backend/libs/openface/Pytorch_Retinaface/models/__init__.py backend/libs/openface/Pytorch_Retinaface/models/net.py backend/libs/openface/Pytorch_Retinaface/models/retinaface.py backend/libs/openface/Pytorch_Retinaface/README.md backend/libs/openface/Pytorch_Retinaface/test_fddb.py backend/libs/openface/Pytorch_Retinaface/test_widerface.py backend/libs/openface/Pytorch_Retinaface/train.py backend/libs/openface/Pytorch_Retinaface/utils/__init__.py backend/libs/openface/Pytorch_Retinaface/utils/box_utils.py backend/libs/openface/Pytorch_Retinaface/utils/nms/__init__.py backend/libs/openface/Pytorch_Retinaface/utils/nms/py_cpu_nms.py backend/libs/openface/Pytorch_Retinaface/utils/timer.py backend/libs/openface/STAR/__init__.py backend/libs/openface/STAR/conf/__init__.py backend/libs/openface/STAR/conf/alignment.py backend/libs/openface/STAR/conf/base.py backend/libs/openface/STAR/config.json backend/libs/openface/STAR/demo.py backend/libs/openface/STAR/evaluate.py backend/libs/openface/STAR/lib/__init__.py backend/libs/openface/STAR/lib/backbone/__init__.py backend/libs/openface/STAR/lib/backbone/core/coord_conv.py backend/libs/openface/STAR/lib/backbone/stackedHGNetV1.py backend/libs/openface/STAR/lib/dataset/__init__.py backend/libs/openface/STAR/lib/dataset/alignmentDataset.py backend/libs/openface/STAR/lib/dataset/augmentation.py backend/libs/openface/STAR/lib/dataset/decoder/__init__.py backend/libs/openface/STAR/lib/dataset/decoder/decoder_default.py backend/libs/openface/STAR/lib/dataset/encoder/__init__.py backend/libs/openface/STAR/lib/dataset/encoder/encoder_default.py backend/libs/openface/STAR/lib/loss/__init__.py backend/libs/openface/STAR/lib/loss/awingLoss.py backend/libs/openface/STAR/lib/loss/smoothL1Loss.py backend/libs/openface/STAR/lib/loss/starLoss_v2.py backend/libs/openface/STAR/lib/loss/starLoss.py backend/libs/openface/STAR/lib/loss/wingLoss.py backend/libs/openface/STAR/lib/metric/__init__.py backend/libs/openface/STAR/lib/metric/accuracy.py backend/libs/openface/STAR/lib/metric/fr_and_auc.py backend/libs/openface/STAR/lib/metric/nme.py backend/libs/openface/STAR/lib/metric/params.py backend/libs/openface/STAR/lib/utility.py backend/libs/openface/STAR/lib/utils/__init__.py backend/libs/openface/STAR/lib/utils/dist_utils.py backend/libs/openface/STAR/lib/utils/meter.py backend/libs/openface/STAR/lib/utils/time_utils.py backend/libs/openface/STAR/lib/utils/vis_utils.py backend/libs/openface/STAR/main.py backend/libs/openface/STAR/README.md backend/libs/openface/STAR/requirements-py310.txt backend/libs/openface/STAR/requirements-py37.txt backend/libs/openface/STAR/tester.py backend/libs/openface/STAR/tools/__init__.py backend/libs/openface/STAR/tools/analysis_motivation.py backend/libs/openface/STAR/tools/infinite_loop_gpu.py backend/libs/openface/STAR/tools/infinite_loop.py backend/libs/openface/STAR/tools/split_wflw.py backend/libs/openface/STAR/tools/testtime_pca.py backend/libs/openface/STAR/trainer.py backend/llm/api.py backend/llm/route.py backend/main.py backend/of_helper.py backend/prompts/__init__.py backend/prompts/addBlock.txt backend/prompts/addFirstBlock.txt backend/prompts/emotion_analysis.py backend/requirements.txt backend/tests/test_main.py backend/tests/test_of.py frontend/package.json frontend/public/data.json frontend/public/index.html frontend/public/manifest.json frontend/public/robots.txt frontend/README.md frontend/src/App.css frontend/src/App.js frontend/src/App.test.js frontend/src/components/AudioTranscriber.js frontend/src/components/ConfusionTrend.jsx frontend/src/components/EmotionDistribution.jsx frontend/src/components/EmotionRadar.jsx frontend/src/components/SentimentShift.jsx frontend/src/components/SSEStreamer.js frontend/src/components/UnifiedAnalyzer.js frontend/src/components/usePerformanceIndex.jsx frontend/src/components/WebcamAnalyzer.js frontend/src/index.css frontend/src/index.js frontend/src/MagicBento.css frontend/src/MagicBento.jsx [2 more files omitted for size] ``` ### Dependencies - backend/requirements.txt: anthropic@>=0.7.0, click@>=8.1.7, dotenv@>=0.9.9, fastapi@>=0.104.0, huggingface-hub@>=0.21.0, imageio@>=2.34.2, matplotlib@>=3.10.1, numpy@>=2.2.6, opencv-python@>=4.8.0, pandas@>=2.2.3, Pillow@>=9.4.0, python-dotenv@>=1.0.0, requests@>=2.32.0, scikit-image@>=0.24.0, scipy@>=1.13.0, seaborn@>=0.13.2, tensorboardX@>=2.6.2.2, timm@>=1.0.15, torch@>=2.9.0, torchvision@>=0.24.0, tqdm@>=4.66.2, uvicorn@>=0.24.0 - frontend/package.json: @testing-library/dom@^10.4.1, @testing-library/jest-dom@^6.9.1, @testing-library/react@^16.3.0, @testing-library/user-event@^13.5.0, gsap@^3.13.0, lucide-react@^0.548.0, react@^19.2.0, react-dom@^19.2.0, react-scripts@5.0.1, recharts@^3.3.0, web-vitals@^2.1.4 ### Recent commits (newest first) - push the emotion radar - Merge pull request #3 from alonr619/radar-graph - fixes - radar graph! - Updated sentiment analysis - Merge pull request #2 from alonr619/multipliers - multiply probs - Merge branch 'main' of https://github.com/alonr619/calhacks - Comprehension rate and performance index - Merge pull request #1 from alonr619/bounding_box - Merge branch 'main' of https://github.com/alonr619/calhacks - update thresholds - fix cognitive load - bounding boxes implemented with a slow and fast request - Change name and threshold - Scrollable live transcript - Merge branch 'main' of https://github.com/alonr619/calhacks - Attention distribution pie chart - clean unifiedanalyzer.js - clean main.py ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/requirements.txt ``` requests>=2.32.0 python-dotenv>=1.0.0 fastapi>=0.104.0 uvicorn>=0.24.0 anthropic>=0.7.0 opencv-python>=4.8.0 torch>=2.9.0 numpy>=2.2.6 matplotlib>=3.10.1 seaborn>=0.13.2 scipy>=1.13.0 scikit-image>=0.24.0 imageio>=2.34.2 pandas>=2.2.3 Pillow>=9.4.0 torchvision>=0.24.0 tqdm>=4.66.2 tensorboardX>=2.6.2.2 timm>=1.0.15 click>=8.1.7 huggingface-hub>=0.21.0 dotenv>=0.9.9 ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "dependencies": { "@testing-library/dom": "^10.4.1", "@testing-library/jest-dom": "^6.9.1", "@testing-library/react": "^16.3.0", "@testing-library/user-event": "^13.5.0", "gsap": "^3.13.0", "lucide-react": "^0.548.0", "react": "^19.2.0", "react-dom": "^19.2.0", "react-scripts": "5.0.1", "recharts": "^3.3.0", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### backend/main.py ```python from fastapi import FastAPI, File, UploadFile, Form, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse, StreamingResponse import os import shutil from fish.route import router as fish_router from llm.route import router as llm_router import cv import logging # Set up logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) app = FastAPI() # Include routers app.include_router(fish_router) app.include_router(llm_router) logger.info("🚀 Starting FastAPI server...") logger.info(f"📁 Working directory: {os.getcwd()}") logger.info(f"🔑 Claude API Key present: {'CLAUDE_API_KEY' in os.environ}") @app.get("/") async def root(): logger.info("🏠 Root endpoint hit!") return {"message": "Server is running!", "status": "ok"} @app.get("/test") async def test(): logger.info("🧪 Test endpoint hit!") return {"status": "ok", "message": "Server is working!"} SAVE_DIR = "frames" if os.path.exists(SAVE_DIR): shutil.rmtree(SAVE_DIR) os.makedirs(SAVE_DIR) cur_frame_slow = 0 cur_frame_fast = 0 MAX_FRAMES = 10 # Add CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:3000", "http://127.0.0.1:3000"], # Add your frontend URLs allow_credentials=True, allow_methods=["*"], # Allows all methods allow_headers=["*"], # Allows all headers ) def save_frame(image, slow=False): global cur_frame_slow, cur_frame_fast if slow: filepath = f"{SAVE_DIR}/frame_slow_{cur_frame_slow}.jpg" cur_frame_slow = (cur_frame_slow + 1) % MAX_FRAMES else: filepath = f"{SAVE_DIR}/frame_fast_{cur_frame_fast}.jpg" cur_frame_fast = (cur_frame_fast + 1) % MAX_FRAMES if os.path.exists(filepath): os.remove(filepath) with open(filepath, "wb") as buffer: shutil.copyfileobj(image.file, buffer) print(f"✅ Saved frame: {filepath}") return filepath @app.post("/analyze") async def analyze_image( image: UploadFile = File(None), ): filepath = save_frame(image, slow=True) json_data = cv.analyze_image_slow(filepath) print(json_data) return json_data @app.post("/analyze-fast") async def analyze_image_fast( image: UploadFile = File(None) ): filepath = save_frame(image, slow=False) json_data = cv.analyze_image_fast(filepath) return json_data if __name__ == "__main__": import uvicorn logger.info("🚀 Starting server on port 8000...") uvicorn.run("main:app", host="127.0.0.1", port=8000, reload=False, log_level="info") ``` ### frontend/src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( // // ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### frontend/src/App.js ```javascript import React, { useState, useEffect, useRef } from "react"; import "./App.css"; import UnifiedAnalyzer from "./components/UnifiedAnalyzer"; import EmotionDistribution from "./components/EmotionDistribution"; import EmotionRadar from "./components/EmotionRadar"; import ConfusionTrend from "./components/ConfusionTrend"; import SentimentShift from "./components/SentimentShift"; import SSEStreamer from "./components/SSEStreamer"; import usePerformanceIndex from "./components/usePerformanceIndex"; import { ParticleCard, GlobalSpotlight } from "./MagicBento"; function App() { const [attention, setAttention] = useState(78); const [cognitiveLoad, setCognitiveLoad] = useState("Moderate"); const [dataList, setDataList] = useState([]); const performanceIndex = usePerformanceIndex(dataList, 0.7); const comprehensionRate = dataList.length > 0 ? Math.round( (dataList[dataList.length - 1]?.cvData?.aggregate_attention ?? 0) * 100 ) : 0; const bentoRef = useRef(null); useEffect(() => { const interval = setInterval(() => { setAttention((a) => Math.max(0, Math.min(100, a + (Math.random() * 6 - 3)))); }, 5000); return () => clearInterval(interval); }, []); const feedback = attention >= 75 ? "Audience attention is high — maintain your pace!" : attention >= 50 ? "Attention dipping slightly — re-engage with examples." : "Critical drop — vary tone and energy immediately!"; return (

SpeakEasy

attention is all you need

Speaker Insights

🎯 Comprehension Rate

{comprehensionRate}%

⚙️ Performance Index

{performanceIndex}/100

🧠 Cognitive Load

{cognitiveLoad}
{/* Row 3 — Main Analytics */}

📷 Webcam Feed

📈 Attention Trend

📉 Sentiment Shift Over Time

Emotion Radar

0 ? dataList[dataList.length - 1]?.cvData : null} />
); } export default App; ``` ### backend/libs/openface/cli.py ```python import os import click from huggingface_hub import snapshot_download from demo import process_image def download_weights_from_hf(repo_id, save_path): """ Downloads an entire folder from Hugging Face Model Hub. Args: repo_id (str): The Hugging Face repo ID. save_path (str): The local path to save the folder contents. """ if not os.path.exists(save_path): os.makedirs(save_path, exist_ok=True) print("Downloading weights from Hugging Face...") snapshot_download(repo_id=repo_id, local_dir=save_path, repo_type="model") print(f"Weights downloaded to {save_path}") else: print("Weights already exist. Skipping download.") @click.group() def cli(): """Command-line interface for OpenFace.""" click.echo("CLI initialized") pass @cli.command(name='download') # Explicitly set command name @click.option("--repo-id", default="nutPace/openface_weights", help="Hugging Face repo ID") @click.option("--output", default="./weights", help="Path to save the weights") def download(repo_id, output): """Download weights from Hugging Face.""" click.echo(f"Starting download from {repo_id}") # Debug line save_path = os.path.abspath(output) download_weights_from_hf(repo_id, save_path) @cli.command() @click.argument('image_path', type=click.Path(exists=True)) @click.option('--output-dir', '-o', default='results', help='Directory to save results') @click.option('--device', '-d', default='cuda', type=click.Choice(['cuda', 'cpu']), help='Device to run inference on') def detect(image_path, output_dir, device): """Process an image and save face analysis results to CSV. IMAGE_PATH: Path to the input image file """ try: output_file = process_image( image_path=image_path, output_dir=output_dir, device=device ) click.echo(f"Results saved to: {output_file}") except Exception as e: click.echo(f"Error: {str(e)}", err=True) raise click.Abort() @cli.command(name='detect-video') @click.argument('video_path', type=click.Path(exists=True)) @click.option('--output-dir', '-o', default='results', help='Directory to save results') @click.option('--device', '-d', default='cuda', type=click.Choice(['cuda', 'cpu']), help='Device to run inference on') def detect_video(video_path, output_dir, device): """Process a video and save per-frame face analysis results to CSV. VIDEO_PATH: Path to the input video file """ from openface.demo import process_video try: output_file = process_video( video_path=video_path, output_dir=output_dir, device=device ) click.echo(f"Results saved to: {output_file}") except Exception as e: click.echo(f"Error: {str(e)}", err=True) raise click.Abort() if __name__ == "__main__": cli() ``` ### backend/libs/openface/STAR/main.py ```python import argparse from trainer import train from tester import test def add_data_options(parser): group = parser.add_argument_group("dataset") group.add_argument("--image_dir", type=str, default=None, help="the directory of image") group.add_argument("--annot_dir", type=str, default=None, help="the directory of annot") group.add_argument("--ckpt_dir", type=str, default=None, help="the output directory of checkpoints and logs") def add_base_options(parser): group = parser.add_argument_group("base") group.add_argument("--mode", type=str, default="train", help="train or test") group.add_argument("--config_name", type=str, default="alignment", help="set configure file name") group.add_argument('--device_ids', type=str, default="0,1,2,3", help="set device ids, -1 means use cpu device, >= 0 means use gpu device") group.add_argument('--data_definition', type=str, default='WFLW', help="COFW, 300W, WFLW") group.add_argument('--learn_rate', type=float, default=0.001, help='learning rate') group.add_argument("--batch_size", type=int, default=128, help="the batch size in train process") group.add_argument('--width', type=int, default=256, help='the width of input image') group.add_argument('--height', type=int, default=256, help='the height of input image') def add_train_options(parser): group = parser.add_argument_group('train') group.add_argument("--train_num_workers", type=int, default=None, help="the num of workers in train process") group.add_argument('--loss_func', type=str, default='STARLoss_v2', help="loss function") group.add_argument("--val_batch_size", type=int, default=None, help="the batch size in val process") group.add_argument("--val_num_workers", type=int, default=None, help="the num of workers in val process") def add_eval_options(parser): group = parser.add_argument_group("eval") group.add_argument("--pretrained_weight", type=str, default=None, help="set pretrained model file name, if ignored then train the network without pretrain model") group.add_argument('--norm_type', type=str, default='default', help='default, ocular, pupil') group.add_argument('--test_file', type=str, default="test.tsv", help='for wflw, test.tsv/test_xx_metadata.tsv') def add_starloss_options(parser): group = parser.add_argument_group('starloss') group.add_argument('--star_w', type=float, default=1, help="regular loss ratio") group.add_argument('--star_dist', type=str, default='smoothl1', help='STARLoss distance function') if __name__ == "__main__": parser = argparse.ArgumentParser(description="Entry Function") add_base_options(parser) add_data_options(parser) add_train_options(parser) add_eval_options(parser) add_starloss_options(parser) args = parser.parse_args() print( "mode is %s, config_name is %s, pretrained_weight is %s, image_dir is %s, annot_dir is %s, device_ids is %s" % ( args.mode, args.config_name, args.pretrained_weight, args.image_dir, args.annot_dir, args.device_ids)) args.device_ids = list(map(int, args.device_ids.split(","))) if args.mode == "train": train(args) elif args.mode == "test": test(args) else: print("unknown running mode") ``` ### backend/cv.py ```python from of_helper import OpenFaceHelper slow_of_helper = OpenFaceHelper() fast_of_helper = OpenFaceHelper() def analyze_image_slow(image_path: str): slow_of_helper.set_image(image_path) slow_of_helper.extract() slow_of_helper.analyze_all_faces() json_data = slow_of_helper.get_slow_json() return json_data def analyze_image_fast(image_path: str): """Fast analysis - only face detection and bounding boxes""" fast_of_helper.set_image(image_path) fast_of_helper.extract() # Only face detection, no emotion analysis json_data = fast_of_helper.get_fast_json() return json_data ``` ### backend/algo.py ```python # algorithms for finding aggregate sentiment etc ''' import numpy as np from collections import deque ZERO_PROBS = {'angry': 0, 'disgust': 0, 'fear': 0, 'happy': 0, 'sad': 0, 'surprise': 0, 'neutral': 0} ALL_EMOTIONS = ("angry", "disgust", "fear", "happy", "sad", "surprise", "neutral") def calculate_gaze_direction(features): x, y, w, h = features["x"], features["y"], features["w"], features["h"] def norm(pt): return np.array([(pt[0] - x) / w, (pt[1] - y) / h]) box_center_norm = norm([x + w / 2, y + h / 2]) nose_norm = norm(features["nose"]) left_eye_norm = norm(features["left_eye"]) right_eye_norm = norm(features["right_eye"]) mouth_left_norm = norm(features["mouth_left"]) mouth_right_norm = norm(features["mouth_right"]) eye_center = (right_eye_norm + left_eye_norm) / 2 mouth_center = (mouth_left_norm + mouth_right_norm) / 2 # ok we get head orientation here, so we shift eyes based on head orientation # head_diff = nose_norm - box_center_norm head_diff = mouth_center - box_center_norm eye_center -= head_diff threshold = 0.1 dx, dy = eye_center[0] - box_center_norm[0], eye_center[1] - box_center_norm[1] if abs(dx) <= threshold: gaze = "Center" else: gaze = "Left" if dx < 0 else "Right" return gaze, (int(x + eye_center[0] * w), int(y + eye_center[1] * h)) def get_dominant_emotion(probs: dict, keys=ALL_EMOTIONS): assert (len(keys) > 0) best = 0 best_key = keys[0] for k, v in probs.items(): if k not in keys: continue if v > best: best = v best_key = k return best_key def get_average_emotions(all_probs: list[dict]): avg_probs = dict(ZERO_PROBS) for probs in all_probs: for k, v in probs.items(): # print(k, v) # print(avg_probs[k]) avg_probs[k] += v / len(all_probs) # print(avg_probs[k]) # print("AVG PROBS", avg_probs) return avg_probs def add_aggregate_sentiment(data): faces = data["faces"] probs = [] for face in faces: emotion_probs = face["emotions"] probs.append(emotion_probs) avg_probs = get_average_emotions(probs) aggregate_dominant_emotions = get_dominant_emotion(avg_probs) data["aggregate_emotions"] = avg_probs data["aggregate_dominant_emotions"] = aggregate_dominant_emotions def get_box(x: int, y: int, s: int) -> (int, int): grid_x = x // s grid_y = y // s return grid_x, grid_y def get_emotion_colormap(data, s, keys=ALL_EMOTIONS): # return an emotion map of (height / s) x (width / s) size eps = 0.1 image_height = data["height"] image_width = data["width"] height = (image_height + s - 1) // s width = (image_width + s - 1) // s mul = 0.9 loc_probs = [[list() for _ in range(width)] for _ in range(height)] grid = [[("none", 0) for _ in range(width)] for _ in range(height)] print(f"Grid initialized with {height} rows, {width} cols") faces = data["faces"] for face_number, face in enumerate(faces): fx, fy, fw, fh = ( face["bounding_box"]["x"], face["bounding_box"]["y"], face["bounding_box"]["w"], face["bounding_box"]["h"] ) x1, x2, y1, y2 = (fx, fx + fw, fy, fy + fh) c1, r1 = get_box(x1, y1, s) c2, r2 = get_box(x2, y2, s) for row in range(r1, r2 + 1): for col in range(c1, c2 + 1): # print(f"Face {face_number + 1} in grid box (row={row}, col={col})") loc_probs[row][col].append(face["emotions"]) q = deque() vis = [[False for _ in range(width)] for _ in range(height)] for row in range(height): for col in range(width): if len(loc_probs[row][col]) == 0: continue avg_probs = get_average_emotions(loc_probs[row][col]) dominant_emotion = get_dominant_emotion(avg_probs, keys=keys) conf = avg_probs[dominant_emotion] / 100.0 if conf < eps: continue grid[row][col] = (dominant_emotion, conf) vis[row][col] = True print(dominant_emotion, avg_probs[dominant_emotion], grid[row][col]) q.append((row, col)) dr = [0, 0, 1, -1] dc = [1, -1, 0, 0] while len(q) > 0: row, col = q.popleft() # print(f"Visiting row={row}, col={col}") for d in range(4): new_row = row + dr[d] new_col = col + dc[d] if new_row < 0 or new_row >= height or new_col < 0 or new_col >= width: continue if vis[new_row][new_col]: continue new_conf = grid[row][col][1] * mul new_emotion = grid[row][col][0] print(new_emotion, new_conf) grid[new_row][new_col] = (new_emotion, new_conf) vis[new_row][new_col] = True q.append((new_row, new_col)) return grid # BFS, voronoi style ''' ``` ### backend/of_helper.py ```python import time import cv2 import numpy as np import os import torch from libs.openface.face_detection import FaceDetector from libs.openface.multitask_model import MultitaskPredictor from pathlib import Path ROOT = Path(__file__).resolve().parent.parent WEIGHTS_DIR = str(ROOT / "backend" / "libs" / "openface" / "weights") DETECTOR_MODEL_PATH = WEIGHTS_DIR + "/Alignment_RetinaFace.pth" MT_MODEL_PATH = WEIGHTS_DIR + "/MTL_backbone.pth" os.environ["OPENFACE_WEIGHTS_DIR"] = WEIGHTS_DIR CONFIDENCE_THRESHOLD = 0.3 EMOTIONS = ["Neutral", "Happy", "Sad", "Surprise", "Fear", "Disgust", "Anger", "Contempt"] def gaze_to_unit_vector(yaw: float, pitch: float): """ Convert yaw (left/right) and pitch (up/down) to a 3D unit vector. yaw: radians, negative=left, positive=right pitch: radians, negative=down, positive=up """ x = np.cos(pitch) * np.sin(-yaw) y = -np.sin(pitch) # z = np.cos(pitch) * np.cos(yaw) vec = np.array([x, y]) return vec / np.linalg.norm(vec) def draw_vector(img, point, vector, length=20, color=(0,255,0)): dx = int(length * vector[0]) dy = int(length * vector[1]) x1, y1 = int(point[0]), int(point[1]) x2, y2 = x1 + dx, y1 + dy cv2.arrowedLine(img, (x1, y1), (x2, y2), color, 2, tipLength=0.2) class OpenFaceHelper: def __init__(self): self.image_path = None self.image = None self.height = 0 self.width = 0 self.num_faces = 0 self.faces = [] self.crops = [] self.attentions = [] self.dominant_emotions = [] self.emotion_probs = [] self.gaze_outputs = [] self.gaze_vectors = [] self.au_outputs = [] print("Initializing Open Face Helper") start_time = time.time() self.detector = FaceDetector(model_path=DETECTOR_MODEL_PATH, device='cpu') self.mt_model = MultitaskPredictor(model_path=MT_MODEL_PATH, device='cpu') end_time = time.time() print(f"Initialization Complete in {end_time - start_time} seconds") def set_image(self, path): self.image_path = path self.image = cv2.imread(self.image_path) self.height, self.width = self.image.shape[:2] @staticmethod def get_bb(face): return tuple(map(int, face[:4])) def get_crop(self, face): x1, y1, x2, y2 = self.get_bb(face) return self.image[y1:y2, x1:x2].copy() def extract(self): print("Extracting faces...") start_time = time.time() _, faces = self.detector.get_face(self.image_path, resize=1.0) self.faces = [] self.crops = [] for face in faces: conf = face[4] if conf < CONFIDENCE_THRESHOLD: continue self.faces.append(face) self.crops.append(self.get_crop(face)) self.num_faces = len(self.faces) end_time = time.time() print(f"Extract Complete in {end_time - start_time} seconds") def analyze_face(self, face_crop: np.ndarray): emotion_logits, gaze_output, au_output = self.mt_model.predict(face_crop) # print("EMOTION LOGITS: ", emotion_logits) # print("Predicted Gaze (Yaw, Pitch): ", gaze_output) # print("Predicted Action Units (Intensities)", au_output) return emotion_logits, gaze_output, au_output def multiply_emotions(self, emotion_probs): emotion_probs[0] *= 1.4 emotion_probs[1] *= 1.5 emotion_probs[2] *= 0.8 emotion_probs[3] *= 0.8 emotion_probs[4] *= 1 emotion_probs[5] *= 0.7 emotion_probs[6] *= 1.1 emotion_probs[7] *= 0.4 s = sum(emotion_probs) emotion_probs /= s def analyze_all_faces(self): if self.num_faces == 0: return print("Analyzing faces...") start_time = time.time() self.emotion_probs = [] self.dominant_emotions = [] self.gaze_outputs = [] self.au_outputs = [] self.attentions = [] for i in range(self.num_faces): face = self.faces[i] crop = self.crops[i] emotion_logits, gaze_output, au_output = self.mt_model.predict(crop) emotion_probs = torch.softmax(emotion_logits, dim=1)[0] self.multiply_emotions(emotion_probs) #emotion_probs = np.array(torch.softmax(emotion_probs, dim=0)) emotion_probs = np.array(emotion_probs) dominant_emotion = EMOTIONS[np.argmax(emotion_probs)] gaze_output = gaze_output[0].tolist() gaze_output[1], gaze_output[0] = gaze_output[0], gaze_output[1] gaze_vector = gaze_to_unit_vector(gaze_output[0], gaze_output[1]) attention = self.get_attention(face, gaze_vector) # print(attention) # print(dominant_emotion) # print(f"Face #{i + 1} emotion probs: {emotion_probs}") # print(f"Face #{i + 1} gaze_output: {gaze_output}") # print(f"Face #{i + 1} au_output: {au_output}") self.emotion_probs.append(emotion_probs) self.dominant_emotions.append(dominant_emotion) self.gaze_outputs.append(gaze_output) self.gaze_vectors.append(gaze_vector) self.au_outputs.append(au_output) self.attentions.append(attention) end_time = time.time() print(f"Analyze Complete in {end_time - start_time} seconds") @staticmethod def get_eye_center(face): left_eye = np.array([face[5], face[6]]) right_eye = np.array([face[7], face[8]]) eye_center = (left_eye + right_eye) / 2 eye_center = (int(eye_center[0]), int(eye_center[1])) return eye_center def get_obs_vector(self, face): eye_center = self.get_eye_center(face) obs_center = np.array([self.width / 2, 4 * self.height / 5]) obs_vector = obs_center - eye_center obs_vector [truncated — 3471 more characters] ``` [90 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]