# Project export: ExpressCall This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Use facial recognition and vocal outbursts to determine when patients with language, significant mobility, or cognitive barriers need immediate medical attention and alert overworked nursing staff. - Devpost: https://devpost.com/software/expresscall - GitHub: https://github.com/lillianyjiang/CalHacks - Video: https://www.youtube.com/embed/bS6ICgiLPLc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — lillianyjiang (5 commits), zach (2 commits) ## Devpost submission (written by the team) ### Inspiration In our current hospital space, 93% nurses surveyed in a study reported they thought hospitals were understaffed. That means without proper care and watch over specific patients with no means of accessing the helpline themselves (i.e those physically unable to move, unaccustomed to communication, or marginalized groups), critical signals that lead to life-or-death decisions may be missed. Observed by one of our own teammates, these patients’ only source of outlet may be vocal outbursts or facial expressions to convey their pain and need for help, which is not necessarily picked up by physical monitors.. Oftentimes, especially in absence of family members, their moans and cries echo unnoticed by nurses in understaffed hospitals – until it is too late This is where ExpressCare comes in, a “call-button” technology that picks up these neglected cues to alert corresponding staff members, morphing hospitals into a more empathetic place. ### What it does Let’s consider a low-income, non-native english speaker that is admitted to an overstaffed hospital without family members. The language barriers alone make a big threat to communication, which is then amplified if the patient is physically constrained to the bed. ExpressCare monitors patients in these states and picks up distress queues to alert nurses if an overall distress index surpasses our threshold. ExpressCare “call-button” device then displays an alert with pain level index and duration on the nurses’ side, informing about a call for help. Our team considered varying needs in hospital spaces, including patients that need psychological support over immediate medical attention. Therefore, an elderly patient admitted for dementia that is exhibiting queues of confusion as the primary emotion may trigger the system as well. Our device monitors this and sends a milder message to corresponding staff members, including hospital volunteers of psychologists that may better address these circumstances. Nurses are able to monitor their patients during normal circumstances as well, as ExpressCare offers a display of levels of discomfort throughout the day. ### How we built it Express Care uses Hume API to configure facial patterns and vocal outbursts to pick up varying emotions. We used batch API and streaming API to process these configurations side-by-side, while isolating the specific negative emotions that resemble discomfort. A distress score is then calculated, which can include anger, pain, confusion, or others as underlying triggers. ### Challenges we ran into Getting the API to process audio and video was the primary challenge. We tried using PyAudio to store audio files and processing them in time intervals alongside video frames, and to get the technology to monitor vocal outbursts and facial expressions at the same time. However, we were unable to make it run on our local devices due to a PyAudio installation error, so decided to pare down the features we wanted to include in this iteration. ### Accomplishments we're proud of This is an incredibly important need that we've identified in the healthcare space, and are proud of the contribution we've made to this area. ### What we learned We learned a lot throughout this project that will be useful to us in the future--both technically and market knowledge of AI, and APIs as a product. ### What's next In the future, we would like to test this with real patients in a SF public hospital to measure the difference in length of distress period with and without ExpressCall, as well as to see if patients need anything else from this service. Additionally, we would like to expand its use case to an iteration where we measure loneliness, sadness, and boredom for the long-term stay patients to see if they need psychological intervention. ## README (from the GitHub repository) # Cal Hacks 10.0 | Hume AI ## Getting Started 1. **Sign up:** Navigate to [https://beta.hume.ai/sign-up](https://beta.hume.ai/sign-up) to sign up and obtain your [API Key](https://dev.hume.ai/docs/quick-start). 2. **Let us know you signed up:** Navigate to [https://link.hume.ai/custom_model_sign_up](https://link.hume.ai/custom_model_sign_up) for access to Custom Models and updates throughout the hackathon. 3. **Documentation:** Explore the Hume API documentation at [dev.hume.ai](https://dev.hume.ai). There you will find Guides, Recipes, FAQs, and our API Reference documentation. 4. **Dataset Tips:** https://dev.hume.ai/docs/data-tips-what-should-your-data-look-like 5. **Support:** You can reach us in the Cal Hacks Hume Slack Channel (`#spons-hume`) or on the [Hume Developer Community Discord Server](https://discord.com/invite/WPRSugvAm6). ## Hume AI Hume provides the AI toolkit to measure, understand, and improve how technology affects human emotion. Our algorithms understand nuanced speech prosody, vocal bursts, facial expression, and tone of language—which, integrated into large language models, will determine how people experience the future of AI. Our APIs can process video, audio, images, or text and can be integrated with LLMs to build better healthcare solutions, digital assistants, communication tools, and more. Learn more about Hume AI, and the science behind the platform: - Hume AI Intro: [The New Science of Expression](https://hume.ai/video/) - [Fundamental Advances in Understanding Nonverbal Behavior](https://youtu.be/4-EEhdqETJY) | Keynote by Alan Cowen | ICML 2022 - Our interactive expression model maps for the voice, face, and language: [https://hume.ai/products/#models](https://hume.ai/products/#models) ## Sample Projects Below are a list of example projects in Python and NodeJS for reference to jumpstart your project development. Visit the `#built-with-hume` channel in the Community section of our [Discord Server](https://discord.com/invite/WPRSugvAm6) for more reference code! **Python** - [Expressions Prompt Engineering](https://github.com/HumeAI/expressive-prompt-engineering/tree/main) | _Developed by our research team, this project is a solution for enhancing LLM prompts with expression via Hume APIs. ChatGPT and OpenAI language embeddings are also integrated to enable deeper experimentation._ - [Humechat](https://github.com/HumeAI/CalHacks/tree/main/humechat) | _Sample project which usees OpenAPI with Hume AI. Also includes an example of how to stringify Hume API expression predictions with OpenAPI._ **NodeJS** - [Hume Raw Text Processor](https://github.com/HumeAI/CalHacks/tree/main/hume-raw-text-processor) | _Sample project on processing raw text with Hume's Batch API. Implements polling for a job status before fetching predictions with exponential backoff._ - [Sandbox](https://github.com/HumeAI/CalHacks/tree/main/sandbox) | _Provides a sample on how to use Hume's Streaming API with your webcam and mic._ - [Hume Chrome Extension](https://github.com/HumeAI/hume-chrome-extension) | _Sample project which demonstrates how our Web Socket API could be used to obtain real-time inference results from our Face model for videos streamed in a web browser._ ## How Businesses and Researchers are utilizing Hume today: - **Health & Wellness** | Clinical diagnosis (e.g., depression, autism); patient monitoring, therap.y - **AI Research/Services** | The next generation of search, recommendation, and content generation. - **Social Networks** | Toxicity detection; health/well-being monitoring; relationship compatibility. - **Call Center Analytics** | Call triaging (e.g., frustration); emergency detection (e.g., pain); training. - **Embedded Devices** | Social robots; AI dashcams; warehouse safety. - **Brand/Financial Analysis** | Sentiment analysis for market forecasting and brand sentiment research. - **Creative Tools** | Character animation; content generation, editing, and curation. - **Digital Assistants** | Conversational AI (e.g., backchanneling); optimization (e.g., ↓ frustration). - **UX/CX Research** | Sentiment analysis of user interviews and tests. - **Gaming & XR** | Animation; virtual characters; moderation (e.g., bullying); optimization. - **Education/Coaching** | Focus/boredom detection; student well-being; leadership coaching. - **Sales/Meeting Analytics** | Sales rep coaching; analyzing customer engagement and sentiment. ## Detected evidence (automated analysis) Indexed codebase: 61 recognized source files, 83 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (73 of 73) ``` .DS_Store .gitignore hands-on/README.md hume-raw-text-processor/.gitignore hume-raw-text-processor/package.json hume-raw-text-processor/README.md hume-raw-text-processor/src/index.ts hume-raw-text-processor/tsconfig.json humechat/chat.py humechat/main.py humechat/README.md humechat/requirements.txt LICENSE README.md sandbox/.gitignore sandbox/.vscode/extensions.json sandbox/.vscode/settings.json sandbox/components/inputs/Button.tsx sandbox/components/inputs/TextArea.tsx sandbox/components/inputs/TextBox.tsx sandbox/components/menu/Auth.tsx sandbox/components/menu/Login.tsx sandbox/components/menu/Nav.tsx sandbox/components/menu/Toolbar.tsx sandbox/components/widgets/AudioWidgets.tsx sandbox/components/widgets/BurstWidgets.tsx sandbox/components/widgets/Descriptor.tsx sandbox/components/widgets/DiscreteTimeline.tsx sandbox/components/widgets/FaceTrackedVideo.tsx sandbox/components/widgets/FaceWidgets.tsx sandbox/components/widgets/LanguageWidgets.tsx sandbox/components/widgets/Loader.tsx sandbox/components/widgets/LoaderSet.tsx sandbox/components/widgets/ProsodyWidgets.tsx sandbox/components/widgets/TopEmotions.tsx sandbox/lib/data/audioPrediction.ts sandbox/lib/data/boundingBox.ts sandbox/lib/data/characterRange.ts sandbox/lib/data/embedding.ts sandbox/lib/data/emotion.ts sandbox/lib/data/facePrediction.ts sandbox/lib/data/languagePrediction.ts sandbox/lib/data/range.ts sandbox/lib/data/timeRange.ts sandbox/lib/data/trackedFace.ts sandbox/lib/hooks/keyPress.ts sandbox/lib/hooks/stability.ts sandbox/lib/hooks/storage.ts sandbox/lib/media/audioRecorder.ts sandbox/lib/media/videoRecorder.ts sandbox/lib/utilities/asyncUtilities.ts sandbox/lib/utilities/blobUtilities.ts sandbox/lib/utilities/embeddingUtilities.ts sandbox/lib/utilities/emotionUtilities.ts sandbox/lib/utilities/environmentUtilities.ts sandbox/lib/utilities/scalingUtilities.ts sandbox/lib/utilities/styleUtilities.ts sandbox/lib/utilities/typeUtilities.ts sandbox/next.config.js sandbox/package.json sandbox/pages/_app.tsx sandbox/pages/burst/index.tsx sandbox/pages/burst/timeline/index.tsx sandbox/pages/face/calibrate/index.tsx sandbox/pages/face/index.tsx sandbox/pages/index.tsx sandbox/pages/language/index.tsx sandbox/pages/prosody/index.tsx sandbox/postcss.config.js sandbox/README.md sandbox/styles/globals.css sandbox/tailwind.config.js sandbox/tsconfig.json ``` ### Dependencies - hume-raw-text-processor/package.json: @types/node@^20.8.7, typescript@^5.2.2 - humechat/requirements.txt: colorama, git@+https://github.com/aarnphm/whispercpp.git, gTTS, hume, hume[stream], numpy, openai, opencv-python, playsound, pvrecorder, pynput, whispercpp - sandbox/package.json: @fontsource/poppins@^4.5.10, @phosphor-icons/react@2.0.5, @types/node@18.11.9, @types/react@18.0.24, autoprefixer@^10.4.13, class-variance-authority@^0.6.0, next@13.0.1, postcss@^8.4.18, prettier@^2.8.8, prettier-plugin-tailwindcss@^0.2.7, react@^18.2.0, react-dom@^18.2.0, react-use@^17.4.0, tailwind-merge@^1.12.0, tailwindcss@^3.2.1, typescript@4.8.4 ### Recent commits (newest first) - final code - restoring to previous version - it's not working - still working - distress score for faces - adds link to hume chrome extension example project - initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### humechat/requirements.txt ``` whispercpp openai pynput playsound pvrecorder numpy git+https://github.com/aarnphm/whispercpp.git gTTS opencv-python hume hume[stream] colorama ``` ### hume-raw-text-processor/package.json ``` { "name": "hume-raw-text-processor", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "start": "tsc && node dist/index.js", "build": "tsc", "lint": "eslint 'src/**/*.{ts,tsx}'", "format": "prettier --write 'src/**/*.{ts,tsx}'" }, "keywords": [], "author": "", "license": "ISC", "devDependencies": { "@types/node": "^20.8.7", "typescript": "^5.2.2" } } ``` ### sandbox/package.json ``` { "name": "sandbox", "version": "0.2.0", "private": true, "scripts": { "dev": "next dev -p 3001", "build": "next build", "start": "next start -p 3001", "lint": "next lint" }, "dependencies": { "@fontsource/poppins": "^4.5.10", "@phosphor-icons/react": "2.0.5", "@types/node": "18.11.9", "@types/react": "18.0.24", "class-variance-authority": "^0.6.0", "next": "13.0.1", "react": "^18.2.0", "react-dom": "^18.2.0", "react-use": "^17.4.0", "tailwind-merge": "^1.12.0", "typescript": "4.8.4" }, "devDependencies": { "autoprefixer": "^10.4.13", "postcss": "^8.4.18", "prettier": "^2.8.8", "prettier-plugin-tailwindcss": "^0.2.7", "tailwindcss": "^3.2.1" } } ``` ### humechat/main.py ```python import threading import asyncio import os import cv2 import time import traceback import websockets import numpy as np from pynput import keyboard from pvrecorder import PvRecorder from whispercpp import Whisper from chat import message, store_emotions from playsound import playsound from hume import HumeStreamClient, HumeClientException from hume.models.config import FaceConfig from gtts import gTTS # Configurations HUME_API_KEY = "VbiYShiDrtyMySWqUodZMSaOZdyf7Tm0vdKIHBvBmaO6IaV9" # paste your API Key here HUME_FACE_FPS = 1 / 3 # 3 FPS TEMP_FILE = 'temp.jpg' TEMP_WAV_FILE = 'temp.wav' # Initialize whisper model, pyttsx3 engine, and pv recorder w = Whisper.from_pretrained("tiny.en") recorder = PvRecorder(device_index=-1, frame_length=512) # Global variables recording = False recording_data = [] # Webcam setup cam = cv2.VideoCapture(0) async def webcam_loop(): while True: try: client = HumeStreamClient(HUME_API_KEY) config = FaceConfig(identify_faces=True) async with client.connect([config]) as socket: print("(Connected to Hume API!)") while True: if not recording: _, frame = cam.read() cv2.imwrite(TEMP_FILE, frame) result = await socket.send_file(TEMP_FILE) store_emotions(result) await asyncio.sleep(1 / 3) except websockets.exceptions.ConnectionClosedError: print("Connection lost. Attempting to reconnect in 1 seconds.") time.sleep(1) except HumeClientException: print(traceback.format_exc()) break except Exception: print(traceback.format_exc()) def start_asyncio_event_loop(loop, asyncio_function): asyncio.set_event_loop(loop) loop.run_until_complete(asyncio_function) def recording_loop(): global recording_data, recording while recording: frame = recorder.read() recording_data.append(frame) recorder.stop() print("(Recording stopped...)") recording_data = np.hstack(recording_data).astype(np.int16).flatten().astype(np.float32) / 32768.0 transcription = w.transcribe(recording_data) response = message(transcription) if response: tts = gTTS(text=response, lang='en') tts.save(TEMP_WAV_FILE) playsound(TEMP_WAV_FILE) os.remove(TEMP_WAV_FILE) def on_press(key): global recording, recording_data, recorder if key == keyboard.Key.space: if recording: recording = False else: recording = True recording_data = [] recorder.start() print("(Recording started...)") threading.Thread(target=recording_loop).start() new_loop = asyncio.new_event_loop() threading.Thread(target=start_asyncio_event_loop, args=(new_loop, webcam_loop())).start() with keyboard.Listener(on_press=on_press) as listener: print("Speak to Joaquin!") print("(Press spacebar to speak. To finish speaking, press spacebar again)") listener.join() ``` ### sandbox/pages/index.tsx ```typescript import { BookOpenText as BookIcon, Ear as EarIcon, Microphone as MicrophoneIcon, SmileySticker as SmileyIcon, } from "@phosphor-icons/react"; import Link from "next/link"; export default function HomePage() { return (
Hume AI Sandbox
Select a modality to try out Hume's models with your webcam and microphone
); } type ModelSectionProps = { iconClass: any; name: string; page: string; }; function ModelSection(props: ModelSectionProps) { return (
{props.name}
); } ``` ### hume-raw-text-processor/src/index.ts ```typescript const BASE_URL = 'https://api.hume.ai/v0/batch/jobs'; // 1. Set your API Key const HUME_API_KEY = ''; // 2. Specify which Language const language: Language = 'en'; // 3. Specify Language Model configuration const languageModelConfig: LanguageModelConfig = {}; // 4. Copy and paste the text you'd like processed here const rawTextInput = ''; // 5. Run `npm run start` to get inference results (predictions) from Hume's Language Model for the rawTextInput. processRawText(rawTextInput, language, languageModelConfig).catch( (error: Error) => console.error('An error occurred:', error) ); /** * Function which starts a job, polls the status of the job until status is `COMPLETED`, and then fetches the * job predictions. */ async function processRawText( rawText: string, language: Language, languageModelConfig: LanguageModelConfig ): Promise { const MAX_RETRIES = 5; // adjust the number of retries here const INITIAL_DELAY_MS = 1000; // starting with 1 second delay let delay = INITIAL_DELAY_MS; const jobId = await startJob(rawText, language, languageModelConfig); // poll with exponential backoff for (let attempt = 0; attempt < MAX_RETRIES; attempt++) { const status = await getJobStatus(jobId); if (status === 'COMPLETED') { console.log('Status is COMPLETED!'); const predictions = await getPredictions(jobId); console.log(JSON.stringify(predictions)); return; } console.log(`Status is ${status}. Retrying in ${delay / 1000} seconds...`); await sleep(delay); delay *= 2; // exponential backoff } console.error('Max retries reached. Giving up.'); } /** * See API Reference for more information on the start job endpoint: https://dev.hume.ai/reference/start_job */ async function startJob( rawText: string, language: Language, languageModelConfig: LanguageModelConfig ): Promise { const body = JSON.stringify({ text: [rawText], models: { language: languageModelConfig }, transcription: { language }, }); const options = { ...buildHumeRequestOptions('POST'), body }; const response = await fetch(BASE_URL, options); if (!response.ok) { throw new Error(`Failed to start job: ${response.statusText}`); } const json = await response.json(); return json.job_id as string; } /** * See API Reference for more information on the get job details endpoint: https://dev.hume.ai/reference/get_job. */ async function getJobStatus(jobId: string): Promise { const options = buildHumeRequestOptions('GET'); const response = await fetch(`${BASE_URL}/${jobId}`, options); if (!response.ok) { throw new Error(`Failed to fetch job status: ${response.statusText}`); } const json = await response.json(); return json.state.status; } /** * See API Reference for more information on the get job predictions endpoint: https://dev.hume.ai/reference/get_job_predictions. */ async function getPredictions(jobId: string): Promise { const options = buildHumeRequestOptions('GET'); const response = await fetch(`${BASE_URL}/${jobId}/predictions`, options); if (!response.ok) { throw new Error(`Failed to fetch job predictions: ${response.statusText}`); } const json = await response.json(); return json; } /** * Helper function for building headers and options for Hume API requests. */ function buildHumeRequestOptions(method: 'GET' | 'POST'): { method: 'GET' | 'POST'; headers: Headers; } { const headers = new Headers(); headers.append('X-Hume-Api-Key', HUME_API_KEY); headers.append('Content-Type', 'application/json'); return { method, headers }; } /** * Helper function to support exponential backoff implementation when polling for job status. */ function sleep(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); } /** * Language (ISO 639-1) Codes used to specify which language is to be processed. See our documentation * for supported languages here: https://dev.hume.ai/docs/supported-languages. */ type Language = | 'zh' // Chinese | 'da' // Danish | 'nl' // Dutch | 'en' // English | 'en-AU' // English (Australia) | 'en-IN' // English (India) | 'en-NZ' // English (New Zealand) | 'en-GB' // English (United Kingdom) | 'fr' // French | 'fr-CA' // French (Canadian) | 'de' // German | 'hi' // Hindi | 'hi-Latn' // Hindi (Roman Script) | 'id' // Indonesian | 'it' // Italian | 'ja' // Japanese | 'ko' // Korean | 'no' // Norwegian | 'pl' // Polish | 'pt' // Portuguese | 'pt-BR' // Portuguese (Brazil) | 'pt-PT' // Portuguese (Portugal) | 'ru' // Russian | 'es' // Spanish | 'es-419' // Spanish (Latin America) | 'sv' // Swedish | 'ta' // Tamil | 'tr' // Turkish | 'uk'; // Ukrainian /** * The granularity at which to generate predictions. `utterance` corresponds to a natural pause or break in conversation, while `conversational_turn` * corresponds to a change in speaker. Granularity will default to `word` if not provided. * * For more information on configuring granularity, check out our documentation here: * https://dev.hume.ai/docs/how-granular-are-the-outputs-of-our-speech-prosody-and-language-models */ type LanguageGranularity = | 'word' | 'sentence' | 'utterance' | 'conversational_turn'; /** * Configuration object which informs how Hume's Language model will process the text, and whether to include predictions from * the Sentiment and Toxicity models. See the start job endpoint for more details on job configuration: * https://dev.hume.ai/reference/start_job. */ type LanguageModelConfig = { granularity?: LanguageGranularity; sentiment?: {}; toxicity?: {}; }; ``` ### sandbox/pages/face/index.tsx ```typescript import { FaceWidgets } from "../../components/widgets/FaceWidgets"; export default function FacePage() { return (
Facial Expression
); } ``` ### sandbox/pages/language/index.tsx ```typescript import { LanguageWidgets } from "../../components/widgets/LanguageWidgets"; export default function LanguagePage() { return (
Written Language
); } ``` ### sandbox/pages/prosody/index.tsx ```typescript import { ProsodyWidgets } from "../../components/widgets/ProsodyWidgets"; export default function ProsodyPage() { return (
Speech Prosody
Speech prosody is not about the words you say, but the way you say them.
); } ``` ### sandbox/pages/burst/index.tsx ```typescript import { BurstWidgets } from "../../components/widgets/BurstWidgets"; export default function BurstPage() { return (
Vocal Burst
Vocal bursts are non-linguistic vocal utterances, including laughs, sighs, oohs, ahhs, umms, gasps, and groans.
); } ``` [49 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]