# Project export: Vocera This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Vocera introduces vocal signatures: a deepfake-resistant authentication system learning how you speak by building a speech representation vector. We protect from scam calls and validate transactions. - Devpost: https://devpost.com/software/vocera - GitHub: https://github.com/akh1lk/vocera - Team: 3 GitHub contributor(s) — asungii (27 commits), Akhil Kagithapu (13 commits), Paul (2 commits) ## Devpost submission (written by the team) ### Inspiration With the rise of generative AI, voice deepfakes have become nearly indistinguishable from real speech—leading to real-world harm. A close friend’s mother was tricked into wiring money to someone impersonating her son. Incidents like this underscore the urgent need for trustworthy voice authentication during sensitive calls. Moreover, recent voice models like ElevenLabs' v3 have become remarkably close to actual speaking patterns, overcoming the Uncanny Valley, while few algorithms exist to identify these deepfaked voices. ### What it does Vocera (vo-cher-uh) is a mobile app that verifies a caller's identity using their unique vocal signature—designed specifically for high-risk phone calls, such as when a friend or family member asks for money, gift cards, or urgent help like bail. To confirm their identity, the caller must speak a secret passphrase known as a Vox Key—a private phrase that's grammatically nonsensical with a contrasting tone, like "The grieving dogs are raining birds," enthusiastically. The unusual structure, and the distinct way each person says it, makes it difficult for deepfake models to mimic. Vocera then authenticates the caller through a three-step process: Passphrase check – Is the correct Vox Key textual phrase being spoken? Speaker verification – Does the voice resemble the registered user's voice? Deepfake detection – Are there signs the voice was AI-generated? In the interest of maximum security, this Vox Key captures your vocal signature—tone, inflection, and other features unique to you. After each use, the Vox Key must be replaced by recording ten ~5 second voice clips to capture low-level vocal traits such as shimmer, inflection, and pause distribution. If the verification passes, you know you’re speaking with the real person—not a cloned voice. ### How we built it Vocera is built with React Native and Expo for cross-platform deployment, styled with NativeWind, and powered by a Zustand context store for state management. We use Supabase for user auth and data storage (Postgres databases + buckets), and Heroku for hosting our models. Each user’s Vox Key is generated by Claude and recorded 10 times to capture unique vocal traits. We extract embeddings using OpenSMILE and store them as part of each user’s voice profile. Our three-stage verification pipeline includes: OpenAI Whisper for passphrase transcription SpeechBrain for speaker verification OpenSMILE + sigmoid scoring to detect nuanced deepfake artifacts To improve robustness, we generated adversarial examples using ElevenLabs voice cloning tools, allowing us to fine-tune the system with both authentic and deepfake data. ### Challenges we ran into There were no pretrained or unified solutions capable of countering cutting-edge generative voice models, so we had to engineer our own from scratch. This included designing evaluation procedures, creating datasets, and testing multiple models to build a security pipeline that even state-of-the-art deepfakes—like ElevenLabs v3—couldn’t fool. While we had prior experience with React, we were new to React Native and mobile development. Finding and learning the ins and outs of suitable audio recording libraries was challenging. Animation tuning and mobile-specific bugs consumed much of our limited 24-hour window. We also ran into delays when FFmpeg, our media conversion tool, corrupted audio files into formats incompatible with our models, stalling integration and testing throughout the hackathon. ### Accomplishments we're proud of We’re proud of building a fully functional mobile app in just 24 hours, despite being new to React Native. We developed a deepfake-resistant voice authentication pipeline that performed better than human judgment in many cases. We successfully integrated OpenAI Whisper, Claude, SpeechBrain, and OpenSMILE into a multi-stage verification system. Our UI is clean, intuitive, and animation-enhanced, offering a user-friendly experience without compromising security. Additionally, we generated synthetic deepfake audio using ElevenLabs to strengthen our model against real-world threats. ### What we learned We learned how to extract and compare voice embeddings using OpenSMILE to detect deepfakes, and how to integrate ML tools like Whisper and SpeechBrain into a real-time mobile app. We gained hands-on experience with React Native, mobile-specific debugging, and building secure, user-friendly interfaces under time pressure. ### What's next We aim to optimize real-time performance, expand to on-device processing for privacy, and position Vocera as a new standard for voice authentication in banking, identity verification, and sensitive communications. ## README (from the GitHub repository) # Vocera - Voice Authentication + DeepFake Detection System A sophisticated voice verification platform combining React Native mobile app, FastAPI backend, and advanced machine learning for secure biometric authentication. ## 🎯 Overview Vocera is a comprehensive voice authentication system that uses multiple AI models to verify user identity through voice analysis. In a world of deepfakes that are difficult for the human eyes and ears to discern, Vocera provides you with a way to verify the world around you. Specifically, it helps you prevent deepfake calls from scammers, who pose as family & friends asking for money. The platform employs a dual-verification approach using both traditional signal processing (openSMILE) and modern deep learning (SpeechBrain ECAPA-VOXCELEB) for robust speaker verification. ## ✨ Features ### 🔊 **Advanced Voice Authentication** - **Dual Verification System**: Combines openSMILE feature extraction with SpeechBrain deep learning - **Anti-Deepfake Protection**: Sophisticated algorithms to detect synthetic voice generation - **Confidence Scoring**: Sigmoid-based confidence calculation with tunable thresholds - **Feature Normalization**: StandardScaler preprocessing for consistent analysis ### 📱 **Cross-Platform Mobile App** - **Universal Support**: iOS, Android, and Web deployment - **Real-time Recording**: High-quality voice capture with waveform visualization - **Auth Integration**: Supabase Email Sign-In authentication - **Cloud Sync**: Supabase backend integration for data persistence ### 🤖 **Machine Learning Pipeline** - **openSMILE Feature Extraction**: 88-dimensional eGeMAPSv02 feature vectors - **SpeechBrain ECAPA-VOXCELEB**: State-of-the-art speaker verification model - **Textual Verification**: OpenAI Whisper transcription with GPT-4 semantic analysis - **Euclidean Distance Analysis**: Normalized distance calculations for authenticity scoring ## 🛠️ Tech Stack ### Frontend (vocera-frontend) + Backend - **Framework**: React Native & Expo - **Styling**: TailwindCSS with NativeWind - **Audio**: Expo Audio for recording and playback - **State Management**: Zustand - **Backend**: Supabase client integration, Supabase Buckets Storage - **AI Integration**: OpenAI & Anthropic APIs ### ML Models (voice-detect) - **Server**: Flask with Python 3.9 - **ML Libraries**: - openSMILE (feature extraction) - SpeechBrain (speaker verification) - scikit-learn (StandardScaler normalization) - NumPy/SciPy (numerical computation) - **Database**: Supabase (PostgreSQL-based) - **AI Services**: OpenAI Whisper, GPT-4 - **Deployment**: Docker containerization ready ## 🚀 Quick Start ### Prerequisites - Node.js 18+ - Python 3.9+ - Expo CLI - iOS Simulator or Android emulator ### Installation 1. **Clone Repository** ```bash git clone cd vocera ``` 2. **Install Dependencies** ```bash # Install all dependencies npm run install:all # Or install individually cd vocera-frontend && npm install cd ../voice-detect && pip install -r requirements.txt cd ../api && pip install -r requirements.txt ``` 3. **Environment Setup** Create `vocera-frontend/.env`: ```env EXPO_PUBLIC_SUPABASE_URL=your_supabase_url EXPO_PUBLIC_SUPABASE_ANON_KEY=your_supabase_key EXPO_PUBLIC_OPENAI_API_KEY=your_openai_key EXPO_PUBLIC_ANTHROPIC_API_KEY=your_anthropic_key ``` Create `voice-detect/.env`: ```env SUPABASE_URL=your_supabase_url SUPABASE_SERVICE_KEY=your_supabase_service_key OPENAI_API_KEY=your_openai_key ``` ### Development **Start Voice Detection Server:** ```bash cd voice-detect python app.py # Server runs on http://localhost:5001 ``` **Start Frontend:** ```bash cd vocera-frontend npm run start # Then choose your platform: # - Press 'i' for iOS simulator # - Press 'a' for Android emulator # - Press 'w' for web browser ``` ``` ## 🔬 Voice Authentication Process ### 1. Calibration Phase ```bash # Calibrate user profile with 10 voice samples curl -X POST http://localhost:5001/calibrate \ -F "user_id=username" \ -F "files=@sample1.wav" \ -F "files=@sample2.wav" \ # ... (all 10 calibration files) ``` ### 2. Verification Phase ```bash # Verify voice sample against user profile curl -X POST http://localhost:5001/verify \ -F "user_id=username" \ -F "files=@test_voice.wav" ``` ## Detected evidence (automated analysis) Indexed codebase: 79 recognized source files, 268 KB. - Anthropic (technology) — detected in the code - C (language) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - Kotlin (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 125) ``` .gitignore ci_test.sh laerdonsampledata/laerdon_profile.json package.json README.md run_full_test.py test_calibration.py test_verification.py vocera-frontend/.env.example vocera-frontend/.gitignore vocera-frontend/android/.gitignore vocera-frontend/android/app/build.gradle vocera-frontend/android/app/debug.keystore vocera-frontend/android/app/proguard-rules.pro vocera-frontend/android/app/src/debug/AndroidManifest.xml vocera-frontend/android/app/src/main/AndroidManifest.xml vocera-frontend/android/app/src/main/java/com/anonymous/voceravoiceverification/MainActivity.kt vocera-frontend/android/app/src/main/java/com/anonymous/voceravoiceverification/MainApplication.kt vocera-frontend/android/app/src/main/res/drawable/ic_launcher_background.xml vocera-frontend/android/app/src/main/res/drawable/rn_edit_text_material.xml vocera-frontend/android/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml vocera-frontend/android/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml vocera-frontend/android/app/src/main/res/values-night/colors.xml vocera-frontend/android/app/src/main/res/values/colors.xml vocera-frontend/android/app/src/main/res/values/strings.xml vocera-frontend/android/app/src/main/res/values/styles.xml vocera-frontend/android/build.gradle vocera-frontend/android/gradle.properties vocera-frontend/android/gradle/wrapper/gradle-wrapper.properties vocera-frontend/android/gradlew vocera-frontend/android/gradlew.bat vocera-frontend/android/settings.gradle vocera-frontend/app.config.js vocera-frontend/app.json vocera-frontend/app/_layout.tsx vocera-frontend/app/(tabs)/_layout.tsx vocera-frontend/app/(tabs)/calls.tsx vocera-frontend/app/(tabs)/index.tsx vocera-frontend/app/(tabs)/record.tsx vocera-frontend/app/(tabs)/settings.tsx vocera-frontend/app/+not-found.tsx vocera-frontend/app/voxkey-wizard.tsx vocera-frontend/babel.config.js vocera-frontend/components/Account.tsx vocera-frontend/components/AppWrapper.tsx vocera-frontend/components/Auth.tsx vocera-frontend/components/Collapsible.tsx vocera-frontend/components/ExternalLink.tsx vocera-frontend/components/GoogleSignIn.tsx vocera-frontend/components/HapticTab.tsx vocera-frontend/components/HelloWave.tsx vocera-frontend/components/InstructionsPanel.tsx vocera-frontend/components/NameInput.tsx vocera-frontend/components/ParallaxScrollView.tsx vocera-frontend/components/Recorder.tsx vocera-frontend/components/ThemedText.tsx vocera-frontend/components/ThemedView.tsx vocera-frontend/components/TranscriptView.tsx vocera-frontend/components/ui/IconSymbol.ios.tsx vocera-frontend/components/ui/IconSymbol.tsx vocera-frontend/components/ui/TabBarBackground.ios.tsx vocera-frontend/components/ui/TabBarBackground.tsx vocera-frontend/components/VoiceVerificationButton.tsx vocera-frontend/components/VoxButton.tsx vocera-frontend/constants/Colors.ts vocera-frontend/constants/FontHelper.ts vocera-frontend/constants/Theme.ts vocera-frontend/eslint.config.js vocera-frontend/global.css vocera-frontend/hooks/useAudioRecorder.ts vocera-frontend/hooks/useColorScheme.ts vocera-frontend/hooks/useColorScheme.web.ts vocera-frontend/hooks/useThemeColor.ts vocera-frontend/ios/.gitignore vocera-frontend/ios/.xcode.env vocera-frontend/ios/Podfile vocera-frontend/ios/Podfile.lock vocera-frontend/ios/Podfile.properties.json vocera-frontend/ios/Vocera.xcodeproj/project.pbxproj vocera-frontend/ios/Vocera.xcodeproj/xcshareddata/xcschemes/Vocera.xcscheme vocera-frontend/ios/Vocera.xcworkspace/contents.xcworkspacedata vocera-frontend/ios/Vocera/AppDelegate.swift vocera-frontend/ios/Vocera/Images.xcassets/AppIcon.appiconset/Contents.json vocera-frontend/ios/Vocera/Images.xcassets/Contents.json vocera-frontend/ios/Vocera/Images.xcassets/SplashScreenBackground.colorset/Contents.json vocera-frontend/ios/Vocera/Images.xcassets/SplashScreenLogo.imageset/Contents.json vocera-frontend/ios/Vocera/Info.plist vocera-frontend/ios/Vocera/PrivacyInfo.xcprivacy vocera-frontend/ios/Vocera/SplashScreen.storyboard vocera-frontend/ios/Vocera/Supporting/Expo.plist vocera-frontend/ios/Vocera/Vocera-Bridging-Header.h vocera-frontend/ios/Vocera/Vocera.entitlements vocera-frontend/lib/supabase.ts vocera-frontend/metro.config.js vocera-frontend/nativewind-env.d.ts vocera-frontend/package.json vocera-frontend/react-native.config.js vocera-frontend/README.md vocera-frontend/scripts/reset-project.js vocera-frontend/services/api.ts vocera-frontend/services/audioUtils.ts vocera-frontend/services/claudeAPI.ts vocera-frontend/services/openaiService.ts vocera-frontend/services/supabaseService.ts vocera-frontend/store/voceraStore.ts vocera-frontend/tailwind.config.js vocera-frontend/tsconfig.json voice-detect/.gitignore voice-detect/app.py voice-detect/Aptfile voice-detect/database.py voice-detect/Dockerfile voice-detect/feature_extractor.py voice-detect/insert_gold_vectors.py voice-detect/Procfile voice-detect/requirements.txt voice-detect/runtime.txt voice-detect/speaker_verification.py voice-detect/test_audio.m4a voice-detect/textual_verification.py [5 more files omitted for size] ``` ### Dependencies - vocera-frontend/package.json: @anthropic-ai/sdk@^0.54.0, @babel/core@^7.25.2, @expo/vector-icons@^14.1.0, @react-native-async-storage/async-storage@2.1.2, @react-native-google-signin/google-signin@^14.0.1, @react-navigation/bottom-tabs@^7.3.10, @react-navigation/elements@^2.3.8, @react-navigation/native@^7.1.6, @supabase/supabase-js@^2.50.0, @types/crypto-js@^4.2.2, @types/react@~19.0.10, axios@^1.10.0, crypto-js@^4.2.0, eslint@^9.25.0, eslint-config-expo@~9.2.0, expo@~53.0.12, expo-audio@~0.4.6, expo-blur@~14.1.5, expo-constants@~17.1.6, expo-file-system@~18.1.10, expo-font@~13.3.1, expo-haptics@~14.1.4, expo-image@~2.3.0, expo-linear-gradient@^14.1.5, expo-linking@~7.1.5, expo-router@~5.1.0, expo-secure-store@~14.2.3, expo-sharing@^13.1.5, expo-splash-screen@~0.30.9, expo-status-bar@~2.2.3, expo-symbols@~0.4.5, expo-system-ui@~5.0.9, expo-web-browser@~14.2.0, fs@^0.0.1-security, lucide-react-native@^0.522.0, moti@^0.30.0, nativewind@^4.1.23, openai@^5.6.0, react@19.0.0, react-dom@19.0.0, react-native@0.79.4, react-native-gesture-handler@~2.24.0, react-native-reanimated@~3.17.4, react-native-safe-area-context@^5.4.0, react-native-screens@~4.11.1, react-native-svg@^15.11.2, react-native-url-polyfill@^2.0.0, react-native-vector-icons@^10.2.0, react-native-web@~0.20.0, react-native-webview@13.13.5, tailwindcss@^3.4.17, typescript@~5.8.3, zustand@^5.0.5 - voice-detect/requirements.txt: Flask, gunicorn, numpy, openai, opensmile, pydub, python-dotenv, requests, scikit-learn, scipy, supabase, Werkzeug ### Recent commits (newest first) - Update README.md - merge pull request #1 from akh1lk/backend - Merge branch 'main' into backend - Create README.md - Final UI + frontend + supabase logic - try again exp - edited sigmoid - test 2 - m4a works - fixed supa issues - Add system dependencies for openSMILE M4A support - Remove audio conversion - work directly with M4A files - replaced tmp with m4a - give sox more info - Add SoX audio conversion for robust M4A handling on Heroku - Trying more aggressive conversion - Simplify audio conversion by assuming all files are M4A format - Fix M4A audio conversion with fallback strategies for problematic metadata - Fix audio format detection: detect format from file content instead of URL extension - rebuilding index ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "vocera", "version": "1.0.0", "description": "Vocera monorepo with frontend app and voice detection service", "private": true, "scripts": { "frontend:dev": "cd vocera-frontend && npm start", "frontend:android": "cd vocera-frontend && npm run android", "frontend:ios": "cd vocera-frontend && npm run ios", "frontend:web": "cd vocera-frontend && npm run web", "voice:dev": "cd voice-detect && npm run dev", "voice:start": "cd voice-detect && npm start", "install:all": "cd vocera-frontend && npm install && cd ../voice-detect && npm install" }, "keywords": [ "vocera", "voice-detection", "react-native", "expo", "monorepo" ], "author": "", "license": "MIT" } ``` ### voice-detect/requirements.txt ``` Flask numpy scipy opensmile Werkzeug gunicorn openai python-dotenv scikit-learn requests supabase pydub ``` ### voice-detect/Dockerfile ``` # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the dependencies file to the working directory COPY requirements.txt . # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir -r requirements.txt # Copy the content of the local src directory to the working directory COPY . . # Make port 5001 available to the world outside this container EXPOSE 5001 # Define environment variable ENV NAME VoceraBackend # Run app.py when the container launches CMD ["gunicorn", "--bind", "0.0.0.0:5001", "app:app"] ``` ### vocera-frontend/package.json ``` { "name": "vocera-frontend", "main": "expo-router/entry", "version": "1.0.0", "scripts": { "start": "expo start", "reset-project": "node ./scripts/reset-project.js", "android": "expo run:android", "ios": "expo run:ios", "web": "expo start --web", "lint": "expo lint" }, "dependencies": { "@anthropic-ai/sdk": "^0.54.0", "@expo/vector-icons": "^14.1.0", "@react-native-async-storage/async-storage": "2.1.2", "@react-native-google-signin/google-signin": "^14.0.1", "@react-navigation/bottom-tabs": "^7.3.10", "@react-navigation/elements": "^2.3.8", "@react-navigation/native": "^7.1.6", "@supabase/supabase-js": "^2.50.0", "axios": "^1.10.0", "crypto-js": "^4.2.0", "expo": "~53.0.12", "expo-audio": "~0.4.6", "expo-blur": "~14.1.5", "expo-constants": "~17.1.6", "expo-file-system": "~18.1.10", "expo-font": "~13.3.1", "expo-haptics": "~14.1.4", "expo-image": "~2.3.0", "expo-linear-gradient": "^14.1.5", "expo-linking": "~7.1.5", "expo-router": "~5.1.0", "expo-secure-store": "~14.2.3", "expo-sharing": "^13.1.5", "expo-splash-screen": "~0.30.9", "expo-status-bar": "~2.2.3", "expo-symbols": "~0.4.5", "expo-system-ui": "~5.0.9", "expo-web-browser": "~14.2.0", "fs": "^0.0.1-security", "lucide-react-native": "^0.522.0", "moti": "^0.30.0", "nativewind": "^4.1.23", "openai": "^5.6.0", "react": "19.0.0", "react-dom": "19.0.0", "react-native": "0.79.4", "react-native-gesture-handler": "~2.24.0", "react-native-reanimated": "~3.17.4", "react-native-safe-area-context": "^5.4.0", "react-native-screens": "~4.11.1", "react-native-svg": "^15.11.2", "react-native-url-polyfill": "^2.0.0", "react-native-vector-icons": "^10.2.0", "react-native-web": "~0.20.0", "react-native-webview": "13.13.5", "tailwindcss": "^3.4.17", "zustand": "^5.0.5" }, "devDependencies": { "@babel/core": "^7.25.2", "@types/crypto-js": "^4.2.2", "@types/react": "~19.0.10", "eslint": "^9.25.0", "eslint-config-expo": "~9.2.0", "typescript": "~5.8.3" }, "private": true } ``` ### vocera-frontend/app/_layout.tsx ```typescript import { DarkTheme, DefaultTheme, ThemeProvider } from '@react-navigation/native'; import { useFonts } from 'expo-font'; import { Stack } from 'expo-router'; import { StatusBar } from 'expo-status-bar'; import 'react-native-reanimated'; import '../global.css'; import { useColorScheme } from '@/hooks/useColorScheme'; import AppWrapper from '../components/AppWrapper'; export default function RootLayout() { const colorScheme = useColorScheme(); const [loaded] = useFonts({ 'SpaceMono-Regular': require('../assets/fonts/SpaceMono-Regular.ttf'), // Georgia Pro fonts 'GeorgiaPro-CondBlack': require('../assets/fonts/GeorgiaPro-CondBlack.ttf'), 'GeorgiaPro-CondRegular': require('../assets/fonts/GeorgiaPro-CondRegular.ttf'), 'GeorgiaPro-Bold': require('../assets/fonts/GeorgiaPro-Bold.ttf'), 'GeorgiaPro-BoldItalic': require('../assets/fonts/GeorgiaPro-BoldItalic.ttf'), 'GeorgiaPro-Black': require('../assets/fonts/GeorgiaPro-Black.ttf'), // Inter fonts 'Inter-Regular': require('../assets/fonts/Inter_28pt-Regular.ttf'), 'Inter-Medium': require('../assets/fonts/Inter_28pt-Medium.ttf'), 'Inter-SemiBold': require('../assets/fonts/Inter_28pt-SemiBold.ttf'), 'Inter-Bold': require('../assets/fonts/Inter_18pt-Bold.ttf'), 'Inter-Black': require('../assets/fonts/Inter_18pt-Black.ttf'), 'Inter-BoldItalic': require('../assets/fonts/Inter_18pt-BoldItalic.ttf'), 'Inter-Italic': require('../assets/fonts/Inter_18pt-Italic.ttf'), 'Inter-SemiBoldItalic': require('../assets/fonts/Inter_28pt-SemiBoldItalic.ttf'), // Legacy font names for compatibility 'Inter_18pt-Black': require('../assets/fonts/Inter_18pt-Black.ttf'), 'Inter_18pt-Bold': require('../assets/fonts/Inter_18pt-Bold.ttf'), 'Inter_18pt-BoldItalic': require('../assets/fonts/Inter_18pt-BoldItalic.ttf'), 'Inter_18pt-Italic': require('../assets/fonts/Inter_18pt-Italic.ttf'), 'Inter_28pt-BoldItalic': require('../assets/fonts/Inter_28pt-BoldItalic.ttf'), 'Inter_28pt-Medium': require('../assets/fonts/Inter_28pt-Medium.ttf'), 'Inter_28pt-SemiBold': require('../assets/fonts/Inter_28pt-SemiBold.ttf'), 'Inter_28pt-SemiBoldItalic': require('../assets/fonts/Inter_28pt-SemiBoldItalic.ttf'), }); if (!loaded) { // Async font loading only occurs in development. return null; } return ( ); } ``` ### voice-detect/app.py ```python import os import json from flask import Flask, request, jsonify import numpy as np from scipy.spatial.distance import euclidean from itertools import combinations import tempfile from sklearn.preprocessing import StandardScaler import requests from dotenv import load_dotenv from pydub import AudioSegment import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Set up file logging file_handler = logging.FileHandler("voice_detect.log") file_handler.setLevel(logging.INFO) file_formatter = logging.Formatter( "%(asctime)s - %(name)s - %(levelname)s - %(message)s" ) file_handler.setFormatter(file_formatter) logger.addHandler(file_handler) from feature_extractor import ( extract_opensmile_features, create_scaler_from_features, serialize_scaler, deserialize_scaler, ) from database import SupabaseDatabase # from speaker_verification import verify_speaker_from_data # COMMENTED OUT FOR MINIMAL DEPLOYMENT from supabase import create_client, Client load_dotenv() SUPABASE_URL = os.getenv("SUPABASE_URL") SUPABASE_SERVICE_KEY = os.getenv("SUPABASE_SERVICE_KEY") app = Flask(__name__) # Initialize Supabase client supabase: Client = create_client(SUPABASE_URL, SUPABASE_SERVICE_KEY) # Use SupabaseDatabase instead of FileDatabase db = SupabaseDatabase(supabase_client=supabase) # Tunable parameters for scoring thresholds # Note: These may need adjustment after normalization OPEN_SMILE_THRESHOLD_MULTIPLIER = 1.8 # Note: No audio conversion needed - working directly with M4A files def download_audio_from_url(audio_url): """ Downloads an audio file from the supabase bucket URL and saves it to a temporary local file. """ try: # Use generic suffix since we handle multiple audio formats temp_file = tempfile.NamedTemporaryFile( delete=False, suffix=".audio", mode="wb" ) response = requests.get(audio_url, stream=True) response.raise_for_status() for chunk in response.iter_content(chunk_size=8192): if chunk: # filter out keep-alive chunks temp_file.write(chunk) temp_file.close() return temp_file.name except requests.exceptions.RequestException as e: print(f"Error: Failed to download file from URL. {e}") return None def download_multiple_files_from_urls(file_urls): """ Downloads multiple audio files from Supabase storage URLs and saves them to temporary local files. Args: file_urls (list): List of Supabase storage URLs to download Returns: list: List of local temporary file paths, or empty list if failed """ if not file_urls or not isinstance(file_urls, list): logger.error("file_urls must be a non-empty list") return [] temp_files = [] failed_downloads = [] try: for i, url in enumerate(file_urls): try: # Download to a temporary file with M4A extension for SoX compatibility temp_file = tempfile.NamedTemporaryFile( delete=False, suffix=".m4a", mode="wb" ) response = requests.get(url, stream=True) response.raise_for_status() bytes_written = 0 for chunk in response.iter_content(chunk_size=8192): if chunk: # filter out keep-alive chunks temp_file.write(chunk) bytes_written += len(chunk) temp_file.close() logger.info( f"Successfully downloaded file {i+1}/{len(file_urls)}, {bytes_written} bytes" ) # Keep files as M4A - no conversion needed logger.info(f"File {i+1} - keeping as M4A format") temp_files.append(temp_file.name) logger.info(f"File {i+1} downloaded and ready as M4A") except requests.exceptions.RequestException as e: logger.error(f"Error downloading file {i+1} from URL {url}: {e}") failed_downloads.append(url) # Clean up the failed temp file if it was created if "temp_file" in locals() and hasattr(temp_file, "name"): try: os.remove(temp_file.name) except: pass if failed_downloads: logger.warning( f"Failed to download {len(failed_downloads)} files: {failed_downloads}" ) if not temp_files: logger.error("No files were successfully downloaded") return [] logger.info( f"Successfully downloaded {len(temp_files)} out of {len(file_urls)} files" ) return temp_files except Exception as e: logger.error(f"Error in download_multiple_files_from_urls: {e}") # Clean up any successfully downloaded files in case of error for temp_file_path in temp_files: try: if os.path.exists(temp_file_path): os.remove(temp_file_path) except: pass return [] def download_files_from_vox_key_folder(vox_key_id, expected_count=10): """ Downloads all audio files from a specific vox_key folder in Supabase storage. Args: vox_key_id (str): The vox_key identifier (e.g., "vox_key_7") expected_count (int): Expected number of files (default 10 for calibration) Returns: list: List of local temporary file paths, or empty list if failed """ try: # List files in the vox_key folder files = supabase.storage.from_("vocera-audiostore").list(vox_key_id) if not files: logger.error(f"No files found in folder {vox_key_id}") return [] # Filter for .wav files and sort them wav_files = [f for f in files if f["name"].endswith(".wav")] [truncated — 17722 more characters] ``` ### vocera-frontend/app/(tabs)/_layout.tsx ```typescript import { Tabs } from 'expo-router'; import React, { useEffect, useState } from 'react'; import { Platform, SafeAreaView } from 'react-native'; import { Ionicons } from '@expo/vector-icons'; import { supabase } from '../../lib/supabase'; import { Session } from '@supabase/supabase-js'; import { LinearGradient } from 'expo-linear-gradient'; import Auth from '../../components/Auth'; export default function TabLayout() { const [session, setSession] = useState(null); useEffect(() => { supabase.auth.getSession().then(({ data: { session } }) => { setSession(session); }); const { data: { subscription }, } = supabase.auth.onAuthStateChange((_event, session) => { setSession(session); }); return () => subscription.unsubscribe(); }, []); // Show auth screen if not authenticated if (!session) { return ( ); } return ( ( ), }} /> ( ), }} /> ( ), }} /> ); } ``` ### vocera-frontend/app/(tabs)/index.tsx ```typescript import React, { useState, useEffect, useRef } from 'react'; import { View, Text, StyleSheet, Alert, SafeAreaView, TouchableOpacity, Dimensions, TextInput, ScrollView } from 'react-native'; import { LinearGradient } from 'expo-linear-gradient'; import Animated, { useSharedValue, useAnimatedStyle, withSpring, withTiming, withSequence, withRepeat, withDelay, Easing, runOnJS, interpolate, FadeIn, cancelAnimation } from 'react-native-reanimated'; import Svg, { Path } from 'react-native-svg'; import { Mic, Lock, X } from 'lucide-react-native'; import { VoxButton } from '../../components/VoxButton'; import { NameInput } from '../../components/NameInput'; import { InstructionsPanel } from '../../components/InstructionsPanel'; import { TranscriptView } from '../../components/TranscriptView'; import { useVoceraStore } from '../../store/voceraStore'; import { supabaseService } from '../../services/supabaseService'; import { openaiService, herokuService } from '../../services/openaiService'; import { audioUtils } from '../../services/audioUtils'; import { useAudioRecorder, AudioModule, RecordingPresets } from 'expo-audio'; type VerificationStep = 'initial' | 'name-entry' | 'ready' | 'recording' | 'processing' | 'result'; // Wave Animation Component const WaveAnimation = ({ isActive }: { isActive: boolean }) => { const waveValue = useSharedValue(0); useEffect(() => { if (isActive) { // Reset to 0 before starting new animation waveValue.value = 0; waveValue.value = withRepeat( withTiming(30, { duration: 1200, easing: Easing.linear }), -1, false ); } else { // Cancel any ongoing animation cancelAnimation(waveValue); waveValue.value = withTiming(0, { duration: 300 }); } // Cleanup function return () => { cancelAnimation(waveValue); }; }, [isActive]); const animatedStyle = useAnimatedStyle(() => ({ transform: [{ translateX: -waveValue.value }], })); return ( ); }; export default function HomeScreen() { const [currentStep, setCurrentStep] = useState('initial'); const [callerName, setCallerName] = useState(''); const [nameError, setNameError] = useState(''); const [verificationResult, setVerificationResult] = useState<{ match: boolean; confidence: number; transcript: string; } | null>(null); const [isAnimated, setIsAnimated] = useState(false); const [showNameInput, setShowNameInput] = useState(false); const [nameSubmitted, setNameSubmitted] = useState(false); const [recordingTimeLeft, setRecordingTimeLeft] = useState(5); const [showRecordingText, setShowRecordingText] = useState(false); // Audio recording const audioRecorder = useAudioRecorder({ extension: '.wav', sampleRate: 44100, numberOfChannels: 1, bitRate: 128000, linearPCMBitDepth: 16, linearPCMIsBigEndian: false, linearPCMIsFloat: false, }); const recordingTimerRef = useRef | null>(null); const currentTargetUserIdRef = useRef(null); // Animation values const logoTranslateY = useSharedValue(0); const logoScale = useSharedValue(1); const fadeOutAnimation = useSharedValue(1); const middleContentOpacity = useSharedValue(1); const nameInputOpacity = useSharedValue(0); const instructionsOpacity = useSharedValue(0); const xButtonOpacity = useSharedValue(0); const xButtonScale = useSharedValue(0.8); const recordingTextOpacity = useSharedValue(0); const transcriptOpacity = useSharedValue(0); const resultOpacity = useSharedValue(0); // Entrance animation values const entranceProgress = useSharedValue(0); const titleOpacity = useSharedValue(0); const subtitleOpacity = useSharedValue(0); const logoOpacity = useSharedValue(0); const micTextOpacity = useSharedValue(0); const trustOpacity = useSharedValue(0); // Ripple animation values const outerRippleScale = useSharedValue(0.8); const middleRippleScale = useSharedValue(0.8); const innerRippleScale = useSharedValue(0.8); const outerRippleOpacity = useSharedValue(0); const middleRippleOpacity = useSharedValue(0); const innerRippleOpacity = useSharedValue(0); const { user, currentTranscript, setCurrentTranscript, addSavedCall, setTargetUserId, targetUserId } = useVoceraStore(); // Entrance animations on mount useEffect(() => { // Title fades in first titleOpacity.value = withTiming(1, { duration: 800, easing: Easing.out(Easing.quad) }); // Subtitle follows subtitleOpacity.value = withDelay(200, withTiming(1, { duration: 800, easing: Easing.out(Easing.quad) })); // Logo scales up and fades in logoOpacity.value = withDelay(400, withTiming(1, { duration: 1000, easing: Easing.out(Easing.cubic) })); // Ripples animate in sequence outerRippleScale.value = withDelay(600, withSpring(1, { damping: 15, stiffness: 100 })); outerRippleOpacity.value = withDelay(600, withTiming(1, { duration: 600 })); middleRippleScale.value = withDelay(700, withSpring(1, { damping: 15, stiffness: 100 })); middleRippleOpacity.value = withDelay(700, withTiming(0.7, { duration: 600 })); innerRippleScale.value = withDelay(800, withSpring(1, { damping: 15, stiffness: 100 })); innerRippleOpacity.value = withDelay(800, withTiming(1, { duration [truncated — 30908 more characters] ``` ### ci_test.sh ```shell #!/bin/bash # ci_test.sh - Automated testing script for CI/CD integration set -e # Exit on any error echo "🎙️ VOCERA VOICE DETECTION - CI/CD TEST RUNNER" echo "================================================" # Check if we're in the right directory if [ ! -d "voice-detect" ] || [ ! -f "run_full_test.py" ]; then echo "❌ Error: Please run this script from the project root directory" exit 1 fi # Install dependencies echo "📦 Installing dependencies..." cd voice-detect pip install -r requirements.txt cd .. # Start server in background echo "🚀 Starting voice detection server..." cd voice-detect python app.py & SERVER_PID=$! cd .. # Function to cleanup on exit cleanup() { echo "🧹 Cleaning up..." if [ ! -z "$SERVER_PID" ]; then kill $SERVER_PID 2>/dev/null || true wait $SERVER_PID 2>/dev/null || true fi } # Set trap to cleanup on script exit trap cleanup EXIT # Wait for server to start echo "⏳ Waiting for server to start..." sleep 10 # Check if server is responding echo "🔍 Checking server status..." max_retries=10 retry_count=0 while [ $retry_count -lt $max_retries ]; do if curl -s http://localhost:5001/get_profile/test > /dev/null 2>&1; then echo "✅ Server is responding" break fi retry_count=$((retry_count + 1)) echo "⏳ Server not ready, retrying... ($retry_count/$max_retries)" sleep 2 done if [ $retry_count -eq $max_retries ]; then echo "❌ Server failed to start properly" exit 1 fi # Run full test suite echo "🧪 Running full test suite..." python run_full_test.py # Capture exit code TEST_RESULT=$? if [ $TEST_RESULT -eq 0 ]; then echo "🎉 All tests passed successfully!" else echo "❌ Some tests failed (exit code: $TEST_RESULT)" fi exit $TEST_RESULT ``` ### test_calibration.py ```python #!/usr/bin/env python3 """ Calibration script for voice detection system. Automatically calibrates user profiles using their calibration audio files. """ import os import sys import requests import json from pathlib import Path def calibrate_user(user_id, calib_dir, base_url="http://localhost:5001"): """Calibrate a user profile using their calibration files.""" calib_path = Path(calib_dir) if not calib_path.exists(): print(f"❌ Calibration directory not found: {calib_dir}") return False # Get all .wav files in calibration directory audio_files = list(calib_path.glob("*.wav")) if len(audio_files) != 10: print( f"❌ Expected 10 calibration files, found {len(audio_files)} in {calib_dir}" ) return False # Sort files to ensure consistent ordering audio_files.sort() print(f"🎯 Calibrating user '{user_id}' with {len(audio_files)} files...") # Prepare files for upload files = [] file_handles = [] try: for audio_file in audio_files: file_handle = open(audio_file, "rb") file_handles.append(file_handle) files.append(("files", (audio_file.name, file_handle, "audio/wav"))) # Make calibration request data = {"user_id": user_id} response = requests.post(f"{base_url}/calibrate", data=data, files=files) if response.status_code == 200: result = response.json() print(f"✅ Calibration successful for {user_id}") print(f" Status: {result.get('status')}") print(f" Files processed: {result.get('files_received')}") return True else: print(f"❌ Calibration failed for {user_id}") print(f" Status code: {response.status_code}") print(f" Response: {response.text}") return False except Exception as e: print(f"❌ Error calibrating {user_id}: {str(e)}") return False finally: # Close all file handles for file_handle in file_handles: file_handle.close() def main(): if len(sys.argv) < 2: print("Usage: python test_calibration.py []") print(" python test_calibration.py all") print("\nExamples:") print(" python test_calibration.py laerdon") print(" python test_calibration.py srikar") print(" python test_calibration.py all") sys.exit(1) user_id = sys.argv[1] if user_id == "all": # Calibrate all users users = [ ("laerdon", "laerdonsampledata/calib"), ("srikar", "srikarsampledata/calib"), ("gold", "goldsampledata/calib"), ] success_count = 0 for uid, calib_dir in users: if calibrate_user(uid, calib_dir): success_count += 1 print() # Empty line between users print( f"🏁 Calibration completed: {success_count}/{len(users)} users successful" ) else: # Calibrate specific user if len(sys.argv) >= 3: calib_dir = sys.argv[2] else: calib_dir = f"{user_id}sampledata/calib" success = calibrate_user(user_id, calib_dir) sys.exit(0 if success else 1) if __name__ == "__main__": main() ``` [69 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]