Project Info

Vocera

This project did not submit a demo video on Devpost.

Inspiration

With the rise of generative AI, voice deepfakes have become nearly indistinguishable from real speech—leading to real-world harm. A close friend’s mother was tricked into wiring money to someone impersonating her son. Incidents like this underscore the urgent need for trustworthy voice authentication during sensitive calls. Moreover, recent voice models like ElevenLabs' v3 have become remarkably close to actual speaking patterns, overcoming the Uncanny Valley, while few algorithms exist to identify these deepfaked voices.

What it does

Vocera (vo-cher-uh) is a mobile app that verifies a caller's identity using their unique vocal signature—designed specifically for high-risk phone calls, such as when a friend or family member asks for money, gift cards, or urgent help like bail. To confirm their identity, the caller must speak a secret passphrase known as a Vox Key—a private phrase that's grammatically nonsensical with a contrasting tone, like "The grieving dogs are raining birds," enthusiastically. The unusual structure, and the distinct way each person says it, makes it difficult for deepfake models to mimic. Vocera then authenticates the caller through a three-step process: Passphrase check – Is the correct Vox Key textual phrase being spoken? Speaker verification – Does the voice resemble the registered user's voice? Deepfake detection – Are there signs the voice was AI-generated? In the interest of maximum security, this Vox Key captures your vocal signature—tone, inflection, and other features unique to you. After each use, the Vox Key must be replaced by recording ten ~5 second voice clips to capture low-level vocal traits such as shimmer, inflection, and pause distribution. If the verification passes, you know you’re speaking with the real person—not a cloned voice.

How we built it

Vocera is built with React Native and Expo for cross-platform deployment, styled with NativeWind, and powered by a Zustand context store for state management. We use Supabase for user auth and data storage (Postgres databases + buckets), and Heroku for hosting our models. Each user’s Vox Key is generated by Claude and recorded 10 times to capture unique vocal traits. We extract embeddings using OpenSMILE and store them as part of each user’s voice profile. Our three-stage verification pipeline includes: OpenAI Whisper for passphrase transcription SpeechBrain for speaker verification OpenSMILE + sigmoid scoring to detect nuanced deepfake artifacts To improve robustness, we generated adversarial examples using ElevenLabs voice cloning tools, allowing us to fine-tune the system with both authentic and deepfake data.

Challenges we ran into

There were no pretrained or unified solutions capable of countering cutting-edge generative voice models, so we had to engineer our own from scratch. This included designing evaluation procedures, creating datasets, and testing multiple models to build a security pipeline that even state-of-the-art deepfakes—like ElevenLabs v3—couldn’t fool. While we had prior experience with React, we were new to React Native and mobile development. Finding and learning the ins and outs of suitable audio recording libraries was challenging. Animation tuning and mobile-specific bugs consumed much of our limited 24-hour window. We also ran into delays when FFmpeg, our media conversion tool, corrupted audio files into formats incompatible with our models, stalling integration and testing throughout the hackathon.

Accomplishments we're proud of

We’re proud of building a fully functional mobile app in just 24 hours, despite being new to React Native. We developed a deepfake-resistant voice authentication pipeline that performed better than human judgment in many cases. We successfully integrated OpenAI Whisper, Claude, SpeechBrain, and OpenSMILE into a multi-stage verification system. Our UI is clean, intuitive, and animation-enhanced, offering a user-friendly experience without compromising security. Additionally, we generated synthetic deepfake audio using ElevenLabs to strengthen our model against real-world threats.

What we learned

We learned how to extract and compare voice embeddings using OpenSMILE to detect deepfakes, and how to integrate ML tools like Whisper and SpeechBrain into a real-time mobile app. We gained hands-on experience with React Native, mobile-specific debugging, and building secure, user-friendly interfaces under time pressure.

What's next

We aim to optimize real-time performance, expand to on-device processing for privacy, and position Vocera as a new standard for voice authentication in banking, identity verification, and sensitive communications.

Vocera - Voice Authentication + DeepFake Detection System

A sophisticated voice verification platform combining React Native mobile app, FastAPI backend, and advanced machine learning for secure biometric authentication.

🎯 Overview

Vocera is a comprehensive voice authentication system that uses multiple AI models to verify user identity through voice analysis. In a world of deepfakes that are difficult for the human eyes and ears to discern, Vocera provides you with a way to verify the world around you. Specifically, it helps you prevent deepfake calls from scammers, who pose as family & friends asking for money. The platform employs a dual-verification approach using both traditional signal processing (openSMILE) and modern deep learning (SpeechBrain ECAPA-VOXCELEB) for robust speaker verification.

✨ Features

🔊 Advanced Voice Authentication

Dual Verification System: Combines openSMILE feature extraction with SpeechBrain deep learning
Anti-Deepfake Protection: Sophisticated algorithms to detect synthetic voice generation
Confidence Scoring: Sigmoid-based confidence calculation with tunable thresholds
Feature Normalization: StandardScaler preprocessing for consistent analysis

📱 Cross-Platform Mobile App

Universal Support: iOS, Android, and Web deployment
Real-time Recording: High-quality voice capture with waveform visualization
Auth Integration: Supabase Email Sign-In authentication
Cloud Sync: Supabase backend integration for data persistence

🤖 Machine Learning Pipeline

openSMILE Feature Extraction: 88-dimensional eGeMAPSv02 feature vectors
SpeechBrain ECAPA-VOXCELEB: State-of-the-art speaker verification model
Textual Verification: OpenAI Whisper transcription with GPT-4 semantic analysis
Euclidean Distance Analysis: Normalized distance calculations for authenticity scoring

🛠️ Tech Stack

Frontend (vocera-frontend) + Backend

Framework: React Native & Expo
Styling: TailwindCSS with NativeWind
Audio: Expo Audio for recording and playback
State Management: Zustand
Backend: Supabase client integration, Supabase Buckets Storage
AI Integration: OpenAI & Anthropic APIs

ML Models (voice-detect)

Server: Flask with Python 3.9
ML Libraries:
- openSMILE (feature extraction)
- SpeechBrain (speaker verification)
- scikit-learn (StandardScaler normalization)
- NumPy/SciPy (numerical computation)
Database: Supabase (PostgreSQL-based)
AI Services: OpenAI Whisper, GPT-4
Deployment: Docker containerization ready

🚀 Quick Start

Prerequisites

Node.js 18+
Python 3.9+
Expo CLI
iOS Simulator or Android emulator

Installation

Clone Repository

git clone <repository-url>
cd vocera

Install Dependencies

# Install all dependencies
npm run install:all

# Or install individually
cd vocera-frontend && npm install
cd ../voice-detect && pip install -r requirements.txt
cd ../api && pip install -r requirements.txt

Environment Setup

Create vocera-frontend/.env:

EXPO_PUBLIC_SUPABASE_URL=your_supabase_url
EXPO_PUBLIC_SUPABASE_ANON_KEY=your_supabase_key
EXPO_PUBLIC_OPENAI_API_KEY=your_openai_key
EXPO_PUBLIC_ANTHROPIC_API_KEY=your_anthropic_key

Create voice-detect/.env:

SUPABASE_URL=your_supabase_url
SUPABASE_SERVICE_KEY=your_supabase_service_key
OPENAI_API_KEY=your_openai_key

Development

Start Voice Detection Server:

cd voice-detect
python app.py
# Server runs on http://localhost:5001

Start Frontend:

cd vocera-frontend
npm run start
# Then choose your platform:
# - Press 'i' for iOS simulator
# - Press 'a' for Android emulator  
# - Press 'w' for web browser


## 🔬 Voice Authentication Process

### 1. Calibration Phase
```bash
# Calibrate user profile with 10 voice samples
curl -X POST http://localhost:5001/calibrate \
  -F "user_id=username" \
  -F "files=@sample1.wav" \
  -F "files=@sample2.wav" \
  # ... (all 10 calibration files)

2. Verification Phase

# Verify voice sample against user profile
curl -X POST http://localhost:5001/verify \
  -F "user_id=username" \
  -F "files=@test_voice.wav"

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CIn code
CSSIn code
FlaskIn code
JavaScriptIn code
KotlinIn code
OpenAIIn code
PythonIn code
ReactIn code
SupabaseIn code
SwiftIn code
Tailwind CSSIn code
TypeScriptIn code

13 of 13 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

268 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.