Project Info

Ventiuno

Inspiration

We went on a trip to Mexico where we got to visit a casino for the first time together. Since we were new to the game, we all played with our phones out—checking the optimal way to play, analyzing our hands, and even trying to count cards. When we later saw the Meta Ray-Bans, we immediately thought about the possibilities of using the camera and mic to give you real-time, optimal move suggestions!

What it does

Ventiuno takes in a live feed from the Meta Ray-Bans and detects when and where cards are present. Using the recognized cards and the information about previously seen ones, it calculates the best move for you in real time. The recommendation is then sent directly through the headset’s built-in speakers.

How we built it

We designed a custom computer vision pipeline that segments the image based on detected cards, then crops, flattens, and enhances each one before passing it into a small vision-language model (VLM) for fast card recognition. Once the cards are identified, we use K-Means clustering to assign each card to the correct player’s hand. The system keeps track of the overall game state and combines previous cards with standard blackjack heuristics to determine the optimal move at any given point.

Challenges we ran into

The pipeline—from video feed to audio suggestion—had to be both very fast and very accurate. Initially, we tried to run everything through a VLM for convenience, but it wasn’t fast enough. We switched to a modified YOLO model for card segmentation, which greatly improved speed. Another major issue was perspective: existing poker/card recognition models perform well from a top-down view but struggle with natural angles. To fix this, we implemented a “flattening” step—grabbing the card’s corner points, unwarping it into its original rectangle, and then passing it into a lightweight VLM for identification. We also used multi-threading in Python to parallelize recognition and drastically reduce latency.

Accomplishments we're proud of

We’re proud that Ventiuno works end-to-end—from real-time card recognition all the way to audio feedback. It’s been tested extensively and even accounts for edge cases like splits, doubles, and other game situations. The final result is a robust, fully functional solution that’s ready to be used in any blackjack setting.

What we learned

We learned a lot about computer vision and how combining classical CV techniques with modern AI models can massively improve both speed and accuracy. At first, we assumed we could just throw everything into a VLM, but incorporating preprocessing steps like perspective correction made a huge difference. We also learned how powerful multi-threading can be for real-time inference—it helped us process video frames, audio output, and model predictions simultaneously with minimal delay.

What's next

We’ve already trained and tested our CV + game analysis + audio system on recorded footage from Meta Ray-Bans. However, we ran into some issues setting up a live stream from the glasses. We’re currently waiting for the official Meta API (coming later this year) to fully integrate real-time video input. Once the API is released, we plan to port everything directly onto the glasses and extend the system to other games—starting with Heads-Up Poker.

Real-time Computer Vision Blackjack Strategy Advisor

A real-time blackjack strategy advisor that combines computer vision, AI-powered card detection, and Game Theory Optimal (GTO) decision-making to provide optimal play recommendations. Features hands-free operation via AirPods, live camera feed processing, and text-to-speech announcements.

Demo

This system enables hands-free blackjack strategy analysis in real-time. Position your camera over a blackjack table, press your AirPod or spacebar to capture the current hand, and receive instant GTO recommendations via visual display and audio announcement. The system detects cards using computer vision, analyzes the game state, and provides mathematically optimal moves based on proven basic strategy.

Key Features:

Automatic card detection from camera feed using Roboflow object detection and Claude Vision API
Real-time GTO strategy recommendations based on dealer and player cards
Hands-free capture using AirPods media controls
Text-to-speech move announcements for eyes-free operation
Live web dashboard for real-time results visualization
Advanced card counting with Hi-Lo and Hi-Opt II systems
Kelly Criterion-based betting recommendations

Screenshots

The system includes three main interfaces:

Live Dashboard (http://localhost:8080/live) - Real-time display of detected cards and optimal moves with color-coded recommendations
AirPod Control Page (http://localhost:8080/control) - Safari-based interface for media key detection enabling hands-free capture
Camera Feed Window - OpenCV window showing live camera feed with capture controls

Example outputs are stored in the card_captures/ directory after each capture session.

Technical Architecture

System Overview

The application is built using a modular architecture with the following components:

Backend (Python/Flask)

Flask REST API server for strategy calculations and live hand updates
Real-time card detection pipeline using computer vision
GTO strategy engine implementing basic strategy matrices
Card counting module with multiple counting systems
Text-to-speech integration using macOS say command

Computer Vision Pipeline

Roboflow Inference SDK for card object detection
Anthropic Claude Vision API for card classification
OpenCV for camera capture and image preprocessing
Perspective correction and image enhancement algorithms
K-means clustering for spatial card grouping (dealer vs. player hands)

Frontend (HTML/JavaScript)

Real-time dashboard with server-sent events for live updates
Safari-based media key detection for AirPod integration
Responsive card input interface for manual testing
Color-coded move recommendations (Hit/Stand/Double/Split/Surrender)

Integration Layer

Shared trigger mechanism using file-based inter-process communication
Callback-based camera capture system
Thread-safe hand state management

Core Algorithms

Card Detection: Multi-stage pipeline involving object detection, classification, spatial clustering, and suit stripping to extract card ranks.

GTO Strategy: Implements standard blackjack basic strategy using lookup tables for hard totals, soft totals, and pair splitting decisions.

Card Counting: Tracks remaining deck composition, calculates running count and true count, implements Illustrious 18 index plays, and provides betting recommendations using the Kelly Criterion.

System Requirements

Operating System: macOS (required for text-to-speech and AirPod integration)
Python: 3.9 or higher
Camera: Webcam or external camera for card capture
Browser: Safari (for AirPod media key detection), any modern browser for live dashboard
Audio: AirPods or speakers for text-to-speech announcements

Installation

1. Clone the Repository

git clone <repository-url>
cd full-workflow

2. Create Virtual Environment

python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

Core Dependencies:

Flask==3.0.0
Werkzeug==3.0.1
requests==2.31.0
opencv-python>=4.8.0
anthropic>=0.25.0
inference-sdk>=0.9.0
scikit-learn>=1.3.0
numpy>=1.24.0
Pillow>=10.0.0

4. Configure API Keys

Create an api_key.txt file in the project root:

echo "your-anthropic-api-key-here" > api_key.txt

API Key Sources:

Anthropic API key: https://console.anthropic.com/
Roboflow API key is pre-configured in the code (public demo key)

5. Camera Permissions

Grant camera access to Terminal:

System Settings → Privacy & Security → Camera → Enable Terminal

Running the Application

Quick Start (Recommended)

The fastest way to run the complete system with AirPod integration:

./start_airpod_workflow.sh

This automated script will:

Activate the virtual environment
Install/verify all dependencies
Start the Flask server on port 8080
Open the AirPod control page in Safari
Open the live dashboard
Launch the camera capture window
Enable hands-free capture via AirPods

After launching, you must click once on the Safari control page to enable media key detection.

Manual Start (Advanced)

For more control over individual components:

Terminal 1: Start Flask Server

source venv/bin/activate
python app.py

The server will start on http://localhost:8080

Terminal 2: Start Card Detection

source venv/bin/activate
python live_card_detector.py

Terminal 3 (Optional): Open Web Interfaces

open -a Safari "http://localhost:8080/control"  # AirPod control
open "http://localhost:8080/live"                # Live dashboard

Usage

Capture Methods:

AirPod (Hands-free): Press the play/pause button on your right AirPod
Keyboard: Press SPACE or C in the camera window
Web Interface: Click "Test Capture" on the control page

Workflow:

Position camera to view blackjack table (dealer cards at top, player cards at bottom)
Trigger capture when cards are dealt
System automatically detects and classifies cards (2-3 seconds)
GTO recommendation is displayed on live dashboard
Text-to-speech announces the optimal move
Repeat for next hand

Camera Controls:

SPACE or C: Capture current frame and analyze
Q: Quit the application

API Documentation

Core Endpoints

POST `/api/recommend`

Get optimal move recommendation for a blackjack hand.

Request:

{
  "dealer_cards": ["K"],
  "player_cards": ["A", "7"]
}

Response:

{
  "success": true,
  "optimal_move": "H (Hit)",
  "move_type": "hit",
  "player_value": 18,
  "is_soft": true,
  "dealer_upcard": 10,
  "is_pair": false,
  "dealer_cards_count": 1,
  "player_cards_count": 2
}

POST `/api/trigger-capture`

Trigger camera capture programmatically (used by AirPod control page).

Response:

{
  "success": true,
  "message": "Capture triggered"
}

GET `/api/latest-hand`

Retrieve the most recently analyzed hand.

Response:

{
  "dealer_cards": ["K"],
  "player_cards": ["A", "7"],
  "recommendation": {
    "optimal_move": "H (Hit)",
    "player_value": 18,
    "is_soft": true
  },
  "timestamp": "2025-10-26T12:34:56.789"
}

POST `/api/latest-hand`

Update the latest hand (used by card detection pipeline).

Request:

{
  "dealer_cards": ["K"],
  "player_cards": ["A", "7"],
  "recommendation": {
    "optimal_move": "H (Hit)",
    "move_type": "hit"
  }
}

GET `/api/health`

Health check endpoint.

Response:

{
  "status": "healthy",
  "service": "Blackjack GTO Advisor"
}

Project Structure

full-workflow/
├── app.py                      # Flask server and API endpoints
├── blackjack_gto.py           # GTO strategy implementation
├── card_counter.py            # Card counting and betting advisor
├── card_detector.py           # Computer vision card detection
├── capture_image.py           # Camera capture with callback support
├── live_card_detector.py      # Real-time detection pipeline
├── tts_announcer.py           # Text-to-speech announcements
├── shared_trigger.py          # IPC trigger mechanism
├── pipeline_integration.py    # Integration testing utilities
├── start_airpod_workflow.sh   # Automated launch script
├── requirements.txt           # Python dependencies
├── api_key.txt               # Anthropic API key (create this)
├── templates/
│   ├── index.html            # Manual card input UI
│   ├── live.html             # Live dashboard
│   ├── control.html          # AirPod control page
│   └── airpod_capture.html   # Alternative capture interface
├── static/
│   └── silent.mp3            # Silent audio for media session
├── card_captures/            # Saved card capture images
└── tests/
    ├── test_blackjack.py     # Strategy algorithm tests
    ├── test_api.py           # API endpoint tests
    └── test_webapp.py        # Web interface tests

Card Format and Strategy

Card Values

Number cards (2-10): Face value
Face cards (J, Q, K): 10 points
Ace (A): 1 or 11 (automatically optimized)

Optimal Moves

H (Hit): Take another card
S (Stand): Keep current hand
D (Double): Double bet and take exactly one more card
P (Split): Split pair into two separate hands
R (Surrender): Forfeit half bet and fold

Strategy Implementation

The system implements standard basic strategy:

Hard Totals:

17+: Always stand
13-16: Stand vs dealer 2-6, hit vs 7+
12: Stand vs 4-6, hit otherwise
11: Always double
10: Double vs dealer 2-9
9: Double vs dealer 3-6
8 or less: Always hit

Soft Totals:

Soft 19+: Stand
Soft 18: Stand vs 2-8, hit vs 9+
Soft 17-: More aggressive (double/hit based on dealer)

Pairs:

Always split: Aces and 8s
Never split: 10s and 5s
Conditional splits based on dealer upcard

Advanced Features

Card Counting

The CardCounter class tracks deck composition and provides:

Running Count: Hi-Lo or Hi-Opt II counting system
True Count: Running count normalized by remaining decks
Illustrious 18: Count-dependent strategy deviations
Kelly Betting: Optimal bet sizing based on advantage
Insurance Decisions: Mathematical insurance recommendations

Usage:

from card_counter import CardCounter

counter = CardCounter(num_decks=2, counting_system='hi-lo')
counter.remove_cards(['K', 'Q', '5', '6'])  # Cards seen
true_count = counter.get_true_count()
bet_size = counter.recommend_bet(min_bet=10, bankroll=1000)

Programmatic Usage

from blackjack_gto import get_optimal_move
from card_detector import detect_cards

# Detect cards from image
dealer_cards, player_cards = detect_cards("poker_table.jpg")

# Strip suits (keep only ranks)
dealer_ranks = [card[:-1] for card in dealer_cards]
player_ranks = [card[:-1] for card in player_cards[0]]

# Get optimal move
move = get_optimal_move(dealer_ranks, player_ranks)
print(f"Optimal move: {move}")

Troubleshooting

Camera Not Opening

# Check camera permissions
System Settings → Privacy & Security → Camera → Enable Terminal

# Test camera detection
python -c "import cv2; print(cv2.VideoCapture(0).isOpened())"

AirPod Trigger Not Working

Ensure Safari control page is open and has been clicked (to enable media session)
Close other media applications (Music, Spotify, etc.)
Test with the "Test Capture" button on the control page first
Check that Flask server is running on port 8080

Card Detection Failures

Improve lighting: Bright, even lighting works best
Camera angle: Position camera directly above table
Card spacing: Ensure cards are not heavily overlapping
Card visibility: All cards should be fully visible
Dealer position: Dealer cards should be in top row of camera view

Server Won't Start

# Check if port 8080 is in use
lsof -i :8080

# Kill existing process if needed
kill -9 $(lsof -t -i:8080)

# Restart server
python app.py

Missing Dependencies

# Reinstall all dependencies
source venv/bin/activate
pip install --upgrade -r requirements.txt

Performance Considerations

Card Detection: 2-3 seconds per capture (includes API calls to Roboflow and Claude)
GTO Calculation: <10ms (lookup table-based)
Total Latency: ~2-3 seconds from capture to recommendation
Debounce Time: 1 second between captures to prevent duplicate triggers

Testing

Run All Tests

source venv/bin/activate

# Test GTO strategy
python test_blackjack.py

# Test API endpoints
python test_api.py

# Test web interface
python test_webapp.py

Test Individual Components

# Test card detection on a sample image
python card_detector.py

# Test GTO calculations
python blackjack_gto.py

# Test integration pipeline
python pipeline_integration.py

Future Work

Enhanced Computer Vision

Support for multiple camera angles and perspectives
Real-time video processing without manual capture triggers
Automatic table boundary detection and tracking
Support for different card deck designs and variations

Advanced Strategy

Implement full Illustrious 18 and Fab 4 indices in real-time play
Add support for different rule variations (H17 vs S17, DAS, etc.)
Multi-hand strategy optimization for simultaneous hands
Bankroll management integration with session tracking

Hardware Integration

Support for smart glasses (Ray-Ban Meta, Mentra, etc.) for truly hands-free operation
Mobile device support (iOS/Android apps)
Bluetooth peripheral support for discrete triggers
Integration with casino player card systems

User Experience

Historical hand tracking and performance analytics
Practice mode with simulated hands and feedback
Customizable strategy variations and house rules
Multi-user support with individual profiles and preferences

Machine Learning

Train custom card detection models for improved accuracy
Adaptive strategy based on observed dealer patterns
Anomaly detection for unusual plays or dealer behaviors
Confidence scoring for detection quality assessment

Security and Privacy

Local-only processing option without external API calls
Encrypted storage of hand history and betting data
Secure API key management with environment variables
HTTPS support for web interfaces

License

This project is for educational and research purposes only. Using card counting or strategy advisors in casinos may violate their terms of service. Always check local laws and casino policies before using any gambling assistance tools.

Acknowledgments

Anthropic Claude Vision API for card classification
Roboflow for object detection infrastructure
OpenCV for computer vision capabilities
Flask for web framework
The blackjack basic strategy research community

Analysis

Compare with all teams

View

Metric

tbagri
6

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

FlaskIn code
HTMLIn code
PythonIn code
ReactClaimed

3 of 4 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

248 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Ventiuno

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments we're proud of

What we learned

What's next

Real-time Computer Vision Blackjack Strategy Advisor

Demo

Screenshots

Technical Architecture

System Overview

Core Algorithms

System Requirements

Installation

1. Clone the Repository

2. Create Virtual Environment

3. Install Dependencies

4. Configure API Keys

5. Camera Permissions

Running the Application

Quick Start (Recommended)

Manual Start (Advanced)

Terminal 1: Start Flask Server

Terminal 2: Start Card Detection

Terminal 3 (Optional): Open Web Interfaces

Usage

API Documentation

Core Endpoints

POST /api/recommend

POST /api/trigger-capture

GET /api/latest-hand

POST /api/latest-hand

GET /api/health

Project Structure

Card Format and Strategy

Card Values

Optimal Moves

Strategy Implementation

Advanced Features

Card Counting

Programmatic Usage

Troubleshooting

Camera Not Opening

AirPod Trigger Not Working

Card Detection Failures

Server Won't Start

Missing Dependencies

Performance Considerations

Testing

Run All Tests

Test Individual Components

Future Work

Enhanced Computer Vision

Advanced Strategy

Hardware Integration

User Experience

Machine Learning

Security and Privacy

License

Acknowledgments

Technology

AI coding agents

Codebase size

Structure

Interface

Application logic

Languages

Dependencies

POST `/api/recommend`

POST `/api/trigger-capture`

GET `/api/latest-hand`

POST `/api/latest-hand`

GET `/api/health`