Project Info
Inspiration
Gambit addresses a critical gap in the $300 billion global video game market by introducing a voice-enabled AI to enhance online game accessibility. Our solution serves the massive, yet underserved, population of gamers with disabilities.
What it does
Gambit listens to a user's voice and then executes the in-game actions, such as moving a character or initiating a sequence of moves, all in real time. By converting voice commands into complex inputs and information, Gambit is positioned not only as a crucial social good but also as a powerful economic driver that unlocks a significant segment of the market.
How we built it
Together, these components form a seamless voice-to-action system adaptable to nearly any game or platform. 🧠 Google Gemini – Bridges the gap between computer vision and spoken commands. Converts spoken instructions into intelligent, context-aware action sequences. 🗣️ Deepgram – Handles accurate, low-latency speech-to-text conversion, with the ability to conversational cues and pauses. 👁️ Computer Vision – Allows Gambit to “see” and interpret on-screen environments dynamically. 🐍 Python Automation Engine – Executes the translated commands by controlling keyboard and mouse inputs directly.
Challenges we ran into
We ran into challenges with Omniparser, the technology we used for image segmentation. Downloading the process was smooth on MacOS, but took multiple hours for Windows.
Accomplishments we're proud of
We’re most proud of Gambit’s adaptability. It can interpret and respond to any visual interface without needing game-specific integrations — a major step toward universal gaming accessibility.
What we learned
We explored new domains across AI vision, AI speech to text technologies, and real-time system control. More importantly, we learned how these technologies can intersect to create meaningful, inclusive innovation that extends beyond entertainment.
What's next
Next, we plan to: Expand Gambit’s compatibility to include a wider range of game genres (e.g., FPS, strategy, racing). Integrate customizable command mappings for different player needs. Partner with game studios and accessibility organizations to scale our impact. Explore mobile and VR adaptations, making hands-free play accessible across devices.
Voice-Controlled Board Game Player (Python)
Prereqs
- Python 3.10+
- API Keys as env vars
DEEPGRAM_API_KEYANTHROPIC_API_KEY
Install
pip install -r requirements.txt
Run the backend server (API + Web UI)
python -m uvicorn server:app --host 127.0.0.1 --port 8000
- Open the web UI at: http://127.0.0.1:8000/
- Endpoints used by the UI:
- GET
/api/connect4/state - POST
/api/connect4/reset - POST
/api/connect4/move/{column}(1–7) - POST
/api/llm/parseand/api/llm/command(for LLM parsing/apply)
- GET
Env vars (recommended):
ANTHROPIC_API_KEYfor LLM parsing on the server
Run the voice controller (optional)
python main.py
- If
DEEPGRAM_API_KEYis set, the app starts a background speech-to-text worker (Deepgram listen.v2) and enqueues transcripts. - Each transcript is sent to the backend via
POST /api/llm/command, so the web UI updates automatically. - If the server isn’t reachable, it falls back to local parsing and engine execution (the web UI will not reflect those local-only changes).
Env vars (optional/required):
DEEPGRAM_API_KEYfor live voice input.ANTHROPIC_API_KEYfor LLM parsing (used by both server and local fallback).
Notes
- Game state is stored in-memory on the backend. Restarting the server resets the board.
- The frontend periodically refreshes state so speech-driven moves appear without clicking.
Analysis
View
Metric
- 27
- 4
- 1
- 1
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- AnthropicIn code
- FastAPIIn code
- HTMLIn code
- JavaScriptIn code
- PythonIn code
- Google GeminiClaimed
- ReactClaimed
5 of 7 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
67 KB
Source files
18
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
nathaniellaurent/Gambit
22 files · 2.8 MB · @ 4d4c025
Structure
Interface
2 files · 9%Screens, components and styles rendered to the user.
Application logic
15 files · 68%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python87%
- JavaScript6%
- HTML4%
- Markdown2%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
requirements.txt
pypi · 11- anthropic
- deepgram-sdk
- fastapi
- google-generativeai
- numpy
- pillow
- pyautogui
- python-chess
- python-dotenv
- sounddevice
- uvicorn[standard]
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.