Eloception

MVP web app for near-real-time spoken feedback on pronunciation and enunciation. Uses streaming ASR (Deepgram), AWS Polly TTS, and an LLM (OpenAI) to generate concise coaching tips.

Repo structure (monorepo)

• /apps/web – Next.js (App Router) + TypeScript frontend
• /apps/server – Flask + Flask-SocketIO backend (Deepgram, OpenAI, AWS Polly)
• /packages/shared – Shared TypeScript types (optional for MVP)

Prerequisites

• Node.js 18+ and npm (or pnpm)
• Python 3.11+
• API keys: Deepgram, OpenAI; AWS credentials for Polly TTS

Setup

1. Backend (Flask server)

cd apps/server
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env and set:
#   DEEPGRAM_API_KEY=...
#   OPENAI_API_KEY=...
#   AWS_ACCESS_KEY_ID=... / AWS_SECRET_ACCESS_KEY=... / AWS_REGION=...
#   AWS_POLLY_VOICE_ID=...   # optional; default is Joanna

Important for Flask-SocketIO: Run the server with eventlet so WebSockets work:

# From apps/server with venv activated:
python app.py
# Server listens on port 5001 by default (set PORT=5002 etc. if needed). On macOS, 5000 is often used by AirPlay.

The app is written for async_mode="eventlet", so ensure eventlet is installed (pip install eventlet is in requirements.txt). If you use flask run, it may use the development server; for production or reliable WebSocket handling, run with the socketio.run(app, ...) pattern above.

2. Frontend (Next.js)

# From repo root
npm install
cp apps/web/.env.local.example apps/web/.env.local
# Edit apps/web/.env.local and set:
#   NEXT_PUBLIC_WS_URL=http://localhost:5001
npm run dev:web

3. Run both (from repo root)

npm install
npm run dev

This runs the backend (Flask on port 5001) and the web app (Next.js on port 3000) concurrently. Ensure apps/server/.env is set as above.

Env vars

Var	Where	Description
DEEPGRAM_API_KEY	server	Deepgram API key for live transcription
OPENAI_API_KEY	server	OpenAI API key for coaching tips
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION	server	AWS credentials for Polly TTS
AWS_POLLY_VOICE_ID	server	Polly voice (e.g. Joanna, Matthew; default in .env.example)
NEXT_PUBLIC_WS_URL	web	Backend URL for Socket.IO (e.g. http://localhost:5001)

Usage

1. Open the web app (e.g. http://localhost:3000).
2. Click Start and allow microphone access.
3. Speak; live captions appear. After each phrase (silence ≥800ms or Deepgram final), you get:
   • Metrics: WPM, filler count, low-confidence words
   • A short coaching tip + drill phrase
   • Spoken feedback via AWS Polly TTS (barge-in: if you speak again while TTS is playing, it stops).

Troubleshooting

• CORS: Backend allows http://localhost:3000 and http://127.0.0.1:3000. If you use another origin, add it in app.py (CORS(app, origins=[...])).
• Mic permissions: Ensure the browser has microphone access and that you’re on HTTPS or localhost.
• Port 5000 in use (macOS): On macOS Monterey and later, port 5000 is often used by AirPlay Receiver. The app defaults to port 5001; set NEXT_PUBLIC_WS_URL=http://localhost:5001 in the web app and run the server with python app.py (or PORT=5002 python app.py to use another port).
• WebSocket / transport: Socket.IO will try WebSocket then polling. If the backend is not the same host as the frontend, set NEXT_PUBLIC_WS_URL to the full backend URL (no trailing slash). For Flask-SocketIO, use eventlet (or gevent) in dev so WebSockets work.
• No transcription: Check DEEPGRAM_API_KEY and that the server is receiving binary audio_chunk events. Server expects 16 kHz mono 16-bit PCM.
• No tips / TTS: Check OPENAI_API_KEY and AWS credentials for Polly. Tips are rate-limited (about once per 2 seconds per phrase).

Tech summary

• Frontend: Next.js 14 (App Router), TypeScript, Socket.IO client, Web Audio API (mic → 16 kHz mono PCM), minimal UI with live transcript, last tip, and metrics.
• Backend: Flask, Flask-SocketIO (eventlet), Deepgram live streaming, phrase-boundary logic (final + 800ms silence, debounced), OpenAI for tip + drill, AWS Polly TTS, barge-in (tts_stop on new audio).

No database; MVP scope only.