# Project export: Why waste time say lot word when few word do trick? This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Real-time conversation assistant with AI-powered response suggestions and emotional state analysis - Devpost: https://devpost.com/software/why-waste-time-say-lot-word-when-few-word-do-trick - GitHub: https://github.com/ianalin123/few-word-do-trick - Video: https://www.youtube.com/embed/MEH357zinV4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Lava: Best Use of Lava Gateway) - Team: 3 GitHub contributor(s) — Mohak Akul Prakash (10 commits), Iana Lin (9 commits), Peicheng Li (2 commits) ## Devpost submission (written by the team) ### Inspiration “Why waste time say lot word when few word do trick.” — Kevin Malone, The Office. Kevin might’ve been joking, but for many people with speech impediments or communication challenges, saying fewer words can actually make communication smoother, faster, and more accessible. We wanted to build something that not only helps people communicate, but also helps them express their personality and emotion. Because real communication isn’t just about words — it’s about who you are. ### What it does Few Words Do Trick is an assistive communication platform designed for people with speech impediments or expressive communication difficulties. Unlike traditional AAC (Augmentative and Alternative Communication) tools that focus purely on transmitting speech, our system adds an emotional and personalized layer using real-time EEG emotion detection and MBTI-based personality modeling. From previous research, giving LLMs a persona using the MBTI framework boosts their conversational intelligence by 17–22%. Thus, our system integrates emotional signals from the user’s EEG headset with their personality profile to generate responses that are not only faster and clearer but also more natural and authentic to who they are. This creates a communication experience that feels genuinely human — reflecting tone, mood, and individuality — rather than robotic or generic. By combining neuroscience, machine learning, and personality theory, Few Words Do Trick bridges the gap between accessibility and emotional expression, helping users communicate efficiently and meaningfully in real time. ### How we built it Our system runs on three main layers: Signal and Emotion Processing, Intelligent Backend, and Frontend Experience. The Signal and Emotion Processing Layer integrates the EEG headset, applies Fourier Transforms and temporal smoothing, and performs emotion classification using power spectrum density analysis and a Random Forest Classifier model. The Intelligent Backend Layer handles speech-to-text and sentence generation using Lava and OpenAI’s GPT-5, as well as text-to-speech synthesis with ElevenLabs (more specifically, Whisper model) for customizable, emotion-aware voices. It’s built with FastAPI and Pydantic for validation, with Vite ensuring a smooth connection between the backend and frontend. The Frontend Experience Layer is built with React and NGROK tunneling. It features a MBTI personality quiz, real-time EEG and voice visualization, and a voice customization dashboard using the ElevenLabs API. The UI is designed to be simple, intuitive, and a little fun — keeping accessibility at the center. ### Challenges we ran into We faced several challenges throughout development. Microphone and EEG data access proved difficult without deployment, and collecting consistent EEG signals for model training required plenty of creative “method acting” to simulate emotional states. Integrating detected emotions into the real-time speech output pipeline was complex, and setting up a server to merge MBTI personality data with generated responses added another layer of difficulty. On top of that, we had to design a user interface that felt approachable, expressive, and even enjoyable to use. ### Accomplishments we're proud of We’re proud to have achieved 90% confidence in our emotion classification using EEG data, as well as successfully integrating multiple APIs across the frontend and backend. We built a fully functional real-time emotion-to-speech pipeline and developed personalized, expressive voice outputs that feel human and authentic. Most importantly, we built something that makes communication more natural and personal — a system that doesn’t just speak for you, but speaks like you. ### What we learned We learned how to process and classify EEG signals in real time, integrate emotional intelligence into speech systems, and design with empathy in mind. We also realized how vital personalization is in communication — even when powered by AI. And of course, we learned that Kevin Malone’s wisdom can be surprisingly relevant at a hackathon. ### What's next Looking ahead, we plan to expand Few Words Do Trick into a tool for everyday use by integrating portable EEG hardware and refining our emotion models with larger datasets. We also hope to add multilingual and cultural context support and eventually release it as an open-source assistive communication platform. Our goal is to bridge technology and empathy to help everyone express themselves — because sometimes, the fewest words make the biggest difference. ## README (from the GitHub repository) # AI Conversation Assistant A real-time conversation assistant that uses AI to generate contextual responses based on keywords, conversation history, and emotional state from EEG data. ## Features - **Real-time Audio Recording**: Browser-based microphone recording with Web Audio API - **Speech-to-Text**: OpenAI Whisper integration via Lava Payments - **AI Response Generation**: GPT-4o-mini for generating contextual responses - **Text-to-Speech**: ElevenLabs integration with sentiment-based voice modulation - **Emotional State Integration**: EEG data visualization and processing - **Multi-sentiment Responses**: Generate calm, neutral, and excited response options ## Tech Stack - **Frontend**: React + Vite - **Backend**: FastAPI (Python) - **AI Services**: OpenAI (GPT-3.5, Whisper) via Lava Payments - **TTS**: ElevenLabs - **Audio**: Web Audio API ## Setup Instructions ### 1. Backend Setup ```bash cd backend python3 -m venv venv source venv/bin/activate pip install -r requirements.txt cp .env.example .env # Edit .env with your API keys python main.py ``` ### 2. Frontend Setup ```bash cd frontend npm install npm run dev ``` ### 3. Environment Variables Create `.env` file in backend directory: ```env LAVA_BASE_URL=your_lava_base_url LAVA_FORWARD_TOKEN=your_lava_forward_token ELEVENLABS_VOICE_ID=your_voice_id ``` ## Usage 1. **Start Recording**: Click the microphone button to start recording 2. **Enter Keywords**: Type keywords in the input field 3. **Generate Responses**: Click "Generate" to get AI responses 4. **Select Response**: Choose from calm, neutral, or excited options 5. **Listen**: The selected response will be spoken with appropriate sentiment ## API Endpoints - `POST /api/speech-to-text` - Convert audio to text - `POST /api/generate-responses` - Generate AI responses - `POST /api/text-to-speech` - Convert text to speech with sentiment - `GET /api/health` - Health check ## Development - Backend runs on `http://localhost:8000` - Frontend runs on `http://localhost:3000` - CORS is configured for development - Use ngrok for microphone access in production ## Hackathon Notes - 24-hour development timeline - Lava Payments integration for OpenAI API calls - EEG emotional state simulation - Real-time conversation flow - Sentiment-based response generation ## Detected evidence (automated analysis) Indexed codebase: 49 recognized source files, 9813 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Firebase (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (77 of 77) ``` .DS_Store .gitignore backend/agents/__init__.py backend/agents/conversation_agent.py backend/diagrams/agent_graph.mmd backend/diagrams/input_modalities.mmd backend/diagrams/render.mmd backend/emotion_model_binary.joblib backend/live_stream.py backend/main.py backend/mbti.py backend/models/__init__.py backend/models/firestore_models.py backend/plots/pca_2d_interactive.html backend/plots/pca_3d_interactive.html backend/pretraining/data/happy_trial1.csv backend/pretraining/data/happy_trial2.csv backend/pretraining/data/happy_trial3.csv backend/pretraining/data/happy_trial4.csv backend/pretraining/data/happy_trial5.csv backend/pretraining/data/neutral_trial1.csv backend/pretraining/data/neutral_trial2.csv backend/pretraining/data/neutral_trial3.csv backend/pretraining/data/neutral_trial4.csv backend/pretraining/data/neutral_trial5.csv backend/pretraining/data/sadness_trial1.csv backend/pretraining/data/sadness_trial2.csv backend/pretraining/data/sadness_trial3.csv backend/pretraining/data/sadness_trial4.csv backend/pretraining/data/sadness_trial5.csv backend/pretraining/eeg_emotion_analysis.py backend/pretraining/features.csv backend/pretraining/label_map.json backend/pretraining/preprocess.py backend/pretraining/record_data.py backend/pretraining/train_binary_classifier.py backend/pretraining/visualize_pca.py backend/requirements.txt backend/services/__init__.py backend/services/firestore_service.py backend/services/speaker_service.py frontend/.DS_Store frontend/index.html frontend/original/index.html frontend/original/src/App.css frontend/original/src/App.jsx frontend/original/src/components/AudioRecorder.jsx frontend/original/src/components/ConversationDisplay.jsx frontend/original/src/components/EmotionalStateDisplay.jsx frontend/original/src/components/FeedbackModal.css frontend/original/src/components/FeedbackModal.jsx frontend/original/src/components/KeywordInput.jsx frontend/original/src/components/ResponseSelector.css frontend/original/src/components/ResponseSelector.jsx frontend/original/src/components/VoiceDashboard.jsx frontend/original/src/index.css frontend/original/src/main.jsx frontend/package.json frontend/public/.DS_Store frontend/src/.DS_Store frontend/src/App.css frontend/src/App.jsx frontend/src/components/AudioRecorder.jsx frontend/src/components/ConversationDisplay.jsx frontend/src/components/EmotionalStateDisplay.jsx frontend/src/components/FeedbackModal.css frontend/src/components/FeedbackModal.jsx frontend/src/components/KeywordInput.jsx frontend/src/components/ResponseSelector.css frontend/src/components/ResponseSelector.jsx frontend/src/components/VoiceDashboard.css frontend/src/components/VoiceDashboard.jsx frontend/src/firebase.js frontend/src/index.css frontend/src/main.jsx frontend/vite.config.js README.md ``` ### Dependencies - backend/requirements.txt: fastapi@==0.103.2, firebase-admin@==6.5.0, httpx@==0.25.2, joblib@>=1.3.0, langchain@==1.2.17, langchain-core@==1.3.3, langchain-openai@==1.2.1, langgraph@==1.1.10, librosa@==0.11.0, muselsl, numpy@>=1.24.0, pydantic@>=2.0,<3.0, pylsl@>=1.16.0, python-dotenv@==1.0.0, python-multipart@==0.0.6, scikit-learn@>=1.3.0, scipy@>=1.11.0, uvicorn@==0.23.2, websockets@>=12.0 - frontend/package.json: @types/react@^18.2.43, @types/react-dom@^18.2.17, @vitejs/plugin-react@^4.2.1, axios@^1.6.2, eslint@^8.55.0, eslint-plugin-react@^7.33.2, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.5, firebase@^12.12.1, react@^18.2.0, react-dom@^18.2.0, react-router-dom@^7.9.4, vite@^5.0.8 ### Recent commits (newest first) - Update README.md - Merge pull request #6 from ianalin123/agentic - fix minor things, full agentic support - Merge pull request #5 from ianalin123/agentic - added langchain, langgraph, firebase, diarization - updated some UI and changed some logic - change UI - duplicated UI, added support for another frontend, preserving old aswell - fixed multiple changes, UI left - Merge pull request #4 from ianalin123/voice-dashboard - Add ElevenLabs voice dashboard with working preview - Merge branch 'myer-briggs' - keep 3000 as port - Merge branch 'myer-briggs' - Updated main.py lava prompt - Merge pull request #2 from ianalin123/mohak-ki-daali - minor changes - Personality quiz button color change - Personality quiz - big aah change ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/requirements.txt ``` fastapi==0.103.2 uvicorn==0.23.2 httpx==0.25.2 python-multipart==0.0.6 python-dotenv==1.0.0 joblib>=1.3.0 numpy>=1.24.0 scikit-learn>=1.3.0 websockets>=12.0 scipy>=1.11.0 pylsl>=1.16.0 pydantic>=2.0,<3.0 python-dotenv==1.0.0 muselsl firebase-admin==6.5.0 langgraph==1.1.10 langchain-core==1.3.3 langchain-openai==1.2.1 langchain==1.2.17 librosa==0.11.0 ``` ### frontend/package.json ``` { "name": "ai-conversation-assistant", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "lint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0", "preview": "vite preview" }, "dependencies": { "axios": "^1.6.2", "firebase": "^12.12.1", "react": "^18.2.0", "react-dom": "^18.2.0", "react-router-dom": "^7.9.4" }, "devDependencies": { "@types/react": "^18.2.43", "@types/react-dom": "^18.2.17", "@vitejs/plugin-react": "^4.2.1", "eslint": "^8.55.0", "eslint-plugin-react": "^7.33.2", "eslint-plugin-react-hooks": "^4.6.0", "eslint-plugin-react-refresh": "^0.4.5", "vite": "^5.0.8" } } ``` ### backend/main.py ```python from fastapi import FastAPI, HTTPException, UploadFile, File, Form, Request, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import StreamingResponse import httpx import os from dotenv import load_dotenv from typing import List, Optional, Set, Dict import json import io import joblib import numpy as np from mbti import get_mbti_communication_style from pydantic import BaseModel from services.firestore_service import init_firestore, create_conversation, append_message from services.speaker_service import speaker_service from models.firestore_models import MessageCreate from agents.conversation_agent import conversation_graph from datetime import datetime, timezone load_dotenv() _FIREBASE_KEY_PATH = os.path.join(os.path.dirname(__file__), "..", "firebase-admin-key.json") init_firestore(os.path.abspath(_FIREBASE_KEY_PATH)) app = FastAPI(title="AI Conversation Assistant", version="1.0.0") # In-memory storage for user settings (in production, use a database) user_settings: Dict[str, Dict] = {} # Default voice settings (can be customized per user) DEFAULT_VOICE_SETTINGS = { "stability": 0.5, # 0.0=very expressive, 1.0=very consistent "similarity_boost": 0.75, # How closely to match original voice (0.0-1.0) "style": 0.0, # Style exaggeration (0.0=neutral, 1.0=max, adds latency) "use_speaker_boost": True # Speaker enhancement (recommended: True) } # Pydantic models for request/response class VoiceSelection(BaseModel): user_id: str voice_id: str voice_name: Optional[str] = None class VoiceSettings(BaseModel): user_id: str stability: float similarity_boost: float style: float use_speaker_boost: bool # Load emotion classification model MODEL_PATH = os.path.join(os.path.dirname(__file__), "emotion_model_binary.joblib") emotion_model = None emotion_scaler = None label_map = None try: model_data = joblib.load(MODEL_PATH) # Extract model, scaler, and label map from the dictionary emotion_model = model_data['model'] emotion_scaler = model_data['scaler'] label_map = model_data.get('label_map', {}) # Reverse the label map (it's currently {'happy': '0', 'sadness': '1'}) # We need {0: 'happy', 1: 'sadness'} reverse_label_map = {int(v): k for k, v in label_map.items()} print(f"✓ Emotion model loaded successfully") print(f" Label mapping: {reverse_label_map}") except Exception as e: print(f"✗ Failed to load emotion model: {e}") emotion_model = None emotion_scaler = None reverse_label_map = {0: 'happy', 1: 'sadness'} def predict_emotion(features): """Simple function to predict emotion from features""" if emotion_model is None or emotion_scaler is None: return None, 0.0 # Scale features features_array = np.array(features).reshape(1, -1) features_scaled = emotion_scaler.transform(features_array) # Predict prediction = emotion_model.predict(features_scaled)[0] proba = emotion_model.predict_proba(features_scaled)[0] confidence = float(np.max(proba)) # Use the actual label map from the model emotion_label = reverse_label_map.get(prediction, 'unknown') # Normalize 'sadness' to 'sad' for frontend emotion = 'sad' if emotion_label == 'sadness' else emotion_label return emotion, confidence # WebSocket connection manager for broadcasting emotions class ConnectionManager: def __init__(self): self.active_connections: Set[WebSocket] = set() async def connect(self, websocket: WebSocket): await websocket.accept() self.active_connections.add(websocket) def disconnect(self, websocket: WebSocket): self.active_connections.discard(websocket) async def broadcast(self, message: dict): """Broadcast message to all connected clients""" disconnected = set() for connection in self.active_connections: try: await connection.send_json(message) except Exception: disconnected.add(connection) # Clean up disconnected clients for conn in disconnected: self.active_connections.discard(conn) manager = ConnectionManager() # CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["*"], # Configure for production allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Environment variables LAVA_BASE_URL = os.getenv("LAVA_BASE_URL") LAVA_FORWARD_TOKEN = os.getenv("LAVA_FORWARD_TOKEN") ELEVENLABS_VOICE_ID = os.getenv("ELEVENLABS_VOICE_ID") ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY") class ConversationCreateRequest(BaseModel): user_id: str @app.get("/api/health") async def health_check(): return {"status": "healthy", "message": "AI Conversation Assistant API"} @app.post("/api/conversations") async def create_conversation_endpoint(req: ConversationCreateRequest): conv_id = create_conversation(req.user_id) return {"conv_id": conv_id} @app.post("/api/conversations/{conv_id}/messages") async def append_message_endpoint(conv_id: str, message: MessageCreate): msg_id = append_message(conv_id, message) return {"msg_id": msg_id} @app.post("/api/user/voice") async def set_user_voice(selection: VoiceSelection): """Set the user's selected voice ID""" user_settings[selection.user_id] = { "voice_id": selection.voice_id, "voice_name": selection.voice_name } print(f"✓ User {selection.user_id} selected voice: {selection.voice_name} ({selection.voice_id})") return {"status": "success", "message": f"Voice set to {selection.voice_name}"} @app.get("/api/user/{user_id}/voice") async def get_user_voice(user_id: str): """Get the user's selected voice ID""" if user_id in user_settings and "voice_id" in user_settings[user_id]: return { "voice_id": user_settings[user_id]["voice_id"], [truncated — 13400 more characters] ``` ### frontend/src/main.jsx ```javascript import React from 'react' import ReactDOM from 'react-dom/client' import { BrowserRouter, Routes, Route } from 'react-router-dom' import VoiceDashboard from './components/VoiceDashboard.jsx' import App from './App.jsx' import './index.css' ReactDOM.createRoot(document.getElementById('root')).render( } /> } /> ) ``` ### frontend/src/App.jsx ```javascript import React, { useState, useEffect, useRef } from 'react' import { collection, onSnapshot, orderBy, query } from 'firebase/firestore' import { db } from './firebase' import AudioRecorder from './components/AudioRecorder' import ConversationDisplay from './components/ConversationDisplay' import ResponseSelector from './components/ResponseSelector' import EmotionalStateDisplay from './components/EmotionalStateDisplay' import KeywordInput from './components/KeywordInput' import FeedbackModal from './components/FeedbackModal' import './App.css' // Get or create a unique user ID function getUserId() { let userId = localStorage.getItem('user_id') if (!userId) { userId = 'user_' + Math.random().toString(36).substring(2, 15) localStorage.setItem('user_id', userId) } return userId } async function postMessage(convId, speaker, text, emotionLabel, emotionConfidence) { const body = { speaker, text, timestamp: new Date().toISOString(), emotion: emotionLabel ? { label: emotionLabel, confidence: emotionConfidence } : null, } await fetch(`/api/conversations/${convId}/messages`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body), }) } function App() { const [convId, setConvId] = useState(null) const [conversation, setConversation] = useState([]) const [emotionalState, setEmotionalState] = useState('neutral') const [userKeywords, setUserKeywords] = useState('') const [generatedResponses, setGeneratedResponses] = useState(null) const [isProcessing, setIsProcessing] = useState(false) const [showFeedbackModal, setShowFeedbackModal] = useState(false) // Add personality state const [personalityType, setPersonalityType] = useState('') const [personalityDescription, setPersonalityDescription] = useState('') const [editableResponse, setEditableResponse] = useState('') const [selectedEnergy, setSelectedEnergy] = useState('') const [uiStep, setUiStep] = useState('input') // 'input' | 'selecting' | 'editing' const convIdRef = useRef(null) const convCreatePromiseRef = useRef(null) const getOrCreateConvId = async () => { if (convIdRef.current) return convIdRef.current if (convCreatePromiseRef.current) return await convCreatePromiseRef.current convCreatePromiseRef.current = (async () => { const res = await fetch('/api/conversations', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ user_id: getUserId() }), }) const data = await res.json() const id = data.conv_id convIdRef.current = id setConvId(id) return id })() return await convCreatePromiseRef.current } const handleNewConversation = () => { convIdRef.current = null convCreatePromiseRef.current = null setConvId(null) setConversation([]) } // Subscribe to Firestore messages for the current conversation useEffect(() => { if (!convId) return const q = query( collection(db, 'conversations', convId, 'messages'), orderBy('timestamp', 'asc') ) const unsubscribe = onSnapshot(q, snapshot => { const msgs = snapshot.docs.map(doc => { const d = doc.data() return { id: d.id, speaker: d.speaker, text: d.text, timestamp: d.timestamp?.toDate?.()?.toLocaleTimeString() ?? '', } }) setConversation(msgs) }) return unsubscribe }, [convId]) // Connect to WebSocket for live emotion updates from EEG useEffect(() => { let ws = null let reconnectTimer = null let isUnmounting = false const connectWebSocket = () => { if (isUnmounting) return try { ws = new WebSocket('ws://localhost:8000/ws/emotions') ws.onopen = () => { console.log('✓ Connected to emotion stream') // Send a ping to keep connection alive ws.send('ping') } ws.onmessage = (event) => { try { const data = JSON.parse(event.data) console.log('📨 Frontend received:', data) if (data.type === 'emotion_update') { console.log(`🎭 UPDATING KEVIN TO: ${data.emotion.toUpperCase()} (${(data.confidence * 100).toFixed(1)}%)`) setEmotionalState(data.emotion) } } catch (error) { console.error('Error parsing WebSocket message:', error) } } ws.onerror = (error) => { console.error('WebSocket error - is backend running?', error) } ws.onclose = () => { console.log('Disconnected from emotion stream') // Attempt to reconnect after 3 seconds if not unmounting if (!isUnmounting) { console.log('Will attempt to reconnect in 3 seconds...') reconnectTimer = setTimeout(connectWebSocket, 3000) } } } catch (error) { console.error('Failed to create WebSocket:', error) if (!isUnmounting) { reconnectTimer = setTimeout(connectWebSocket, 3000) } } } // Initial connection connectWebSocket() // Cleanup on unmount return () => { isUnmounting = true if (reconnectTimer) { clearTimeout(reconnectTimer) } if (ws && ws.readyState === WebSocket.OPEN) { ws.close() } } }, []) const handleAudioTranscription = (_transcription) => { // STT endpoint now persists the diarized message directly; Firestore listener updates UI. } const handleGenerateResponses = async () => { if (!userKeywords.trim()) { alert('Please enter some keywords') return } setIsProcessing(true) try { const conversationText = conversation .map(msg => `${msg.speaker}: ${msg.text}`) .join('\n') const response = await fetch('/api/generate-responses', { method: 'POST', headers: { [truncated — 6967 more characters] ``` ### frontend/original/src/main.jsx ```javascript import React from 'react' import ReactDOM from 'react-dom/client' import { BrowserRouter, Routes, Route } from 'react-router-dom' import VoiceDashboard from './components/VoiceDashboard.jsx' import App from './App.jsx' import './index.css' ReactDOM.createRoot(document.getElementById('root')).render( } /> } /> ) ``` ### frontend/original/src/App.jsx ```javascript import React, { useState, useEffect } from 'react' import AudioRecorder from './components/AudioRecorder' import ConversationDisplay from './components/ConversationDisplay' import ResponseSelector from './components/ResponseSelector' import EmotionalStateDisplay from './components/EmotionalStateDisplay' import KeywordInput from './components/KeywordInput' import FeedbackModal from './components/FeedbackModal' import './App.css' // Get or create a unique user ID function getUserId() { let userId = localStorage.getItem('user_id') if (!userId) { userId = 'user_' + Math.random().toString(36).substring(2, 15) localStorage.setItem('user_id', userId) } return userId } function App() { const [conversation, setConversation] = useState([]) const [emotionalState, setEmotionalState] = useState('neutral') const [userKeywords, setUserKeywords] = useState('') const [generatedResponses, setGeneratedResponses] = useState(null) const [isProcessing, setIsProcessing] = useState(false) const [showFeedbackModal, setShowFeedbackModal] = useState(false) // Add personality state const [personalityType, setPersonalityType] = useState('') const [personalityDescription, setPersonalityDescription] = useState('') // Connect to WebSocket for live emotion updates from EEG useEffect(() => { let ws = null let reconnectTimer = null let isUnmounting = false const connectWebSocket = () => { if (isUnmounting) return try { ws = new WebSocket('ws://localhost:8000/ws/emotions') ws.onopen = () => { console.log('✓ Connected to emotion stream') // Send a ping to keep connection alive ws.send('ping') } ws.onmessage = (event) => { try { const data = JSON.parse(event.data) console.log('📨 Frontend received:', data) if (data.type === 'emotion_update') { console.log(`🎭 UPDATING KEVIN TO: ${data.emotion.toUpperCase()} (${(data.confidence * 100).toFixed(1)}%)`) setEmotionalState(data.emotion) } } catch (error) { console.error('Error parsing WebSocket message:', error) } } ws.onerror = (error) => { console.error('WebSocket error - is backend running?', error) } ws.onclose = () => { console.log('Disconnected from emotion stream') // Attempt to reconnect after 3 seconds if not unmounting if (!isUnmounting) { console.log('Will attempt to reconnect in 3 seconds...') reconnectTimer = setTimeout(connectWebSocket, 3000) } } } catch (error) { console.error('Failed to create WebSocket:', error) if (!isUnmounting) { reconnectTimer = setTimeout(connectWebSocket, 3000) } } } // Initial connection connectWebSocket() // Cleanup on unmount return () => { isUnmounting = true if (reconnectTimer) { clearTimeout(reconnectTimer) } if (ws && ws.readyState === WebSocket.OPEN) { ws.close() } } }, []) const handleAudioTranscription = (transcription) => { const newMessage = { id: Date.now(), speaker: 'OTHER', text: transcription, timestamp: new Date().toLocaleTimeString() } setConversation(prev => [...prev, newMessage]) } const handleGenerateResponses = async () => { if (!userKeywords.trim()) { alert('Please enter some keywords') return } setIsProcessing(true) try { const conversationText = conversation .map(msg => `${msg.speaker}: ${msg.text}`) .join('\n') const response = await fetch('/api/generate-responses', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ user_keywords: userKeywords, previous_conversation: conversationText, emotional_state: emotionalState, personality_type: personalityType, // Send personality type personality_description: personalityDescription // Send description }) }) if (!response.ok) { throw new Error('Failed to generate responses') } const data = await response.json() setGeneratedResponses(data) } catch (error) { console.error('Error generating responses:', error) alert('Failed to generate responses. Please try again.') } finally { setIsProcessing(false) } } // Add function to handle personality results from quiz const handlePersonalityResult = (type, description) => { setPersonalityType(type) setPersonalityDescription(description) } const handleResponseSelect = async (response, energy) => { // Changed 'sentiment' to 'energy' // Add user's selected response to conversation const userMessage = { id: Date.now(), speaker: 'USER', text: response, timestamp: new Date().toLocaleTimeString() } setConversation(prev => [...prev, userMessage]) // Play text-to-speech with energy level and emotional state try { const userId = getUserId() const formData = new FormData() formData.append('text', response) formData.append('energy', energy) // NEW: low/medium/high/contradictory formData.append('emotional_state', emotionalState) // NEW: happy/sad from EEG formData.append('user_id', userId) // NEW: user ID for voice preference console.log(`🎤 TTS: energy=${energy}, emotion=${emotionalState}, user=${userId}`) const response_audio = await fetch('/api/text-to-speech', { method: 'POST', body: formData }) if (response_audio.ok) { const audioBlob = await response_audio.blob() const audioUrl = URL.createObjectURL(audioBlob) const audio = ne [truncated — 3060 more characters] ``` ### frontend/index.html ```html AI Conversation Assistant
``` ### frontend/vite.config.js ```javascript import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' export default defineConfig({ plugins: [react()], server: { port: 3000, proxy: { '/api': { target: 'http://localhost:8000', changeOrigin: true } }, allowedHosts: ['localhost', '127.0.0.1', '0.0.0.0', 'conjoinedly-snapless-sid.ngrok-free.dev'] } }) ``` ### backend/live_stream.py ```python """ Live EEG streaming from Muse headset with real-time emotion classification Connects to Muse via LSL and sends predictions via WebSocket """ import asyncio import websockets import json import numpy as np from pylsl import StreamInlet, resolve_byprop, resolve_streams from scipy import signal from collections import deque import sys # Import preprocessing parameters from preprocess.py ORIGINAL_SFREQ = 256 # Hz (Muse sampling rate) TARGET_SFREQ = 128 # Hz (downsampled) WINDOW_SIZE = 2.0 # seconds WINDOW_OVERLAP = 0.5 # 50% overlap (1 second step) # Filter parameters BANDPASS_LOW = 1.0 BANDPASS_HIGH = 45.0 NOTCH_FREQ = 60.0 FILTER_ORDER = 5 # Frequency bands FREQ_BANDS = { 'theta': (4, 8), 'alpha': (8, 13), 'beta': (13, 30), 'gamma': (30, 45) } # Muse channel names (standard order) CHANNEL_NAMES = ['TP9', 'AF7', 'AF8', 'TP10', 'Right AUX'] class LiveEEGProcessor: """Real-time EEG processing and feature extraction""" def __init__(self, buffer_duration=3.0): """ Initialize live EEG processor Args: buffer_duration: How many seconds of data to keep in buffer """ self.buffer_duration = buffer_duration self.buffer_size = int(buffer_duration * ORIGINAL_SFREQ) # Create circular buffer for each channel self.n_channels = len(CHANNEL_NAMES) self.buffer = deque(maxlen=self.buffer_size) # Processing flags self.last_process_time = 0 self.process_interval = (1 - WINDOW_OVERLAP) * WINDOW_SIZE # 1 second def add_sample(self, sample): """Add a new EEG sample to the buffer""" self.buffer.append(sample) def preprocess(self, data): """Apply preprocessing pipeline to raw EEG data""" # 1. Bandpass filter (1-45 Hz) nyquist = ORIGINAL_SFREQ / 2 low = BANDPASS_LOW / nyquist high = BANDPASS_HIGH / nyquist sos_bandpass = signal.butter(FILTER_ORDER, [low, high], btype='band', output='sos') data_filtered = signal.sosfiltfilt(sos_bandpass, data, axis=1) # 2. Notch filter (60 Hz - power line noise) Q = 30.0 b_notch, a_notch = signal.iirnotch(NOTCH_FREQ, Q, fs=ORIGINAL_SFREQ) data_notched = signal.filtfilt(b_notch, a_notch, data_filtered, axis=1) # 3. Downsample to 128 Hz downsample_factor = int(ORIGINAL_SFREQ / TARGET_SFREQ) data_downsampled = signal.decimate(data_notched, downsample_factor, axis=1) return data_downsampled def compute_band_power(self, data, sfreq=TARGET_SFREQ): """Compute power in each frequency band""" band_powers = {band: [] for band in FREQ_BANDS.keys()} for ch_idx in range(data.shape[0]): freqs, psd = signal.welch(data[ch_idx, :], fs=sfreq, nperseg=min(256, data.shape[1])) for band_name, (low_freq, high_freq) in FREQ_BANDS.items(): freq_mask = (freqs >= low_freq) & (freqs <= high_freq) band_power = np.trapz(psd[freq_mask], freqs[freq_mask]) band_powers[band_name].append(band_power) return band_powers def compute_differential_entropy(self, data, sfreq=TARGET_SFREQ): """Compute differential entropy for each frequency band""" de_values = {band: [] for band in FREQ_BANDS.keys()} for ch_idx in range(data.shape[0]): for band_name, (low_freq, high_freq) in FREQ_BANDS.items(): nyquist = sfreq / 2 low = low_freq / nyquist high = high_freq / nyquist sos = signal.butter(4, [low, high], btype='band', output='sos') band_signal = signal.sosfiltfilt(sos, data[ch_idx, :]) variance = np.var(band_signal) if variance > 0: de = 0.5 * np.log(2 * np.pi * np.e * variance) else: de = 0.0 de_values[band_name].append(de) return de_values def extract_features(self, segment, sfreq=TARGET_SFREQ): """Extract all 52 features from a 2-second segment""" features = [] # 1. Band Power (20 features: 4 bands × 5 channels) band_powers = self.compute_band_power(segment, sfreq) for band_name in ['theta', 'alpha', 'beta', 'gamma']: for ch_idx in range(len(CHANNEL_NAMES)): features.append(band_powers[band_name][ch_idx]) # 2. Differential Entropy (20 features: 4 bands × 5 channels) de_values = self.compute_differential_entropy(segment, sfreq) for band_name in ['theta', 'alpha', 'beta', 'gamma']: for ch_idx in range(len(CHANNEL_NAMES)): features.append(de_values[band_name][ch_idx]) # 3. Asymmetry Features (2 features) # Frontal Alpha Asymmetry: log(AF8_alpha) - log(AF7_alpha) af7_idx = 1 # AF7 af8_idx = 2 # AF8 af7_alpha = band_powers['alpha'][af7_idx] af8_alpha = band_powers['alpha'][af8_idx] frontal_asymmetry = np.log(af8_alpha + 1e-10) - np.log(af7_alpha + 1e-10) features.append(frontal_asymmetry) # Temporal Alpha Asymmetry: log(TP10_alpha) - log(TP9_alpha) tp9_idx = 0 # TP9 tp10_idx = 3 # TP10 tp9_alpha = band_powers['alpha'][tp9_idx] tp10_alpha = band_powers['alpha'][tp10_idx] temporal_asymmetry = np.log(tp10_alpha + 1e-10) - np.log(tp9_alpha + 1e-10) features.append(temporal_asymmetry) # 4. Power Ratios (10 features: 2 ratios × 5 channels) for ch_idx in range(len(CHANNEL_NAMES)): # Beta/Alpha ratio beta_power = band_powers['beta'][ch_idx] alpha_power = band_powers['alpha'][ch_idx] features.append(beta_power / (alpha_power + 1e-10)) # Gamma/Theta ratio gamma_power = band_powers['gamma'][ch_idx] theta_power = band_powers['theta'][ch_idx] [truncated — 5228 more characters] ``` [35 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]