AI Assistant Hub 🤖

A powerful multi-modal AI assistant platform combining voice interaction and vision capabilities with persistent memory.

🌟 Features

Voice Agent MVP (Port 8091)

• 🎙️ Natural voice conversations
• 🧠 Persistent memory across sessions
• 🗣️ High-quality text-to-speech
• 📝 Real-time transcription

Vision System (Port 8093)

• 📸 Image upload and analysis
• 🎥 Webcam integration
• 🔍 Powered by Claude Vision
• 📊 Detailed visual feedback

Home Server (Port 8080)

• 🏠 Unified interface
• 🚦 Service health monitoring
• 🔄 Real-time status updates
• 🎯 Easy navigation

🚀 Quick Start

1. Clone the repository

   git clone https://github.com/yourusername/ai-assistant-hub.git
   cd ai-assistant-hub
   

2. Set up environment

   python -m venv venv
   source venv/bin/activate  # Linux/Mac
   venv\Scripts\activate     # Windows
   pip install -r requirements.txt
   

3. Configure API keys

   cp .env.example .env
   # Edit .env with your API keys
   

4. Start services

   # Terminal 1: Start Letta Server
   letta serve --port 8283
   
   # Terminal 2: Start Home Server
   python home_server.py
   
   # Terminal 3: Start Voice Agent
   python main.py
   
   # Terminal 4: Start Vision System
   python simple_vision_server.py
   

5. Access the application

   • Open http://localhost:8080 in your browser
   • Navigate to Voice or Vision services
   • Start interacting!

📚 Documentation

• DevPost Submission
• Technical Documentation
• API Documentation

🛠️ Development

Using Docker

# Build and start all services
docker-compose up --build

# Stop services
docker-compose down

Running Tests

pytest tests/

📋 Requirements

• Python 3.9+
• OpenAI API key
• Anthropic API key
• Letta API key

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Open a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• OpenAI for Whisper and TTS
• Anthropic for Claude Vision
• Letta for conversation management
• FastAPI team for the awesome framework

🎤 Voice Agent MVP

A full-stack AI-powered voice conversation application with persistent memory

🌟 Features

• 🎙️ Real-time Voice Recording - Record audio directly in the browser
• 📁 Audio File Upload - Support for MP3, WAV, WebM, and other formats
• 🗣️ Speech-to-Text - Powered by OpenAI Whisper API
• 🤖 AI Conversation - Intelligent responses using OpenAI GPT models
• 🔊 Text-to-Speech - Natural voice responses with multiple voice options
• 🧠 Persistent Memory - Conversation history using Letta memory agent
• 💬 Text Chat - Alternative text-based interaction
• 📊 Session Management - Track conversations and user sessions
• 🔄 Real-time Health Monitoring - Component status tracking

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Frontend      │    │   FastAPI       │    │   AI Services   │
│   (HTML/CSS/JS) │◄──►│   Backend       │◄──►│   (OpenAI/Letta)│
└─────────────────┘    └─────────────────┘    └─────────────────┘
        │                       │                       │
        │                       │                       │
        ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Voice Input   │    │   SQLite DB     │    │   Memory Agent  │
│   Audio Upload  │    │   Sessions      │    │   Conversation  │
└─────────────────┘    └─────────────────┘    └─────────────────┘

🚀 Quick Start

Prerequisites

• Python 3.8+
• OpenAI API Key
• Modern web browser with microphone support

Installation

1. Clone the repository

git clone https://github.com/Pavanmahaveer7/voice-agent-mvp.git
cd voice-agent-mvp

2. Create virtual environment

python -m venv venv

# Windows
.\venv\Scripts\activate

# macOS/Linux
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Set up environment variables

# Create .env file
echo "OPENAI_API_KEY=your_openai_api_key_here" > .env

5. Start Letta server (in a separate terminal)

letta server --port 8283

6. Run the application

python -m uvicorn main:app --host 0.0.0.0 --port 8091 --reload

7. Open your browser

http://localhost:8091

🎯 Usage

Voice Recording

1. Click "Start Recording" button
2. Speak your message
3. Click "Stop Recording"
4. Audio is automatically transcribed and sent to the AI

File Upload

1. Click the upload area or drag & drop an audio file
2. Supported formats: MP3, WAV, WebM, M4A, FLAC, AAC, OGG
3. Click "Send Audio" to process

Text Chat

1. Type your message in the text input
2. Press Enter or click the send button
3. Receive both text and audio responses

Voice Settings

• Choose from 6 different AI voices (Alloy, Echo, Fable, Onyx, Nova, Shimmer)
• Toggle auto-play for audio responses
• Adjust voice settings in real-time

🔧 Configuration

Environment Variables

Create a .env file in the root directory:

# Required
OPENAI_API_KEY=your_openai_api_key_here

# Optional
LETTA_SERVER_URL=http://localhost:8283
DATABASE_PATH=voice_agent_mvp.db
DEBUG=false

Audio Settings

Configure audio processing in src/config/settings.py:

ALLOWED_AUDIO_FORMATS = ['wav', 'mp3', 'm4a', 'flac', 'aac', 'webm', 'ogg']
MAX_AUDIO_SIZE_MB = 25
TTS_MODEL = "tts-1"
TTS_VOICE = "alloy"

📚 API Documentation

Once running, visit:

• Interactive API Docs: http://localhost:8091/docs
• ReDoc Documentation: http://localhost:8091/redoc
• Health Check: http://localhost:8091/health

Key Endpoints

• GET / - Web interface
• POST /voice-chat - Process voice messages
• POST /chat - Text-based chat
• GET /health - System health status
• GET /session/info - Session information
• POST /session/new - Create new session

🧠 Memory System

The application uses Letta (formerly MemGPT) for persistent conversation memory:

• Session Persistence - Conversations saved across sessions
• User Context - Remembers user preferences and history
• Long-term Memory - Maintains context over extended conversations
• Mock Mode - Fallback when Letta server is unavailable

🔍 Monitoring & Health

Health Check Components

• ✅ Whisper Handler - Speech recognition
• ✅ TTS Handler - Text-to-speech
• ✅ Audio Utils - Audio processing
• ✅ Conversation Handler - Memory management
• ✅ OpenAI API - External API connectivity
• ✅ Letta Agent - Memory agent status

Logs

Check application logs for detailed debugging:

# View real-time logs
tail -f logs/voice_agent.log

🧪 Testing

Run the test suite:

# Install test dependencies
pip install pytest pytest-asyncio

# Run tests
pytest tests/

# Run with coverage
pytest --cov=src tests/

🚀 Deployment

Docker (Coming Soon)

docker build -t voice-agent-mvp .
docker run -p 8091:8091 voice-agent-mvp

Production Considerations

• Use environment variables for secrets
• Set up proper CORS origins
• Configure HTTPS
• Use production ASGI server (Gunicorn + Uvicorn)
• Set up proper logging and monitoring

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• OpenAI - Whisper and GPT APIs
• Letta Team - Memory agent framework
• FastAPI - Modern web framework
• The AI Community - Inspiration and support

🐛 Troubleshooting

Common Issues

"Connection Failed" in interface:

• Ensure FastAPI server is running on port 8091
• Check browser console for JavaScript errors
• Verify static files are being served correctly

Letta agent errors:

• Start Letta server: letta server --port 8283
• Check Letta server logs
• Application works in mock mode without Letta

Audio not working:

• Grant microphone permissions in browser
• Check supported audio formats
• Verify OpenAI API key is set

Dependencies issues:

• Update pip: pip install --upgrade pip
• Reinstall requirements: pip install -r requirements.txt --force-reinstall

Getting Help

• 📧 Issues: GitHub Issues
• 📖 Documentation: API Docs
• 💬 Discussions: GitHub Discussions

---

Made with ❤️ and AI | Star ⭐ this repo if you found it helpful!