# Project export: DevOps & Documentation Copilot This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: DevOps Copilot instantly turns your team’s documentation into an AI-powered knowledge base, so anyone can ask a question and get accurate, source-cited answers in Slack or on the web. - Devpost: https://devpost.com/software/devops-documentation-copilot - GitHub: https://github.com/JaiPrathikReddySoda/DevOps-Documentation-Copilot - Video: https://www.youtube.com/embed/navD7Fwy2jo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — JaiPrathikReddySoda (4 commits) ## Devpost submission (written by the team) ### Inspiration Every developer and DevOps engineer knows the frustration of searching through countless wiki pages, README files, and outdated documentation just to find a simple answer. We wanted to fix this problem and make documentation as easy to access as asking a question in chat with reliable, source-cited answers. Our inspiration came from daily pain points and a desire to empower teams with AI that works for them, not against them. ### What it does DevOps & Documentation Copilot transforms any team’s documentation into an AI-powered, searchable knowledge base. Users can upload Markdown, PDF, Word, and text files (or even point to URLs and GitHub repos), and the system will: Chunk and embed documents using semantic AI models Store embeddings in a fast local FAISS vector database Answer questions via web interface or Slack bot, always with clear citations Retrieve context from the docs, generate an answer with a language model (OpenAI, Groq, or Anthropic), and show exactly where the answer came from How I built it Document Processing: Parsed and chunked a wide range of doc types (Markdown, PDF, TXT, DOCX, URLs, GitHub). Semantic Embeddings: Used sentence-transformers to convert doc chunks into high-dimensional vectors. Vector Search: Leveraged FAISS for fast, scalable similarity search. RAG Engine: Built a Retrieval-Augmented Generation pipeline, retrieving top matches and generating answers with LLMs. User Interfaces: Streamlit for web upload & interactive Q&A Slack bot for seamless team chat integration Streamlit for web upload & interactive Q&A Slack bot for seamless team chat integration Source Attribution: Every answer includes document citations and highlighted text snippets for full transparency. Challenges Dependency Hell: Pinning compatible versions of sentence-transformers, transformers, and huggingface_hub took a lot of trial and error. Performance at Scale: Keeping the system fast and memory-efficient with large document sets and long files. Slack API Growing Pains: Navigating changes in Slack’s developer UI and permission systems while getting Socket Mode and bot tokens working. Reducing AI Hallucinations: Careful prompt engineering and smart chunking were required to ensure the model stayed grounded in real docs. Accomplishments that I am proud of End-to-End Working MVP: From uploading a doc to getting instant, source-cited answers in both the web app and Slack. Multi-provider Support: Swappable LLMs (OpenAI, Groq, Anthropic) with a single config. User Trust: Every answer is backed by proof, no more guessing where info came from. User-Centric Design: Clean, accessible UI for both web and Slack. My learning's AI and IR (Information Retrieval) are a perfect match for internal knowledge bases—when done right, you get accuracy, speed, and transparency. Dependency management in Python’s ML ecosystem is critical for reliable hackathon projects. Human-centered AI design (easy UIs, source citations, multi-modal access) is just as important as smart algorithms. ### What's next More Integrations: Support for Google Docs, Confluence, Notion, and code repositories. Semantic Search for Code: Enable code snippet retrieval, inline explanations, and API documentation Q&A. Usage Analytics: Insights on popular questions and knowledge gaps. Enterprise-Ready: Add authentication, user management, and cloud deployment options. Try DevOps Copilot— and never get lost in your docs again! ## README (from the GitHub repository) # DevOps & Documentation Copilot A comprehensive AI-powered documentation assistant that processes various document types, stores them as embeddings in a FAISS vector database, and provides intelligent answers through a web interface or Slack bot using RAG. This Documentation Copilot transforms your documentation into an intelligent knowledge base that can: - **Process Multiple Document Types**: Markdown, PDF, Word docs, text files, web URLs, and GitHub files - **Create Smart Embeddings**: Break documents into meaningful chunks and store them as vectors - **Provide Intelligent Answers**: Use RAG to find relevant information and generate contextual responses - **Show Source Attribution**: Always cite which documents were used to generate answers - **Work Everywhere**: Access through a web interface or directly in Slack ## System Architecture ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Documents │ │ Document │ │ Vector │ │ (PDF, MD, │───▶│ Processor │───▶│ Store │ │ URLs, etc.) │ │ (Chunking) │ │ (FAISS) │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ ┌─────────────────┐ ┌─────────────────┐ │ │ User Query │───▶│ RAG Engine │◀──────────┘ │ (Web/Slack) │ │ (LLM + RAG) │ └─────────────────┘ └─────────────────┘ ``` ## 📋 Prerequisites Before you begin, ensure you have: - **Python 3.8+** installed on your system - **Git** for cloning the repository - **API Keys** for at least one LLM provider (OpenAI, Groq, or Anthropic) - **Slack Workspace** (optional, for Slack bot functionality) ## 🛠️ Installation & Setup ### Step 1: Clone and Navigate to Project ```bash # Clone the repository (if not already done) git clone cd doc-copilot # Or if you're already in the project directory pwd # Should show your project path ``` ### Step 2: Create Virtual Environment ```bash # Create a virtual environment python -m venv .venv # Activate the virtual environment # On macOS/Linux: source .venv/bin/activate # On Windows: # .venv\Scripts\activate ``` ### Step 3: Install Dependencies ```bash # Install all required packages pip install -r requirements.txt ``` ### Step 4: Set Up Environment Variables Create a `.env` file in your project root with your API keys: ```bash # Create .env file touch .env ``` Add the following content to your `.env` file: ```env # Required: At least one LLM provider API key OPENAI_API_KEY=your_openai_api_key_here # OR GROQ_API_KEY=your_groq_api_key_here # OR ANTHROPIC_API_KEY=your_anthropic_api_key_here # Optional: Slack bot configuration (only if using Slack) SLACK_BOT_TOKEN=your_slack_bot_token_here SLACK_APP_TOKEN=your_slack_app_token_here # Optional: Default configuration DEFAULT_LLM_PROVIDER=openai DEFAULT_MODEL=gpt-3.5-turbo DEFAULT_TEMPERATURE=0.1 DEFAULT_MAX_TOKENS=1000 ``` **How to Get API Keys:** 1. **OpenAI API Key**: - Go to [OpenAI Platform](https://platform.openai.com/api-keys) - Sign up/login and create a new API key 2. **Groq API Key**: - Visit [Groq Console](https://console.groq.com/) - Sign up and generate an API key 3. **Anthropic API Key**: - Go to [Anthropic Console](https://console.anthropic.com/) - Sign up and create an API key ### Step 5: Quick Setup (Optional) Run the automated setup script to verify everything is working: ```bash python quick_start.py ``` This script will: - Validate your API keys - Test document processing - Create a sample vector store - Verify the RAG system ## Running the System ### Option 1: Web Interface (Recommended for First Use) Start the Streamlit web application: ```bash streamlit run app.py ``` The web interface will open at `http://localhost:8501` **Using the Web Interface:** 1. **Upload Documents**: - Click "Browse files" to upload individual files - Or enter a folder path to process all documents in that folder - Supported formats: PDF, Markdown, Word docs, text files 2. **Process Web Content**: - Enter a URL to scrape and process web content - Or enter a GitHub URL to process repository files 3. **Ask Questions**: - Type your question in the chat interface - Select your preferred LLM provider and model - Get answers with source citations ### Option 2: Slack Bot (For Team Collaboration) #### Step 1: Create Slack App 1. Go to [Slack API Apps](https://api.slack.com/apps) 2. Click "Create New App" → "From scratch" 3. Name your app (e.g., "Documentation Copilot") 4. Select your workspace #### Step 2: Configure Slack App 1. **Enable Socket Mode**: - Go to "Socket Mode" in the left sidebar - Enable Socket Mode - Generate an App-Level Token (starts with `xapp-`) 2. **Add Bot Token Scopes**: - Go to "OAuth & Permissions" - Add these Bot Token Scopes: - `commands` (for slash commands) - `chat:write` (to send messages) - `app_mentions:read` (to respond to mentions) 3. **Install App to Workspace**: - Click "Install to Workspace" - Copy the Bot User OAuth Token (starts with `xoxb-`) 4. **Create Slash Commands**: - Go to "Slash Commands" - Create these commands: - `/ask` - Ask questions about documents - `/docs-status` - Check system status - `/docs-help` - Show help information #### Step 3: Update Environment Variables Add your Slack tokens to your `.env` file: ```env SLACK_BOT_TOKEN=xoxb-your-bot-token-here SLACK_APP_TOKEN=xapp-your-app-token-here ``` #### Step 4: Run the Slack Bot ```bash python slack_bot.py ``` **Using the Slack Bot:** 1. **Invite the bot** to a channel: `@your-bot-name` 2. **Check status**: Type `/docs-status` 3. **Ask questions**: Type `/ask What is this project about?` 4. **Get help**: Type `/docs-help` ## 📚 Supported Document Types | Format | Extension | Features | |--------|-----------|----------| | **Markdown** | `.md` | Direct parsing, metadata extraction | | **PDF** | `.pdf` | Text extraction, layout preservation | | **Word** | `.docx` | Full text and formatting | | **Text** | `.txt` | Simple text processing | | **Web URLs** | - | Content scraping, metadata | | **GitHub Files** | - | Direct repository access | ## 🔧 Configuration Options ### LLM Provider Settings You can configure different LLM providers in the web interface or modify defaults in your `.env` file: ```env # Default LLM Configuration DEFAULT_LLM_PROVIDER=openai # openai, groq, anthropic DEFAULT_MODEL=gpt-3.5-turbo # Model name for the provider DEFAULT_TEMPERATURE=0.1 # Creativity level (0.0-1.0) DEFAULT_MAX_TOKENS=1000 # Maximum response length ``` ### Vector Store Settings The system automatically manages: - **Chunk Size**: 1000 characters per chunk - **Overlap**: 200 characters between chunks - **Similarity Threshold**: 0.5 (configurable) - **Top-K Results**: 5 (configurable) ## Testing the System ### Automated Testing Run the comprehensive test suite: ```bash python test_system.py ``` This will test: - Document processing - Vector storage - RAG functionality - API integrations - Error handling ### Manual Testing 1. **Test Document Processing**: ```bash python -c " from document_processor import DocumentProcessor processor = DocumentProcessor() result = processor.process_file('sample_docs/README.md') print(f'Processed {len(result)} chunks') " ``` 2. **Test Vector Store**: ```bash python -c " from vector_store import VectorStore store = VectorStore() stats = store.get_stats() print(f'Total vectors: {stats[\"total_vectors\"]}') " ``` 3. **Test RAG Engine**: ```bash python -c " from rag_engine import RAGEngine from vector_store import VectorStore store = VectorStore() store.load() rag = RAGEngine(store) result = rag.answer_question('What is this project about?') print(f'Answer: {result[\"answer\"]}') " ``` ## Monitoring and [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 12 recognized source files, 111 KB. - LangChain (technology) — detected in the code - Python (language) — detected in the code - Streamlit (technology) — detected in the code ## Codebase structure (from repository index) ### Files (17 of 17) ``` .gitattributes .gitignore app.py document_processor.py quick_start.py rag_engine.py README.md requirements.txt sample_docs/sample_documentation.md sample_docs/sample_text.txt setup.py slack_bot.py slack_manifest.yaml test_system.py utils.py vector_index/config.json vector_store.py ``` ### Dependencies - requirements.txt: beautifulsoup4@>=4.12.0, faiss-cpu@>=1.7.4, langchain@>=0.0.350, langchain-anthropic@>=0.0.1, langchain-groq@>=0.0.1, langchain-openai@>=0.0.2, markdown@>=3.5.0, numpy@>=1.26.0, pandas@>=2.0.0, pypdf2@>=3.0.1, python-docx@>=0.8.11, python-dotenv@>=1.0.0, requests@>=2.31.0, sentence-transformers@>=2.2.0, slack-bolt@>=1.18.0, streamlit@>=1.28.0, tiktoken@>=0.5.0 ### Recent commits (newest first) - edit in .gitignore - Remove __pycache__ and .pyc files from repo - Add .env to .gitignore - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### sample_docs/sample_documentation.md ```markdown # Sample Documentation This is a sample markdown document for testing the Documentation Copilot. ## Features - Document processing - Vector embeddings - RAG implementation - Multiple LLM support ## Usage 1. Upload your documents 2. Ask questions 3. Get AI-powered answers ## Configuration Set your API keys in the environment variables: - OPENAI_API_KEY - GROQ_API_KEY - ANTHROPIC_API_KEY ``` ### requirements.txt ``` streamlit>=1.28.0 langchain>=0.0.350 langchain-openai>=0.0.2 langchain-groq>=0.0.1 langchain-anthropic>=0.0.1 faiss-cpu>=1.7.4 pypdf2>=3.0.1 python-docx>=0.8.11 markdown>=3.5.0 beautifulsoup4>=4.12.0 requests>=2.31.0 python-dotenv>=1.0.0 slack-bolt>=1.18.0 numpy>=1.26.0 pandas>=2.0.0 tiktoken>=0.5.0 sentence-transformers>=2.2.0 ``` ### app.py ```python """ Documentation Copilot - Streamlit Web Application This is the main web interface for the Documentation Copilot MVP. It provides a user-friendly way to: - Upload and process documents - Ask questions about the documents - Get AI-powered answers with source attribution - Configure LLM settings """ import streamlit as st import os import tempfile from pathlib import Path import time from typing import List, Dict, Any from dotenv import load_dotenv # Load environment variables from .env file load_dotenv() # Import our custom modules from document_processor import DocumentProcessor from vector_store import VectorStore from rag_engine import RAGEngine from utils import ( validate_file_path, validate_folder_path, validate_url, validate_github_url, get_supported_file_extensions, is_supported_file, create_sample_documents, validate_api_keys, get_environment_info, format_file_size ) # Page configuration st.set_page_config( page_title="Documentation Copilot", page_icon="📚", layout="wide", initial_sidebar_state="expanded" ) # Custom CSS for better styling st.markdown(""" """, unsafe_allow_html=True) # Initialize session state if 'vector_store' not in st.session_state: st.session_state.vector_store = None if 'rag_engine' not in st.session_state: st.session_state.rag_engine = None if 'documents_loaded' not in st.session_state: st.session_state.documents_loaded = False if 'chat_history' not in st.session_state: st.session_state.chat_history = [] def initialize_components(): """Initialize vector store and RAG engine components.""" try: # Initialize vector store if st.session_state.vector_store is None: st.session_state.vector_store = VectorStore() # Try to load existing index if st.session_state.vector_store.load(): st.session_state.documents_loaded = True st.success("Loaded existing document index!") # Initialize RAG engine if st.session_state.rag_engine is None: st.session_state.rag_engine = RAGEngine( vector_store=st.session_state.vector_store, llm_provider=st.session_state.get('llm_provider', 'openai'), model_name=st.session_state.get('model_name', 'gpt-3.5-turbo'), temperature=st.session_state.get('temperature', 0.1), max_tokens=st.session_state.get('max_tokens', 1000) ) except Exception as e: st.error(f"Error initializing components: {str(e)}") def process_uploaded_files(uploaded_files: List) -> List[Dict[str, Any]]: """Process uploaded files and return chunks.""" processor = DocumentProcessor() all_chunks = [] with st.spinner("Processing uploaded files..."): for uploaded_file in uploaded_files: try: # Save uploaded file to temporary location with tempfile.NamedTemporaryFile(delete=False, suffix=uploaded_file.name) as tmp_file: tmp_file.write(uploaded_file.getvalue()) tmp_path = tmp_file.name # Process the file if is_supported_file(tmp_path): chunks = processor.process_file(tmp_path) all_chunks.extend(chunks) st.success(f"Processed {uploaded_file.name}: {len(chunks)} chunks") else: st.warning(f"Skipped {uploaded_file.name}: Unsupported file type") # Clean up temporary file os.unlink(tmp_path) except Exception as e: st.error(f"Error processing {uploaded_file.name}: {str(e)}") return all_chunks def process_folder_path(folder_path: str) -> List[Dict[str, Any]]: """Process all files in a folder.""" processor = DocumentProcessor() with st.spinner(f"Processing folder: {folder_path}"): try: chunks = processor.process_folder(folder_path) st.success(f"Processed folder: {len(chunks)} chunks from {folder_path}") return chunks except Exception as e: st.error(f"Error processing folder: {str(e)}") return [] def process_url(url: str) -> List[Dict[str, Any]]: """Process content from a URL.""" processor = DocumentProcessor() with st.spinner(f"Processing URL: {url}"): try: chunks = processor.process_url(url) st.success(f"Processed URL: {len(chunks)} chunks from {url}") return chunks except Exception as e: st.error(f"Error processing URL: {str(e)}") return [] def process_github_file(repo_url: str, file_path: str) -> List[Dict[str, Any]]: """Process a file from GitHub.""" processo [truncated — 15778 more characters] ``` ### slack_manifest.yaml ```yaml display_information: name: Doc Copilot description: Ask questions about your documentation directly from Slack. background_color: "#2c3e50" features: bot_user: display_name: Doc Copilot always_online: true slash_commands: - command: /ask description: Ask a question about your documents. usage_hint: "[your question]" should_escape: false - command: /docs-status description: Show the status of the documentation copilot. should_escape: false - command: /docs-help description: Show help information for the Documentation Copilot bot. should_escape: false oauth_config: scopes: bot: - commands - chat:write - app_mentions:read - im:history - im:read - users:read settings: event_subscriptions: bot_events: - app_mention - message.im interactivity: is_enabled: true org_deploy_enabled: false socket_mode_enabled: true token_rotation_enabled: false ``` ### setup.py ```python """ Setup Script for Documentation Copilot This script sets up the Documentation Copilot MVP: 1. Installs required dependencies 2. Creates necessary directories 3. Sets up environment configuration 4. Runs initial tests """ import os import sys import subprocess import shutil from pathlib import Path def check_python_version(): """Check if Python version is compatible.""" print("🔧 Checking Python version...") if sys.version_info < (3, 8): print("❌ Python 3.8 or higher is required") print(f" Current version: {sys.version}") return False print(f"✅ Python {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}") return True def install_dependencies(): """Install required dependencies.""" print("\n📦 Installing dependencies...") try: # Install from requirements.txt subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", "requirements.txt"]) print("✅ Dependencies installed successfully") return True except subprocess.CalledProcessError as e: print(f"❌ Failed to install dependencies: {str(e)}") return False def create_directories(): """Create necessary directories.""" print("\n📁 Creating directories...") directories = [ "vector_index", "sample_docs", "logs" ] for directory in directories: Path(directory).mkdir(exist_ok=True) print(f"✅ Created directory: {directory}") def create_env_file(): """Create .env file if it doesn't exist.""" print("\n⚙️ Setting up environment configuration...") env_file = Path(".env") if env_file.exists(): print("✅ .env file already exists") return True # Create .env file from template env_template = """# Documentation Copilot Environment Variables # Fill in your API keys below # LLM API Keys (set at least one) OPENAI_API_KEY=your_openai_api_key_here GROQ_API_KEY=your_groq_api_key_here ANTHROPIC_API_KEY=your_anthropic_api_key_here # Slack Bot Configuration (optional) SLACK_BOT_TOKEN=your_slack_bot_token_here SLACK_APP_TOKEN=your_slack_app_token_here # Default LLM Configuration DEFAULT_LLM_PROVIDER=openai DEFAULT_MODEL=gpt-3.5-turbo DEFAULT_TEMPERATURE=0.1 DEFAULT_MAX_TOKENS=1000 # Vector Store Configuration VECTOR_STORE_PATH=vector_index EMBEDDING_MODEL=all-MiniLM-L6-v2 # Document Processing Configuration CHUNK_SIZE=1000 CHUNK_OVERLAP=200 """ try: with open(env_file, 'w') as f: f.write(env_template) print("✅ Created .env file") print(" Please edit .env and add your API keys") return True except Exception as e: print(f"❌ Failed to create .env file: {str(e)}") return False def run_tests(): """Run system tests.""" print("\n🧪 Running system tests...") try: # Import test modules to check if everything is working from test_system import main as run_tests run_tests() print("✅ All tests passed") return True except Exception as e: print(f"⚠️ Some tests failed: {str(e)}") print(" This is normal if API keys are not set") return False def show_next_steps(): """Show next steps for the user.""" print("\n🎉 Setup completed!") print("\n📋 Next Steps:") print("1. 🔑 Set up API keys:") print(" - Edit the .env file") print(" - Add your OpenAI, Groq, or Anthropic API key") print("\n2. 🚀 Start the web interface:") print(" streamlit run app.py") print("\n3. 🤖 Start the Slack bot (optional):") print(" python slack_bot.py") print("\n4. 🧪 Run quick start:") print(" python quick_start.py") print("\n5. 📚 Add your documents:") print(" - Upload files through the web interface") print(" - Process folders with documentation") print(" - Add web URLs or GitHub files") def main(): """Main setup function.""" print("🚀 Documentation Copilot - Setup") print("=" * 50) # Check Python version if not check_python_version(): sys.exit(1) # Install dependencies if not install_dependencies(): print("❌ Setup failed. Please check the error messages above.") sys.exit(1) # Create directories create_directories() # Create environment file if not create_env_file(): print("❌ Failed to create environment file.") sys.exit(1) # Run tests run_tests() # Show next steps show_next_steps() print("\n✨ Setup completed successfully!") print(" You can now start using the Documentation Copilot.") if __name__ == "__main__": main() ``` ### quick_start.py ```python """ Quick Start Script for Documentation Copilot This script helps you get started with the Documentation Copilot MVP: 1. Sets up the environment 2. Creates sample documents 3. Processes and embeds the documents 4. Demonstrates the RAG functionality """ import os import sys from pathlib import Path from dotenv import load_dotenv # Load environment variables load_dotenv() # Import our modules from document_processor import DocumentProcessor from vector_store import VectorStore from rag_engine import RAGEngine from utils import validate_api_keys, create_sample_documents def check_environment(): """Check if the environment is properly set up.""" print("🔧 Checking environment...") # Check Python version if sys.version_info < (3, 8): print("❌ Python 3.8 or higher is required") return False print(f"✅ Python {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}") # Check API keys api_keys = validate_api_keys() available_providers = [k for k, v in api_keys.items() if v] if not available_providers: print("⚠️ No API keys found. You can still test document processing and vector storage.") print(" To use RAG functionality, set one of: OPENAI_API_KEY, GROQ_API_KEY, ANTHROPIC_API_KEY") return False print(f"✅ API keys available for: {', '.join(available_providers)}") return True def create_sample_data(): """Create sample documents for testing.""" print("\n📝 Creating sample documents...") try: sample_files = create_sample_documents("sample_docs") print(f"✅ Created {len(sample_files)} sample documents:") for file_path in sample_files: print(f" - {Path(file_path).name}") return sample_files except Exception as e: print(f"❌ Failed to create sample documents: {str(e)}") return [] def process_documents(sample_files): """Process the sample documents.""" print("\n🔄 Processing documents...") try: processor = DocumentProcessor(chunk_size=1000, chunk_overlap=200) all_chunks = [] for file_path in sample_files: print(f" Processing {Path(file_path).name}...") chunks = processor.process_file(file_path) all_chunks.extend(chunks) print(f" ✅ Created {len(chunks)} chunks") print(f"✅ Total chunks created: {len(all_chunks)}") return all_chunks except Exception as e: print(f"❌ Failed to process documents: {str(e)}") return [] def setup_vector_store(chunks): """Set up the vector store with processed chunks.""" print("\n🗄️ Setting up vector store...") try: vector_store = VectorStore() vector_store.add_documents(chunks) vector_store.save() stats = vector_store.get_stats() print(f"✅ Vector store created with {stats['total_vectors']} vectors") print(f" Sources: {stats['unique_sources']}") print(f" File types: {list(stats['file_types'].keys())}") return vector_store except Exception as e: print(f"❌ Failed to set up vector store: {str(e)}") return None def test_rag_functionality(vector_store): """Test the RAG functionality.""" print("\n🤖 Testing RAG functionality...") api_keys = validate_api_keys() if not any(api_keys.values()): print("⚠️ Skipping RAG test - no API keys available") return None try: # Initialize RAG engine provider = list(api_keys.keys())[0] model = "gpt-3.5-turbo" if provider == "openai" else "llama2-70b-4096" rag_engine = RAGEngine( vector_store=vector_store, llm_provider=provider, model_name=model, temperature=0.1, max_tokens=500 ) print(f"✅ RAG engine initialized with {provider}") # Test questions test_questions = [ "What is the Documentation Copilot?", "What file types are supported?", "How do I use the system?" ] print("\n🧪 Testing sample questions:") for question in test_questions: print(f"\nQ: {question}") try: result = rag_engine.answer_question(question, k=3, threshold=0.3) print(f"A: {result['answer'][:150]}...") print(f" Sources: {len(result['sources'])}") except Exception as e: print(f" ❌ Error: {str(e)}") return rag_engine except Exception as e: print(f"❌ Failed to test RAG functionality: {str(e)}") return None def show_next_steps(): """Show next steps for the user.""" print("\n🎉 Quick start completed!") print("\n📋 Next Steps:") print("1. 🚀 Start the web interface:") print(" streamlit run app.py") print("\n2. 🤖 Start the Slack bot (optional):") print(" python slack_bot.py") print("\n3. 🧪 Run system tests:") print(" python test_system.py") print("\n4. 📚 Add your own documents:") print(" - Upload files through the web interface") print(" - Process folders with documentation") print(" - Add web URLs or GitHub files") print("\n5. ⚙️ Configure LLM settings:") print(" - Choose your preferred provider") print(" - Adjust model parameters") print(" - Set temperature and token limits") def main(): """Main quick start function.""" print("🚀 Documentation Copilot - Quick Start") print("=" * 50) # Check environment has_api_keys = check_environment() # Create sample data sample_files = create_sample_data() if not sample_files: print("❌ Failed to create sample data. Exiting.") return # Process documents chunks = process_documents [truncated — 540 more characters] ``` ### test_system.py ```python """ Test Script for Documentation Copilot This script tests all major components of the Documentation Copilot system: - Document processing - Vector store operations - RAG engine functionality - Utility functions """ import os import sys from pathlib import Path from dotenv import load_dotenv # Load environment variables load_dotenv() # Import our modules from document_processor import DocumentProcessor from vector_store import VectorStore from rag_engine import RAGEngine from utils import ( validate_api_keys, get_environment_info, create_sample_documents, validate_file_path, validate_url, get_supported_file_extensions ) def test_environment(): """Test environment setup.""" print("🔧 Testing Environment Setup...") # Check Python version print(f"Python version: {sys.version}") # Check environment info env_info = get_environment_info() print(f"Platform: {env_info['platform']}") print(f"Working directory: {env_info['working_directory']}") # Check API keys api_keys = validate_api_keys() print("API Key Status:") for provider, status in api_keys.items(): status_icon = "✅" if status else "❌" print(f" {status_icon} {provider.title()}: {'Available' if status else 'Not Set'}") if not any(api_keys.values()): print("⚠️ Warning: No API keys found. Some tests will be skipped.") print() def test_utilities(): """Test utility functions.""" print("🔧 Testing Utility Functions...") # Test file validation assert validate_file_path(__file__) == True assert validate_file_path("nonexistent_file.txt") == False print("✅ File validation tests passed") # Test URL validation assert validate_url("https://example.com") == True assert validate_url("not_a_url") == False print("✅ URL validation tests passed") # Test supported extensions extensions = get_supported_file_extensions() assert '.md' in extensions assert '.pdf' in extensions print("✅ Supported extensions test passed") print() def test_document_processor(): """Test document processor.""" print("📄 Testing Document Processor...") processor = DocumentProcessor(chunk_size=500, chunk_overlap=100) # Create test content test_content = """# Test Document This is a test document for the Documentation Copilot. ## Features - Document processing - Vector embeddings - RAG implementation ## Usage 1. Upload documents 2. Ask questions 3. Get answers This document contains multiple paragraphs to test chunking functionality. The processor should create appropriate chunks based on the content structure. """ # Test chunk creation chunks = processor._create_chunks(test_content, { 'source': 'test.md', 'type': 'markdown', 'filename': 'test.md' }) assert len(chunks) > 0 print(f"✅ Created {len(chunks)} chunks from test content") # Test chunk statistics stats = processor.get_chunk_stats(chunks) assert stats['total_chunks'] == len(chunks) assert stats['total_tokens'] > 0 print(f"✅ Chunk statistics: {stats['total_chunks']} chunks, {stats['total_tokens']} tokens") print() def test_vector_store(): """Test vector store operations.""" print("🗄️ Testing Vector Store...") # Initialize vector store vector_store = VectorStore(index_path="test_vector_index") # Create test chunks test_chunks = [ { 'content': 'This is a test document about machine learning.', 'metadata': { 'source': 'test1.md', 'type': 'markdown', 'filename': 'test1.md', 'chunk_id': 0, 'chunk_size': 50, 'token_count': 10 } }, { 'content': 'Machine learning is a subset of artificial intelligence.', 'metadata': { 'source': 'test2.md', 'type': 'markdown', 'filename': 'test2.md', 'chunk_id': 0, 'chunk_size': 60, 'token_count': 12 } } ] # Add documents to vector store vector_store.add_documents(test_chunks) print(f"✅ Added {len(test_chunks)} chunks to vector store") # Test search results = vector_store.search("machine learning", k=2, threshold=0.1) assert len(results) > 0 print(f"✅ Search returned {len(results)} results") # Test statistics stats = vector_store.get_stats() assert stats['total_vectors'] == len(test_chunks) print(f"✅ Vector store stats: {stats['total_vectors']} vectors") # Test save and load vector_store.save() print("✅ Vector store saved") # Create new instance and load new_vector_store = VectorStore(index_path="test_vector_index") if new_vector_store.load(): print("✅ Vector store loaded successfully") new_stats = new_vector_store.get_stats() assert new_stats['total_vectors'] == len(test_chunks) else: print("❌ Failed to load vector store") # Clean up import shutil if os.path.exists("test_vector_index"): shutil.rmtree("test_vector_index") print() def test_rag_engine(): """Test RAG engine (requires API key).""" print("🤖 Testing RAG Engine...") api_keys = validate_api_keys() if not any(api_keys.values()): print("⚠️ Skipping RAG engine tests - no API keys available") print() return # Initialize components vector_store = VectorStore(index_path="test_rag_index") # Add test documents test_chunks = [ { 'content': 'The Documentation Copilot is an AI-powered tool for document Q&A.', 'metadata': { 'source': 'docs.md', 'type': 'markdown', 'filename': 'docs.md', [truncated — 3062 more characters] ``` ### utils.py ```python """ Utility Functions This module contains utility functions used across the Documentation Copilot system: - File validation and handling - URL validation - Text processing - Configuration management """ import os import re import tempfile import shutil from pathlib import Path from typing import List, Dict, Any, Optional, Union from urllib.parse import urlparse import zipfile import mimetypes def validate_file_path(file_path: str) -> bool: """ Validate if a file path exists and is accessible. Args: file_path: Path to the file Returns: True if file is valid, False otherwise """ try: path = Path(file_path) return path.exists() and path.is_file() except Exception: return False def validate_folder_path(folder_path: str) -> bool: """ Validate if a folder path exists and is accessible. Args: folder_path: Path to the folder Returns: True if folder is valid, False otherwise """ try: path = Path(folder_path) return path.exists() and path.is_dir() except Exception: return False def validate_url(url: str) -> bool: """ Validate if a URL is properly formatted. Args: url: URL to validate Returns: True if URL is valid, False otherwise """ try: result = urlparse(url) return all([result.scheme, result.netloc]) except Exception: return False def validate_github_url(url: str) -> bool: """ Validate if a URL is a GitHub repository URL. Args: url: URL to validate Returns: True if it's a valid GitHub URL, False otherwise """ if not validate_url(url): return False parsed = urlparse(url) return 'github.com' in parsed.netloc def get_supported_file_extensions() -> List[str]: """ Get list of supported file extensions. Returns: List of supported file extensions """ return ['.md', '.pdf', '.docx', '.txt'] def is_supported_file(file_path: str) -> bool: """ Check if a file is supported by the system. Args: file_path: Path to the file Returns: True if file is supported, False otherwise """ if not validate_file_path(file_path): return False file_extension = Path(file_path).suffix.lower() return file_extension in get_supported_file_extensions() def get_file_size_mb(file_path: str) -> float: """ Get file size in megabytes. Args: file_path: Path to the file Returns: File size in MB """ try: size_bytes = os.path.getsize(file_path) return size_bytes / (1024 * 1024) except Exception: return 0.0 def sanitize_filename(filename: str) -> str: """ Sanitize a filename by removing or replacing invalid characters. Args: filename: Original filename Returns: Sanitized filename """ # Remove or replace invalid characters sanitized = re.sub(r'[<>:"/\\|?*]', '_', filename) # Remove leading/trailing spaces and dots sanitized = sanitized.strip('. ') # Limit length if len(sanitized) > 255: sanitized = sanitized[:255] return sanitized def create_temp_directory() -> str: """ Create a temporary directory for file processing. Returns: Path to the temporary directory """ temp_dir = tempfile.mkdtemp(prefix="doc_copilot_") return temp_dir def cleanup_temp_directory(temp_dir: str) -> None: """ Clean up a temporary directory. Args: temp_dir: Path to the temporary directory """ try: if os.path.exists(temp_dir): shutil.rmtree(temp_dir) except Exception as e: print(f"Warning: Could not clean up temp directory {temp_dir}: {e}") def extract_zip_file(zip_path: str, extract_to: str) -> List[str]: """ Extract a ZIP file and return list of extracted file paths. Args: zip_path: Path to the ZIP file extract_to: Directory to extract to Returns: List of extracted file paths """ extracted_files = [] try: with zipfile.ZipFile(zip_path, 'r') as zip_ref: zip_ref.extractall(extract_to) for root, dirs, files in os.walk(extract_to): for file in files: file_path = os.path.join(root, file) if is_supported_file(file_path): extracted_files.append(file_path) except Exception as e: print(f"Error extracting ZIP file: {e}") return extracted_files def get_file_type_info(file_path: str) -> Dict[str, Any]: """ Get information about a file type. Args: file_path: Path to the file Returns: Dictionary with file type information """ file_path = Path(file_path) if not file_path.exists(): return {'error': 'File not found'} file_info = { 'name': file_path.name, 'extension': file_path.suffix.lower(), 'size_mb': get_file_size_mb(str(file_path)), 'is_supported': is_supported_file(str(file_path)), 'mime_type': mimetypes.guess_type(str(file_path))[0] } return file_info def format_file_size(size_bytes: int) -> str: """ Format file size in human-readable format. Args: size_bytes: Size in bytes Returns: Formatted size string """ if size_bytes == 0: return "0 B" size_names = ["B", "KB", "MB", "GB", "TB"] i = 0 while size_bytes >= 1024 and i < len(size_names) - 1: size_bytes /= 1024.0 i += 1 return f"{size_bytes:.1f} {size_names[i]}" def count_tokens(text: str) -> int: """ Count the number of tokens in text using tiktoken. [truncated — 4944 more characters] ``` [4 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]