# Project export: Open Record This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Journalism’s first end-to-end automated research assistant - Devpost: https://devpost.com/software/fairplay-ai - GitHub: https://github.com/niralishahh/calhackssu25 - Video: https://www.youtube.com/embed/rUDcYAdE3f8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — niralishahh (4 commits), Garima Upadhyay (1 commits) ## Devpost submission (written by the team) ### Inspiration Open Record was inspired by the growing crisis in local journalism. While national outlets continue to expand, smaller newspapers and student-run publications are shutting down or burning out—struggling to keep up with the overwhelming volume of public information they’re expected to monitor. Since 2005, over 2,500 local newspapers in the U.S. have closed. One of the biggest bottlenecks is the time and effort required to review and analyze public meeting recordings, budget documents, and long transcripts. We wanted to build a tool that not only helps journalists read and summarize content—but actually helps them uncover the stories that matter. ### What it does Open Record is more than just a chatbot; it’s a proactive research assistant designed specifically for local journalists. Users can upload audio or video files, such as city council meetings, select one or more recordings, and ask complex questions. The system can summarize discussions, identify key issues, pull direct quotes with timestamps, and even analyze trends across multiple meetings. Journalists can also save important chatbot messages for easy reference. But Open Record goes beyond reactive Q&A: it’s a system that finds the questions worth asking. Instead of spending hours combing through transcripts, journalists are handed the most relevant insights automatically. ### How we built it We used the Groq Whisper API to transcribe city council recordings with high accuracy and speaker diarization. These transcripts were then segmented and embedded using Google Cloud’s Vertex AI Embeddings API. We stored those embeddings in Supabase, which served as our vector database for efficient retrieval. User queries are handled through a Retrieval-Augmented Generation (RAG) system powered by Anthropic’s Claude 3, which generates contextual answers grounded in the retrieved transcript chunks. This full-stack pipeline allows for fast, accurate, and traceable responses across large civic datasets. ### Challenges we ran into We encountered several technical challenges throughout development. One major hurdle was orchestrating a seamless workflow across multiple services: transcription, vector embedding, storage, and LLM querying all had to be coordinated carefully to function as a unified system. We also ran into difficulties with prompt engineering—specifically, how to guide Claude 3 to stay grounded in the retrieved transcript data. We had to experiment with different chunking strategies to balance semantic accuracy with latency and cost, especially when working with long, multi-speaker transcripts. Finally, ensuring speaker attribution remained accurate through diarization and preprocessing was a nuanced problem that required fine-tuning. ### Accomplishments we're proud of We’re proud to have built a fully functional, end-to-end RAG system capable of operating across multiple long-form documents and audio files. Open Record handles messy, real-world civic content and delivers structured, journalist-ready insights. Along the way, we gained experience working with cutting-edge tools like Groq, Claude, Supabase, and Vertex AI—all while solving a problem with real social impact. ### What we learned Building Open Record gave us a deep, hands-on understanding of Retrieval-Augmented Generation systems—from embedding generation and vector storage to prompt tuning and output evaluation. We learned how to preprocess and clean noisy, multi-speaker data for better semantic results. We explored the trade-offs between chunk size, latency, and answer quality in a RAG pipeline. Most importantly, we learned how to design an AI system not just for accuracy, but for usefulness—delivering insights that real people, like student journalists, can act on. ### What's next Next, we plan to integrate semantic search so users can explore all their previously uploaded files by topic. We also hope to incorporate Google Document AI to analyze structured documents like city budgets, giving users an even broader view of public decision-making. Ultimately, our vision is to build a fully automated pipeline that ingests new public meeting data overnight and delivers a tailored morning briefing—helping local journalists cover more ground with less effort, and empowering communities to hold their governments accountable. ## README (from the GitHub repository) # 🤖 Claude AI Research Agent An intelligent AI research agent powered by Anthropic's Claude that uses advanced planning and tool selection to analyze documents, extract direct quotes with page numbers, and answer questions through natural conversation. ## ✨ Features - **🧠 Intelligent Planning**: AI agent that plans and chooses the best tools for each request - **🔧 Tool-Based Architecture**: Specialized tools for different types of analysis - **📄 Direct Quote Extraction**: Extract exact quotes with page numbers for citations - **📋 Multiple Summary Types**: Comprehensive, bullet points, and executive summaries - **🔍 Contextual Analysis**: Answer questions with deep document understanding - **📊 Batch Processing**: Process multiple documents simultaneously - **📝 Text Analysis**: Direct text input and analysis - **💬 Natural Conversation**: Upload a document and ask questions naturally - **📚 PDF Support**: Extract and analyze text from PDF documents - **🔄 Multi-Format Support**: PDF, TXT, MD, PY, JS, HTML, CSS, JSON, XML, CSV - **🎨 Modern Web Interface**: Beautiful, responsive UI with real-time feedback - **🔌 RESTful API**: Full API for integration with other applications - **⚡ Claude AI Integration**: Leverages Anthropic's Claude for intelligent document processing ## 🏗️ Architecture ``` calhackssu25/ ├── backend/ │ └── claude/ │ ├── claude.py # Main AI agent implementation │ ├── app.py # Flask API server │ └── example.py # Usage examples ├── app/ │ └── index.html # Web interface ├── requirements.txt # Python dependencies ├── start.sh # Startup script └── README.md # This file ``` ## 🤖 AI Agent Capabilities This system functions as an intelligent AI research agent with advanced planning and tool selection capabilities: ### 🧠 Intelligent Planning System The AI agent automatically plans and executes the best approach for each request: - **Tool Selection**: Chooses the most appropriate tools based on user intent - **Multi-Step Planning**: Can combine multiple tools for complex requests - **Context Awareness**: Considers document type and content when planning - **Fallback Logic**: Graceful degradation when planning fails ### 🔧 Available Tools The agent has access to specialized tools for different tasks: #### 📄 Document Summarization Tool - **Purpose**: Generate comprehensive summaries with key points - **Input**: Document path and summary type (comprehensive/bullet_points/executive) - **Output**: Structured summary with analysis #### 🗣️ Quote Extraction Tool - **Purpose**: Extract direct quotes with page numbers and context - **Input**: Document path and specific query - **Output**: Formatted quotes with page numbers, speakers, and relevance #### ❓ Question Answering Tool - **Purpose**: Answer specific questions about document content - **Input**: Document path and question - **Output**: Contextual answer with document references #### 📊 Text Analysis Tool - **Purpose**: Analyze raw text content for various insights - **Input**: Text content and analysis type (summary/key_points/sentiment/topics) - **Output**: Structured analysis results ### 🔍 Intelligent Analysis The AI agent can: - **Plan Complex Requests**: Automatically determine which tools to use - **Combine Multiple Tools**: Execute multi-step analyses when needed - **Adapt to Context**: Choose different approaches based on document type - **Provide Reasoning**: Explain why certain tools were chosen - **Handle Ambiguity**: Use fallback strategies when planning is uncertain ### 💡 Natural Language Understanding Users can ask questions naturally: - "Summarize this document" - "What are the main topics discussed?" - "Give me quotes about budget discussions" - "Who were the key speakers and what did they say?" - "Analyze the decision-making process" The agent will automatically: 1. **Analyze the request** to understand user intent 2. **Plan the execution** by selecting appropriate tools 3. **Execute the plan** using the chosen tools 4. **Combine results** into a coherent response ## 🚀 Quick Start ### Prerequisites 1. **Python 3.8+** 2. **Anthropic API Key** - Get one from [Anthropic Console](https://console.anthropic.com/) ### Installation 1. **Clone the repository** ```bash git clone cd calhackssu25 ``` 2. **Create and activate virtual environment** ```bash python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` 3. **Install dependencies** ```bash pip install -r requirements.txt ``` 4. **Set up environment variables** ```bash export ANTHROPIC_API_KEY="your-anthropic-api-key" ``` Or create a `.env` file: ```env ANTHROPIC_API_KEY=your-anthropic-api-key ``` ### Running the Application #### Option 1: Using the startup script (Recommended) ```bash chmod +x start.sh ./start.sh ``` #### Option 2: Manual startup 1. **Start the backend server** ```bash cd backend/claude python app.py ``` The API server will start on `http://localhost:5002` 2. **Open the web interface** Open `app/index.html` in your browser or serve it with a local server: ```bash cd app python -m http.server 8000 ``` Then visit `http://localhost:8000` ## 📖 Usage ### Web Interface The web interface provides three main modes: 1. **Single Document**: Upload and summarize a single document 2. **Batch Processing**: Process multiple documents at once 3. **Direct Text**: Paste text directly for analysis ### API Endpoints #### Health Check ```bash GET http://localhost:5002/health ``` #### Get Agent Status ```bash GET http://localhost:5002/status ``` #### Summarize Single Document ```bash POST http://localhost:5002/summarize Content-Type: application/json { "file_path": "/path/to/document.pdf", "summary_type": "comprehensive" } ``` #### Batch Summarize Documents ```bash POST http://localhost:5002/batch-summarize Content-Type: application/json { "file_paths": ["/path/to/doc1.pdf", "/path/to/doc2.txt"], "summary_type": "bullet_points" } ``` #### Analyze Text ```bash POST http://localhost:5003/analyze Content-Type: application/json { "text": "Your text content here...", "summary_type": "executive" } ``` #### Ask Questions About Document ```bash POST http://localhost:5003/ask Content-Type: application/json { "file_path": "/path/to/document.pdf", "question": "What was the general consensus on the housing budget?" } ``` ### Python API ```python from backend.claude.claude import ClaudeAgent # Initialize the agent agent = ClaudeAgent() # Summarize a PDF document result = agent.summarize_document("path/to/document.pdf", "comprehensive") print(f"Summary: {result.summary}") print(f"Key points: {result.key_points}") # Batch summarize multiple documents (mix of formats) results = agent.batch_summarize(["doc1.pdf", "doc2.txt", "doc3.md"], "bullet_points") for result in results: print(f"Document summary: {result.summary}") ``` ## 🔧 Configuration ### Summary Types - **comprehensive**: Detailed summary with main ideas, supporting information, and context - **bullet_points**: Concise bullet-point format for quick scanning - **executive**: High-level summary suitable for executive review ### Supported File Types - **`.pdf`** - PDF documents (requires PyPDF2 and pdfplumber) - **`.txt`** - Plain text files - **`.md`** - Markdown files - **`.py`** - Python files - **`.js`** - JavaScript files - **`.html`** - HTML files - **`.css`** - CSS files - **`.json`** - JSON files - **`.xml`** - XML files - **`.csv`** - CSV files ### PDF Processing The application uses two libraries for PDF processing: - **pdfplumber**: Primary PDF text extraction (better quality) - **PyPDF2**: Fallback PDF processing PDF files are processed page by page, with page numbers included in the extracted text for better context. ## 🛠️ Development ### Running Examples [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 38 recognized source files, 346 KB. - Anthropic (technology) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code ## Codebase structure (from repository index) ### Files (43 of 43) ``` .gitignore backend/.DS_Store backend/claude/app.py backend/claude/claude.py backend/claude/clear_documents.py backend/claude/example.py backend/claude/fix_schema.py backend/claude/rag_agent.py backend/claude/rag_app.py backend/claude/run_sql_setup.py backend/claude/supabase_setup.sql calhackssu25-9b150c2b63a2.json mycode/app/chat_interface.html mycode/app/index.html mycode/app/view_response.html mycode/app/view_transcription.html mycode/backend/api_server.py mycode/backend/audio_chunker.py mycode/backend/chat_client.py mycode/backend/example_usage.py mycode/backend/inspect_chunks.py mycode/backend/migration_add_saved_feature.sql mycode/backend/rag_handler.py mycode/backend/supabase_client.py mycode/backend/supabase_setup.sql mycode/backend/test_chunking.py mycode/backend/test_rag_components.py mycode/backend/transcription_agent.py mycode/CHAT_SETUP.md mycode/README.md mycode/requirements.txt mycode/supabase_chat_schema.sql mycode/test_supabase.py mycode/unified_schema.sql RAG_README.md README.md requirements.txt start_all.sh start_rag.sh start.sh test_agentic.py test_server.py test_vertexai.py ``` ### Dependencies - mycode/requirements.txt: aiofiles@>=23.0.0, anthropic@>=0.25.0, fastapi@>=0.115.0, google-auth@>=2.23.4, google-cloud-aiplatform@>=1.38.1, google-cloud-storage@>=2.10.0, google-generativeai@>=0.3.0, groq@>=0.4.0, numpy@>=1.20.0, pdfplumber@>=0.10.3, pydantic@>=2.0.0, PyPDF2@>=3.0.1, python-dotenv, python-multipart@>=0.0.6, requests@>=2.31.0, supabase@>=2.15.0, typing-extensions@>=4.7.0, uuid@>=1.30, uvicorn@>=0.24.0 - requirements.txt: anthropic@>=0.25.0, fastapi@==0.104.1, google-auth@>=2.23.4, google-cloud-aiplatform@>=1.38.1, google-cloud-storage@>=2.10.0, numpy, pdfplumber@==0.10.3, pydantic@>=2.0.0, PyPDF2@==3.0.1, python-dotenv@>=1.0.0, python-multipart@==0.0.6, requests@>=2.31.0, supabase@>=2.15.0, typing-extensions@>=4.7.0, uuid@==1.30, uvicorn@==0.24.0 ### Recent commits (newest first) - final version (in mycode) - garimas code bfr merging - got rag working - hello - setup - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### RAG_README.md ```markdown # Vertex AI RAG Document Analysis System A scalable Retrieval-Augmented Generation (RAG) system built with Google Vertex AI, Supabase, and Anthropic Claude for efficient document analysis and question answering. ## 🚀 Features - **Scalable Architecture**: Handles large documents by chunking them into smaller pieces - **Vector Search**: Uses Google Vertex AI embeddings for semantic similarity search - **Multi-Document Support**: Query across multiple documents simultaneously - **Real-time Processing**: Fast response times with optimized vector search - **PDF Support**: Full PDF text extraction and processing - **Modern UI**: Clean, responsive web interface - **Google Cloud Integration**: Leverages Vertex AI's powerful embedding models ## 🏗️ Architecture ### Phase 1: Document Ingestion 1. **File Upload**: Users upload documents through the web interface 2. **Text Extraction**: Documents are processed to extract text content 3. **Chunking**: Text is split into 300-word chunks with 50-word overlap 4. **Embedding Generation**: Each chunk is converted to a vector using Vertex AI 5. **Database Storage**: Chunks and embeddings are stored in Supabase with pgvector ### Phase 2: Query Processing 1. **Query Input**: Users ask questions and select relevant documents 2. **Query Embedding**: The question is converted to a vector using Vertex AI 3. **Semantic Search**: Vector similarity search finds relevant chunks 4. **Context Assembly**: Top 8 most relevant chunks are combined 5. **AI Response**: Claude generates answers based on the retrieved context ## 🛠️ Tech Stack - **Backend**: FastAPI (Python) - **Database**: Supabase (PostgreSQL with pgvector) - **Embeddings**: Google Vertex AI Text Embedding API - **LLM**: Anthropic Claude 3 Haiku - **Frontend**: HTML/CSS/JavaScript - **PDF Processing**: pdfplumber, PyPDF2 - **Cloud Platform**: Google Cloud Platform ## 📋 Prerequisites 1. **Supabase Account**: Create a project and get your URL and service role key 2. **Google Cloud Account**: Set up a project and enable Vertex AI API 3. **Anthropic Account**: Get API key for Claude access 4. **Python 3.8+**: For running the backend 5. **Google Cloud Authentication**: Service account key or gcloud auth ## 🚀 Quick Start ### 1. Environment Setup Set the required environment variables: ```bash export ANTHROPIC_API_KEY="your-anthropic-api-key" export SUPABASE_URL="your-supabase-project-url" export SUPABASE_SERVICE_ROLE_KEY="your-supabase-service-role-key" export GOOGLE_CLOUD_PROJECT_ID="your-google-cloud-project-id" export GOOGLE_CLOUD_LOCATION="us-central1" # Optional, defaults to us-central1 export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account-key.json" # Optional if using gcloud auth ``` ### 2. Google Cloud Setup 1. **Enable APIs**: Enable the Vertex AI API in your Google Cloud project 2. **Authentication**: Either: - Use service account: `export GOOGLE_APPLICATION_CREDENTIALS="path/to/key.json"` - Or use gcloud: `gcloud auth applicat [truncated — 4986 more characters] ``` ### mycode/CHAT_SETUP.md ```markdown # Chat System Setup Guide This guide explains how to set up the chat system for the AI Transcription Agent with user-specific chats, responses, and file management. ## Database Schema Setup ### 1. Run the SQL Schema First, execute the SQL schema in your Supabase database. You can do this through the Supabase dashboard: 1. Go to your Supabase project dashboard 2. Navigate to the SQL Editor 3. Copy and paste the contents of `supabase_chat_schema.sql` 4. Execute the script This will create: - `users` table (if not exists) - `chats` table for managing user conversations - `chat_responses` table for storing AI responses - `chat_files` table for associating files with chats - Updated `transcriptions` table with user_id field - Row Level Security (RLS) policies - Helper functions for common operations ### 2. Environment Variables Make sure your `.env` file includes: ```env SUPABASE_URL=your_supabase_url SUPABASE_ANON_KEY=your_supabase_anon_key GOOGLE_API_KEY=your_google_api_key GROQ_API_KEY=your_groq_api_key ``` ## Backend Setup ### 1. Install Dependencies The chat system uses the existing dependencies plus the new chat client: ```bash pip install -r requirements.txt ``` ### 2. Start the Backend ```bash cd backend python api_server.py ``` The server will run on `http://localhost:8000` and includes all the new chat endpoints. ## Frontend Setup ### 1. Start the Frontend Server ```bash cd app python -m http.server 3000 ``` ### 2. Access the Chat Interface Open your browser and go to: - **Original Interface**: `http://localhost:3000/index.html` - **New Chat Interface**: `http://localhost:3000/chat_interface.html` ## Chat System Features ### Three-Panel Layout 1. **Left Panel - Saved Responses** - Shows chat history for the current conversation - Click on any response to view it in the chat - Responses are automatically saved when you chat 2. **Middle Panel - Chat Interface** - Real-time chat with AI about your files - Requires files to be selected before chatting - Auto-saves all conversations 3. **Right Panel - File Management** - Upload new audio/video files - View all files uploaded by the current user - Add files to the current chat conversation - View transcription details ### User Management Currently, the system uses a demo user ID (`demo-user-123`). In a production environment, you would: 1. Implement proper authentication (e.g., Supabase Auth) 2. Replace the hardcoded `currentUserId` with the actual authenticated user's ID 3. Update the RLS policies to work with Supabase Auth ### File Operations - **Upload**: Files are automatically transcribed and stored with the user ID - **Add to Chat**: Files can be added to specific chat conversations - **View**: Click "View" to see detailed transcription with segments and timestamps - **Remove**: Files can be removed from chat conversations ### Chat Features - **Automatic Chat Creation**: A new chat is created when you first load the interface - * [truncated — 3914 more characters] ``` ### requirements.txt ``` fastapi==0.104.1 uvicorn==0.24.0 anthropic>=0.25.0 python-multipart==0.0.6 pdfplumber==0.10.3 PyPDF2==3.0.1 supabase>=2.15.0 google-cloud-aiplatform>=1.38.1 google-cloud-storage>=2.10.0 google-auth>=2.23.4 python-dotenv>=1.0.0 numpy pydantic>=2.0.0 typing-extensions>=4.7.0 requests>=2.31.0 uuid==1.30 ``` ### mycode/requirements.txt ``` # Combined requirements for Chat and RAG functionality # FastAPI and web server fastapi>=0.115.0 uvicorn>=0.24.0 python-multipart>=0.0.6 # AI and LLM SDKs google-generativeai>=0.3.0 google-cloud-aiplatform>=1.38.1 groq>=0.4.0 anthropic>=0.25.0 # Database and Cloud supabase>=2.15.0 google-cloud-storage>=2.10.0 google-auth>=2.23.4 # PDF and File Handling pdfplumber>=0.10.3 PyPDF2>=3.0.1 aiofiles>=23.0.0 # Utilities python-dotenv>=1.0.0 pydantic>=2.0.0 numpy>=1.20.0 requests>=2.31.0 uuid>=1.30 typing-extensions>=4.7.0 python-dotenv ``` ### backend/claude/app.py ```python from fastapi import FastAPI, File, UploadFile, HTTPException, Form from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse from pydantic import BaseModel import os import logging from dotenv import load_dotenv from claude import ClaudeAgent, DocumentSummary import traceback import tempfile from typing import List, Optional import uvicorn from contextlib import asynccontextmanager import time from pathlib import Path # Load environment variables load_dotenv() # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Global agent instance agent = None # Configure upload settings UPLOAD_FOLDER = "uploads" ALLOWED_EXTENSIONS = {'.txt', '.md', '.py', '.js', '.html', '.css', '.json', '.xml', '.csv', '.pdf'} # Get the absolute path to the uploads directory SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) UPLOAD_FOLDER_ABS = os.path.join(SCRIPT_DIR, UPLOAD_FOLDER) # Ensure uploads directory exists os.makedirs(UPLOAD_FOLDER_ABS, exist_ok=True) def allowed_file(filename: str) -> bool: """Check if file extension is allowed""" if not filename: return False return Path(filename).suffix.lower() in ALLOWED_EXTENSIONS def initialize_agent(): """Initialize the Claude agent""" global agent try: agent = ClaudeAgent() logger.info("Claude agent initialized successfully") return True except Exception as e: logger.error(f"Failed to initialize agent: {e}") return False @asynccontextmanager async def lifespan(app: FastAPI): """Lifespan event handler for FastAPI""" # Startup initialize_agent() yield # Shutdown logger.info("Shutting down FastAPI server...") # Pydantic models for request bodies class SummarizeRequest(BaseModel): file_path: str summary_type: str = "comprehensive" class BatchSummarizeRequest(BaseModel): file_paths: List[str] summary_type: str = "comprehensive" class AnalyzeRequest(BaseModel): text: str summary_type: str = "comprehensive" class DocumentQARequest(BaseModel): file_path: str question: str class QuoteExtractionRequest(BaseModel): file_path: str query: str class AgenticRequest(BaseModel): file_path: str request: str # Initialize FastAPI app app = FastAPI( title="Claude AI Document Summarizer", description="An intelligent AI agent that uses Claude to summarize documents", version="1.0.0", lifespan=lifespan ) # Add CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["*"], # In production, specify your frontend domain allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) @app.get("/health") async def health_check(): """Health check endpoint""" return { "status": "healthy", "agent_initialized": agent is not None } @app.get("/status") async def get_status(): """Get agent status""" if agent is None: raise HTTPException(status_code=500, detail="Agent not initialized") try: status = agent.get_agent_status() return status except Exception as e: logger.error(f"Error getting status: {e}") raise HTTPException(status_code=500, detail=str(e)) @app.post("/upload") async def upload_file(file: UploadFile = File(...)): """Upload a single file""" if agent is None: raise HTTPException(status_code=500, detail="Agent not initialized") try: # Check if file is present if not file: raise HTTPException(status_code=400, detail="No file provided") if not file.filename or file.filename == '': raise HTTPException(status_code=400, detail="No file selected") if not allowed_file(file.filename): raise HTTPException( status_code=400, detail=f"File type not allowed. Allowed types: {', '.join(ALLOWED_EXTENSIONS)}" ) # Save file filename = file.filename file_path = os.path.join(UPLOAD_FOLDER_ABS, filename) with open(file_path, "wb") as buffer: content = await file.read() buffer.write(content) logger.info(f"File uploaded: {file_path}") return { "message": "File uploaded successfully", "file_path": file_path, "filename": filename } except HTTPException: raise except Exception as e: logger.error(f"Error uploading file: {e}") raise HTTPException(status_code=500, detail=str(e)) @app.post("/upload-batch") async def upload_batch(files: List[UploadFile] = File(...)): """Upload multiple files""" if agent is None: raise HTTPException(status_code=500, detail="Agent not initialized") try: # Check if files are present if not files: raise HTTPException(status_code=400, detail="No files provided") uploaded_files = [] for file in files: if file and file.filename and allowed_file(file.filename): filename = file.filename file_path = os.path.join(UPLOAD_FOLDER_ABS, filename) with open(file_path, "wb") as buffer: content = await file.read() buffer.write(content) uploaded_files.append(file_path) logger.info(f"File uploaded: {file_path}") else: logger.warning(f"Skipping invalid file: {file.filename if file else 'None'}") if not uploaded_files: raise HTTPException(status_code=400, detail="No valid files uploaded") return { "message": f"{len(uploaded_files)} files uploaded successfully", "file_paths": uploaded_files } except HTTPException: [truncated — 14780 more characters] ``` ### test_agentic.py ```python #!/usr/bin/env python3 """ Test script for the agentic AI system """ import requests import json def test_agentic_execution(): """Test the agentic execution endpoint""" base_url = "http://localhost:5003" # Test 1: Check if server is running try: response = requests.get(f"{base_url}/health") print(f"Health check: {response.status_code}") print(f"Response: {response.json()}") except Exception as e: print(f"Health check failed: {e}") return # Test 2: Test agentic text analysis try: response = requests.post(f"{base_url}/test-agentic") print(f"\nAgentic test: {response.status_code}") print(f"Response: {json.dumps(response.json(), indent=2)}") except Exception as e: print(f"Agentic test failed: {e}") # Test 3: Test with a simple question (if you have a test file) # This would require uploading a file first print("\nTo test with a document, upload a file and then use the agentic-execute endpoint") if __name__ == "__main__": test_agentic_execution() ``` ### start_all.sh ```shell #!/bin/bash # Get the directory of the script SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" PROJECT_DIR="$SCRIPT_DIR" # Activate virtual environment VENV_PATH="$PROJECT_DIR/venv/bin/activate" if [ -f "$VENV_PATH" ]; then source "$VENV_PATH" echo "Virtual environment activated." else echo "Warning: Virtual environment not found at $VENV_PATH. Make sure to set it up." fi # Start the FastAPI backend server echo "--- Starting FastAPI Backend on port 8000 ---" osascript </dev/null kill $FRONTEND_PID 2>/dev/null echo "✅ Servers stopped" exit 0 } # Set up signal handlers trap cleanup SIGINT SIGTERM # Wait for background processes wait ``` ### start_rag.sh ```shell #!/bin/bash echo "🚀 Starting Vertex AI RAG Document Analysis System..." # Check if we're in the right directory if [ ! -f "backend/claude/rag_app.py" ]; then echo "❌ Error: Please run this script from the project root directory" exit 1 fi # Check if required environment variables are set if [ -z "$ANTHROPIC_API_KEY" ]; then echo "❌ Error: ANTHROPIC_API_KEY environment variable is required" echo "Please set it with: export ANTHROPIC_API_KEY='your-key-here'" exit 1 fi if [ -z "$SUPABASE_URL" ]; then echo "❌ Error: SUPABASE_URL environment variable is required" echo "Please set it with: export SUPABASE_URL='your-supabase-url'" exit 1 fi if [ -z "$SUPABASE_SERVICE_ROLE_KEY" ]; then echo "❌ Error: SUPABASE_SERVICE_ROLE_KEY environment variable is required" echo "Please set it with: export SUPABASE_SERVICE_ROLE_KEY='your-service-role-key'" exit 1 fi if [ -z "$GOOGLE_CLOUD_PROJECT_ID" ]; then echo "❌ Error: GOOGLE_CLOUD_PROJECT_ID environment variable is required" echo "Please set it with: export GOOGLE_CLOUD_PROJECT_ID='your-project-id'" exit 1 fi # Check if Google Cloud credentials are set if [ -z "$GOOGLE_APPLICATION_CREDENTIALS" ]; then echo "⚠️ Warning: GOOGLE_APPLICATION_CREDENTIALS not set" echo "Please set it with: export GOOGLE_APPLICATION_CREDENTIALS='path/to/service-account-key.json'" echo "Or use: gcloud auth application-default login" fi echo "✅ Environment variables are set" # Install dependencies if needed echo "📦 Checking dependencies..." pip install -r requirements.txt # Create uploads directory if it doesn't exist mkdir -p backend/claude/uploads # Start the RAG server echo "🌐 Starting Vertex AI RAG server on port 5005..." cd backend/claude python rag_app.py ``` ### test_server.py ```python #!/usr/bin/env python3 """ Simple test script to verify the FastAPI server """ import requests import json import time def test_server(): base_url = "http://localhost:5003" print("Testing FastAPI server...") # Test health endpoint try: response = requests.get(f"{base_url}/health") print(f"Health check: {response.status_code}") if response.status_code == 200: print(f"Response: {response.json()}") else: print(f"Error: {response.text}") except Exception as e: print(f"Health check failed: {e}") return False # Test status endpoint try: response = requests.get(f"{base_url}/status") print(f"Status check: {response.status_code}") if response.status_code == 200: print(f"Response: {response.json()}") else: print(f"Error: {response.text}") except Exception as e: print(f"Status check failed: {e}") return False # Test upload endpoint with a simple text file try: # Create a simple test file with open("test.txt", "w") as f: f.write("This is a test document for summarization.") with open("test.txt", "rb") as f: files = {"file": f} response = requests.post(f"{base_url}/upload", files=files) print(f"Upload test: {response.status_code}") if response.status_code == 200: upload_data = response.json() print(f"Upload response: {upload_data}") # Test summarize endpoint summarize_data = { "file_path": upload_data["file_path"], "summary_type": "comprehensive" } response = requests.post( f"{base_url}/summarize", headers={"Content-Type": "application/json"}, data=json.dumps(summarize_data) ) print(f"Summarize test: {response.status_code}") if response.status_code == 200: print("✅ Summarization successful!") result = response.json() print(f"Summary: {result['summary'][:100]}...") else: print(f"❌ Summarization failed: {response.text}") return False else: print(f"Upload failed: {response.text}") return False except Exception as e: print(f"Upload/summarize test failed: {e}") return False return True if __name__ == "__main__": success = test_server() if success: print("\n✅ All tests passed!") else: print("\n❌ Some tests failed!") ``` [28 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]