# Project export: MonitorMachine
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 12.0
- Tagline: Automating clinical trial verification to cut costs, save time, bring life-saving treatments to patients faster, and spark more innovation in clinical research.
- Devpost: https://devpost.com/software/monitormachine
- GitHub: https://github.com/sulabhkatila/MonitorMachine
- Video: https://www.youtube.com/embed/BOXuzT-mpNw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Regeneron: Honorable Mention)
- Team: 1 GitHub contributor(s) — Sulabh Katila (40 commits)
## Devpost submission (written by the team)
### Inspiration
Clinical trials are the bridge between scientific discovery and real-world cures -- but that bridge, currently, is slow, fragile, and expensive. While studying how trials operate, I was struck by how much time and money go into one repetitive task: verifying and reviewing data by hand. Monitors spend weeks cross-checking PDFs, lab reports, and CRFs. It a process that consumes up to 30% of a trial’s cost and delays life-saving medicines from reaching patients. I wanted to change that. My inspiration was simple: what if an AI agent could handle this repetitive verification work -- allowing researchers to focus on science, safety, and innovation instead? By automating Source Data Verification and Review, I aim to help trials move faster, cut costs, save more lives, and invite more innovation into the clinical space -- so that life-saving treatments reach people sooner.
### What it does
This platform automates and simplifies clinical trial monitoring by connecting directly to the technologies already used in research — such as Veeva Vault, Medidata Rave, and EHR/EDC systems — eliminating the need to copy or migrate data across tools. It includes a Protocol Analyzer, powered by Fetch.ai and Gemini, which reads the clinical trial protocol and automatically generates a Monitoring Plan based on the study’s endpoints, schedule of activities, and key compliance requirements. Once attached to a study, this plan enables autonomous monitoring by AI agents that follow the defined schedule, retrieve data, and execute tasks automatically. The platform also supports Source Data Verification (SDV) on demand — extracting relevant information from CRFs, eCRFs, or connected EHR/EDC systems, gaining access to source documents, and cross-referencing the two to verify accuracy. When discrepancies are detected, it flags them for review and can take predefined corrective actions — providing faster, smarter, and more integrated clinical trial monitoring.
### How we built it
I built the platform from the ground up as a full-stack web application designed to integrate seamlessly with existing clinical-trial systems. The frontend was built with React.js, CSS, and JavaScript, giving it a modern UI and smooth transitions. The login and sign-up flow includes both standard authentication and SSO integrations with Google, Veeva Vault, and Medidata Rave (mocks), enabling secure, role-based access for sponsors and investigators. The backend uses Flask (Python) for the API layer, with REST endpoints that manage user sessions, study creation, and file intake. Each uploaded or fetched file—protocols, eSource PDFs, and CRFs—is stored securely in a database (ChromaDB). I integrated Fetch.ai agents and Gemini and Claude models to power the Protocol Analyzer. The analyzer parses trial protocols, identifies endpoints, study arms, inclusion/exclusion criteria, and the schedule of activities, then automatically generates a Monitoring Plan tailored to the study design. Along with this I built a monitor agent that is fluent in clinical trail processes, protocols, addressing schedule of events, and more. Once attached to a study, the monitoring plan serves as a guide for autonomous AI agents. These agents connect to linked EDC or EHR systems through APIs, retrieve CRF and source data, and perform Source Data Verification when triggered. The verification engine uses a mix of document parsing, entity matching, and cross-referencing logic to detect inconsistencies between eCRFs and source data. When discrepancies are found, the system flags them in real time and can trigger notifications or corrective actions, all recorded in an audit trail.
### Challenges we ran into
One of the biggest challenges I faced was finding usable data. Real clinical trial data is highly protected for privacy and compliance reasons, and the public examples I found were incomplete, inconsistent, or missing key components like full protocols, CRFs, or source documents. To move forward, I had to study what “complete” datasets look like in real trials — learning the structure and expectations for each type of document: protocols, CRFs, eCRFs, source data, and monitoring reports. From there, I handcrafted realistic mock studies from scratch — writing detailed protocols, building CRF templates, and fabricating example source data that reflected how actual investigators document patient visits. This process gave me a much deeper understanding of clinical data standards and what monitors actually look for, which shaped how I designed the platform’s data models, extraction logic, and AI validation pipeline.
### Accomplishments we're proud of
I built a fully functional AI agent capable of autonomously understanding clinical trial protocols, generating monitoring plans, and verifying study data against its sources. I designed a complete workflow that connects directly to existing EDC platforms like Veeva Vault and Medidata Rave, eliminating the need to manually move data between systems — a real bottleneck in current clinical operations. I also developed a Protocol Analyzer and Monitor agnet that interprets endpoints, inclusion criteria, and schedules of activities, translating them into actionable monitoring tasks for AI agents. The system demonstrates how agentic AI can lead to a future where clinical trials are faster, more connected, and more efficient — cutting costs, reducing human error, and accelerating the delivery of life-saving treatments.
### What we learned
I learned how complex and interconnected the clinical trial process really is — and how much of it depends on accurate, traceable data. Even small verification errors or delays can cost millions of dollars and slow down access to life-saving treatments. Through building this project, I gained a deep understanding of how protocols, CRFs, and source data interact, and how crucial monitoring is for maintaining data integrity and patient safety. Most importantly, I learned how AI and automation can help reduce this burden — not by replacing people, but by empowering researchers to focus on innovation, science, and patient outcomes instead of repetitive manual work.
### What's next
I plan to build and fine-tune my own AI models trained specifically on clinical trial documents — including protocols, CRFs, monitoring plans, and site communications — to make the system smarter and more context-aware. I want to make the platform bulletproof and production-ready, capable of handling real-world variability in data formats, terminologies, and regulatory requirements. Future versions will focus on improving autonomous agent reliability, secure integrations with EHR/EDC APIs, and deeper alignment with CDISC and GCP standards. The ultimate goal is to create a robust, trusted AI monitoring assistant that can operate safely in live trials — saving time, cutting costs, and pushing the boundaries of what’s possible in clinical research.
## README (from the GitHub repository)
# MonitorMachine
Project for CalHacks 12.0.
## Demo and Backgroud
[](https://www.youtube.com/watch?v=BOXuzT-mpNw)
Prototype for clinical trial monitoring automation.
## Detected evidence (automated analysis)
Indexed codebase: 78 recognized source files, 787 KB.
- CSS (language) — detected in the code
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (96 of 96)
```
.gitignore
ACCESS_MODAL_DESIGN.md
agents2/agent1q04wra7vxw_data.json
agents2/agent1qf4au6rzaa_data.json
agents2/agent1qwejw388s7_data.json
agents2/agent1qww3lnjv3c_data.json
agents2/basic_gemini_agent.py
agents2/clinical_trial_agent.py
agents2/demo_source_data_review.py
agents2/demo_trial_monitor.py
agents2/demo_updated_agent.py
agents2/patient_data_validator_agent.py
agents2/PATIENT_DATA_VALIDATOR_README.md
agents2/README.md
agents2/requirements.txt
agents2/source_data_review_agent.py
agents2/trial_monitor_agent.py
agents2/TRIAL_MONITOR_README.md
agents2/USAGE_EXAMPLE.md
AI_AGENT_INTEGRATION.md
backend/API_DOCUMENTATION.md
backend/app.py
backend/chroma_db/chroma.sqlite3
backend/config.py
backend/database_manager.py
backend/demo_setup.py
backend/mock_data.py
backend/models.py
backend/README.md
backend/requirements.txt
backend/setup_database.py
backend/start_server.py
backend/test_auto_setup.py
backend/test_endpoints.py
chat_with_agent.py
DATA_DIRECTORY_ANALYSIS.md
FRONTEND_BACKEND_INTEGRATION.md
INTEGRATION_STATUS.md
mock_trial_site/ACCESS_RESPONSE_FLOW.md
mock_trial_site/backend/app.py
mock_trial_site/backend/requirements.txt
mock_trial_site/config.json
mock_trial_site/data/Patient001/README.md
mock_trial_site/frontend/package.json
mock_trial_site/frontend/public/config.json
mock_trial_site/frontend/public/index.html
mock_trial_site/frontend/src/App.css
mock_trial_site/frontend/src/App.js
mock_trial_site/frontend/src/index.css
mock_trial_site/frontend/src/index.js
mock_trial_site/mock_trial_site/config.json
mock_trial_site/mock_trial_site/frontend/public/config.json
mock_trial_site/mock_trial_site/SETUP_PORTS.md
mock_trial_site/README.md
mock_trial_site/SETUP_PORTS.md
mock_trial_site/STATUS.md
mock_trial_site/test_access_flow.py
mock_trial_site/test_complete_flow.py
mock_trial_site/test_response_flow.py
mock_trial_site/WEBSOCKET_FLOW.md
MOCK_USERS_REMOVAL_SUMMARY.md
README.md
sdvsdr/.gitignore
sdvsdr/package.json
sdvsdr/public/index.html
sdvsdr/public/manifest.json
sdvsdr/public/robots.txt
sdvsdr/README.md
sdvsdr/src/App.css
sdvsdr/src/App.js
sdvsdr/src/components/BackendTest.js
sdvsdr/src/components/LoginIsland.css
sdvsdr/src/components/LoginIsland.js
sdvsdr/src/components/Navbar.css
sdvsdr/src/components/Navbar.js
sdvsdr/src/components/SSOModals.js
sdvsdr/src/data/studies.js
sdvsdr/src/data/User.js
sdvsdr/src/hooks/useUsers.js
sdvsdr/src/index.js
sdvsdr/src/pages/CreateStudy.css
sdvsdr/src/pages/CreateStudy.js
sdvsdr/src/pages/Login.js
sdvsdr/src/pages/StudiesDashboard.css
sdvsdr/src/pages/StudiesDashboard.js
sdvsdr/src/services/api.js
sdvsdr/src/services/dataService.js
sdvsdr/src/services/fileScrapingService.js
sdvsdr/src/utils/testIntegration.js
sdvsdr/src/utils/userStorage.js
test_demo_workflow.py
test_frontend_backend.html
test_integration.py
test_protocol_analysis.py
test_trialmonitor_rest.py
TRIALMONITOR_REST_SETUP.md
```
### Dependencies
- agents2/requirements.txt: google-genai@>=0.2.0, python-dotenv@>=1.0.0, typing-extensions@>=4.0.0, uagents@>=1.0.0
- backend/requirements.txt: chromadb@==0.4.18, Flask@==2.3.3, pandas@==2.1.1, PyPDF2@==3.0.1, python-dotenv@==1.0.0, Werkzeug@==2.3.7
- mock_trial_site/backend/requirements.txt: Flask@==2.3.3, flask-cors@==4.0.0, Werkzeug@==2.3.7
- mock_trial_site/frontend/package.json: react@^18.2.0, react-dom@^18.2.0, react-scripts@5.0.1
- sdvsdr/package.json: @testing-library/dom@^10.4.1, @testing-library/jest-dom@^6.9.1, @testing-library/react@^16.3.0, @testing-library/user-event@^13.5.0, react@^19.2.0, react-dom@^19.2.0, react-router-dom@^7.9.4, react-scripts@5.0.1, web-vitals@^2.1.4
### Recent commits (newest first)
- Add initial project description to README
- push
- lssdlfj
- sl=k
- asDLFkj
- dsfl
- sdlf-
- update
- attach
- monitor stategy
- monitoring plan
- update create study functionality
- add more agent behaviour
- dsf
- update propmt
- update for pdf
- preview
- pdf
- Update UI
- add edc file viewer
## Key source files (fetched from GitHub, selected and truncated for size)
### MOCK_USERS_REMOVAL_SUMMARY.md
```markdown
# Mock Users Removal Summary
## ✅ **Successfully Removed Mock Users from Frontend**
The frontend no longer uses local mock user data and instead fetches users directly from the backend API.
## 🔧 **Changes Made:**
### 1. **Created `useUsers` Hook** (`src/hooks/useUsers.js`)
- **Purpose**: Fetches users from backend API and groups them by company
- **Features**:
- Fetches all users from `/api/users` endpoint
- Groups users by company (Google, Veera Vault, Medidata)
- Provides loading and error states
- Handles API failures gracefully
### 2. **Updated SSO Modals** (`src/components/SSOModals.js`)
- **Removed**: Import of `mockUsers` from local file
- **Added**: Import of `useUsers` hook
- **Updated**: All three SSO modals (Google, Veera, Medidata) to use backend data
- **Added**: Loading states with "Loading users..." message
- **Fixed**: JSX syntax for conditional rendering
### 3. **Deleted Mock Users File** (`src/data/mockUsers.js`)
- **Removed**: Entire file containing hardcoded user data
- **Result**: No more duplicate user data between frontend and backend
## 🔄 **New Data Flow:**
```
Backend API → useUsers Hook → SSO Modals → User Selection → Login
```
### Before:
```
Frontend mockUsers.js → SSO Modals → User Selection → Login
Backend API → Studies/Data Management
```
### After:
```
Backend API → useUsers Hook → SSO Modals → User Selection → Login
Backend API → Studies/Data Management
```
## 🎯 **Benefits Achieved:**
### ✅ **Single Source of Truth**
- All user data now comes from backend API
- No more data duplication between frontend and backend
- Consistent user information across the entire application
### ✅ **Real-time Data**
- User changes in backend are immediately reflected in login
- No need to update frontend when users change
- Dynamic user management
### ✅ **Better Architecture**
- Clean separation of concerns
- Reusable `useUsers` hook
- Proper loading and error states
### ✅ **Maintainability**
- One place to manage user data (backend)
- Easier to add/remove users
- Consistent data structure
## 📊 **Current Data Sources:**
### ✅ **Backend API** (Single Source)
- **Users**: 12 users across 3 companies
- **Studies**: 4+ studies with real-time updates
- **Investigators**: Dynamic assignment and management
### ❌ **Removed Frontend Mock Data**
- ~~`mockUsers.js`~~ - **DELETED**
- ~~`mockStudies`~~ - **REMOVED** (from studies.js)
## 🧪 **Testing Results:**
### ✅ **Backend API Tests**
```
✅ Get all users: 200 (12 users)
✅ Users by company: Working correctly
✅ User data structure: Consistent with frontend
```
### ✅ **Frontend Build Tests**
```
✅ Build successful: No errors
✅ No import errors: mockUsers.js removed cleanly
✅ JSX syntax: Fixed conditional rendering
```
### ✅ **Integration Tests**
```
✅ useUsers hook: Fetches and groups users correctly
✅ SSO modals: Display backend users with loading states
✅ Login flow: Works with b
[truncated — 1252 more characters]
```
### DATA_DIRECTORY_ANALYSIS.md
```markdown
# Data Directory Analysis
## ❌ **Cannot Remove the Entire Data Directory**
The `src/data/` directory **cannot be completely removed** because it contains essential class definitions that are still actively used by the frontend-backend integration.
## 📁 **Current Data Directory Contents**
```
src/data/
├── User.js ✅ STILL NEEDED
├── studies.js ✅ STILL NEEDED (classes only)
└── mockUsers.js ✅ STILL NEEDED
```
## 🔍 **Usage Analysis**
### 1. **`User.js`** - ✅ **KEEP**
**Used by:**
- `src/services/dataService.js` - Converts backend user JSON to frontend User instances
- `src/utils/userStorage.js` - For localStorage operations
**Purpose:** Provides the User class with methods like `getFullName()`, `getInitials()`, etc.
### 2. **`studies.js`** - ✅ **KEEP** (Classes Only)
**Used by:**
- `src/services/dataService.js` - Converts backend study JSON to frontend Study, Site, StudyFile instances
**Purpose:** Provides Study, Site, and StudyFile classes with methods like:
- `Study.getTotalSites()`, `Study.getActiveSites()`, `Study.hasPrincipalInvestigator()`
- `Site.addESourceFile()`, `Site.addCRFFile()`
- `StudyFile` for file metadata
**✅ REMOVED:** `mockStudies` export (no longer used)
### 3. **`mockUsers.js`** - ✅ **KEEP**
**Used by:**
- `src/components/SSOModals.js` - Shows available user accounts for login
**Purpose:** Provides the list of users that can be selected during SSO login
## 🎯 **What Was Removed**
### ✅ **Removed Mock Studies Data**
- **Before:** `studies.js` contained `mockStudies` array with hardcoded study data
- **After:** `mockStudies` removed, only class definitions remain
- **Reason:** Studies now come from backend API via `dataService.getStudies()`
## 🔄 **Data Flow Architecture**
```
Backend API (JSON) → dataService.js → Frontend Classes (User.js, studies.js) → React Components
```
### Example Flow:
1. **Backend** returns study JSON: `{id: "STD-001", title: "Study Title", ...}`
2. **dataService.js** calls `this.convertToStudy(backendStudy)`
3. **studies.js** creates new `Study()` instance with methods
4. **React Components** use study methods like `study.hasPrincipalInvestigator()`
## 🚫 **Why We Can't Remove the Classes**
### 1. **Method Functionality**
The frontend components rely on class methods:
```javascript
// StudiesDashboard.js
{
study.hasPrincipalInvestigator() ? (
Has PI: {study.principalInvestigator.name}
) : (
Missing Investigator
);
}
```
### 2. **Data Conversion**
The dataService needs these classes to convert backend JSON to frontend objects:
```javascript
// dataService.js
convertToStudy(backendStudy) {
const study = new Study(
backendStudy.id,
backendStudy.title,
// ... other properties
);
return study; // Now has methods like hasPrincipalInvestigator()
}
```
### 3. **Type Safety & Consistency**
Classes provide consistent interfaces and prevent errors:
```jav
[truncated — 1326 more characters]
```
### backend/requirements.txt
```
Flask==2.3.3
chromadb==0.4.18
PyPDF2==3.0.1
pandas==2.1.1
python-dotenv==1.0.0
Werkzeug==2.3.7
```
### agents2/requirements.txt
```
# Core dependencies for agents
python-dotenv>=1.0.0
google-genai>=0.2.0
uagents>=1.0.0
# Optional but recommended
typing-extensions>=4.0.0
```
### sdvsdr/package.json
```
{
"name": "sdvsdr",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/dom": "^10.4.1",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.0",
"@testing-library/user-event": "^13.5.0",
"react": "^19.2.0",
"react-dom": "^19.2.0",
"react-router-dom": "^7.9.4",
"react-scripts": "5.0.1",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### mock_trial_site/backend/requirements.txt
```
Flask==2.3.3
flask-cors==4.0.0
Werkzeug==2.3.7
```
### mock_trial_site/frontend/package.json
```
{
"name": "mock-trial-site-frontend",
"version": "0.1.0",
"private": true,
"dependencies": {
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-scripts": "5.0.1"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### sdvsdr/src/index.js
```javascript
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
const root = ReactDOM.createRoot(document.getElementById("root"));
root.render(
,
);
```
### sdvsdr/src/App.js
```javascript
import React from "react";
import {
Navigate,
Route,
BrowserRouter as Router,
Routes,
} from "react-router-dom";
import "./App.css";
import CreateStudy from "./pages/CreateStudy";
import Login from "./pages/Login";
import StudiesDashboard from "./pages/StudiesDashboard";
import { getUser } from "./utils/userStorage.js";
// Protected Route Component
const ProtectedRoute = ({ children }) => {
const user = getUser();
return user ? children : ;
};
// Public Route Component (redirect to studies if already logged in)
const PublicRoute = ({ children }) => {
const user = getUser();
return user ? : children;
};
function App() {
return (
);
}
export default App;
```
### backend/app.py
```python
import os
import uuid
from flask import Flask, request, jsonify, g
from werkzeug.utils import secure_filename
import chromadb
from chromadb.api import ClientAPI
from chromadb.api.models.Collection import Collection
import PyPDF2
import pandas as pd
from io import BytesIO
import json
from config import IS_LOCAL, CHROMA_API_KEY, CHROMA_TENANT, CHROMA_DATABASE
from database_manager import (
get_all_users,
get_users_by_company,
get_users_by_role,
get_study_by_id,
get_studies_by_sponsor,
get_studies_by_status,
add_investigator_to_study,
get_database_statistics,
db_manager,
)
app = Flask(__name__)
# Enable CORS for all routes
@app.after_request
def after_request(response):
response.headers.add("Access-Control-Allow-Origin", "*")
response.headers.add("Access-Control-Allow-Headers", "Content-Type,Authorization")
response.headers.add("Access-Control-Allow-Methods", "GET,PUT,POST,DELETE,OPTIONS")
return response
# Configuration
UPLOAD_FOLDER = "uploads"
# Initialize database on startup
def initialize_database():
"""Initialize the database with mock data on server startup."""
try:
print("Initializing database...")
db_manager.ensure_data_loaded()
stats = get_database_statistics()
print(f"✅ Database initialized successfully!")
print(f" - Users: {stats['total_users']}")
print(f" - Studies: {stats['total_studies']}")
print(f" - Studies without PI: {stats['studies_without_investigator']}")
except Exception as e:
print(f"❌ Error initializing database: {e}")
# Initialize database when the app starts
initialize_database()
ALLOWED_EXTENSIONS = {"pdf", "csv"}
MAX_CONTENT_LENGTH = 16 * 1024 * 1024 # 16MB max file size
app.config["UPLOAD_FOLDER"] = UPLOAD_FOLDER
app.config["MAX_CONTENT_LENGTH"] = MAX_CONTENT_LENGTH
# Create upload directory if it doesn't exist
os.makedirs(UPLOAD_FOLDER, exist_ok=True)
# Global variable for local/cloud configuration
islocal = IS_LOCAL
def get_chroma_client() -> ClientAPI:
"""Get ChromaDB client (local or cloud based on islocal flag)."""
if islocal:
if "chroma_client" not in g:
g.chroma_client = chromadb.PersistentClient(path="./chroma_db")
return g.chroma_client
else:
if "chroma_client" not in g:
g.chroma_client = chromadb.CloudClient(
api_key=CHROMA_API_KEY, tenant=CHROMA_TENANT, database=CHROMA_DATABASE
)
return g.chroma_client
def get_chroma_collection() -> Collection:
"""Get ChromaDB collection (local or cloud based on islocal flag)."""
chroma_client = get_chroma_client()
if "chroma_collection" not in g:
g.chroma_collection = chroma_client.get_or_create_collection(
name="documents", metadata={"hnsw:space": "cosine"}
)
return g.chroma_collection
def allowed_file(filename):
"""Check if the uploaded file has an allowed extension."""
return "." in filename and filename.rsplit(".", 1)[1].lower() in ALLOWED_EXTENSIONS
def extract_text_from_pdf(file_content):
"""Extract text content from PDF file."""
try:
pdf_reader = PyPDF2.PdfReader(BytesIO(file_content))
text = ""
for page in pdf_reader.pages:
text += page.extract_text() + "\n"
return text.strip()
except Exception as e:
raise Exception(f"Error extracting text from PDF: {str(e)}")
def extract_text_from_csv(file_content):
"""Extract text content from CSV file."""
try:
# Read CSV content
df = pd.read_csv(BytesIO(file_content))
# Convert DataFrame to text representation
text_content = (
f"CSV Data with {len(df)} rows and {len(df.columns)} columns:\n\n"
)
text_content += f"Columns: {', '.join(df.columns.tolist())}\n\n"
# Add first few rows as text
text_content += "Sample data:\n"
text_content += df.head(10).to_string(index=False)
# Add summary statistics for numeric columns
numeric_cols = df.select_dtypes(include=["number"]).columns
if len(numeric_cols) > 0:
text_content += "\n\nSummary statistics:\n"
text_content += df[numeric_cols].describe().to_string()
return text_content
except Exception as e:
raise Exception(f"Error processing CSV file: {str(e)}")
def process_file(file_content, filename):
"""Process uploaded file and extract text content."""
file_extension = filename.rsplit(".", 1)[1].lower()
if file_extension == "pdf":
return extract_text_from_pdf(file_content)
elif file_extension == "csv":
return extract_text_from_csv(file_content)
else:
raise Exception(f"Unsupported file type: {file_extension}")
@app.route("/upload", methods=["POST"])
def upload_file():
"""Handle file upload and store in ChromaDB."""
try:
# Check if file is present in request
if "file" not in request.files:
return jsonify({"error": "No file provided"}), 400
file = request.files["file"]
# Check if file is selected
if file.filename == "":
return jsonify({"error": "No file selected"}), 400
# Check if file has allowed extension
if not allowed_file(file.filename):
return (
jsonify(
{
"error": f'File type not allowed. Allowed types: {", ".join(ALLOWED_EXTENSIONS)}'
}
),
400,
)
# Read file content
file_content = file.read()
# Process file and extract text
text_content = process_file(file_content, file.filename)
if not text_content.strip():
return (
jsonify({"error": "No text content could be extracted from the file"}),
400,
)
# Generate unique ID for the document
[truncated — 16998 more characters]
```
[64 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]