# Project export: Agent Ricky This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Monitoring the Health and Safety of First Responders and Military Personnel - Devpost: https://devpost.com/software/jarvis-mhfy78 - GitHub: https://github.com/nipunbatra8/healthbot - Demo: https://github.com/nipunbatra8/mapagent - Video: https://www.youtube.com/embed/H3TpX43lDUs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (DAIN Labs: AI Agent Excellence & Innovation Awards ($5k Cash [1st] & $2.5K Cash [2nd] & $500 Cash [3rd] + Agent Launchpad Invitation)) - Team: 2 GitHub contributor(s) — Adarsh Ashok (2 commits), nipunbatra8 (1 commits) ## Devpost submission (written by the team) ### Inspiration Our team wanted to use this opportunity at TreeHacks to create something bigger than ourselves, something that serves others. Who better can we serve than the first responders and the military personnel who have devoted their lives to protect and serve us every day? These first responders and military personnel must respond quickly because they are placed in very delicate situations. Thus, we realized that, we needed an AI that can be proactive and create real-time decision-making. Unlike rule-based automation or passive AI that waits for user input, agentic AI goes beyond this traditional AI by enabling autonomous decision-making in critical situations where speed and efficiency are very important. ### What it does Agent Ricky is a custom AI agent tailor-made for first responders and military personnel to monitor the status, safety, and health of soldiers, squadrons, and first responders in training and in action. It is essentially another member of the team. Live biometric and geolocation data collection enables Agent Ricky to detect abnormalities and dangers in real time by providing data for a series of health metrics. These health metrics are then utilized by a custom DAIN-driven AI agent to provide timely alerts, statistics, and information that could prove to be the key to save many lives. ### How we built it Languages: Python · TypeScript Frameworks and Tools: DAIN · Terra API · Nvidia Brev.dev · TensorFlow 1. Biometric and Geolocation Data Collection Originally, we began with the goal of using Terra's PPG waveform data to aid an AI agent in recognizing arrhythmia in order to monitor heart health. However, once we implemented a prototype system, we thought of other data that is pertinent to first responders and military personnel that could also be analyzed, monitored, and reported through the same system. As such, the data collection step now involves collection of a variety of biometric and geolocation data. 2. Heart Rhythms Neural Network Classification PPG waveform data provided by Terra API allowed us to train a keras sequential neural network from scratch using Nvidia's Brev.dev. The neural network is designed to differentiate between regular, irregular, and atrial fibrillation (afib) heart rhythms and provides crucial data for the health of first responders and military personnel. 3. Data Processing We took a different approach for processing other important biometric and geolocation data. For example, we calculated important metrics such as respiratory rate, SpO~2~, HRV, stress estimation, sleep quality, pulse wave analysis, and peak detection. These calculations could then be used for detecting alarming conditions. In addition, the preprocessed data can also be passed to the AI agent for visualization purposes. 4. Custom DAIN-powered AI Alerts Agent DAIN is a custom SDK that allows us to build custom AI agents on top of LLMs to communicate with them in an efficient manner. We designed a DAIN-powered AI service the provides timely alerts to keep safe. It takes in the variety of detected regularities or irregularities in biometric and geolocation data and determines if an alert is necessary to notify leaders and supervisors of. 5. Custom DAIN-powered AI Visualization Agent We also designed a DAIN-powered AI service that provides important visualization information to help leaders stay better informed about training and combat situations. For example, respiratory rates or location over time can prove to be crucial information. ### Challenges we ran into One area that we spent a significant time in was learning how to use DAIN and all its possibilities. Although DAIN draws on the power of LLMs to create AI agents, DAIN's functionality and SDK is quite different from any LLMs we had previously worked with. However, after poking and prodding our way through documentation and experimentation, we discovered the interesting potential of DAIN. Learning through the demo services, the UI manipulation, and background processes took a lot of experimentation, but also offered a novel learning experience in creating and launching AI services. ### Accomplishments we're proud of Creating our own Agentic AI that has the potential to make a real impact. Coming into the Hackathon, agentic AI systems were novel to all our team members. Yet, hackathons are the best places to explore new topics, new ideas outside of comfort zones. As such, we are really proud that we, 4 strangers, not only met each other, but synergized into a cohesive unit that collectively explored agentic AI through DAIN's products. In addition, together we were able to build Agent Ricky, an agentic AI system that has the ability to provide first responders and military personnel and their leaders with the crucial information to keep them safe in training, service, and combat. ### What we learned The next powerful wave of AI may come from Agentic AI that leverages the abilities of LLMs to more effectively and efficiently perform specific, precise, and detailed tasks. ### What's next The potential of Agent Ricky only grows the more data there is available. With an expanded user base and many more data points, we see Agent Ricky as having real potential in greatly improving the safety of first responders and military personnel. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 3 recognized source files, 33 KB. - Python (language) — detected in the code - TypeScript (language) — detected in the code - TensorFlow (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (21 of 21) ``` .env.development .gitignore best_model.keras dain.json health_processing.py package.json src/best_model.onnx src/heart_predictor.py src/index.ts src/regular.csv src/test_1.csv src/test_2.csv src/test_3.csv src/test_4.csv src/test_5.csv temp/file_1.csv temp/file_2.csv temp/file_3.csv temp/file_4.csv temp/file_5.csv tsconfig.json ``` ### Dependencies - package.json: @dainprotocol/cli@^1.0.31, @dainprotocol/service-sdk@^1.0.93, @dainprotocol/utils@^0.0.48, @types/express@^4.17.13, @types/node@^22.5.4, axios@^1.7.5, hono@^4.6.3, papaparse@^5.5.2, ts-node@^10.4.0, typescript@^5.5.4, zod@^3.23.8 ### Recent commits (newest first) - first - new health features - first ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "healthbot", "version": "1.0.0", "description": "A Dain Protocol project", "main": "src/index.ts", "scripts": { "start": "ts-node src/index.ts", "dev": "dain dev", "build": "dain build", "deploy": "dain deploy" }, "dependencies": { "@dainprotocol/cli": "^1.0.31", "@dainprotocol/service-sdk": "^1.0.93", "@dainprotocol/utils": "^0.0.48", "@types/express": "^4.17.13", "@types/node": "^22.5.4", "axios": "^1.7.5", "hono": "^4.6.3", "papaparse": "^5.5.2", "ts-node": "^10.4.0", "typescript": "^5.5.4", "zod": "^3.23.8" } } ``` ### src/index.ts ```typescript import { number, z } from "zod"; import { defineDAINService, ToolConfig } from "@dainprotocol/service-sdk"; import { AlertUIBuilder, CardUIBuilder, DainResponse } from "@dainprotocol/utils"; import test from "node:test"; const askForHelp = { description: "Ask for help when critical health readings are detected", parameters: z.object({ question: z.string().describe("The alert that needs attention") }), execute: async ({ question }, processId, processHandler, callSummary) => { // Log the help request callSummary.steps.push({ speaker: "ask_for_help", text: question, timestamp: new Date().toISOString() }); try { // Request human intervention const stepId = await processHandler.requestHumanAction(processId, { message: question, ui: new CardUIBuilder() .title("Critical Health Alert") .content(`I need help with: ${question}`) .build(), actions: [ { id: "check-soldier", title: "Soldier checked - Status OK", requiresResponse: true }, { id: "medical-needed", title: "Medical attention required", requiresResponse: true } ], timeoutMs: 30 * 1000 }); // Wait for response const response = await processHandler.waitForHumanAction( processId, stepId, 30 * 1000 ); // Handle responses if (response.actionId === "medical-needed") { callSummary.steps.push({ speaker: "supervisor", text: `Medical attention required: ${response.responseText}`, timestamp: new Date().toISOString() }); return "Medical attention has been requested: " + response.responseText; } if (response.actionId === "check-soldier") { callSummary.steps.push({ speaker: "supervisor", text: response.responseText || "Soldier status confirmed OK", timestamp: new Date().toISOString() }); return "Soldier has been checked: " + response.responseText; } } catch (error) { return "No response received from supervisor. Please escalate if condition persists."; } } }; const Papa = require("papaparse"); const fs = require("fs"); let csvData = []; let currentIndex = 1; // Start from 1 to skip the header row // Load the CSV file once function loadCSV() { const csvFile = fs.readFileSync( "/Users/adarshashok/Downloads/treehacks/healthbot/src/test_1.csv", "utf8" ); // Parse the CSV data const results = Papa.parse(csvFile, { header: false, dynamicTyping: true, }); csvData = results.data; } // Call loadCSV() once when the script starts loadCSV(); async function createPPGData(number_of_soldiers) { const createdFiles = []; const regularFilePath = '/Users/adarshashok/Downloads/treehacks/healthbot/src/regular.csv'; const tempDir = path.join(process.cwd(), 'temp'); // Ensure the 'temp' directory exists fs.mkdirSync(tempDir, { recursive: true }); try { // Read the first 501 lines from regular.csv const fileStream = fs.createReadStream(regularFilePath); const rl = readline.createInterface({ input: fileStream, crlfDelay: Infinity }); const lines = []; for await (const line of rl) { if (lines.length < 501) { lines.push(line); } else { break; } } if (lines.length === 0) { throw new Error('regular.csv is empty or does not have 501 lines.'); } // Ensure the last line ends with a newline const fileContent = lines.join('\n') + '\n'; // Create the soldier files with the first 501 lines from regular.csv for (let i = 0; i < number_of_soldiers; i++) { const fileName = `file_${i + 1}.csv`; const filePath = path.join(tempDir, fileName); // Write the collected lines to the new file fs.writeFileSync(filePath, fileContent, 'utf8'); createdFiles.push(fileName); console.log(`Created file: ${fileName} with 501 lines.`); } } catch (error) { console.error(`Error: ${error.message}`); } return createdFiles; } let lastCopiedLine = 500; async function addData(number_of_soldiers: number) { for (let i = 0; i < number_of_soldiers; i++) { const fileName = `/Users/adarshashok/Downloads/treehacks/healthbot/temp/file_${i + 1}.csv`; const testName = `/Users/adarshashok/Downloads/treehacks/healthbot/src/test_${i + 1}.csv` await copyDataPoints(testName, fileName, lastCopiedLine); } lastCopiedLine += 10; } const { spawn } = require("child_process"); const path = require("path"); /** * Runs a Python script on multiple CSV files and logs the output. * @param {string[]} csvFiles - Array of CSV file paths. */ function processCSVFiles(csvFiles) { return new Promise((resolve, reject) => { const pythonScript = path.join(__dirname, "heart_predictor.py"); // Path to Python script const pythonProcess = spawn("python3", [pythonScript, ...csvFiles]); let outputData = ''; // Handle stdout (Python output) pythonProcess.stdout.on("data", (data) => { console.log(`Python Output: ${data.toString().trim()}`); outputData += data.toString(); }); // Handle stderr (errors) pythonProcess.stderr.on("data", (data) => { console.error(`Python Error: ${data.toString().trim()}`); }); // Handle process exit pythonProcess.on("close", (code) => { console.log(`Python process exited with code ${code}`); if (code === 0) { // Split output into an array, assuming Python output is comma-separated resolve(outputData.trim().split(/\s*,\s*/)); } else { reject(new Error(`Python script exited with code ${code}`)); } }); }); } const readline = require('readline'); /** * Copies 10 data points from the source CSV and appends to the target CSV, * incrementing the start [truncated — 9217 more characters] ``` ### health_processing.py ```python import torch import torch.nn as nn import torch.nn.functional as F import numpy as np import pandas as pd import os import matplotlib.pyplot as plt import math from scipy.signal import butter, filtfilt, find_peaks from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split # ============================================ # Section 1: Data Preprocessing & PyTorch Model Prediction # ============================================ def process_single_file(file_path, window_size=750, overlap=0.5): """Process a single CSV file and return windowed data (as numpy array).""" try: # Read data using pandas (assumes no header or a header row we skip) data = pd.read_csv(file_path, header=None) numerical_data = data.iloc[1:].values # Skip header if present numerical_data = numerical_data.astype(float) # Separate features: first 3 columns are PPG signals, next 3 are accelerometer data sensor_readings = numerical_data[:, :3] accelerations = numerical_data[:, 3:6] # Normalize features scaler = StandardScaler() sensor_normalized = scaler.fit_transform(sensor_readings) accelerations_normalized = scaler.fit_transform(accelerations) # Combine normalized features features = np.concatenate([sensor_normalized, accelerations_normalized], axis=1) # Create windows with overlap stride = int(window_size * (1 - overlap)) windows = [features[i:i + window_size] for i in range(0, len(features) - window_size, stride)] return np.array(windows) except Exception as e: print(f"Error processing file {file_path}: {str(e)}") return None def predict_heart_rhythm(model, file_path, window_size=500, device='cpu'): """ Predict the heart rhythm class for a new CSV file. Returns: predicted class and confidence. """ preprocessed_data = process_single_file(file_path, window_size) if preprocessed_data is None: return None, None # Convert numpy array to torch tensor # Expected shape: (num_windows, window_size, num_features) input_tensor = torch.tensor(preprocessed_data, dtype=torch.float32).to(device) model.eval() with torch.no_grad(): outputs = model(input_tensor) # Apply softmax to obtain probabilities probabilities = F.softmax(outputs, dim=1) # Average predictions over windows avg_prediction = probabilities.mean(dim=0).cpu().numpy() class_names = ['AFib', 'Irregular', 'Regular'] predicted_class = class_names[np.argmax(avg_prediction)] confidence = np.max(avg_prediction) return predicted_class, confidence # ============================================ # Section 2: PyTorch Model Architecture (Equivalent to TensorFlow model) # ============================================ class HealthModel(nn.Module): def __init__(self, input_shape=(500, 6)): """ input_shape: (sequence_length, num_features) """ super(HealthModel, self).__init__() # Our input is (batch, seq_len, features). For Conv1d, we'll transpose to (batch, channels, seq_len) self.conv1 = nn.Conv1d(in_channels=input_shape[1], out_channels=64, kernel_size=3, padding=1) self.bn1 = nn.BatchNorm1d(64) self.relu = nn.ReLU() self.maxpool1 = nn.MaxPool1d(kernel_size=2) self.conv2 = nn.Conv1d(64, 128, kernel_size=3, padding=1) self.bn2 = nn.BatchNorm1d(128) self.maxpool2 = nn.MaxPool1d(kernel_size=2) # After two poolings, the sequence length becomes input_shape[0] // 4 (e.g., 500//4 = 125) self.lstm1 = nn.LSTM(input_size=128, hidden_size=128, batch_first=True) self.dropout1 = nn.Dropout(0.3) self.lstm2 = nn.LSTM(input_size=128, hidden_size=64, batch_first=True) self.dropout2 = nn.Dropout(0.3) self.fc1 = nn.Linear(64, 64) self.bn3 = nn.BatchNorm1d(64) self.fc2 = nn.Linear(64, 32) self.fc3 = nn.Linear(32, 3) def forward(self, x): # x: (batch, seq_len, features) # Permute to (batch, features, seq_len) for Conv1d x = x.permute(0, 2, 1) x = self.conv1(x) x = self.bn1(x) x = self.relu(x) x = self.maxpool1(x) x = self.conv2(x) x = self.bn2(x) x = self.relu(x) x = self.maxpool2(x) # Permute back for LSTM: (batch, seq_len_new, channels) x = x.permute(0, 2, 1) # Now shape: (batch, seq_len/4, 128) x, _ = self.lstm1(x) x = self.dropout1(x) x, _ = self.lstm2(x) x = self.dropout2(x) # Use the last time step's output x = x[:, -1, :] x = self.fc1(x) # For BatchNorm1d on FC layers, input shape should be (batch, features) x = self.bn3(x) x = self.relu(x) x = self.fc2(x) x = self.relu(x) x = self.fc3(x) # No softmax here; it will be applied later in prediction return x # ============================================ # Section 3: PPG Signal Analysis Functions (Unchanged) # ============================================ def bandpass_filter_np(signal, lowcut, highcut, fs, order=3): nyq = 0.5 * fs low = lowcut / nyq high = highcut / nyq b, a = butter(order, [low, high], btype='band') return filtfilt(b, a, signal) def lowpass_filter_np(signal, cutoff, fs, order=3): nyq = 0.5 * fs normal_cutoff = cutoff / nyq b, a = butter(order, normal_cutoff, btype='low') return filtfilt(b, a, signal) def detect_beats(ppg_signal, fs, distance_sec=0.5): distance_samples = int(distance_sec * fs) peaks, _ = find_peaks(ppg_signal, distance=distance_samples) return peaks def compute_heart_rate(ppg_signal, fs): peaks = detect_beats(ppg_signal, fs) if len(peaks) < 2: return None, peaks i [truncated — 9401 more characters] ``` ### src/heart_predictor.py ```python import onnxruntime as ort import numpy as np import pandas as pd from sklearn.preprocessing import StandardScaler import sys # Load ONNX model onnx_model_path = "/Users/adarshashok/Downloads/treehacks/healthbot/src/best_model.onnx" ort_session = ort.InferenceSession(onnx_model_path) # Data preprocessing for a single file def process_single_file(file_path, window_size=500, overlap=0.75): """Process a single CSV file and return windowed data""" try: # Read data using pandas data = pd.read_csv(file_path, header=None) # Extract numerical columns (skipping the first row) numerical_data = data.iloc[1:].values.astype(float) # Separate features sensor_readings = numerical_data[:, :3] # green, red, IR readings accelerations = numerical_data[:, 3:6] # acc_x, acc_y, acc_z # Normalize features scaler = StandardScaler() sensor_normalized = scaler.fit_transform(sensor_readings) accelerations_normalized = scaler.fit_transform(accelerations) # Combine normalized features features = np.concatenate([sensor_normalized, accelerations_normalized], axis=1) # Create overlapping windows stride = int(window_size * (1 - overlap)) windows = [features[i:i + window_size] for i in range(0, len(features) - window_size, stride)] return np.array(windows).astype(np.float32) except Exception as e: print(f"Error processing file {file_path}: {str(e)}") return None # Prediction function def predict_heart_rhythm(file_path, window_size=500): """Predict the heart rhythm class using ONNX model""" preprocessed_data = process_single_file(file_path, window_size) if preprocessed_data is None: return None, None # Get input name for ONNX model input_name = ort_session.get_inputs()[0].name # Run inference predictions = ort_session.run(None, {input_name: preprocessed_data})[0] # Class labels class_names = ['AFib', 'Irregular', 'Regular'] # Average predictions across windows avg_prediction = predictions.mean(axis=0) predicted_class = class_names[np.argmax(avg_prediction)] confidence = np.max(avg_prediction) return predicted_class, confidence def main(): if len(sys.argv) < 2: print("Usage: python3 heart_predictor.py ...") sys.exit(1) file_paths = sys.argv[1:] for file_path in file_paths: predicted_class, confidence = predict_heart_rhythm(file_path) if predicted_class is not None: print(predicted_class) else: print(f"{file_path}: Prediction failed.") if __name__ == "__main__": main() ```