# Project export: sura.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Your Autonomous Multi-Agentic Infrastructure Immune System - Devpost: https://devpost.com/software/sura-ai - GitHub: https://github.com/ananya-arv/sura-ai - Video: https://www.youtube.com/embed/2i-3o5J2dII?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) β€” ananya-arv (28 commits), shruthi (5 commits), Saishr-ya (1 commits) ## Devpost submission (written by the team) ### Inspiration The project was inspired by the critical need to eliminate human-in-the-loop dependencies during major infrastructure outages. We aimed to build a system capable of handling complex, cascading failuresβ€”like the simulated "CrowdStrike-style faulty update" and "AWS-style availability zone failure" scenarios in our testing pipelineβ€”by providing instantaneous, intelligent response. The core idea is to go beyond simple rule-based automation and leverage LLMs for high-quality root cause analysis. ### What it does SuraAI orchestrates four distinct, self-registering Fetch.ai uAgents that communicate via the Agentverse Mailbox: Canary Agent: Tests new software updates on a small subset of systems (1% canary population) to detect potential flaws before wide deployment. It uses AI analysis to decide whether to DEPLOY or initiate an automatic ROLLBACK. The system successfully prevented a simulated faulty kernel update scenario. Monitoring Agent: Continuously polls a mock infrastructure API for real-time metrics (CPU, Memory, Errors) and applies advanced baselining to identify anomalies. It generates AnomalyAlerts which trigger the autonomous response loop. Response Agent: Receives alerts and immediately consults Claude Sonnet 3.5 via the Lava Gateway for a definitive diagnosis and action recommendation (e.g., SCALE_UP, RESTART, ROLLBACK). It then executes the appropriate automated runbook. The system demonstrated a highly effective 97.6% action success rate (41 incidents resolved out of 42 actions taken) and intelligent alert deduplication. Communication Agent: Upon resolution, it publishes a real-time StatusUpdate to a status page file and sends notifications to stakeholders, ensuring everyone is informed instantly without human intervention. ### How we built it Agent Logic: We structured the system using Fetch.ai uAgents, implementing the core logic across four decoupled Python agents. All agents were configured to use the Agentverse Mailbox for robust, asynchronous message routing. AI Integration: We leveraged the Lava Gateway (Lava Payments) to route requests to the Claude Sonnet 3.5 LLM. This enabled us to inject incident data into a tailored prompt, receiving structured JSON output for AI-driven root cause analysis and action recommendation within the Response Agent. Simulation Environment: A FastAPI application was developed as a mock production infrastructure, featuring 100 simulated systems with endpoints for metric polling, failure injection, and system recovery (rollback). This simulation is entirely managed and displayed via the creao.ai dashboard, which allows users to visualize the agents' lifecycle and a live demo of the test pipeline. Testing and Validation: A comprehensive end-to-end testing pipeline was created to orchestrate all agents and the mock infrastructure through four disaster scenarios, validating the entire communication flow and autonomous recovery capabilities. ### Challenges we ran into Decoupled Agent Communication: A major hurdle was ensuring robust asynchronous message passing between the four Python agents. We addressed this by writing specialized diagnostic tools and implementing fixes to ensure all agents correctly utilized the Fetch.ai ctx.send() method for reliable Mailbox routing. LLM API Complexity: Correctly setting up the Lava Gateway connection, including proper authentication headers and adherence to the underlying Anthropic API's message and response formats, required dedicated debugging scripts to isolate and fix token and payload issues. Test Stability: The complexity of running multiple asynchronous agents and HTTP clients simultaneously led to aiohttp "unclosed connection" warnings. We created a fix-up script to implement a single shared HTTP session for the Monitoring Agent and Orchestrator, ensuring long-running test stability. ### What's next Real-world Observability Integration: Transition the Monitoring Agent from the mock API to real observability platforms like Prometheus or Datadog. Preemptive Recovery: Integrate failure prediction logic into the Monitoring Agent to enable sura.ai to take preemptive actions (e.g., auto-scaling) before an incident becomes critical. Decentralized Consensus: Extend the Response Agent to a cluster of competing agents operating across the Agentverse, where they reach a consensus on the optimal recovery action via a voting mechanism before execution. Dynamic Runbooks: Allow the AI to dynamically generate and validate runbook steps based on the incident context and historical resolution data, rather than relying solely on a hardcoded set of responses. ## README (from the GitHub repository) # SuraAI: Autonomous Multi-Agentic Infrastructure Immune System

SuraAI Architecture Diagram

SuraAI is an autonomous, AI-powered system designed to be the "immune system for your infrastructure". It uses four cooperating Fetch.ai uAgents to detect, diagnose, and auto-remediate production failures in real-time. *** ## πŸ’‘ Inspiration The project was inspired by the critical need to eliminate human-in-the-loop dependencies during major infrastructure outages. We aimed to build a system capable of handling complex, cascading failuresβ€”like the simulated "CrowdStrike-style faulty update" and "AWS-style availability zone failure" scenarios in our testing pipelineβ€”by providing instantaneous, intelligent response. The core idea is to go beyond simple rule-based automation and leverage LLMs for high-quality root cause analysis. *** ## πŸ€– What it Does SuraAI orchestrates four distinct, self-registering **Fetch.ai uAgents** that communicate via the **Agentverse Mailbox**: * **Canary Agent:** Tests new software updates on a small subset of systems (1% canary population) to detect potential flaws before wide deployment. It uses AI analysis to decide whether to **DEPLOY** or initiate an automatic **ROLLBACK**. The system successfully prevented a simulated faulty kernel update scenario. * **Monitoring Agent:** Continuously polls a mock infrastructure API for real-time metrics (CPU, Memory, Errors) and applies advanced baselining to identify anomalies. It generates `AnomalyAlerts` which trigger the autonomous response loop. * **Response Agent:** Receives alerts and immediately consults **Claude Sonnet 3.5** via the **Lava Gateway** for a definitive diagnosis and action recommendation (e.g., `SCALE_UP`, `RESTART`, `ROLLBACK`). It then executes the appropriate automated runbook. The system demonstrated a highly effective **97.6% action success rate** (41 incidents resolved out of 42 actions taken) and intelligent alert deduplication. * **Communication Agent:** Upon resolution, it publishes a real-time `StatusUpdate` to a status page file and sends notifications to stakeholders, ensuring everyone is informed instantly without human intervention. *** ## πŸ› οΈ How We Built It * **Agent Logic:** We structured the system using Fetch.ai uAgents, implementing the core logic across four decoupled Python agents. All agents were configured to use the Agentverse Mailbox for robust, asynchronous message routing. * **AI Integration:** We leveraged the **Lava Gateway (Lava Payments)** to route requests to the Claude Sonnet 3.5 LLM. This enabled us to inject incident data into a tailored prompt, receiving structured JSON output for AI-driven root cause analysis and action recommendation within the Response Agent. * **Simulation Environment:** A **FastAPI** application was developed as a mock production infrastructure, featuring 100 simulated systems with endpoints for metric polling, failure injection, and system recovery (rollback). This simulation is entirely managed and displayed via the creao.ai dashboard. * **Testing and Validation:** A comprehensive end-to-end testing pipeline was created to orchestrate all agents and the mock infrastructure through four disaster scenarios, validating the entire communication flow and autonomous recovery capabilities. *** ## πŸ”­ What’s Next for SuraAI? * **Real-world Observability Integration:** Transition the Monitoring Agent from the mock API to real observability platforms like Prometheus or Datadog. * **Preemptive Recovery:** Integrate failure prediction logic into the Monitoring Agent to enable SuraAI to take preemptive actions (e.g., auto-scaling) before an incident becomes critical. * **Decentralized Consensus:** Extend the Response Agent to a cluster of competing agents operating across the Agentverse, where they reach a consensus on the optimal recovery action via a voting mechanism before execution. * **Dynamic Runbooks:** Allow the AI to dynamically generate and validate runbook steps based on the incident context and historical resolution data, rather than relying solely on a hardcoded set of responses. *** ## πŸ”— Built With | Category | Technology | | :--- | :--- | | **Agent Framework** | Fetch.ai uAgents | | **AI Gateway** | Lava Gateway (Lava Payments) | | **LLM Model** | Claude Sonnet 3.5 | | **Dashboard/Demo** | creao.ai | | **APIs & Web** | FastAPI, aiohttp | | **Language** | Python | | **Communication** | Agentverse Mailbox | ## Detected evidence (automated analysis) Indexed codebase: 45 recognized source files, 209 KB. - FastAPI (technology) β€” detected in the code - Python (language) β€” detected in the code ## Codebase structure (from repository index) ### Files (69 of 69) ``` .gitignore agent_data/agent_registry.json agent_data/agent1q03dhrelys_data.json agent_data/agent1q0sx9t9aqp_data.json agent_data/agent1qg92f9k4tj_data.json agent_data/agent1qvgnwew95l_data.json agent_registry.json agents/__init__.py agents/base_agent.py agents/canary/__init__.py agents/canary/canary_agent.py agents/communication/__init__.py agents/communication/communication_agent.py agents/messages.py agents/monitoring/__init__.py agents/monitoring/monitoring_agent.py agents/registry.py agents/response/__init__.py agents/response/intelligent_response_agent.py config.py dashboard/dashboard_api.py e2e_test_pipeline.py experiments/__init__.py experiments/common/__init__.py experiments/common/llm_client.py experiments/common/scenarios.py experiments/common/stats.py experiments/exp1_rule_based/__init__.py experiments/exp1_rule_based/rule_based_response_agent.py experiments/exp1_rule_based/run_experiment1.py experiments/exp2_timing/__init__.py experiments/exp2_timing/instrumented_pipeline.py experiments/exp2_timing/plot_latency.py experiments/exp2_timing/run_experiment2.py experiments/exp3_extended/__init__.py experiments/exp3_extended/run_experiment3.py experiments/exp3_extended/steady_state.py experiments/exp4_canary/__init__.py experiments/exp4_canary/plot_boundary.py experiments/exp4_canary/run_experiment4.py main.py README.md requirements.txt results/_suite_run.log results/exp1_rule_based/aggregate.json results/exp1_rule_based/raw_trials.jsonl results/exp1_rule_based/run.log results/exp1_rule_based/summary.csv results/exp1_rule_based/summary.md results/exp2_timing/_last_status.json results/exp2_timing/aggregate.json results/exp2_timing/events.jsonl results/exp2_timing/run.log results/exp2_timing/summary.md results/exp3_extended/results.json results/exp3_extended/steady_state_events.jsonl results/exp3_extended/summary.md results/exp4_canary/raw_trials.jsonl results/exp4_canary/results.json results/exp4_canary/summary.md results/exp5_cot/raw_trials.jsonl results/exp5_cot/results.json results/exp5_cot/summary.md results/FINAL_REPORT.md services/__init__.py services/lava_service.py services/mock_infrastructure.py setup_e2e_test.sh status_page.json ``` ### Dependencies - requirements.txt: aiohttp, cosmpy, fastapi, groq, loguru, prometheus-client, psutil, pydantic, pytest, pytest-asyncio, python-dotenv, requests, uagents, uvicorn ### Recent commits (newest first) - removed exp5 - git ignore - iterations of experimetns for sura ai extended - Update README.md with project details and structure - Revise README for clarity and structure - final - Update README.md - Update README.md - Update README.md - Update README.md - deleted files - creao removed - Add initial README.md for SuraAI project - creao ai - creao - backend - full system - lava integration - working agents - new lava based agent ## Key source files (fetched from GitHub, selected and truncated for size) ### results/FINAL_REPORT.md ```markdown # Sura.ai β€” Paper Extension: Experiment Results Model: **claude-sonnet-4-6** (the paper's `claude-3-5-sonnet-20240620` was retired 2025-10-28; its same-tier successor was evaluated through the same Lava gateway). Seed **42**, direct in-process harness, Wilson 95% CIs. All original agent/ scenario code is untouched; new code lives under `experiments/`. > **Experiment 1 (rule-based vs LLM) is excluded from this analysis by author > decision.** Its artifacts remain on disk (`results/exp1_rule_based/`) because > Experiment 3's Table 2 sources the LLM Response-Agent numbers from that run. > **⚠️ Cross-cutting caveat (Zone Failure & Cascading).** The harness delivers the > Response Agent a *single* AnomalyAlert per incident. The paper's activation > sequence for those two scenarios has the agent *"gather zone context"* before > the LLM decides β€” which this harness does not provide. Consequently the LLM > cannot recommend the zone-level actions the ground truth requires (`FAILOVER`, > `ISOLATE`) and scores ~0% on S2/S4 in Table 2 (Exp 3A), which in turn depresses > the Exp 5 accuracy/calibration numbers. This is a harness limitation, not > necessarily an LLM limitation. Fixing it (injecting zone context) is the > recommended follow-up. --- ## Experiment 2 β€” Pipeline timing distributions **What ran:** 50 incidents mixed across the four scenarios, each driven through an instrumented pipeline that reuses the *real* components β€” the real detection threshold logic, the real Lava LLM call, and the real runbook coroutines (with their actual `asyncio.sleep` timings). Every stage boundary timestamped with a monotonic clock. **Results** (`results/exp2_timing/summary.md`, figure `latency_distributions.png`): | Stage | median | p99 | |---|--:|--:| | Detection (compute) | 65 Β΅s | 231 Β΅s | | LLM latency (Lava Gateway) | 3.57 s | 6.45 s | | Runbook execution | 2.01 s | 2.01 s | | **End-to-end** | **5.55 s** | **8.46 s** | **Surprising finding:** measured end-to-end resolution is **~5.6 s**, versus the paper's "~2–5 minutes" estimate β€” an overestimate of **30–50Γ—**. The pipeline is dominated by two real components: the Lava/LLM call (~3.6 s) and runbook execution (~2 s); all orchestration stages are sub-millisecond. Live *time-to-detect* additionally carries the 5 s monitoring poll interval, reported separately rather than folded in. --- ## Experiment 3 β€” Extended trials + steady-state false positives ### Part A β€” Updated Table 2 (LLM Response Agent, N=50, 95% Wilson CI) Sourced from the Exp 1 LLM run (same config; no repeated calls). | Scenario | Correct action(s) | Success (95% CI) | Miss | |---|---|--:|--:| | S1 Faulty Update | ROLLBACK | 0.980 [0.895, 0.996] | 0.020 | | S2 Zone Failure | FAILOVER | 0.000 [0.000, 0.071] | 0.020 | | S3 Memory Leak | RESTART/SCALE_UP | 0.680 [0.542, 0.792] | 0.320 | | S4 Cascading Failure | ISOLATE/FAILOVER | 0.000 [0.000, 0.071] | 0.020 | | **Aggregate** | β€” | **0.415 [0.349, 0.484]** | 0.095 | The 0% on S2/S4 is the s [truncated β€” 5119 more characters] ``` ### results/exp4_canary/summary.md ```markdown # Experiment 4 - Expanded Canary Testing - 10 trials/case Γ— 5 cases = 50 total, seed `42`, model `claude-sonnet-4-6` ## Decision by case type (block = ROLLBACK or INVESTIGATE) | Case | error-rate band | LLM block-rate | LLM rollback | LLM mean conf | Rule block-rate | |---|---|--:|--:|--:|--:| | clean | 0.000–0.005 | 0.60 | 0.00 | 0.78 | 0.00 | | borderline | 0.005–0.010 | 1.00 | 0.00 | 0.74 | 0.00 | | faulty_mild | 0.011–0.030 | 1.00 | 0.30 | 0.84 | 1.00 | | faulty_moderate | 0.030–0.080 | 1.00 | 1.00 | 0.93 | 1.00 | | faulty_severe | 0.150–0.300 | 1.00 | 1.00 | 0.97 | 1.00 | ## Headline rates (95% Wilson CI) | Agent | Prevention rate (faultyβ†’blocked) | False-rollback rate (cleanβ†’blocked) | |---|---|---| | rule | 1.000 [0.886, 1.000] (30/30) | 0.000 [0.000, 0.278] (0/10) | | llm | 1.000 [0.886, 1.000] (30/30) | 0.600 [0.313, 0.832] (6/10) | ## Empirical LLM decision boundary - Stated canary threshold: **error rate 0.01** - Highest error rate the LLM still DEPLOYED: **0.0027438661080822803** - Lowest error rate the LLM BLOCKED: **0.00025308155754520336** See `boundary.png` for the DEPLOY/BLOCK decision vs error rate (LLM & rule). ``` ### requirements.txt ``` # Fetch.ai uAgents uagents cosmpy # LLM & AI groq # API & Web fastapi uvicorn requests aiohttp # Monitoring & Data psutil prometheus-client # Utilities python-dotenv pydantic loguru # Testing pytest pytest-asyncio ``` ### main.py ```python import asyncio import subprocess import time from loguru import logger import sys class SuraAIOrchestrator: """Orchestrates all SuraAI agents""" def __init__(self): self.agents = { "canary": "agents/canary/canary_agent.py", "monitoring": "agents/monitoring/monitoring_agent.py", "response": "agents/response/intelligent_response_agent.py", "communication": "agents/communication/communication_agent.py" } self.processes = {} def start_all_agents(self): """Start all agents in separate processes""" logger.info("πŸš€ Starting SuraAI - Autonomous Disaster Recovery Network") logger.info("=" * 60) for name, script in self.agents.items(): logger.info(f"Starting {name} agent...") process = subprocess.Popen( [sys.executable, script], stdout=subprocess.PIPE, stderr=subprocess.PIPE ) self.processes[name] = process time.sleep(2) # Give each agent time to start logger.info("=" * 60) logger.info("βœ… All agents started successfully!") logger.info(f"🐦 Canary Agent: Testing updates before deployment") logger.info(f"πŸ‘οΈ Monitoring Agent: Watching all systems 24/7") logger.info(f"πŸš‘ Response Agent: Ready for autonomous recovery") logger.info(f"πŸ“’ Communication Agent: Status updates active") logger.info("=" * 60) def monitor_agents(self): """Monitor agent health""" try: while True: time.sleep(5) for name, process in self.processes.items(): if process.poll() is not None: logger.error(f"❌ {name} agent crashed! Restarting...") self.restart_agent(name) except KeyboardInterrupt: logger.info("\nπŸ›‘ Shutting down SuraAI...") self.stop_all_agents() def restart_agent(self, name: str): """Restart a crashed agent""" script = self.agents[name] process = subprocess.Popen( [sys.executable, script], stdout=subprocess.PIPE, stderr=subprocess.PIPE ) self.processes[name] = process logger.info(f"βœ… {name} agent restarted") def stop_all_agents(self): """Stop all agents""" for name, process in self.processes.items(): logger.info(f"Stopping {name} agent...") process.terminate() process.wait() logger.info("πŸ‘‹ All agents stopped") def main(): orchestrator = SuraAIOrchestrator() orchestrator.start_all_agents() orchestrator.monitor_agents() if __name__ == "__main__": main() ``` ### config.py ```python import os from dotenv import load_dotenv load_dotenv() class Config: # Groq GROQ_API_KEY = os.getenv("GROQ_API_KEY") # Fetch.ai Agentverse CANARY_SEED_PHRASE = os.getenv("CANARY_SEED_PHRASE") MONITORING_SEED_PHRASE = os.getenv("MONITORING_SEED_PHRASE") RESPONSE_SEED_PHRASE = os.getenv("RESPONSE_SEED_PHRASE") COMMUNICATION_SEED_PHRASE = os.getenv("COMMUNICATION_SEED_PHRASE") # Application ENVIRONMENT = os.getenv("ENVIRONMENT", "development") LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO") API_PORT = int(os.getenv("API_PORT", 8000)) # Agent Settings CANARY_PERCENTAGE = 0.001 MONITORING_INTERVAL = 5 ALERT_THRESHOLD = 0.8 config = Config() ``` ### setup_e2e_test.sh ```shell #!/bin/bash # SuraAI E2E Test Pipeline Setup - AGENTVERSE MAILBOX MODE (CORRECTED) # All critical fixes applied echo "πŸš€ SuraAI End-to-End Test Pipeline Setup" echo "========================================" # Colors RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' BLUE='\033[0;34m' NC='\033[0m' # No Color # Set PYTHONPATH export PYTHONPATH=$(pwd):${PYTHONPATH} echo -e "\n${GREEN}πŸ”§ PYTHONPATH set to: $(pwd)${NC}" # Check if agent_registry.json has Agentverse addresses echo -e "\n${YELLOW}Checking agent registry...${NC}" if [ ! -f "agent_registry.json" ]; then echo -e "${RED}❌ agent_registry.json not found!${NC}" echo "" echo "You need to:" echo "1. Start each agent individually" echo "2. Connect them to Agentverse mailbox" echo "3. Update agent_registry.json with their addresses" echo "" echo "Run: ./reconnect_agents.sh" exit 1 fi # Verify addresses look like Agentverse addresses SAMPLE_ADDR=$(python3 -c "import json; d=json.load(open('agent_registry.json')); print(list(d.values())[0]['address'])" 2>/dev/null) if [[ ! $SAMPLE_ADDR == agent1q* ]]; then echo -e "${RED}❌ Registry contains invalid addresses!${NC}" echo "Addresses should start with 'agent1q...'" echo "Found: $SAMPLE_ADDR" echo "" echo "Run: ./reconnect_agents.sh to get correct addresses" exit 1 fi echo -e "${GREEN}βœ… Registry looks good (Agentverse addresses)${NC}" # Cleanup old logs echo -e "\n${YELLOW}Cleaning up old logs...${NC}" rm -f logs/*.log mkdir -p logs # Start mock infrastructure echo -e "\n${GREEN}1️⃣ Starting Mock Infrastructure...${NC}" python services/mock_infrastructure.py > logs/mock_infra.log 2>&1 & MOCK_PID=$! echo " PID: $MOCK_PID" sleep 3 # Check if mock is running curl -s http://localhost:8000/health > /dev/null if [ $? -eq 0 ]; then echo -e " ${GREEN}βœ… Mock infrastructure running${NC}" else echo -e " ${RED}❌ Mock infrastructure failed to start${NC}" echo " Check logs/mock_infra.log for errors" exit 1 fi # Start agents in MAILBOX MODE echo -e "\n${GREEN}2️⃣ Starting Agents (Agentverse Mailbox Mode)...${NC}" echo -e "${YELLOW} Note: Agents will connect to Agentverse, not communicate locally${NC}" echo " Starting Canary Agent..." python agents/canary/canary_agent.py > logs/canary_agent.log 2>&1 & CANARY_PID=$! sleep 4 echo " Starting Monitoring Agent..." python agents/monitoring/monitoring_agent.py > logs/monitoring_agent.log 2>&1 & MONITORING_PID=$! sleep 4 echo " Starting Response Agent..." python agents/response/intelligent_response_agent.py > logs/response_agent.log 2>&1 & RESPONSE_PID=$! sleep 4 echo " Starting Communication Agent..." python agents/communication/communication_agent.py > logs/communication_agent.log 2>&1 & COMMUNICATION_PID=$! sleep 4 echo -e "\n ${GREEN}βœ… All agents started${NC}" echo " Canary PID: $CANARY_PID" echo " Monitoring PID: $MONITORING_PID" echo " Response PID: $RESPONSE_PID" echo " Communication PID: $COMMUNICATION_PID" # CRITICAL FIX #3: Longer wait time for Almanac registration echo -e "\n${YELLOW}3️⃣ Waiting for agents to initialize...${NC}" echo -e "${BLUE} (Almanac registration can take 10-30 seconds)${NC}" sleep 15 # Check if agents are still running echo -e "\n${YELLOW}Verifying agents are alive...${NC}" AGENTS_OK=true for pid in $CANARY_PID $MONITORING_PID $RESPONSE_PID $COMMUNICATION_PID; do if ps -p $pid > /dev/null 2>&1; then echo -e " ${GREEN}βœ… PID $pid is running${NC}" else echo -e " ${RED}❌ PID $pid has died${NC}" AGENTS_OK=false fi done if [ "$AGENTS_OK" = false ]; then echo -e "\n${RED}Some agents crashed. Check logs:${NC}" echo " logs/canary_agent.log" echo " logs/monitoring_agent.log" echo " logs/response_agent.log" echo " logs/communication_agent.log" echo -e "\n${YELLOW}Showing last errors:${NC}" for log in logs/*_agent.log; do if [ -f "$log" ]; then echo -e "\n${YELLOW}=== $(basename $log) ===${NC}" tail -10 "$log" | grep -i "error\|exception\|failed" || tail -5 "$log" fi done kill $MOCK_PID $CANARY_PID $MONITORING_PID $RESPONSE_PID $COMMUNICATION_PID 2>/dev/null exit 1 fi # Wait additional time for Almanac registration to complete echo -e "\n${YELLOW}⏳ Waiting additional 15 seconds for Almanac registration...${NC}" sleep 15 # Verify Agentverse connection (check logs for successful registration) echo -e "\n${YELLOW}4️⃣ Checking Agentverse connection status...${NC}" # Look for successful Almanac registration ALMANAC_SUCCESS=0 for log in logs/*_agent.log; do if grep -q "Registering on almanac contract...complete" "$log" 2>/dev/null; then ALMANAC_SUCCESS=$((ALMANAC_SUCCESS + 1)) AGENT_NAME=$(basename "$log" .log) echo -e " ${GREEN}βœ… $AGENT_NAME registered on Almanac${NC}" fi done if [ $ALMANAC_SUCCESS -lt 4 ]; then echo -e "\n ${YELLOW}⚠️ Only $ALMANAC_SUCCESS/4 agents registered on Almanac${NC}" echo " This is OK if you manually connected them to Agentverse mailbox" fi # CRITICAL FIX #6: Manual verification step echo -e "\n${BLUE}═══════════════════════════════════════════════════════════${NC}" echo -e "${YELLOW}⚠️ IMPORTANT: Manual Verification Required${NC}" echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}" echo "" echo " Before continuing, verify on Agentverse dashboard:" echo -e " ${GREEN}https://agentverse.ai/agents${NC}" echo "" echo " All 4 agents should show:" echo " βœ… Status: Connected" echo " βœ… Green indicator next to agent name" echo " βœ… 'Mailbox' in the connection type" echo "" echo -e "${YELLOW} If agents are NOT connected:${NC}" echo " 1. Find each agent by address on Agentverse" echo " 2. Click 'Connect to Mailbox' button" echo " 3. Wait for 'Connected' status" echo " 4. Come back and continue this script" echo "" echo -e "${BLUE}════════════════════════════════════ [truncated β€” 3966 more characters] ``` ### agents/messages.py ```python from uagents import Model from typing import List # ============================================================================ # CANARY AGENT MESSAGES # ============================================================================ class UpdatePackage(Model): """Represents a software update to be tested""" update_id: str version: str description: str target_systems: List[str] timestamp: float class CanaryTestResult(Model): """Result of canary testing""" update_id: str success: bool affected_systems: int error_rate: float latency_impact: float recommendation: str # "DEPLOY", "ROLLBACK", "INVESTIGATE" details: str # ============================================================================ # MONITORING AGENT MESSAGES # ============================================================================ class SystemMetrics(Model): """Real-time system metrics""" system_id: str cpu_usage: float memory_usage: float disk_usage: float network_latency: float error_count: int timestamp: float class AnomalyAlert(Model): """Alert when anomaly detected""" alert_id: str severity: str # "LOW", "MEDIUM", "HIGH", "CRITICAL" system_id: str metric_type: str current_value: float expected_value: float confidence: float timestamp: float recommendation: str # ============================================================================ # RESPONSE AGENT MESSAGES # ============================================================================ class ResponseAction(Model): """Action taken by response agent""" action_id: str action_type: str target_systems: List[str] reason: str status: str timestamp: float lava_request_id: str = "" # Track Lava usage # ============================================================================ # COMMUNICATION AGENT MESSAGES # ============================================================================ class StatusUpdate(Model): """Status update for stakeholders""" incident_id: str status: str # "INVESTIGATING", "MITIGATING", "RESOLVED" title: str description: str affected_services: List[str] timestamp: float ``` ### agents/base_agent.py ```python from uagents import Agent, Context, Model from loguru import logger import os from typing import Optional, Dict, Any, List from agents.registry import register_agent, get_agent_address, registry from dotenv import load_dotenv class BaseAgentConfig(Model): """Base configuration for all agents""" agent_name: str agent_description: str seed_phrase: Optional[str] = None class AgentMessage(Model): """Standard message format between agents""" sender: str receiver: str message_type: str payload: Dict[str, Any] timestamp: float class BaseSuraAgent: def __init__( self, name: str, seed: str, port: int, capabilities: List[str] = None, endpoint: Optional[str] = None ): load_dotenv() self.agent = Agent( name=name, seed=seed, port=port, mailbox=True # Enable Agentverse mailbox ) self.name = name self.capabilities = capabilities or [] # Setup logging logger.add( f"logs/{name}.log", rotation="100 MB", retention="7 days", level="INFO" ) logger.info(f"βœ… {name} initialized (Mailbox mode)") logger.info(f" Address: {self.agent.address}") logger.info(f" Port: {port} (local backup)") logger.info(f" πŸ“¬ Mailbox: ENABLED - Register this address on Agentverse!") # Auto-register in registry self.register_self() def register_self(self): """Register this agent in the global registry""" try: register_agent( name=self.name, address=str(self.agent.address), port=self.agent._port, capabilities=self.capabilities ) logger.info(f"πŸ“ Registered {self.name} in agent registry") except Exception as e: logger.error(f"Failed to register agent: {e}") def get_peer_address(self, peer_name: str) -> Optional[str]: """Get another agent's address from registry""" address = get_agent_address(peer_name) if address: logger.debug(f"Found {peer_name} at {address[:20]}...") else: logger.warning(f"Agent {peer_name} not found in registry") return address async def send_to_peer(self, ctx: Context, peer_name: str, message: Model) -> bool: address = self.get_peer_address(peer_name) if not address: logger.error(f"❌ Cannot send to {peer_name} - not in registry") return False try: # Use standard ctx.send - it handles all routing automatically await ctx.send(address, message) logger.info(f"βœ… Sent {message.__class__.__name__} to {peer_name}") return True except Exception as e: logger.error(f"❌ Failed to send to {peer_name}: {e}") return False def get_agent(self): return self.agent def get_address(self): return self.agent.address ``` ### services/mock_infrastructure.py ```python from fastapi import FastAPI, HTTPException from pydantic import BaseModel from typing import List, Dict import random import time from loguru import logger app = FastAPI(title="SuraAI Mock Infrastructure") # Simulated systems systems = {f"server-{i}": { "status": "healthy", "cpu": random.uniform(20, 40), "memory": random.uniform(30, 50), "version": "1.0.0", "last_update": None } for i in range(100)} class UpdateRequest(BaseModel): update_id: str version: str target_systems: List[str] class SystemStatus(BaseModel): system_id: str status: str metrics: Dict[str, float] @app.get("/") def root(): return {"service": "SuraAI Mock Infrastructure", "systems": len(systems)} @app.get("/systems") def get_systems(): """Get all systems""" return {"systems": list(systems.keys()), "total": len(systems)} @app.get("/system/{system_id}") def get_system(system_id: str): """Get specific system status""" if system_id not in systems: raise HTTPException(status_code=404, detail="System not found") return systems[system_id] @app.post("/deploy") def deploy_update(update: UpdateRequest): """Deploy update to systems""" logger.info(f"Deploying {update.update_id} to {len(update.target_systems)} systems") # Simulate deployment deployed = [] failed = [] for sys_id in update.target_systems: if sys_id in systems: # 5% chance of failure for demo if random.random() < 0.05: systems[sys_id]["status"] = "failed" failed.append(sys_id) else: systems[sys_id]["version"] = update.version systems[sys_id]["last_update"] = time.time() deployed.append(sys_id) return { "update_id": update.update_id, "deployed": len(deployed), "failed": len(failed), "failed_systems": failed } @app.post("/rollback/{system_id}") def rollback_system(system_id: str): """Rollback a system""" if system_id not in systems: raise HTTPException(status_code=404, detail="System not found") systems[system_id]["version"] = "1.0.0" systems[system_id]["status"] = "healthy" logger.info(f"Rolled back {system_id}") return {"system_id": system_id, "status": "rolled_back"} @app.post("/simulate-failure/{system_id}") def simulate_failure(system_id: str): """Simulate a failure for testing""" if system_id not in systems: raise HTTPException(status_code=404, detail="System not found") systems[system_id]["status"] = "failed" systems[system_id]["cpu"] = 95.0 systems[system_id]["memory"] = 98.0 logger.warning(f"Simulated failure on {system_id}") return {"system_id": system_id, "status": "failure_simulated"} @app.get("/health") def health(): """Health check""" healthy = sum(1 for s in systems.values() if s["status"] == "healthy") return { "total_systems": len(systems), "healthy": healthy, "unhealthy": len(systems) - healthy } if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=8000) ``` ### agents/registry.py ```python from typing import Dict, Optional, List from dataclasses import dataclass, field import json import os from loguru import logger from dotenv import load_dotenv from pathlib import Path @dataclass class AgentInfo: name: str address: str port: int capabilities: List[str] status: str = "active" last_seen: float = 0.0 class AgentRegistry: """ Central registry for agent discovery and communication Supports both in-memory and file-based persistence """ def __init__(self, registry_file: str = "agent_registry.json"): self.agents: Dict[str, AgentInfo] = {} self.registry_file = registry_file self.load_registry() def register(self, name: str, address: str, port: int, capabilities: List[str]) -> None: """Register an agent in the system""" self.agents[name] = AgentInfo( name=name, address=address, port=port, capabilities=capabilities ) logger.info(f"βœ… Registered: {name} at {address[:20]}... (port {port})") self.save_registry() def get_agent(self, name: str) -> Optional[AgentInfo]: """Get agent info by name""" return self.agents.get(name) def get_agent_address(self, name: str) -> Optional[str]: """Get agent address by name (convenience method)""" agent = self.get_agent(name) return agent.address if agent else None def get_all_agents(self) -> Dict[str, AgentInfo]: """Get all registered agents""" return self.agents def get_agents_by_capability(self, capability: str) -> List[AgentInfo]: """Find agents with specific capability""" return [ agent for agent in self.agents.values() if capability in agent.capabilities ] def save_registry(self) -> None: """Persist registry to file""" try: data = { name: { "name": info.name, "address": info.address, "port": info.port, "capabilities": info.capabilities, "status": info.status } for name, info in self.agents.items() } with open(self.registry_file, 'w') as f: json.dump(data, f, indent=2) except Exception as e: logger.error(f"Failed to save registry: {e}") def load_registry(self) -> None: """Load registry from file""" if not os.path.exists(self.registry_file): logger.info("No existing registry found, starting fresh") return try: with open(self.registry_file, 'r') as f: data = json.load(f) for name, info in data.items(): self.agents[name] = AgentInfo(**info) logger.info(f"βœ… Loaded {len(self.agents)} agents from registry") except Exception as e: logger.error(f"Failed to load registry: {e}") def clear_registry(self) -> None: """Clear all registered agents""" self.agents.clear() self.save_registry() logger.info("πŸ—‘οΈ Registry cleared") def print_registry(self) -> None: """Print all registered agents""" print("\n" + "="*70) print("πŸ“‹ AGENT REGISTRY") print("="*70) if not self.agents: print("No agents registered") return for name, info in self.agents.items(): print(f"\nπŸ€– {name.upper()}") print(f" Address: {info.address}") print(f" Port: {info.port}") print(f" Capabilities: {', '.join(info.capabilities)}") print(f" Status: {info.status}") print("\n" + "="*70) # Global registry instance registry = AgentRegistry() # Convenience functions for agents to use def register_agent(name: str, address: str, port: int, capabilities: List[str]) -> None: """Register this agent in the global registry""" registry.register(name, address, port, capabilities) def get_agent_address(name: str) -> Optional[str]: """Get another agent's address""" return registry.get_agent_address(name) def discover_agents(capability: str) -> List[str]: """Find all agents with a specific capability""" agents = registry.get_agents_by_capability(capability) return [agent.address for agent in agents] ``` [23 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]