A sophisticated, production-ready testing framework for the Automatos AI multi-agent orchestration platform. This repository provides comprehensive testing capabilities including API testing, performance monitoring, security validation, user journey validation, and continuous integration support.
- Isolated Component Testing: Test agents, workflows, context engineering, and memory systems independently
- Integration Testing: End-to-end testing across all system components
- Performance Testing: Load testing, stress testing, and performance benchmarking
- Security Testing: Vulnerability scanning, penetration testing, and security validation
- Multi-Agent System Testing: Collaboration, coordination, and consensus mechanisms
- Context Engineering Validation: RAG systems, embeddings, and mathematical foundations
- Memory System Testing: Hierarchical memory, consolidation, and optimization
- Field Theory Integration: Mathematical field operations and context interactions
- Automated Test Execution: Scheduled and trigger-based testing
- Continuous Monitoring: Real-time system health and performance tracking
- Alert Management: Automated notifications via Slack, email, and webhooks
- Reporting Integration: Automatic report generation and distribution
- Multi-Format Reports: HTML, JSON, JUnit XML for CI/CD integration
- Performance Metrics: Response times, throughput, and resource utilization
- Security Analysis: Vulnerability reports and security compliance
- Trend Analysis: Historical data and performance trending
This testing suite validates the complete Automatos AI ecosystem, including:
- Agent Management: Lifecycle, skills, and execution testing
- Workflow Orchestration: Pattern management and execution flows
- Document Management: Processing, analytics, and retrieval
- Context Engineering: RAG system and vector embeddings
- Performance Analytics: System metrics and monitoring
automatos-testing/
βββ framework/ # Core testing framework
β βββ config.py # Configuration management
β βββ base_test.py # Base test classes and utilities
β βββ test_runner.py # Test orchestration and execution
β βββ reporting.py # Report generation and formatting
βββ tests/ # Test suites
β βββ test_agents.py # Agent management testing
β βββ test_workflows.py # Workflow orchestration testing
β βββ test_context_engineering.py # Context and RAG testing
β βββ test_multi_agent.py # Multi-agent systems testing
β βββ test_memory_systems.py # Memory management testing
β βββ test_performance_security.py # Performance and security testing
βββ testing_suites/
β βββ comprehensive_5phase/ # Main test suite
β β βββ run_all_tests.py # Master test runner
β β βββ shared_config.yaml # Test configuration
β β βββ shared_utils.py # Common utilities
β β βββ phase1_agent_management/
β β βββ phase2_workflow_orchestration/
β β βββ phase3_document_management/
β β βββ phase4_context_engineering/
β β βββ phase5_performance_analytics/
β βββ journey_tests/ # User journey tests
βββ n8n/ # N8N automation workflows
β βββ automatos-testing-workflow.json # Automated test execution
β βββ continuous_monitoring.json # System monitoring
βββ reports/ # Generated test reports
βββ results/ # Test results (generated)
βββ logs/ # Execution logs (generated)
βββ test_data/ # Test data and fixtures
βββ scripts/ # Utility scripts
- Python 3.8+ (Python 3.9+ recommended)
- Network access to
https://api.automatos.app
- Required Python packages (see requirements.txt)
- Optional: N8N for workflow automation (http://localhost:5678)
# Clone the testing framework
git clone git@github.com:AutomatosAI/automatos-testing.git
cd automatos-testing
# Install dependencies
pip install -r requirements.txt
# Set up environment (optional)
cp .env.example .env
# Edit .env with your configuration
# Run all tests (comprehensive 5-phase suite)
cd testing_suites/comprehensive_5phase
python3 run_all_tests.py
# Alternative: Use the main test runner
python run_tests.py
# Run specific test suites
python run_tests.py --filter agents
python run_tests.py --filter "multi_agent"
# Run by test level
python run_tests.py --level integration
python run_tests.py --level performance
# Run against different environments
python run_tests.py --environment staging
python run_tests.py --environment production
# Generate comprehensive reports
python run_tests.py --reports --parallel
# Run modules in fixed order and write per-module JSON to reports/<module>.json
python3 run_tests.py --module-sequence --json reports/sequence.json
# Stop on first module failure
python3 run_tests.py --module-sequence --fail-fast
# Point runner at a different API base
python3 run_tests.py --api-url https://api.automatos.app --reports
- Import JSONs from
n8n/
:n8n/workflow_backend_first.json
(backend-first, gate, review/fix/deploy)n8n/workflow_parallel.json
(parallel module tests β single review/fix/deploy)
Set environment variables in n8n:
APP_DIR
,API_BASE
,CANARY_WORKFLOW_ID
, and (optionally)REVIEWER_PROMPT
.
# Agent Management
cd testing_suites/comprehensive_5phase/phase1_agent_management/scripts
python3 comprehensive_agent_test.py
# Workflow Orchestration
cd testing_suites/comprehensive_5phase/phase2_workflow_orchestration/scripts
python3 comprehensive_workflow_test.py
# Document Management
cd testing_suites/comprehensive_5phase/phase3_document_management/scripts
python3 comprehensive_document_test.py
# Context Engineering
cd testing_suites/comprehensive_5phase/phase4_context_engineering/scripts
python3 comprehensive_context_test.py
# Performance Analytics
cd testing_suites/comprehensive_5phase/phase5_performance_analytics/scripts
python3 comprehensive_performance_test.py
Test Phase | Status | Coverage | Endpoints Tested |
---|---|---|---|
Agent Management | β Ready | Agent CRUD, Skills, Execution | /api/agents/* |
Workflow Orchestration | β Ready | Patterns, Templates, Execution | /api/workflows/* |
Document Management | β Ready | Upload, Processing, Analytics | /api/documents/* |
Context Engineering | β Ready | RAG, Embeddings, Retrieval | /api/context/* |
Performance Analytics | β Ready | Metrics, Health, Monitoring | /api/system/* |
- 70% API Success Rate required for each phase to pass
- Real API validation with no fake success reporting
- Comprehensive logging of all requests and responses
- Professional result reporting in JSON and Markdown formats
- Agent CRUD operations
- Skills and patterns management
- Agent execution and coordination
- Performance monitoring
- Health checks
- Workflow creation and management
- Sequential, parallel, and conditional workflows
- Real-time progress monitoring
- N8N integration
- Advanced execution strategies
- Information theory (entropy, mutual information)
- Vector operations (embeddings, similarity, clustering)
- Statistical analysis and optimization
- RAG system testing
- Knowledge graph operations
- Bayesian inference
- Collaborative reasoning sessions
- Consensus mechanisms and conflict resolution
- Agent coordination strategies
- Behavior monitoring and analysis
- System optimization
- Load balancing and scalability
- Hierarchical memory structures
- Working memory operations
- Episodic memory management
- Memory consolidation and optimization
- Associative retrieval
- Backup and restore
- API response time testing
- Concurrent load testing
- Memory usage monitoring
- Throughput measurement
- Security vulnerability scanning
- Authentication and authorization
- Input validation and sanitization
- HTTPS:
https://api.automatos.app
(Recommended) - Backup:
http://206.81.0.227:8000
(Direct IP access)
curl https://api.automatos.app/health
# Expected: {"status":"healthy","service":"automotas-ai-api"}
# Agent Management
GET /api/agents/ # List all agents
POST /api/agents/ # Create new agent
GET /api/agents/{id} # Get agent details
POST /api/agents/{id}/skills # Add agent skills
# System Health
GET /health # Basic health check
GET /api/system/health # Detailed health info
GET /api/system/metrics # Performance metrics
# Context & Analytics
GET /api/context/stats # Context statistics
GET /api/workflows/active # Active workflows
GET /api/documents/analytics/overview # Document analytics
automatos-testing/
βββ README.md # This file
βββ requirements.txt # Python dependencies
βββ TESTING_INDEX.md # Testing overview
βββ .gitignore # Git ignore file
βββ LICENSE # MIT License
βββ testing_suites/
β βββ comprehensive_5phase/ # Main test suite
β β βββ run_all_tests.py # Master test runner
β β βββ shared_config.yaml # Test configuration
β β βββ shared_utils.py # Common utilities
β β βββ phase1_agent_management/
β β βββ phase2_workflow_orchestration/
β β βββ phase3_document_management/
β β βββ phase4_context_engineering/
β β βββ phase5_performance_analytics/
β βββ journey_tests/ # User journey tests
βββ results/ # Test results (generated)
βββ logs/ # Execution logs (generated)
βββ scripts/ # Utility scripts
# Core Configuration
TEST_ENVIRONMENT=development
API_BASE_URL=https://api.automatos.app
API_KEY=your_api_key_here
# Database
TEST_DB_URL=postgresql://postgres:test@localhost:5432/test_db
CLEANUP_DB=true
SEED_TEST_DATA=true
# N8N Integration
N8N_BASE_URL=http://localhost:5678
WEBHOOK_BASE_URL=http://localhost:3001
# Testing Options
PARALLEL_TESTS=true
GENERATE_REPORTS=true
REPORT_FORMAT=html
Edit testing_suites/comprehensive_5phase/shared_config.yaml
:
api:
base_url: "https://api.automatos.app"
fallback_urls:
- "http://206.81.0.227:8000"
timeout: 30
max_retries: 3
performance:
success_threshold: 0.7 # 70% success rate required
from framework.config import test_config
# Access configuration
print(f"API URL: {test_config.api.base_url}")
print(f"Environment: {test_config.environment}")
print(f"Parallel Tests: {test_config.parallel_tests}")
- Master Reports:
MASTER_SUMMARY_*.md
andMASTER_TEST_REPORT_*.json
- Individual Results:
phase*/results/*.json
- API Logs:
phase*/responses/*.json
- Execution Logs:
*.log
files with timestamps
Tests use real API validation with these criteria:
- API Response Validation: Must receive valid HTTP responses
- Content Verification: Response data must be properly formatted
- Performance Thresholds: Response times under 1000ms preferred
- Success Rate: 70% of API calls must succeed for phase to pass
π AGENT MANAGEMENT TESTING COMPLETED
π Total API Calls: 15
β
Successful: 12
β Failed: 3
π Success Rate: 80.0%
π― Required Threshold: 70.0%
β
AGENT MANAGEMENT TESTS PASSED - Meeting minimum threshold
The framework generates comprehensive reports in multiple formats:
- Visual Dashboard: Interactive charts and graphs
- Test Results: Detailed pass/fail status with error details
- Performance Metrics: Response times, throughput, resource usage
- Trend Analysis: Historical performance data
- Machine-Readable: Perfect for CI/CD integration
- Detailed Results: Complete test execution data
- API Integration: Easy integration with external systems
- CI/CD Integration: Compatible with Jenkins, GitHub Actions, etc.
- Test Management: Integration with test management tools
- Continuous Monitoring: Automated test result tracking
The framework includes N8N workflows for automated testing:
- System Health Check: Verify system availability
- Test Suite Execution: Run comprehensive test suites
- Result Analysis: Analyze test results and generate reports
- Notification: Send alerts via Slack, email, or webhooks
Real-time system monitoring with automated alerts:
- Every 15 minutes: System health checks
- Performance tracking: Response times and resource usage
- Alert conditions: Automatic notifications for failures
- Metric logging: Historical data collection
# Import workflows into N8N
# 1. Open N8N (http://localhost:5678)
# 2. Go to Workflows > Import from file
# 3. Import n8n/automatos-testing-workflow.json
# 4. Import n8n/continuous_monitoring.json
# 5. Configure webhook URLs and credentials
# 6. Activate workflows
from framework.base_test import APITest
class TestCustomFeature(APITest):
def __init__(self):
super().__init__("CustomFeature")
async def setup(self):
await self.setup_session()
# Custom setup logic
async def test_custom_functionality(self):
response = await self.make_request("GET", "/api/custom/endpoint")
assert response["status_code"] == 200
# Custom assertions
async def cleanup(self):
# Custom cleanup logic
await self.cleanup_session()
# Run performance tests with custom parameters
python run_tests.py \
--filter performance \
--environment production \
--api-url https://api.automatos.app \
--verbose
# Run comprehensive security tests
python run_tests.py \
--filter security \
--level security \
--reports
- Response Time: Average API response time (target: <100ms)
- Throughput: Requests per second (target: >1000 RPS)
- Success Rate: Percentage of successful requests (target: >99.9%)
- Error Rate: Percentage of failed requests (target: <0.1%)
- Agent Availability: Percentage of agents online and responsive
- Memory Usage: System memory utilization
- CPU Usage: Processor utilization
- Database Performance: Query response times and connection health
- Vulnerability Count: Number of security vulnerabilities detected
- Authentication Success Rate: Successful authentication attempts
- Failed Login Attempts: Potential security threats
- Security Alert Count: Number of security events triggered
Test Failures Due to API Unavailability
# Check if Automatos AI backend is running
curl https://api.automatos.app/health
# Verify configuration
python -c "from framework.config import test_config; print(test_config.api.base_url)"
Database Connection Issues
# Check database connection
python -c "from framework.config import test_config; print(test_config.database.test_db_url)"
# Test database connectivity
psql postgresql://postgres:test@localhost:5432/test_db -c "SELECT 1;"
N8N Integration Issues
# Verify N8N is running
curl http://localhost:5678/rest/active-workflows
# Check webhook endpoints
curl http://localhost:3001/webhook/test-webhook
# Run tests with verbose logging
python run_tests.py --verbose
# Run single test for debugging
python run_tests.py --filter "test_specific_function" --verbose
- β
Agent listing and details (
/api/agents/*
) - β
System health and metrics (
/api/system/*
) - β
Context statistics (
/api/context/stats
) - β
Workflow active status (
/api/workflows/active
) - β
Document analytics (
/api/documents/analytics/*
)
β οΈ Some workflow pattern endpoints return 404β οΈ File upload testing limited (multipart uploads)β οΈ Some advanced context operations not implemented
- Test Coverage: Maintain >95% test coverage
- Documentation: Document all new test cases
- Performance: Ensure tests complete within reasonable time limits
- Security: Follow security best practices in test development
- Create test file in
tests/
directory - Inherit from appropriate base class (
APITest
,DatabaseTest
, etc.) - Implement setup, test methods, and cleanup
- Add comprehensive assertions and error handling
- Update documentation
- Test files:
test_<component>.py
- Test classes:
Test<Component>
- Test methods:
test_<specific_functionality>
- Clone the repository
- Install dependencies:
pip install -r requirements.txt
- Run tests:
python3 run_all_tests.py
- Review results in generated reports
- Use GitHub Issues for bug reports
- Include test logs and error messages
- Specify which phase/endpoint failed
- Provide system information (Python version, OS)
This testing framework is part of the Automatos AI project and is licensed under the MIT License. See LICENSE file for details.
- GitHub Issues: Report bugs and request features
- Discussions: Community discussions and Q&A
- Documentation: Comprehensive guides and tutorials
- Priority Support: 24/7 support with guaranteed response times
- Custom Test Development: Tailored test suites for specific requirements
- Training and Consulting: Expert guidance on testing best practices
- SLA Guarantees: Performance and reliability commitments
Contact: testing-support@automatos.ai
Built with β€οΈ by the Automatos AI Team
Ensuring reliability and performance through comprehensive testing
Note: This testing suite provides real validation of the Automatos AI platform. Results reflect actual system functionality and API health. No fake or simulated results are generated.
For more information, visit Automatos.ai