A complete AWX-based network device upgrade management system designed for managing firmware upgrades across 1000+ heterogeneous network devices with comprehensive validation, security, and monitoring.
This system provides automated firmware upgrade capabilities for:
- Cisco NX-OS (Nexus Switches) with ISSU support
- Cisco IOS-XE (Enterprise Routers/Switches) with Install Mode
- Metamako MOS (Ultra-Low Latency Switches) with Application Management
- Opengear (Console Servers/Smart PDUs) with multi-architecture support
- FortiOS (Fortinet Firewalls) with HA coordination
Status: Production ready for all platforms. See Platform Implementation Status for detailed status.
- Phase 1: Image Loading (business hours safe)
- Phase 2: Image Installation (maintenance window)
- Complete rollback capabilities
- Server-Initiated PUSH Transfers Only - All firmware pushed from upgrade server to devices
- Zero Device-Initiated Operations - No device-to-server connections for firmware retrieval
- SSH Key Authentication Priority - SSH keys preferred over password authentication
- SHA512 Hash Verification - Complete integrity validation for all firmware images
- Cryptographic Signature Verification - Where supported by platform
- Complete Security Audit Trail - All operations logged and verified
- Pre/post upgrade network state comparison
- BGP, BFD, IGMP/multicast, routing validation
- IPSec tunnel and VPN connectivity validation
- Interface optics and transceiver health monitoring
- Protocol convergence timing with baseline comparison
- Native systemd service deployment (AWX and NetBox)
- Pre-existing NetBox integration
- InfluxDB v2 metrics integration
- ✅ Complete Grafana dashboard automation with multi-environment support
- ✅ Real-time operational monitoring with 15-second refresh dashboards
- Existing monitoring system integration
# 1. Install base system
./install/setup-system.sh
# 2. Setup AWX with native services
./install/setup-awx.sh
# 3. Setup NetBox with native services
./install/setup-netbox.sh
# 4. Configure monitoring integration
./install/configure-telegraf.sh
# 5. Set up SSL certificates
./install/setup-ssl.sh
# 6. Start all services
./install/create-services.sh
# 7. Deploy Grafana dashboards
cd integration/grafana
export INFLUXDB_TOKEN="your_token_here"
./provision-dashboards.shComprehensive testing capabilities for Mac/Linux development without physical devices:
- ✅ Syntax Validation: 100% CLEAN - All 69+ Ansible files pass syntax checks
- ✅ Security Validation: 100% COMPLIANT - All secure transfer tests pass (10/10)
- ✅ Test Suite Pass Rate: 100% - All 23 test suites passing cleanly ✅
- ✅ Container Integration: SUCCESS - Multi-architecture images (amd64/arm64) available
- ✅ Molecule Testing: 5/9 ROLES - Critical roles configured with Docker testing
- ✅ Container Tests: OPTIMIZED - Parallel execution in 18 minutes (6 test suites)
# Syntax validation (100% clean)
ansible-playbook --syntax-check ansible-content/playbooks/main-upgrade-workflow.yml
# Mock device testing (all 5 platforms)
ansible-playbook -i tests/mock-inventories/all-platforms.yml --check \
ansible-content/playbooks/main-upgrade-workflow.yml
# Complete test suite
./tests/run-all-tests.sh
# Molecule testing (requires Docker)
cd tests/molecule-tests && molecule test
# Container testing (production ready)
docker run --rm ghcr.io/garryshtern/network-device-upgrade-system:latest
podman run --rm ghcr.io/garryshtern/network-device-upgrade-system:latest- Mock Inventory Testing - Simulated device testing for all platforms ✅
- Variable Validation - Requirements and constraint validation ✅
- Template Rendering - Jinja2 template testing without connections ✅
- Workflow Logic - Decision path and conditional testing ✅
- Error Handling - Error condition and recovery validation ✅
- Integration Testing - Complete workflow with mock devices ✅
- Performance Testing - Execution time and resource measurement ✅
- Molecule Testing - Container-based advanced testing ✅
- Platform-Specific Testing - Vendor-specific comprehensive testing ✅
- YAML/JSON Validation - File syntax and structure validation ✅
- CI/CD Integration - GitHub Actions automated testing ✅
See comprehensive guide: Testing Framework Guide
Complete documentation with architectural diagrams and implementation guides:
- 📖 Documentation Hub - Start here for comprehensive guides
- ⚙️ Installation Guide - Step-by-step deployment
- 🔄 Workflow Guide - Upgrade process and safety mechanisms
- 🏗️ Platform Guide - Technical implementation details
- 📊 Implementation Status - Current completion analysis
- 🧪 Testing Framework Guide - Comprehensive testing without physical devices
- 🧪 Molecule Testing Guide - Role-specific container testing
graph TD
A[AWX Services<br/>Job Control<br/>systemd] --> B[Ansible Engine<br/>Playbook Execution<br/>Role-Based]
B --> C[Network Devices<br/>1000+ Supported<br/>Multi-Vendor]
D[NetBox<br/>Inventory DB<br/>Pre-existing] --> B
E[Telegraf<br/>Metrics Agent<br/>Collection] --> F[InfluxDB v2<br/>Time Series<br/>Existing]
C --> F
F --> H[Grafana<br/>Dashboards<br/>Existing]
C -.-> I[Cisco NX-OS]
C -.-> J[Cisco IOS-XE]
C -.-> K[FortiOS]
C -.-> L[Metamako MOS]
C -.-> M[Opengear]
style A fill:#e1f5fe
style C fill:#f3e5f5
style F fill:#e8f5e8
style H fill:#fff3e0
Alternative System Flow:
| Component | Function | Integration |
|---|---|---|
| AWX Services (systemd) | Job orchestration and workflow control | → Ansible Engine |
| Ansible Engine | Playbook execution and device automation | → Network Devices |
| NetBox (Pre-existing) | Device inventory and IPAM management | → Ansible Engine |
| Telegraf | Metrics collection agent | → InfluxDB v2 |
| Network Devices | Target devices for upgrades | → Metrics Export |
| InfluxDB v2 | Time-series metrics storage | → Grafana |
| Grafana | Monitoring dashboards and visualization | Final consumer |
flowchart TD
U[User Request] --> A[AWX Web UI]
A --> B[Job Templates]
B --> C[Workflows]
B --> D[Dynamic Inventory]
D --> E[NetBox<br/>Device Data<br/>Variables]
C --> F[Ansible Execution]
D --> F
F --> G[Network Devices]
G --> H[Metrics Collection]
H --> I[InfluxDB]
I --> J[Grafana<br/>Dashboards]
subgraph "Job Templates"
B1[Health Check]
B2[Image Load]
B3[Validation]
end
subgraph "Workflows"
C1[Phase 1: Load]
C2[Phase 2: Install]
C3[Phase 3: Verify]
end
style U fill:#ffeb3b
style G fill:#f3e5f5
style I fill:#e8f5e8
style J fill:#fff3e0
Simplified Data Flow:
- User Request → AWX Web Interface
- AWX → Executes Ansible playbooks
- Ansible → Connects to network devices via SSH/API
- NetBox → Provides device inventory to Ansible
- Network Devices → Export metrics during operations
- Telegraf → Collects metrics and sends to InfluxDB
- InfluxDB → Stores time-series data for Grafana
- Grafana → Displays dashboards and reports to users
- OS: RHEL/CentOS 8+ or Ubuntu 20.04+
- CPU: 4 cores minimum
- RAM: 8GB minimum
- Storage: 100GB+ for firmware and logs
- Network: Reliable connectivity to all managed devices
- Python: 3.13 with pip
- Ansible: 11.9.0 (ansible-core 2.19.1) - Latest stable version
- Git: Latest stable version
- Single Server Deployment: No clustering required
- Container-based AWX: Podman/Docker container deployment
- Pre-existing NetBox: Uses existing NetBox installation
- SystemD User Services: Native Linux user service management for base components
network-upgrade-system/
├── deployment/ # Service-based deployment structure
│ ├── system/ # Base system setup (SSL, system config)
│ ├── services/ # Individual service deployments
│ │ ├── awx/ # AWX automation platform
│ │ ├── netbox/ # NetBox IPAM & device inventory
│ │ ├── grafana/ # ✅ Complete dashboard automation
│ │ ├── telegraf/ # Metrics collection
│ │ └── redis/ # Caching & job queue
│ └── scripts/ # General deployment scripts
├── ansible-content/ # Ansible automation content
│ ├── playbooks/ # Main orchestration playbooks
│ ├── roles/ # Vendor-specific upgrade roles
│ └── collections/ # Ansible collection requirements
├── tests/ # Comprehensive test suites
├── docs/ # Complete documentation
├── tools/ # Development and utility tools
└── .claude/ # Claude Code commands and workflows
- 📘 Installation Guide - Complete setup instructions
- 🔄 Upgrade Workflow Guide - Upgrade process and safety mechanisms
- 🏗️ Platform Implementation Guide - Technical implementation details
- 📊 Grafana Integration - Dashboard automation and monitoring
- 📖 Documentation Hub - Complete documentation index
For technical support and questions:
- Check the Installation Guide troubleshooting section
- Review platform-specific procedures in Platform Implementation Guide
- Examine log files in
$HOME/.local/share/network-upgrade/logs/ - Use the built-in health check:
./scripts/system-health.sh
This project is licensed under the MIT License - see the LICENSE file for details.