upskill-event-manager/docs/MONITORING-SYSTEMS.md

# HVAC Plugin Monitoring Systems

This document describes the comprehensive enterprise-level monitoring and reliability systems implemented in the HVAC Community Events plugin.

## Overview

The plugin includes four integrated monitoring systems:

1. **Health Monitor** - Automated health checks and system validation
2. **Error Recovery** - Automatic error recovery and graceful degradation
3. **Security Monitor** - Real-time threat detection and response
4. **Performance Monitor** - Performance tracking and optimization alerts

## Health Monitor

### Features
- 8 different health check types
- Automated hourly checks with email alerts
- Admin dashboard integration
- REST API endpoints for external monitoring
- WP-CLI integration

### Health Check Types
- **Database Connectivity** - Tests database connection and table integrity
- **Cache System** - Validates WordPress object cache functionality
- **User Authentication** - Verifies role system and user capabilities
- **Event Management** - Checks The Events Calendar integration
- **Certificate System** - Validates certificate page existence and permissions
- **Background Jobs** - Monitors background job queue health
- **File Permissions** - Checks critical directory permissions
- **Third Party Integrations** - Validates external plugin dependencies

### Usage

#### Admin Interface
Navigate to `Tools > HVAC Health` to view comprehensive health status.

#### WP-CLI
```bash
wp hvac health
```

#### REST API
```
GET /wp-json/hvac/v1/health
```

### Configuration
Health checks run automatically every hour. Critical issues trigger immediate email alerts to the admin email address.

## Error Recovery System

### Features
- 4 recovery strategies for different failure scenarios
- Circuit breaker pattern for external services
- Emergency mode activation for critical failures
- Comprehensive error tracking and statistics

### Recovery Strategies

#### 1. Retry with Exponential Backoff
Used for: Database queries, temporary failures
- Max attempts: 3
- Backoff multiplier: 2

#### 2. Fallback Operations
Used for: Cache operations, non-critical services
- Falls back to safe alternatives
- Skips functionality gracefully

#### 3. Circuit Breaker
Used for: External APIs, third-party services
- Opens after 5 failures
- 5-minute timeout period
- Uses cached data when available

#### 4. Graceful Failure
Used for: File operations, optional features
- Logs errors and continues operation
- Returns safe default values

### Usage

#### Programmatic Usage
```php
$result = HVAC_Error_Recovery::execute_with_recovery(
    'database_query',
    function() {
        // Your database operation
        return $wpdb->get_results("SELECT * FROM table");
    }
);
```

#### Admin Interface
Navigate to `Tools > HVAC Error Recovery` to view error statistics and manage emergency mode.

### Emergency Mode
Automatically activated on fatal errors. Disables problematic functionality and sends immediate email alerts.

## Security Monitor

### Features
- Real-time threat detection
- Automatic IP blocking for malicious activity
- Comprehensive security event logging
- File integrity monitoring
- Database query analysis

### Monitored Threats
- **Failed Login Attempts** - Brute force attack detection
- **SQL Injection** - Pattern detection in requests and queries
- **XSS Attempts** - Cross-site scripting pattern detection
- **File Modification** - Critical plugin file integrity checks
- **Privilege Escalation** - Unauthorized admin actions
- **Suspicious Activity** - Plugin/theme installation monitoring

### Security Settings
```php
$settings = [
    'max_failed_logins' => 5,
    'lockout_duration' => 900,      // 15 minutes
    'monitor_file_changes' => true,
    'scan_requests' => true,
    'alert_threshold' => 3,
    'auto_block_ips' => true
];
```

### Usage

#### Admin Interface
Navigate to `Tools > HVAC Security` to view security events, blocked IPs, and threat statistics.

#### WP-CLI
```bash
wp hvac security stats
wp hvac security events
```

#### REST API
```
GET /wp-json/hvac/v1/security/stats
```

### IP Blocking
Automatic IP blocking triggers on:
- 5+ failed login attempts in 1 hour
- SQL injection attempts
- Critical threat patterns

## Performance Monitor

### Features
- Real-time performance tracking
- Automated performance benchmarks
- Memory usage monitoring
- Database query analysis
- Cache performance tracking

### Performance Metrics
- **Page Load Time** - Full request processing time
- **Memory Usage** - Peak memory consumption
- **Database Queries** - Query count and slow query detection
- **Cache Hit Rate** - Object cache effectiveness
- **File I/O Performance** - Disk operation speed

### Thresholds
```php
const THRESHOLDS = [
    'slow_query_time' => 2.0,        // 2 seconds
    'memory_usage_mb' => 128,        // 128 MB
    'page_load_time' => 3.0,         // 3 seconds
    'db_query_count' => 100,         // 100 queries per request
    'cache_hit_rate' => 70           // 70% cache hit rate
];
```

### Usage

#### Admin Interface
Navigate to `Tools > HVAC Performance` to view performance statistics and run benchmarks.

#### Admin Bar Integration
Performance stats appear in the admin bar for logged-in administrators.

#### WP-CLI
```bash
wp hvac performance stats
wp hvac performance benchmark
```

#### REST API
```
GET /wp-json/hvac/v1/performance/stats
```

### Benchmarking
Automated daily benchmarks test:
- Database query performance
- Memory allocation speed
- Cache read/write operations
- File I/O performance

Performance degradation detection compares current benchmarks with previous results and alerts on 50%+ degradation.

## Deployment Validation

### Features
- 8 critical deployment tests
- Pre-deployment validation
- Performance benchmarks during validation
- Security configuration checks

### Validation Tests
1. **Plugin Activation** - Verifies plugin is active with correct version
2. **Database Connectivity** - Tests database connection and queries
3. **Required Pages** - Checks all plugin pages exist with templates
4. **User Roles** - Validates HVAC trainer roles and capabilities
5. **Essential Functionality** - Tests shortcodes, background jobs, health monitoring
6. **Third Party Integrations** - Verifies The Events Calendar and theme integration
7. **Performance Benchmarks** - Runs performance tests during deployment
8. **Security Configurations** - Checks file permissions, nonce system, debug settings

### Usage

#### Command Line
```bash
php /path/to/plugin/scripts/deployment-validator.php
```

#### WP-CLI
```bash
wp hvac deployment
```

### Integration with Deployment Scripts
Add to your deployment scripts:
```bash
# Run deployment validation
if ! wp hvac deployment; then
    echo "Deployment validation failed!"
    exit 1
fi
```

## Integration and Architecture

### Singleton Pattern
All monitoring classes use the singleton pattern to prevent duplicate initialization:
```php
HVAC_Health_Monitor::init();
HVAC_Error_Recovery::init();
HVAC_Security_Monitor::init();
HVAC_Performance_Monitor::init();
```

### WordPress Integration
- **Cron Jobs** - Automated scheduling for health checks and benchmarks
- **Admin Menus** - Integrated admin interfaces under Tools menu
- **REST API** - RESTful endpoints for external monitoring
- **WP-CLI** - Command-line interface for automation
- **Admin Bar** - Real-time performance stats

### Database Storage
- Uses WordPress options table for configuration and metrics
- Automatic cleanup prevents database bloat
- Transient caching for frequently accessed data

### Error Handling
- Comprehensive error logging through HVAC_Logger
- Fail-safe mechanisms prevent monitoring from breaking site
- Graceful degradation when monitoring systems fail

## Configuration

### Health Monitor Settings
```php
update_option('hvac_health_settings', [
    'check_frequency' => 'hourly',
    'alert_email' => 'admin@example.com',
    'cache_duration' => 300
]);
```

### Security Monitor Settings
```php
update_option('hvac_security_settings', [
    'max_failed_logins' => 5,
    'lockout_duration' => 900,
    'monitor_file_changes' => true,
    'auto_block_ips' => true
]);
```

### Performance Monitor Settings
```php
update_option('hvac_performance_settings', [
    'email_alerts' => true,
    'alert_threshold' => 3,
    'benchmark_frequency' => 'daily'
]);
```

## Troubleshooting

### Common Issues

#### Health Checks Failing
1. Check database connectivity
2. Verify file permissions
3. Ensure The Events Calendar is active
4. Check WordPress cron system

#### Security Alerts Not Working
1. Verify admin email setting
2. Check email delivery system
3. Review security event logs
4. Test manual alert trigger

#### Performance Monitoring Inactive
1. Ensure monitoring conditions are met
2. Check if request should be monitored
3. Verify performance thresholds
4. Review performance event logs

### Debug Mode
Enable debug logging for detailed monitoring information:
```php
define('HVAC_DEBUG_MONITORING', true);
```

### Log Files
Monitor logs for system health:
- WordPress debug.log
- HVAC_Logger entries
- Server error logs

## Best Practices

### Production Deployment
1. Always run deployment validation before going live
2. Monitor health checks for first 24 hours post-deployment
3. Review security events regularly
4. Set up external monitoring for REST API endpoints

### Performance Optimization
1. Enable object caching for better cache hit rates
2. Monitor slow query logs and optimize problematic queries
3. Use performance benchmarks to identify degradation trends
4. Configure appropriate performance thresholds

### Security Hardening
1. Enable automatic IP blocking
2. Monitor file integrity checks
3. Review security events weekly
4. Configure security alert thresholds appropriately

### Maintenance
1. Review and clean old monitoring data monthly
2. Update performance thresholds based on site growth
3. Test emergency recovery procedures quarterly
4. Document any custom monitoring configurations

## API Reference

### Health Monitor API
```php
// Run all health checks
$results = HVAC_Health_Monitor::run_all_checks($force_refresh = false);

// Check overall health status
$status = $results['overall_status']; // 'healthy', 'warning', 'critical'
```

### Error Recovery API
```php
// Execute with recovery
$result = HVAC_Error_Recovery::execute_with_recovery($type, $callback, $args);

// Check emergency mode
$is_emergency = HVAC_Error_Recovery::is_emergency_mode();
```

### Security Monitor API
```php
// Get security statistics
$stats = HVAC_Security_Monitor::get_security_stats();

// Trigger emergency lockdown
HVAC_Security_Monitor::emergency_lockdown();
```

### Performance Monitor API
```php
// Get performance statistics
$stats = HVAC_Performance_Monitor::get_performance_stats();

// Run benchmark
HVAC_Performance_Monitor::run_performance_benchmark();
```

## Support and Maintenance

This monitoring system is designed to be self-maintaining with automatic cleanup and intelligent alerting. For issues or questions:

1. Check the admin interfaces for immediate insights
2. Review log files for detailed error information
3. Use WP-CLI commands for automation and testing
4. Consult this documentation for configuration options

The system is designed to fail gracefully - if monitoring systems encounter issues, they will not impact the main plugin functionality.