HKIA Content Aggregation System - Complete content scraping and markdown generation for 5 sources (WordPress, MailChimp RSS, Podcast RSS, YouTube, Instagram)
dc57ce80d5
- Created SystemMonitor class for health check monitoring - Implemented system metrics collection (CPU, memory, disk, network) - Added application metrics monitoring (scrapers, logs, data sizes) - Built alert system with configurable thresholds - Developed HTML dashboard generator with real-time charts - Added systemd services for automated monitoring (15-min intervals) - Created responsive web dashboard with Bootstrap and Chart.js - Implemented automatic cleanup of old metric files - Added comprehensive documentation and troubleshooting guide Features: - Real-time system resource monitoring - Scraper performance tracking and alerts - Interactive dashboard with trend charts - Email-ready alert notifications - Systemd integration for production deployment - Security hardening with minimal privileges - Auto-refresh dashboard every 5 minutes - 7-day metric retention with automatic cleanup Alert conditions: - Critical: CPU >80%, Memory >85%, Disk >90% - Warning: Scraper inactive >24h, Log files >100MB - Error: Monitoring failures, configuration issues 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| config | ||
| docs | ||
| monitoring | ||
| src | ||
| systemd | ||
| test_data | ||
| tests | ||
| .gitignore | ||
| .python-version | ||
| capture_tiktok_backlog.py | ||
| CLAUDE.md | ||
| claude.md | ||
| debug_wordpress.py | ||
| debug_wordpress_raw.py | ||
| debug_youtube_detailed.py | ||
| debug_youtube_videos.py | ||
| detailed_monitor.py | ||
| install.sh | ||
| install_production.sh | ||
| main.py | ||
| monitor_backlog.py | ||
| pyproject.toml | ||
| requirements.txt | ||
| requirements_new.txt | ||
| run_production.py | ||
| status.md | ||
| test_instagram_debug.py | ||
| test_instagram_fix.py | ||
| test_markitdown_fix.py | ||
| test_production_deployment.py | ||
| test_real_data.py | ||
| test_sources_simple.py | ||
| test_tiktok_advanced.py | ||
| test_tiktok_scrapling.py | ||
| uv.lock | ||