ben/hvac-kia-content
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
hvac-kia-content/CLAUDE.md
Ben Reed daab901e35 refactor: Update naming convention from hvacknowitall to hkia 
Major Changes:
- Updated all code references from hvacknowitall/hvacnkowitall to hkia
- Renamed all existing markdown files to use hkia_ prefix
- Updated configuration files, scrapers, and production scripts
- Modified systemd service descriptions to use HKIA
- Changed NAS sync path to /mnt/nas/hkia

Files Updated:
- 20+ source files updated with new naming convention
- 34 markdown files renamed to hkia_* format
- All ScraperConfig brand_name parameters now use 'hkia'
- Documentation updated to reflect new naming

Rationale:
- Shorter, cleaner filenames
- Consistent branding across all outputs
- Easier to type and reference
- Maintains same functionality with improved naming

Next Steps:
- Deploy updated services to production
- Update any external references to old naming
- Monitor scrapers to ensure proper operation

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-19 13:35:23 -03:00

4.3 KiB
Raw Blame History

HKIA Content Aggregation System

Project Overview

Complete content aggregation system that scrapes 6 sources (WordPress, MailChimp RSS, Podcast RSS, YouTube, Instagram, TikTok), converts to markdown, and runs twice daily with incremental updates.

Architecture

  • Base Pattern: Abstract scraper class with common interface
  • State Management: JSON-based incremental update tracking
  • Parallel Processing: 5 sources run in parallel, TikTok separate (GUI requirement)
  • Output Format: hkia_[source]_[timestamp].md
  • Archive System: Previous files archived to timestamped directories
  • NAS Sync: Automated rsync to /mnt/nas/hkia/

Key Implementation Details

Instagram Scraper (src/instagram_scraper.py)

  • Uses instaloader with session persistence
  • Aggressive rate limiting: 15-30 second delays, extended breaks every 5 requests
  • Session file: instagram_session_hkia1.session
  • Authentication: Username hkia1, password I22W5YlbRl7x

TikTok Scraper (src/tiktok_scraper_advanced.py)

  • Advanced anti-bot detection using Scrapling + Camofaux
  • Requires headed browser with DISPLAY=:0
  • Stealth features: geolocation spoofing, OS randomization, WebGL support
  • Cannot be containerized due to GUI requirements

YouTube Scraper (src/youtube_scraper.py)

  • Uses yt-dlp for metadata extraction
  • Channel: @HVACKnowItAll
  • Fetches video metadata without downloading videos

RSS Scrapers

  • MailChimp: https://us10.campaign-archive.com/feed?u=d1a98c3e62003104038942e21&id=2205dbf985
  • Podcast: https://feeds.libsyn.com/568690/spotify

WordPress Scraper (src/wordpress_scraper.py)

  • Direct API access to hkia.com
  • Fetches blog posts with full content

Technical Stack

  • Python: 3.11+ with UV package manager
  • Key Dependencies:
    • instaloader (Instagram)
    • scrapling[all] (TikTok anti-bot)
    • yt-dlp (YouTube)
    • feedparser (RSS)
    • markdownify (HTML conversion)
  • Testing: pytest with comprehensive mocking

Deployment Strategy

⚠️ IMPORTANT: systemd Services (Not Kubernetes)

Originally planned for Kubernetes deployment but TikTok requires headed browser with DISPLAY=:0, making containerization impossible.

Production Setup

# Service files location
/etc/systemd/system/hvac-scraper.service
/etc/systemd/system/hvac-scraper.timer
/etc/systemd/system/hvac-scraper-nas.service  
/etc/systemd/system/hvac-scraper-nas.timer

# Installation directory
/opt/hvac-kia-content/

# Environment setup
export DISPLAY=:0
export XAUTHORITY="/run/user/1000/.mutter-Xwaylandauth.90WDB3"

Schedule

  • Main Scraping: 8AM and 12PM Atlantic Daylight Time
  • NAS Sync: 30 minutes after each scraping run
  • User: ben (requires GUI access for TikTok)

Environment Variables

# Required in /opt/hvac-kia-content/.env
INSTAGRAM_USERNAME=hkia1
INSTAGRAM_PASSWORD=I22W5YlbRl7x
YOUTUBE_CHANNEL=@HVACKnowItAll
TIKTOK_USERNAME=hkia
NAS_PATH=/mnt/nas/hkia
TIMEZONE=America/Halifax
DISPLAY=:0
XAUTHORITY="/run/user/1000/.mutter-Xwaylandauth.90WDB3"

Commands

Testing

# Test individual sources
uv run python test_real_data.py --source [youtube|instagram|tiktok|wordpress|mailchimp|podcast]

# Test backlog processing  
uv run python test_real_data.py --type backlog --items 50

# Full test suite
uv run pytest tests/ -v

Production Operations

# Run orchestrator manually
uv run python -m src.orchestrator

# Run specific sources
uv run python -m src.orchestrator --sources youtube instagram

# NAS sync only
uv run python -m src.orchestrator --nas-only

# Check service status
sudo systemctl status hvac-scraper.service
sudo journalctl -f -u hvac-scraper.service

Critical Notes

  1. TikTok GUI Requirement: Must run on desktop environment with DISPLAY=:0
  2. Instagram Rate Limiting: 100 requests/hour with exponential backoff
  3. State Files: Located in state/ directory for incremental updates
  4. Archive Management: Previous files automatically moved to timestamped archives
  5. Error Recovery: All scrapers handle rate limits and network failures gracefully

Project Status: ✅ COMPLETE

  • All 6 sources working and tested
  • Production deployment ready via systemd
  • Comprehensive testing completed (68+ tests passing)
  • Real-world data validation completed
  • Full backlog processing capability verified