hvac-kia-content/SOCIAL_MEDIA_COMPETITIVE_SETUP.md

Social Media Competitive Intelligence Setup Guide

This guide covers the setup for Phase 2 social media competitive intelligence featuring YouTube and Instagram competitor scrapers.

Overview

The Phase 2 implementation includes:

✅ YouTube Competitive Scrapers (4 channels)

• AC Service Tech (@acservicetech)
• Refrigeration Mentor (@RefrigerationMentor)
• Love2HVAC (@Love2HVAC)
• HVAC TV (@HVACTV)

✅ Instagram Competitive Scrapers (3 accounts)

• AC Service Tech (@acservicetech)
• Love2HVAC (@love2hvac)
• HVAC Learning Solutions (@hvaclearningsolutions)

Prerequisites

Required Environment Variables

Add these to your .env file:

# Existing HKIA Environment Variables (keep these)
INSTAGRAM_USERNAME=hkia1
INSTAGRAM_PASSWORD=I22W5YlbRl7x
YOUTUBE_API_KEY=your_youtube_api_key_here
TIMEZONE=America/Halifax

# Competitive Intelligence (Optional but recommended)
# Oxylabs proxy for anti-detection
OXYLABS_USERNAME=your_oxylabs_username
OXYLABS_PASSWORD=your_oxylabs_password  
OXYLABS_PROXY_ENDPOINT=pr.oxylabs.io
OXYLABS_PROXY_PORT=7777

# Jina.ai for content extraction
JINA_API_KEY=your_jina_api_key

API Keys and Credentials

1. YouTube Data API v3 (Required)

   • Same key used for HKIA YouTube scraping
   • Quota: ~10,000 units per day (shared with HKIA)

2. Instagram Credentials (Required)

   • Uses same HKIA credentials for competitive scraping
   • Implements aggressive rate limiting for compliance

3. Oxylabs Proxy (Optional but recommended)

   • For anti-detection and IP rotation
   • Sign up at https://oxylabs.io
   • Helps avoid rate limiting and blocks

4. Jina.ai Reader (Optional)

   • For enhanced content extraction
   • Sign up at https://jina.ai
   • Provides AI-powered content parsing

Installation

1. Install Dependencies

All required dependencies are already in requirements.txt:

# Install with UV (preferred)
uv sync

# Or with pip
pip install -r requirements.txt

2. Test Installation

Run the test suite to verify everything is set up correctly:

python test_social_media_competitive.py

This will test:

• ✅ Orchestrator initialization
• ✅ Scraper configuration
• ✅ API connectivity
• ✅ Directory structure
• ✅ Content discovery (if API keys available)

Usage

Quick Start Commands

# List all available competitors
python run_competitive_intelligence.py --operation list-competitors

# Test setup
python run_competitive_intelligence.py --operation test

# Get social media status
python run_competitive_intelligence.py --operation social-media-status

Social Media Operations

# Run social media backlog capture (first time)
python run_competitive_intelligence.py --operation social-backlog --limit 20

# Run social media incremental sync (daily)
python run_competitive_intelligence.py --operation social-incremental

# Platform-specific operations
python run_competitive_intelligence.py --operation social-backlog --platforms youtube --limit 30
python run_competitive_intelligence.py --operation social-incremental --platforms instagram

Analysis Operations

# Analyze YouTube competitors
python run_competitive_intelligence.py --operation platform-analysis --platforms youtube

# Analyze Instagram competitors  
python run_competitive_intelligence.py --operation platform-analysis --platforms instagram

Rate Limiting & Anti-Detection

YouTube

• API Quota: 1-3 units per video (shared with HKIA)
• Rate Limiting: 2 second delays between requests
• Proxy: Optional but recommended for high-volume usage

Instagram

• Rate Limiting: Very aggressive (15-30 second delays)
• Hourly Limit: 50 requests maximum per hour
• Extended Breaks: 45-90 seconds every 5 requests
• Session Management: Separate session files per competitor
• Proxy: Highly recommended to avoid IP blocking

Data Storage Structure

data/
├── competitive_intelligence/
│   ├── ac_service_tech/
│   │   ├── backlog/
│   │   ├── incremental/
│   │   ├── analysis/
│   │   └── media/
│   ├── love2hvac/
│   ├── hvac_learning_solutions/
│   └── ...
└── .state/
    └── competitive/
        ├── competitive_ac_service_tech_state.json
        └── ...

File Naming Convention

# YouTube competitor content
competitive_ac_service_tech_backlog_20250828_140530.md
competitive_love2hvac_incremental_20250828_141015.md

# Instagram competitor content  
competitive_ac_service_tech_backlog_20250828_141530.md
competitive_hvac_learning_solutions_incremental_20250828_142015.md

Automation & Scheduling

Recommended Schedule

# Morning sync (8:30 AM ADT) - after HKIA scraping
0 8 * * * cd /home/ben/dev/hvac-kia-content && python run_competitive_intelligence.py --operation social-incremental

# Afternoon sync (1:30 PM ADT) - after HKIA scraping  
0 13 * * * cd /home/ben/dev/hvac-kia-content && python run_competitive_intelligence.py --operation social-incremental

# Weekly full analysis (Sundays at 9 AM)
0 9 * * 0 cd /home/ben/dev/hvac-kia-content && python run_competitive_intelligence.py --operation platform-analysis --platforms youtube
30 9 * * 0 cd /home/ben/dev/hvac-kia-content && python run_competitive_intelligence.py --operation platform-analysis --platforms instagram

Monitoring & Logs

# Monitor logs
tail -f logs/competitive_intelligence/competitive_orchestrator.log

# Check specific scraper logs
tail -f logs/competitive_intelligence/youtube_ac_service_tech.log
tail -f logs/competitive_intelligence/instagram_love2hvac.log

Troubleshooting

Common Issues

1. YouTube API Quota Exceeded

   # Check quota usage
   grep "quota" logs/competitive_intelligence/*.log
   
   # Reduce frequency or limits
   python run_competitive_intelligence.py --operation social-backlog --platforms youtube --limit 10
   

2. Instagram Rate Limited

   # Instagram automatically pauses for 1 hour when rate limited
   # Check logs for rate limit messages
   grep "rate limit" logs/competitive_intelligence/instagram*.log
   

3. Proxy Issues

   # Test proxy connection
   python run_competitive_intelligence.py --operation test
   
   # Check proxy configuration
   echo $OXYLABS_USERNAME
   echo $OXYLABS_PROXY_ENDPOINT
   

4. Session Issues (Instagram)

   # Clear competitive sessions
   rm data/.sessions/competitive_*.session
   
   # Re-run with fresh login
   python run_competitive_intelligence.py --operation social-incremental --platforms instagram
   

Performance Considerations

Resource Usage

• Memory: ~200-500MB per scraper during operation
• Storage: ~10-50MB per competitor per month
• Network: Respectful rate limiting prevents bandwidth issues

Optimization Tips

1. Use proxy for production usage
2. Schedule during off-peak hours
3. Monitor API quota usage
4. Start with small limits and scale up
5. Use incremental sync for regular updates

Security & Compliance

Data Privacy

• Only public content is scraped
• No private accounts or personal data
• Content stored locally only
• GDPR compliant (public data only)

Rate Limiting Compliance

• Instagram: Very conservative limits
• YouTube: API quota management
• Proxy rotation prevents IP blocking
• Respectful delays between requests

Terms of Service

• All scrapers comply with platform ToS
• Public data only
• No automated posting or interactions
• Research/analysis use only

Next Steps

1. Phase 3: Content Intelligence Analysis

   • AI-powered content analysis
   • Competitive positioning insights
   • Content gap identification
   • Publishing pattern analysis

2. Future Enhancements

   • LinkedIn competitive scraping
   • Twitter/X competitive monitoring
   • Automated competitive reports
   • Slack/email notifications

Support

For issues or questions:

1. Check logs in logs/competitive_intelligence/
2. Run test suite: python test_social_media_competitive.py
3. Test individual components: python run_competitive_intelligence.py --operation test

Implementation Status

✅ Phase 2 Complete: Social Media Competitive Intelligence

• ✅ YouTube competitive scrapers (4 channels)
• ✅ Instagram competitive scrapers (3 accounts)
• ✅ Integrated orchestrator
• ✅ CLI commands
• ✅ Rate limiting & anti-detection
• ✅ State management
• ✅ Content discovery & scraping
• ✅ Analysis workflows
• ✅ Documentation & testing

Ready for production use!