hvac-kia-content/docs/CONTENT_CLASSIFICATION.md

# HVAC Content Classification System

## Overview
The Content Classification System uses Claude Haiku AI to analyze and structure HVAC content from multiple sources into concise JSON files. These files provide structured metadata, summaries, and classifications for use in content creation projects.

## Features

### Structured Classification
Each content item is analyzed and classified with:
- **URL**: Original content location
- **Date Published**: Publication date
- **Author**: Content creator
- **Word Count**: Content length
- **Summary**: 1-3 sentence summary of main points
- **Key Learnings**: 3-10 bullet point takeaways
- **Content Type**: technical/business/educational/marketing/troubleshooting/installation/maintenance
- **Application**: Residential/Commercial/Industrial/Automotive/Marine
- **Categories**: Technical categories and tags
- **Brands Mentioned**: HVAC brands, manufacturers, tools referenced
- **Tools Mentioned**: Specific HVAC equipment and software
- **Topics**: Technical topics (refrigeration, heat pumps, ductwork, etc.)
- **Meta Information**:
  - Difficulty level (beginner/intermediate/advanced)
  - Target audience (homeowner/technician/contractor/engineer)
  - Actionable content flag
  - Troubleshooting focus flag
- **Classification Confidence**: AI confidence score

## Architecture

### Core Components

#### 1. Content Parser (`src/content_analysis/content_parser.py`)
- Extracts individual content items from aggregated markdown files
- Handles all content sources: WordPress, YouTube, Instagram, Podcast, MailChimp
- Validates content structure and extracts metadata
- Returns structured `ContentItem` objects

#### 2. Content Classifier (`src/content_analysis/content_classifier.py`)
- Uses Claude Haiku API for cost-effective AI classification
- Processes content with structured JSON prompts
- Implements rate limiting and retry logic:
  - 1 second delay between requests
  - Exponential backoff on failures
  - 5 retry attempts per item
- Returns `ClassifiedContent` objects with all metadata

#### 3. Markdown Consolidator (`consolidate_markdown_sources.py`)
- Deduplicates content across multiple markdown files
- Keeps most recent version of each content item by ID
- Consolidates from 53,000+ items to ~3,000 unique items
- Handles case variations in source names

#### 4. Classification Runner (`classify_youtube_podcast_only.py`)
- Focused script for classifying specific sources
- Sequential processing to avoid rate limit conflicts
- Progress tracking and error handling
- Saves results as clean JSON files

## Data Flow

```
1. Raw Markdown Files (multiple versions per source)
   ↓
2. Consolidation & Deduplication
   ↓
3. Consolidated Markdown (5 files: blog, podcast, youtube, instagram, mailchimp)
   ↓
4. Content Parsing & Validation
   ↓
5. Claude Haiku Classification
   ↓
6. Structured JSON Output
   ↓
7. NAS Storage for Distribution
```

## File Structure

### Input Files
- `data/consolidated/hkia_blog_consolidated.md` - WordPress blog posts
- `data/consolidated/hkia_podcast_consolidated.md` - Podcast episodes (431 items)
- `data/consolidated/hkia_youtube_consolidated.md` - YouTube videos (447 items)
- `data/consolidated/hkia_instagram_consolidated.md` - Instagram posts
- `data/consolidated/hkia_mailchimp_consolidated.md` - Newsletter content

### Output Files
- `data/clean_classified/blog.json` - Classified blog content
- `data/clean_classified/podcast.json` - Classified podcast episodes
- `data/clean_classified/youtube.json` - Classified YouTube videos
- `data/clean_classified/instagram.json` - Classified Instagram posts
- `data/clean_classified/mailchimp.json` - Classified newsletter content

### NAS Sync
- Files automatically synced to: `/mnt/nas/hkia/clean_classified/`

## Usage

### Full Consolidation and Classification
```bash
# Step 1: Consolidate markdown files with deduplication
uv run python consolidate_markdown_sources.py

# Step 2: Classify specific sources (YouTube & Podcast)
export ANTHROPIC_API_KEY="your-api-key"
uv run python classify_youtube_podcast_only.py

# Step 3: Sync to NAS
rsync -av data/clean_classified/ /mnt/nas/hkia/clean_classified/
```

### Classification Only (if consolidated files exist)
```bash
# Run focused classification
export ANTHROPIC_API_KEY="your-api-key"
uv run python classify_youtube_podcast_only.py
```

## API Configuration

### Claude Haiku Settings
- **Model**: claude-3-haiku-20240307
- **Max Tokens**: 1000 per request
- **Temperature**: 0.1 (low for consistent classification)
- **Rate Limiting**: 
  - 80,000 output tokens per minute limit
  - ~80 requests per minute maximum
  - 1 second delay between requests

### Cost Estimation
- **Input**: $0.25 per million tokens
- **Output**: $1.25 per million tokens
- **Typical Cost**: ~$1.30 for 878 items (447 YouTube + 431 Podcast)

## Performance

### Processing Times
- **With 1-second rate limiting**: ~3 seconds per item
- **YouTube (447 videos)**: ~22 minutes
- **Podcast (431 episodes)**: ~22 minutes
- **Total for all sources**: ~45 minutes

### Success Rates
- Typical success rate: >99%
- Automatic retry on JSON parsing errors
- Exponential backoff on API rate limits

## Error Handling

### Rate Limiting
- Base delay: 1 second between requests
- Exponential backoff: 2x multiplier on retry
- Maximum retries: 5 attempts per item

### JSON Parsing Errors
- Automatic retry with backoff
- Fallback JSON extraction from response text
- Logged errors for debugging

## Monitoring

### Progress Tracking
- Console output every 10 items
- Shows current item ID and number
- Success/failure counts
- Estimated time remaining

### Log Files
- Detailed logging with timestamps
- Error messages and stack traces
- API response debugging

## Integration

### Claude Desktop Projects
The classified JSON files are optimized for use in Claude Desktop projects:
- Massively reduced file sizes (KB instead of MB)
- Structured data for easy parsing
- Rich metadata for content filtering
- Summaries and key learnings for quick reference

### Use Cases
- Content gap analysis
- Topic research and planning
- Content repurposing
- Competitive analysis
- Training material development
- SEO optimization

## Maintenance

### Updating Classifications
1. Re-run consolidation if new markdown files added
2. Re-classify specific sources as needed
3. Sync to NAS for distribution

### Adding New Sources
1. Add source pattern to `consolidate_markdown_sources.py`
2. Update content parser if needed
3. Run consolidation and classification

### API Key Management
- Store in `.env` file as `ANTHROPIC_API_KEY`
- Never commit API keys to repository
- Use environment variables in production

## Troubleshooting

### Common Issues

#### Rate Limit Errors (429)
- Solution: Increase delay between requests
- Current setting: 1 second (optimal for 80k tokens/min)

#### JSON Parsing Errors
- Usually caused by malformed API responses
- Automatic retry handles most cases
- Check logs for persistent failures

#### Missing Content
- Verify markdown consolidation captured all files
- Check case sensitivity in source patterns
- Ensure NAS sync completed successfully

## Future Enhancements

### Planned Features
- Batch processing optimization
- Parallel classification with rate limit management
- Incremental updates for new content
- Custom classification templates per source
- Advanced deduplication strategies

### Potential Improvements
- Switch to newer Claude models when available
- Implement caching for unchanged content
- Add quality scoring metrics
- Create summary reports and analytics