web-crawler

# Rust Web Crawler (rcrawler)

High-performance web crawler built in **pure Rust** with
production-grade features for fast, reliable site crawling.

## When to Use This Skill

Use this skill when the user requests:

- Web crawling or site mapping
- Sitemap discovery and analysis
- Link extraction and validation
- Site structure visualization
- robots.txt compliance checking
- Performance-critical web scraping
- Generating interactive web reports with graph visualization

## Core Capabilities

### 🚀 Performance

- **60+ pages/sec** throughput with async Tokio runtime
- **<50ms startup** time - Near-instant initialization
- **~50MB memory** usage - Efficient resource consumption
- **5.4 MB binary** - Single executable, no dependencies

### 🤖 Intelligence

- **Sitemap discovery**: Automatically finds and parses sitemap.xml (3 standard locations)
- **robots.txt compliance**: Respects crawling rules with per-domain caching
- **Smart filtering**: Auto-excludes images, CSS, JS, PDFs by default
- **Domain auto-detection**: Extracts and restricts to base domain automatically

### 🔒 Safety

- **Rate limiting**: Token bucket algorithm (default 2 req/s)
- **Configurable timeout**: 30 second default
- **Memory safe**: Rust's ownership system prevents crashes
- **Graceful shutdown**: 2-second grace period for pending requests

### 📊 Output

- **Multiple formats**: JSON, Markdown, HTML, CSV, Links, Text
- **LLM-ready Markdown**: Clean content with YAML frontmatter
- **Interactive HTML report**: Dashboard with graph visualization
- **Stealth mode**: User-agent rotation and realistic headers
- **Content filtering**: Remove nav, ads, scripts for clean data
- **Real-time progress**: Updates every 5 seconds during crawl

### 📝 Monitoring

- **Structured logging**: tracing with timestamps and log levels
- **Progress tracking**: `[Progress] Pages: X/Y | Active jobs: Z | Errors: N`
- **Detailed statistics**: Pages found, crawled, external links, errors, duration

## Installation & Se