structured-logging

# SQLite for Structured Data

## STOP - Before You Start

**Before writing any data analysis code, answer these questions:**

1. Will I query this data more than once? → **Use SQLite**
2. Do I need GROUP BY, COUNT, AVG, or JOIN? → **Use SQLite**
3. Am I about to write Python/jq parsing code? → **Use SQLite instead**
4. Is the dataset >100 records? → **Use SQLite**

If you answered YES to any question above, use SQLite. Don't write custom parsing code.

## Core Principle

**SQLite is just a file** - no server, no setup, zero dependencies. Use it when you'd otherwise write custom parsing code or re-process data for each query.

## Common Misconception

❌ **"Databases are too complex for small datasets"**

Reality: SQLite = simpler than writing JSON parsing code.

```bash
# This is "complex" (custom code for every query):
cat data.json | jq '.[] | select(.status=="failed")' | jq -r '.error_type' | sort | uniq -c

# This is "simple" (SQL does the work):
sqlite3 data.db "SELECT error_type, COUNT(*) FROM errors WHERE status='failed' GROUP BY error_type"
```

**Setup cost:** `sqlite3 file.db` - that's it. It's just a file like JSON.

## When to Use SQLite

Use when ANY of these apply:

- **>100 records** - JSON/grep becomes unwieldy
- **Multiple aggregations** - Need to GROUP BY, COUNT, AVG, etc.
- **Multiple queries** - Will ask follow-up questions about same data
- **Correlation needed** - Joining data from multiple sources
- **State tracking** - Need queryable progress/status over time

## When NOT to Use SQLite

Don't use when ALL of these are true:

- <50 records total
- Single simple query
- No aggregations needed
- Won't have follow-up questions

→ For tiny datasets with simple access, JSON/grep is fine.

## Red Flags - Use SQLite Instead

STOP and use SQLite if you're about to:

- Write Python/Node code to parse JSON/CSV for analysis
- Run same jq/grep command with slight variations
- Write custom aggregation logic (COUNT, AVG, GROUP BY in code)
- Manually correlat