python-data-engineering

# Python Data Engineering

Comprehensive patterns for building production-grade data pipelines, dimensional models, and API integrations using SQLAlchemy 2.0+ and modern data warehousing practices.

## When to Use This Skill

Apply this skill when building **custom Python data pipelines** with SQLAlchemy and PostgreSQL.

**Use this skill for:**
- Building ETL/ELT pipelines with Python
- Transforming API responses to database models
- Designing dimensional warehouses (fact/dimension tables)
- Implementing schema evolution strategies
- Creating reusable data transformation layers
- Integrating external systems (SaaS APIs, ERPs) with PostgreSQL

**Consider frameworks FIRST for common SaaS integrations:**
- **Airbyte** (600+ connectors, open-source) - NetSuite, Salesforce, Stripe, etc.
- **Meltano/Singer** (300+ taps, code-centric) - Reproducible, Git-based pipelines
- **Fivetran** (500+ connectors, managed) - Fast time-to-market, higher cost
- **dbt** (SQL transformations only) - Post-load transformations, not extraction

**Build custom Python when:**
- No existing connector for your source system
- Complex business logic required during transformation
- Unique incremental sync requirements
- Strategic differentiator justifying maintenance cost
- Existing connectors don't support your customizations

### Build vs Buy Decision Tree

```
Need to integrate external data source?
│
├─ Check existing connectors (Airbyte, Fivetran, Meltano)
│  │
│  ├─ Connector exists + meets requirements
│  │  └─ [USE FRAMEWORK] - Faster, maintained, documented
│  │
│  └─ No connector OR custom logic needed
│     │
│     ├─ Simple transformation (SQL only)
│     │  └─ [USE dbt] - Post-load SQL transformations
│     │
│     └─ Complex transformation OR custom logic
│        └─ [BUILD CUSTOM PYTHON] - Full control, use patterns in this skill
│
└─ Consider:
   • Maintenance cost (custom code = ongoing maintenance)
   • Time to market (framework = faster initial setup)
   • Customization needs (