schema-e2e-validation

# Schema E2E Validation

## When to Use

- Validating schema changes before commit
- Verifying YAML schema matches live ClickHouse Cloud
- Regenerating Python types, DDL, or docs
- Running full schema workflow validation

## Prerequisites

### Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

```bash
colima start
```

**Check if running:**

```bash
docker ps  # Should not error
```

### Doppler Access (For validation targets)

Required for `+test-schema-validate` and `+test-schema-e2e`:

```bash
doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd
```

### Earthly Installation

```bash
brew install earthly
```

---

## Quick Commands

### Generation only (no secrets)

```bash
cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate
```

### Full E2E with validation (requires Doppler)

```bash
cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e
```

### All non-secret targets

```bash
cd /Users/terryli/eon/gapless-network-data
earthly +all
```

---

## Artifacts

After running `+test-schema-generate` or `+test-schema-e2e`, check `./earthly-artifacts/`:

| Path                       | Contents                    |
| -------------------------- | --------------------------- |
| `types/blocks.py`          | Pydantic + TypedDict models |
| `types/__init__.py`        | Package init                |
| `ddl/ethereum_mainnet.sql` | ClickHouse DDL              |
| `docs/ethereum_mainnet.md` | Markdown documentation      |

For E2E, artifacts are under `e2e/types/`, `e2e/ddl/`, `e2e/docs/`.

---

## Earthfile Targets Reference

| Target                  | Secrets | Purpose                    |
| ----------------------- | ------- | -------------------------- |
| `+deps`                 | No      | Install uv + dependencies  |
| `+build`                | No