stream-processing

# Stream Processing

Patterns and technologies for real-time data processing, event streaming, and stream analytics.

## When to Use This Skill

- Designing real-time data pipelines
- Choosing stream processing frameworks
- Implementing event-driven architectures
- Building real-time analytics
- Understanding streaming vs batch trade-offs

## Batch vs Streaming

### Comparison

| Aspect | Batch | Streaming |
| ------ | ----- | --------- |
| Latency | Minutes to hours | Milliseconds to seconds |
| Data | Bounded (finite) | Unbounded (infinite) |
| Processing | Process all at once | Process as it arrives |
| State | Recompute each run | Maintain continuously |
| Complexity | Lower | Higher |
| Cost | Often lower | Often higher |

### When to Use Streaming

```text
Use streaming when:
- Real-time responses required (<1 minute)
- Events need immediate action (fraud, alerts)
- Data arrives continuously
- Users expect live updates
- Time-sensitive business decisions

Use batch when:
- Daily/hourly reports sufficient
- Complex transformations needed
- Cost optimization priority
- Historical analysis
- One-time processing
```

## Stream Processing Concepts

### Event Time vs Processing Time

```text
Event Time: When event actually occurred
Processing Time: When event is processed

Example:
┌─────────────────────────────────────────────────────────┐
│ Event: Purchase at 10:00:00 (event time)                │
│ Network delay: 5 seconds                                │
│ Processing: 10:00:05 (processing time)                  │
└─────────────────────────────────────────────────────────┘

Why it matters:
- Late events need handling
- Ordering not guaranteed
- Watermarks track progress
```

### Watermarks

```text
Watermark = "All events before this time have arrived"

Event stream:
──[10:01]──[10:02]──[10:00]──[10:03]──[Watermark: 10:00]──

Allows system to:
- Know when window is complete
- Handle late events
- Balance latency vs completeness
```

### Windows

```text
Tumbling