deepgram-incident-runbook

# Deepgram Incident Runbook

## Overview
Standardized procedures for responding to Deepgram-related incidents in production.

## Quick Reference

| Resource | URL |
|----------|-----|
| Deepgram Status | https://status.deepgram.com |
| Deepgram Console | https://console.deepgram.com |
| Support | support@deepgram.com |
| Discord | https://discord.gg/deepgram |

## Incident Severity Levels

| Level | Definition | Response Time | Examples |
|-------|------------|---------------|----------|
| SEV1 | Complete outage | Immediate | All transcriptions failing |
| SEV2 | Major degradation | < 15 min | 50%+ error rate |
| SEV3 | Minor degradation | < 1 hour | Elevated latency |
| SEV4 | Minor issue | < 24 hours | Single feature affected |

## Incident Response Procedures

### Initial Triage (First 5 Minutes)

```bash
#!/bin/bash
# scripts/triage.sh - Quick assessment script

echo "=== Deepgram Incident Triage ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""

# 1. Check Deepgram status page
echo "1. Checking Deepgram Status..."
curl -s https://status.deepgram.com/api/v2/status.json | jq '.status.indicator'

# 2. Check our error rate
echo ""
echo "2. Recent Error Rate (last 5 min)..."
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(deepgram_transcription_requests_total{status="error"}[5m]))/sum(rate(deepgram_transcription_requests_total[5m]))' \
  | jq '.data.result[0].value[1]'

# 3. Check latency
echo ""
echo "3. P95 Latency (last 5 min)..."
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.95,sum(rate(deepgram_transcription_latency_seconds_bucket[5m]))by(le))' \
  | jq '.data.result[0].value[1]'

# 4. Quick connectivity test
echo ""
echo "4. API Connectivity Test..."
curl -s -o /dev/null -w "Status: %{http_code}, Time: %{time_total}s\n" \
  -X GET 'https://api.deepgram.com/v1/projects' \
  -H "Authorization: Token $DEEPGRAM_API_KEY"
```

### SEV1: Complete Outage

**Symptoms:**
- 100% tra