perplexity-incident-runbook

# Perplexity Incident Runbook

## Overview
Rapid incident response procedures for Perplexity-related outages.

## Prerequisites
- Access to Perplexity dashboard and status page
- kubectl access to production cluster
- Prometheus/Grafana access
- Communication channels (Slack, PagerDuty)

## Severity Levels

| Level | Definition | Response Time | Examples |
|-------|------------|---------------|----------|
| P1 | Complete outage | < 15 min | Perplexity API unreachable |
| P2 | Degraded service | < 1 hour | High latency, partial failures |
| P3 | Minor impact | < 4 hours | Webhook delays, non-critical errors |
| P4 | No user impact | Next business day | Monitoring gaps |

## Quick Triage

```bash
# 1. Check Perplexity status
curl -s https://status.perplexity.com | jq

# 2. Check our integration health
curl -s https://api.yourapp.com/health | jq '.services.perplexity'

# 3. Check error rate (last 5 min)
curl -s localhost:9090/api/v1/query?query=rate(perplexity_errors_total[5m])

# 4. Recent error logs
kubectl logs -l app=perplexity-integration --since=5m | grep -i error | tail -20
```

## Decision Tree

```
Perplexity API returning errors?
├─ YES: Is status.perplexity.com showing incident?
│   ├─ YES → Wait for Perplexity to resolve. Enable fallback.
│   └─ NO → Our integration issue. Check credentials, config.
└─ NO: Is our service healthy?
    ├─ YES → Likely resolved or intermittent. Monitor.
    └─ NO → Our infrastructure issue. Check pods, memory, network.
```

## Immediate Actions by Error Type

### 401/403 - Authentication
```bash
# Verify API key is set
kubectl get secret perplexity-secrets -o jsonpath='{.data.api-key}' | base64 -d

# Check if key was rotated
# → Verify in Perplexity dashboard

# Remediation: Update secret and restart pods
kubectl create secret generic perplexity-secrets --from-literal=api-key=NEW_KEY --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/perplexity-integration
```

### 429 - Rate Limited
```bash
# Che