exa-incident-runbook

Execute Exa incident response procedures with triage, mitigation, and postmortem.Use when responding to Exa-related outages, investigating errors,or running post-incident reviews for Exa integration failures.Trigger with phrases like "exa incident", "exa outage","exa down", "exa on-call", "exa emergency", "exa broken".

⭐939stars🔀114forks📁View on GitHub🕐Updated Jan 11, 2026⬇️1downloads

When & Why to Use This Skill

The Exa Incident Runbook skill provides a comprehensive, automated framework for managing Exa integration outages and errors. It streamlines the entire SRE lifecycle—from initial triage and health monitoring using kubectl and curl to active mitigation of authentication and rate-limiting issues—ensuring minimal downtime and structured post-incident reporting.

Use Cases

  • Real-time Triage: Use automated commands to check Exa status pages and internal health endpoints to quickly identify the root cause of service disruptions.
  • Automated Mitigation: Rapidly resolve 401/403 authentication errors by rotating Kubernetes secrets or handle 429 rate limits by dynamically enabling request queuing.
  • Post-Incident Analysis: Automatically collect debug bundles, export relevant logs, and capture Prometheus metrics to populate standardized postmortem templates.
  • Stakeholder Communication: Utilize pre-defined internal Slack and external status page templates to maintain clear communication during critical P1/P2 incidents.
nameexa-incident-runbook
description|
allowed-toolsRead, Grep, Bash(kubectl:*), Bash(curl:*)
version1.0.0
licenseMIT
authorJeremy Longshore <jeremy@intentsolutions.io>

Exa Incident Runbook

Overview

Rapid incident response procedures for Exa-related outages.

Prerequisites

  • Access to Exa dashboard and status page
  • kubectl access to production cluster
  • Prometheus/Grafana access
  • Communication channels (Slack, PagerDuty)

Severity Levels

Level Definition Response Time Examples
P1 Complete outage < 15 min Exa API unreachable
P2 Degraded service < 1 hour High latency, partial failures
P3 Minor impact < 4 hours Webhook delays, non-critical errors
P4 No user impact Next business day Monitoring gaps

Quick Triage

# 1. Check Exa status
curl -s https://status.exa.com | jq

# 2. Check our integration health
curl -s https://api.yourapp.com/health | jq '.services.exa'

# 3. Check error rate (last 5 min)
curl -s localhost:9090/api/v1/query?query=rate(exa_errors_total[5m])

# 4. Recent error logs
kubectl logs -l app=exa-integration --since=5m | grep -i error | tail -20

Decision Tree

Exa API returning errors?
├─ YES: Is status.exa.com showing incident?
│   ├─ YES → Wait for Exa to resolve. Enable fallback.
│   └─ NO → Our integration issue. Check credentials, config.
└─ NO: Is our service healthy?
    ├─ YES → Likely resolved or intermittent. Monitor.
    └─ NO → Our infrastructure issue. Check pods, memory, network.

Immediate Actions by Error Type

401/403 - Authentication

# Verify API key is set
kubectl get secret exa-secrets -o jsonpath='{.data.api-key}' | base64 -d

# Check if key was rotated
# → Verify in Exa dashboard

# Remediation: Update secret and restart pods
kubectl create secret generic exa-secrets --from-literal=api-key=NEW_KEY --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/exa-integration

429 - Rate Limited

# Check rate limit headers
curl -v https://api.exa.com 2>&1 | grep -i rate

# Enable request queuing
kubectl set env deployment/exa-integration RATE_LIMIT_MODE=queue

# Long-term: Contact Exa for limit increase

500/503 - Exa Errors

# Enable graceful degradation
kubectl set env deployment/exa-integration EXA_FALLBACK=true

# Notify users of degraded service
# Update status page

# Monitor Exa status for resolution

Communication Templates

Internal (Slack)

🔴 P1 INCIDENT: Exa Integration
Status: INVESTIGATING
Impact: [Describe user impact]
Current action: [What you're doing]
Next update: [Time]
Incident commander: @[name]

External (Status Page)

Exa Integration Issue

We're experiencing issues with our Exa integration.
Some users may experience [specific impact].

We're actively investigating and will provide updates.

Last updated: [timestamp]

Post-Incident

Evidence Collection

# Generate debug bundle
./scripts/exa-debug-bundle.sh

# Export relevant logs
kubectl logs -l app=exa-integration --since=1h > incident-logs.txt

# Capture metrics
curl "localhost:9090/api/v1/query_range?query=exa_errors_total&start=2h" > metrics.json

Postmortem Template

## Incident: Exa [Error Type]
**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** P[1-4]

### Summary
[1-2 sentence description]

### Timeline
- HH:MM - [Event]
- HH:MM - [Event]

### Root Cause
[Technical explanation]

### Impact
- Users affected: N
- Revenue impact: $X

### Action Items
- [ ] [Preventive measure] - Owner - Due date

Instructions

Step 1: Quick Triage

Run the triage commands to identify the issue source.

Step 2: Follow Decision Tree

Determine if the issue is Exa-side or internal.

Step 3: Execute Immediate Actions

Apply the appropriate remediation for the error type.

Step 4: Communicate Status

Update internal and external stakeholders.

Output

  • Issue identified and categorized
  • Remediation applied
  • Stakeholders notified
  • Evidence collected for postmortem

Error Handling

Issue Cause Solution
Can't reach status page Network issue Use mobile or VPN
kubectl fails Auth expired Re-authenticate
Metrics unavailable Prometheus down Check backup metrics
Secret rotation fails Permission denied Escalate to admin

Examples

One-Line Health Check

curl -sf https://api.yourapp.com/health | jq '.services.exa.status' || echo "UNHEALTHY"

Resources

Next Steps

For data handling, see exa-data-handling.