retellai-incident-runbook

Execute Retell AI incident response procedures with triage, mitigation, and postmortem.Use when responding to Retell AI-related outages, investigating errors,or running post-incident reviews for Retell AI integration failures.Trigger with phrases like "retellai incident", "retellai outage","retellai down", "retellai on-call", "retellai emergency", "retellai broken".

⭐939stars🔀114forks📁View on GitHub🕐Updated Jan 11, 2026⬇️1downloads

When & Why to Use This Skill

The Retell AI Incident Runbook skill is a comprehensive automation tool designed for DevOps and SRE teams to manage the full lifecycle of service disruptions. It streamlines the response process—from initial triage and technical mitigation using kubectl and curl to automated evidence collection and postmortem generation—ensuring minimal downtime for Retell AI integrations.

Use Cases

  • Rapid Triage: Quickly determining if a service failure originates from Retell AI's infrastructure or internal configuration by analyzing status pages and real-time error logs.
  • Automated Remediation: Executing immediate fixes for common API issues such as 401/403 authentication failures, 429 rate limiting, and 500-level server errors.
  • Incident Communication: Generating standardized internal Slack updates and external status page notifications to keep stakeholders informed during an outage.
  • Post-Incident Analysis: Automating the collection of debug bundles, metrics, and logs to populate postmortem templates for root cause analysis (RCA).
nameretellai-incident-runbook
description|
allowed-toolsRead, Grep, Bash(kubectl:*), Bash(curl:*)
version1.0.0
licenseMIT
authorJeremy Longshore <jeremy@intentsolutions.io>

Retell AI Incident Runbook

Overview

Rapid incident response procedures for Retell AI-related outages.

Prerequisites

  • Access to Retell AI dashboard and status page
  • kubectl access to production cluster
  • Prometheus/Grafana access
  • Communication channels (Slack, PagerDuty)

Severity Levels

Level Definition Response Time Examples
P1 Complete outage < 15 min Retell AI API unreachable
P2 Degraded service < 1 hour High latency, partial failures
P3 Minor impact < 4 hours Webhook delays, non-critical errors
P4 No user impact Next business day Monitoring gaps

Quick Triage

# 1. Check Retell AI status
curl -s https://status.retellai.com | jq

# 2. Check our integration health
curl -s https://api.yourapp.com/health | jq '.services.retellai'

# 3. Check error rate (last 5 min)
curl -s localhost:9090/api/v1/query?query=rate(retellai_errors_total[5m])

# 4. Recent error logs
kubectl logs -l app=retellai-integration --since=5m | grep -i error | tail -20

Decision Tree

Retell AI API returning errors?
├─ YES: Is status.retellai.com showing incident?
│   ├─ YES → Wait for Retell AI to resolve. Enable fallback.
│   └─ NO → Our integration issue. Check credentials, config.
└─ NO: Is our service healthy?
    ├─ YES → Likely resolved or intermittent. Monitor.
    └─ NO → Our infrastructure issue. Check pods, memory, network.

Immediate Actions by Error Type

401/403 - Authentication

# Verify API key is set
kubectl get secret retellai-secrets -o jsonpath='{.data.api-key}' | base64 -d

# Check if key was rotated
# → Verify in Retell AI dashboard

# Remediation: Update secret and restart pods
kubectl create secret generic retellai-secrets --from-literal=api-key=NEW_KEY --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/retellai-integration

429 - Rate Limited

# Check rate limit headers
curl -v https://api.retellai.com 2>&1 | grep -i rate

# Enable request queuing
kubectl set env deployment/retellai-integration RATE_LIMIT_MODE=queue

# Long-term: Contact Retell AI for limit increase

500/503 - Retell AI Errors

# Enable graceful degradation
kubectl set env deployment/retellai-integration RETELLAI_FALLBACK=true

# Notify users of degraded service
# Update status page

# Monitor Retell AI status for resolution

Communication Templates

Internal (Slack)

🔴 P1 INCIDENT: Retell AI Integration
Status: INVESTIGATING
Impact: [Describe user impact]
Current action: [What you're doing]
Next update: [Time]
Incident commander: @[name]

External (Status Page)

Retell AI Integration Issue

We're experiencing issues with our Retell AI integration.
Some users may experience [specific impact].

We're actively investigating and will provide updates.

Last updated: [timestamp]

Post-Incident

Evidence Collection

# Generate debug bundle
./scripts/retellai-debug-bundle.sh

# Export relevant logs
kubectl logs -l app=retellai-integration --since=1h > incident-logs.txt

# Capture metrics
curl "localhost:9090/api/v1/query_range?query=retellai_errors_total&start=2h" > metrics.json

Postmortem Template

## Incident: Retell AI [Error Type]
**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** P[1-4]

### Summary
[1-2 sentence description]

### Timeline
- HH:MM - [Event]
- HH:MM - [Event]

### Root Cause
[Technical explanation]

### Impact
- Users affected: N
- Revenue impact: $X

### Action Items
- [ ] [Preventive measure] - Owner - Due date

Instructions

Step 1: Quick Triage

Run the triage commands to identify the issue source.

Step 2: Follow Decision Tree

Determine if the issue is Retell AI-side or internal.

Step 3: Execute Immediate Actions

Apply the appropriate remediation for the error type.

Step 4: Communicate Status

Update internal and external stakeholders.

Output

  • Issue identified and categorized
  • Remediation applied
  • Stakeholders notified
  • Evidence collected for postmortem

Error Handling

Issue Cause Solution
Can't reach status page Network issue Use mobile or VPN
kubectl fails Auth expired Re-authenticate
Metrics unavailable Prometheus down Check backup metrics
Secret rotation fails Permission denied Escalate to admin

Examples

One-Line Health Check

curl -sf https://api.yourapp.com/health | jq '.services.retellai.status' || echo "UNHEALTHY"

Resources

Next Steps

For data handling, see retellai-data-handling.