retellai-incident-runbook
Execute Retell AI incident response procedures with triage, mitigation, and postmortem.Use when responding to Retell AI-related outages, investigating errors,or running post-incident reviews for Retell AI integration failures.Trigger with phrases like "retellai incident", "retellai outage","retellai down", "retellai on-call", "retellai emergency", "retellai broken".
When & Why to Use This Skill
The Retell AI Incident Runbook skill is a comprehensive automation tool designed for DevOps and SRE teams to manage the full lifecycle of service disruptions. It streamlines the response process—from initial triage and technical mitigation using kubectl and curl to automated evidence collection and postmortem generation—ensuring minimal downtime for Retell AI integrations.
Use Cases
- Rapid Triage: Quickly determining if a service failure originates from Retell AI's infrastructure or internal configuration by analyzing status pages and real-time error logs.
- Automated Remediation: Executing immediate fixes for common API issues such as 401/403 authentication failures, 429 rate limiting, and 500-level server errors.
- Incident Communication: Generating standardized internal Slack updates and external status page notifications to keep stakeholders informed during an outage.
- Post-Incident Analysis: Automating the collection of debug bundles, metrics, and logs to populate postmortem templates for root cause analysis (RCA).
| name | retellai-incident-runbook |
|---|---|
| description | | |
| allowed-tools | Read, Grep, Bash(kubectl:*), Bash(curl:*) |
| version | 1.0.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
Retell AI Incident Runbook
Overview
Rapid incident response procedures for Retell AI-related outages.
Prerequisites
- Access to Retell AI dashboard and status page
- kubectl access to production cluster
- Prometheus/Grafana access
- Communication channels (Slack, PagerDuty)
Severity Levels
| Level | Definition | Response Time | Examples |
|---|---|---|---|
| P1 | Complete outage | < 15 min | Retell AI API unreachable |
| P2 | Degraded service | < 1 hour | High latency, partial failures |
| P3 | Minor impact | < 4 hours | Webhook delays, non-critical errors |
| P4 | No user impact | Next business day | Monitoring gaps |
Quick Triage
# 1. Check Retell AI status
curl -s https://status.retellai.com | jq
# 2. Check our integration health
curl -s https://api.yourapp.com/health | jq '.services.retellai'
# 3. Check error rate (last 5 min)
curl -s localhost:9090/api/v1/query?query=rate(retellai_errors_total[5m])
# 4. Recent error logs
kubectl logs -l app=retellai-integration --since=5m | grep -i error | tail -20
Decision Tree
Retell AI API returning errors?
├─ YES: Is status.retellai.com showing incident?
│ ├─ YES → Wait for Retell AI to resolve. Enable fallback.
│ └─ NO → Our integration issue. Check credentials, config.
└─ NO: Is our service healthy?
├─ YES → Likely resolved or intermittent. Monitor.
└─ NO → Our infrastructure issue. Check pods, memory, network.
Immediate Actions by Error Type
401/403 - Authentication
# Verify API key is set
kubectl get secret retellai-secrets -o jsonpath='{.data.api-key}' | base64 -d
# Check if key was rotated
# → Verify in Retell AI dashboard
# Remediation: Update secret and restart pods
kubectl create secret generic retellai-secrets --from-literal=api-key=NEW_KEY --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/retellai-integration
429 - Rate Limited
# Check rate limit headers
curl -v https://api.retellai.com 2>&1 | grep -i rate
# Enable request queuing
kubectl set env deployment/retellai-integration RATE_LIMIT_MODE=queue
# Long-term: Contact Retell AI for limit increase
500/503 - Retell AI Errors
# Enable graceful degradation
kubectl set env deployment/retellai-integration RETELLAI_FALLBACK=true
# Notify users of degraded service
# Update status page
# Monitor Retell AI status for resolution
Communication Templates
Internal (Slack)
🔴 P1 INCIDENT: Retell AI Integration
Status: INVESTIGATING
Impact: [Describe user impact]
Current action: [What you're doing]
Next update: [Time]
Incident commander: @[name]
External (Status Page)
Retell AI Integration Issue
We're experiencing issues with our Retell AI integration.
Some users may experience [specific impact].
We're actively investigating and will provide updates.
Last updated: [timestamp]
Post-Incident
Evidence Collection
# Generate debug bundle
./scripts/retellai-debug-bundle.sh
# Export relevant logs
kubectl logs -l app=retellai-integration --since=1h > incident-logs.txt
# Capture metrics
curl "localhost:9090/api/v1/query_range?query=retellai_errors_total&start=2h" > metrics.json
Postmortem Template
## Incident: Retell AI [Error Type]
**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** P[1-4]
### Summary
[1-2 sentence description]
### Timeline
- HH:MM - [Event]
- HH:MM - [Event]
### Root Cause
[Technical explanation]
### Impact
- Users affected: N
- Revenue impact: $X
### Action Items
- [ ] [Preventive measure] - Owner - Due date
Instructions
Step 1: Quick Triage
Run the triage commands to identify the issue source.
Step 2: Follow Decision Tree
Determine if the issue is Retell AI-side or internal.
Step 3: Execute Immediate Actions
Apply the appropriate remediation for the error type.
Step 4: Communicate Status
Update internal and external stakeholders.
Output
- Issue identified and categorized
- Remediation applied
- Stakeholders notified
- Evidence collected for postmortem
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Can't reach status page | Network issue | Use mobile or VPN |
| kubectl fails | Auth expired | Re-authenticate |
| Metrics unavailable | Prometheus down | Check backup metrics |
| Secret rotation fails | Permission denied | Escalate to admin |
Examples
One-Line Health Check
curl -sf https://api.yourapp.com/health | jq '.services.retellai.status' || echo "UNHEALTHY"
Resources
Next Steps
For data handling, see retellai-data-handling.