disaster-recovery

timequity's avatarfrom timequity

Backup strategies, disaster recovery planning, and business continuity.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 18, 2025
[IT Helpdesk and Runbooks]

When & Why to Use This Skill

This Disaster Recovery skill provides a comprehensive framework for building resilient systems and ensuring business continuity. It guides technical teams through defining critical RTO/RPO metrics, implementing automated backup strategies for PostgreSQL and MySQL, and executing structured failover/failback runbooks. By leveraging Infrastructure as Code (Terraform) and multi-region architecture patterns, it helps organizations minimize downtime and protect data integrity during critical infrastructure failures.

Use Cases

  • Establishing Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) to align technical recovery capabilities with business requirements.
  • Implementing automated database backups and cross-region replication using Terraform for cloud-native AWS environments.
  • Executing a standardized 6-step failover runbook (Detect, Assess, Decide, Execute, Verify, Communicate) during primary site outages.
  • Scheduling and performing periodic disaster recovery drills and backup restoration tests to ensure operational readiness and compliance.
namedisaster-recovery
descriptionBackup strategies, disaster recovery planning, and business continuity.

Disaster Recovery

RTO vs RPO

Metric Definition Example
RTO Recovery Time Objective 4 hours
RPO Recovery Point Objective 1 hour

RTO: How long until service is restored? RPO: How much data can we lose?

DR Strategies

Strategy RTO Cost
Backup & Restore Hours $
Pilot Light Minutes $$
Warm Standby Minutes $$$
Multi-Site Active Seconds $$$$

Backup Strategy

Database

# PostgreSQL
pg_dump -h host -U user dbname | gzip > backup.sql.gz

# MySQL
mysqldump -h host -u user -p dbname | gzip > backup.sql.gz

Automated Backups

# AWS RDS
resource "aws_db_instance" "main" {
  backup_retention_period = 7
  backup_window           = "03:00-04:00"

  # Cross-region replica for DR
  replicate_source_db = aws_db_instance.primary.arn
}

DR Runbook

Failover Steps

  1. Detect - Monitor alerts for primary failure
  2. Assess - Confirm failure, estimate recovery
  3. Decide - Failover if RTO exceeded
  4. Execute - Run failover procedure
  5. Verify - Test functionality
  6. Communicate - Update stakeholders

Failback Steps

  1. Verify primary is healthy
  2. Sync data from secondary
  3. Switch traffic back
  4. Monitor closely

Testing

Test Type Frequency
Backup restore Monthly
Failover drill Quarterly
Full DR test Annually

Multi-Region

Primary (us-east-1)          Secondary (us-west-2)
┌─────────────────┐          ┌─────────────────┐
│    App + DB     │ ──sync── │   DB Replica    │
└─────────────────┘          └─────────────────┘
        │                            │
        └──────── Route 53 ──────────┘
                (failover)