chaos-engineering

dralgorhythm's avatarfrom dralgorhythm

Test system resilience through controlled failures. Use when validating fault tolerance, disaster recovery, or system reliability. Covers chaos experiments.

⭐1stars🔀0forks📁View on GitHub🕐Updated Dec 20, 2025
[Testing and QA]

When & Why to Use This Skill

The Chaos Engineering skill is a specialized framework designed for SREs and DevOps teams to proactively improve system resilience. It enables the execution of controlled chaos experiments—such as network latency injection, resource exhaustion, and service disruption—to validate fault tolerance and disaster recovery protocols. By identifying architectural weaknesses before they lead to outages, this skill helps maintain high availability and system reliability in complex distributed environments.

Use Cases

  • Resilience Validation: Simulating network packet loss or high latency to ensure the application handles unstable connectivity without cascading failures.
  • Resource Limit Testing: Stressing CPU, memory, and disk I/O to verify that auto-scaling policies and resource quotas function correctly under extreme load.
  • High Availability Verification: Randomly terminating service instances or containers to confirm that the infrastructure automatically recovers and maintains service continuity.
  • Dependency Fault Injection: Blocking access to external APIs or databases to test the effectiveness of circuit breakers, timeouts, and graceful degradation strategies.
  • Disaster Recovery Drills: Using the experiment templates to conduct 'Game Days' that prepare teams for real-world incident response and system recovery.
namechaos-engineering
descriptionTest system resilience through controlled failures. Use when validating fault tolerance, disaster recovery, or system reliability. Covers chaos experiments.
allowed-toolsRead, Write, Bash, Glob, Grep

Chaos Engineering

Principles

  1. Build a Hypothesis: Define expected behavior
  2. Minimize Blast Radius: Start small
  3. Run in Production: Real conditions matter
  4. Automate: Make experiments repeatable
  5. Minimize Impact: Have abort conditions

Experiment Process

  1. Steady State: Define normal metrics
  2. Hypothesis: "System will maintain X under condition Y"
  3. Introduce Variables: Inject failure
  4. Observe: Compare to steady state
  5. Analyze: Confirm or disprove hypothesis

Common Experiments

Network Failures

# Add latency
tc qdisc add dev eth0 root netem delay 100ms

# Packet loss
tc qdisc add dev eth0 root netem loss 10%

# Remove
tc qdisc del dev eth0 root

Resource Exhaustion

# CPU stress
stress --cpu 4 --timeout 60s

# Memory stress
stress --vm 2 --vm-bytes 1G --timeout 60s

# Disk fill
dd if=/dev/zero of=/tmp/fill bs=1M count=1024

Service Failures

  • Kill processes
  • Restart containers
  • Terminate instances
  • Block dependencies

Chaos Tools

  • Chaos Monkey: Random instance termination
  • Gremlin: Comprehensive chaos platform
  • Litmus: Kubernetes chaos engineering
  • Chaos Mesh: Cloud-native chaos

Experiment Template

## Experiment: [Name]

### Hypothesis
If [condition], then [expected behavior].

### Steady State
- Metric A: [baseline value]
- Metric B: [baseline value]

### Method
1. [Step 1]
2. [Step 2]
3. [Step 3]

### Abort Conditions
- If [condition], stop immediately

### Results
[What happened]

### Findings
[What we learned]

Safety Rules

  1. Start in non-production
  2. Have rollback ready
  3. Monitor continuously
  4. Communicate with team
  5. Document everything