shift-right-testing

Testing in production with feature flags, canary deployments, synthetic monitoring, and chaos engineering. Use when validating real-world behavior, implementing safe deployments, or ensuring production resilience.

⭐23stars🔀7forks📁View on GitHub🕐Updated Dec 8, 2025

When & Why to Use This Skill

This Claude skill facilitates Shift-Right Testing by providing comprehensive frameworks for production validation, including feature flags, canary deployments, and chaos engineering. It helps engineering teams ensure production resilience and implement safe, progressive delivery strategies by monitoring real-world behavior and proactively detecting issues before they impact the broader user base.

Use Cases

  • Implementing progressive feature rollouts (e.g., 1% to 100%) using feature flags to mitigate deployment risks and enable instant rollbacks.
  • Executing canary deployments to compare real-time performance metrics and error rates between new and legacy versions before a full release.
  • Setting up 24/7 synthetic monitoring to simulate critical user journeys and detect outages or regressions proactively.
  • Conducting chaos engineering experiments, such as injecting network latency or service failures, to validate and improve system resilience.
  • Converting production incident data into automated regression tests to close the feedback loop between production and pre-production environments.
nameshift-right-testing
description"Testing in production with feature flags, canary deployments, synthetic monitoring, and chaos engineering. Use when implementing production observability or progressive delivery."
categorytesting-methodologies
priorityhigh
tokenEstimate1000
agents[qe-production-intelligence, qe-chaos-engineer, qe-performance-tester, qe-quality-analyzer]
implementation_statusoptimized
optimization_version1.0
last_optimized2025-12-02
dependencies[]
quick_reference_cardtrue
tags[shift-right, production-testing, canary, feature-flags, chaos-engineering, monitoring]

Shift-Right Testing

When testing in production or implementing progressive delivery: 1. IMPLEMENT feature flags for progressive rollout (1% → 10% → 50% → 100%) 2. DEPLOY with canary releases (compare metrics before full rollout) 3. MONITOR with synthetic tests (proactive) + RUM (reactive) 4. INJECT failures with chaos engineering (build resilience) 5. ANALYZE production data to improve pre-production testing

Quick Shift-Right Techniques:

  • Feature flags → Control who sees what, instant rollback
  • Canary deployment → 5% traffic, compare error rates
  • Synthetic monitoring → Simulate users 24/7, catch issues before users
  • Chaos engineering → Netflix-style failure injection
  • RUM (Real User Monitoring) → Actual user experience data

Critical Success Factors:

  • Production is the ultimate test environment
  • Ship fast with safety nets, not slow with certainty
  • Use production data to improve shift-left testing

Quick Reference Card

When to Use

  • Progressive feature rollouts
  • Production reliability validation
  • Performance monitoring at scale
  • Learning from real user behavior

Shift-Right Techniques

Technique Purpose When
Feature Flags Controlled rollout Every feature
Canary Compare new vs old Every deployment
Synthetic Monitoring Proactive detection 24/7
RUM Real user metrics Always on
Chaos Engineering Resilience validation Regularly
A/B Testing User behavior validation Feature decisions

Progressive Rollout Pattern

1% → 10% → 25% → 50% → 100%
↓      ↓      ↓      ↓
Check  Check  Check  Monitor

Key Metrics to Monitor

Metric SLO Target Alert Threshold
Error rate < 0.1% > 1%
p95 latency < 200ms > 500ms
Availability 99.9% < 99.5%
Apdex > 0.95 < 0.8

Feature Flags

// Progressive rollout with LaunchDarkly/Unleash pattern
const newCheckout = featureFlags.isEnabled('new-checkout', {
  userId: user.id,
  percentage: 10,  // 10% of users
  allowlist: ['beta-testers']
});

if (newCheckout) {
  return <NewCheckoutFlow />;
} else {
  return <LegacyCheckoutFlow />;
}

// Instant rollback on issues
await featureFlags.disable('new-checkout');

Canary Deployment

# Flagger canary config
apiVersion: flagger.app/v1beta1
kind: Canary
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  progressDeadlineSeconds: 60
  analysis:
    interval: 1m
    threshold: 5      # Max failed checks
    maxWeight: 50     # Max traffic to canary
    stepWeight: 10    # Increment per interval
    metrics:
      - name: request-success-rate
        threshold: 99
      - name: request-duration
        threshold: 500

Synthetic Monitoring

// Continuous production validation
await Task("Synthetic Tests", {
  endpoints: [
    { path: '/health', expected: 200, interval: '30s' },
    { path: '/api/products', expected: 200, interval: '1m' },
    { path: '/checkout', flow: 'full-purchase', interval: '5m' }
  ],
  locations: ['us-east', 'eu-west', 'ap-south'],
  alertOn: {
    statusCode: '!= 200',
    latency: '> 500ms',
    contentMismatch: true
  }
}, "qe-production-intelligence");

Chaos Engineering

// Controlled failure injection
await Task("Chaos Experiment", {
  hypothesis: 'System handles database latency gracefully',
  steadyState: {
    metric: 'error_rate',
    expected: '< 0.1%'
  },
  experiment: {
    type: 'network-latency',
    target: 'database',
    delay: '500ms',
    duration: '5m'
  },
  rollback: {
    automatic: true,
    trigger: 'error_rate > 5%'
  }
}, "qe-chaos-engineer");

Production → Pre-Production Feedback Loop

// Convert production incidents to regression tests
await Task("Incident Replay", {
  incident: {
    id: 'INC-2024-001',
    type: 'performance-degradation',
    conditions: { concurrent_users: 500, cart_items: 10 }
  },
  generateTests: true,
  addToRegression: true
}, "qe-production-intelligence");

// Output: New test added to prevent recurrence

Agent Coordination Hints

Memory Namespace

aqe/shift-right/
├── canary-results/*      - Canary deployment metrics
├── synthetic-tests/*     - Monitoring configurations
├── chaos-experiments/*   - Experiment results
├── production-insights/* - Issues → test conversions
└── rum-analysis/*        - Real user data patterns

Fleet Coordination

const shiftRightFleet = await FleetManager.coordinate({
  strategy: 'shift-right-testing',
  agents: [
    'qe-production-intelligence',  // RUM, incident replay
    'qe-chaos-engineer',           // Resilience testing
    'qe-performance-tester',       // Synthetic monitoring
    'qe-quality-analyzer'          // Metrics analysis
  ],
  topology: 'mesh'
});


Remember

Production is the ultimate test environment. Feature flags enable instant rollback. Canary catches issues before 100% rollout. Synthetic monitoring detects problems before users. Chaos engineering builds resilience. RUM shows real user experience.

With Agents: Agents monitor production, replay incidents as tests, run chaos experiments, and convert production insights to pre-production tests. Use agents to maintain continuous production quality.