mutation-testing
Validate test effectiveness with mutation testing using Stryker (TypeScript/JavaScript) and mutmut (Python). Find weak tests that pass despite code mutations. Use to improve test quality.
When & Why to Use This Skill
This Claude skill enables advanced test validation through mutation testing, utilizing industry-standard tools like Stryker (for TypeScript/JavaScript) and mutmut (for Python). It goes beyond traditional code coverage by intentionally introducing small bugs (mutants) into your source code to verify if your existing test suite can detect and 'kill' them. This process helps developers identify weak assertions and gaps in logic, ensuring that tests are truly effective at catching regressions and improving overall software reliability.
Use Cases
- Identifying 'weak' tests that contribute to high code coverage but fail to actually validate the correctness of the logic.
- Strengthening test suites for critical business logic where a 'survived mutant' could represent a potential production bug.
- Optimizing test quality during refactoring by ensuring that the test suite remains robust enough to catch unintended side effects.
- Benchmarking the effectiveness of a testing strategy by targeting a high mutation score (80%+) rather than just line coverage.
- Detecting missing boundary condition tests by analyzing mutants that survive changes to relational or arithmetic operators.
| name | mutation-testing |
|---|---|
| description | Validate test effectiveness with mutation testing using Stryker (TypeScript/JavaScript) and mutmut (Python). Find weak tests that pass despite code mutations. Use to improve test quality. |
| allowed-tools | Bash, Read, Edit, Write, Grep, Glob, TodoWrite |
Mutation Testing
Expert knowledge for mutation testing - validating that your tests actually catch bugs by introducing deliberate code mutations.
Core Concept
- Mutants: Small code changes introduced automatically
- Killed: Test fails with mutation (good - test caught the bug)
- Survived: Test passes with mutation (bad - weak test)
- Score: Percentage of mutants killed (aim for 80%+)
TypeScript/JavaScript (Stryker)
Installation
# Using Bun
bun add -d @stryker-mutator/core @stryker-mutator/vitest-runner
# Using npm
npm install -D @stryker-mutator/core @stryker-mutator/vitest-runner
Configuration
// stryker.config.mjs
export default {
packageManager: 'bun',
reporters: ['html', 'clear-text', 'progress'],
testRunner: 'vitest',
coverageAnalysis: 'perTest',
mutate: ['src/**/*.ts', '!src/**/*.test.ts'],
thresholds: { high: 80, low: 60, break: 60 },
incremental: true,
}
Running Stryker
# Run mutation testing
bunx stryker run
# Incremental mode (only changed files)
bunx stryker run --incremental
# Specific files
bunx stryker run --mutate "src/utils/**/*.ts"
# Open HTML report
open reports/mutation/html/index.html
Example: Weak Test
// Source code
function calculateDiscount(price: number, percentage: number): number {
return price - (price * percentage / 100)
}
// ❌ WEAK: Test passes even if we mutate calculation
test('applies discount', () => {
expect(calculateDiscount(100, 10)).toBeDefined() // Too weak!
})
// ✅ STRONG: Test catches mutation
test('applies discount correctly', () => {
expect(calculateDiscount(100, 10)).toBe(90)
expect(calculateDiscount(100, 20)).toBe(80)
expect(calculateDiscount(50, 10)).toBe(45)
})
Python (mutmut)
Installation
uv add --dev mutmut
Running mutmut
# Run mutation testing
uv run mutmut run
# Show results
uv run mutmut results
# Show specific mutant
uv run mutmut show 1
# Generate HTML report
uv run mutmut html
open html/index.html
Common Mutation Types
// Arithmetic Operator
// Original: a + b → a - b, a * b, a / b
// Relational Operator
// Original: a > b → a >= b, a < b, a <= b
// Logical Operator
// Original: a && b → a || b
// Boolean Literal
// Original: true → false
Mutation Score Targets
| Score | Quality | Action |
|---|---|---|
| 90%+ | Excellent | Maintain quality |
| 80-89% | Good | Small improvements |
| 70-79% | Acceptable | Focus on weak areas |
| < 60% | Poor | Major improvements needed |
Improving Weak Tests
Pattern: Insufficient Assertions
// Before: Mutation survives
test('calculates sum', () => {
expect(sum([1, 2, 3])).toBeGreaterThan(0) // Weak!
})
// After: Mutation killed
test('calculates sum correctly', () => {
expect(sum([1, 2, 3])).toBe(6)
expect(sum([0, 0, 0])).toBe(0)
expect(sum([])).toBe(0)
})
Pattern: Boundary Conditions
// After: Tests boundaries
test('validates age boundaries', () => {
expect(isValidAge(18)).toBe(true) // Min valid
expect(isValidAge(17)).toBe(false) // Just below
expect(isValidAge(100)).toBe(true) // Max valid
expect(isValidAge(101)).toBe(false) // Just above
})
Best Practices
- Start with core business logic modules
- Ensure 80%+ coverage before mutation testing
- Run incrementally (only changed files)
- Focus on important files first
- Don't expect 100% mutation score (equivalent mutants exist)
Workflow
# 1. Ensure good coverage first
bun test --coverage
# Target: 80%+ coverage
# 2. Run mutation testing
bunx stryker run
# 3. Check report
open reports/mutation/html/index.html
# 4. Fix survived mutants
# 5. Re-run incrementally
bunx stryker run --incremental
# or: npx stryker run --incremental
See Also
vitest-testing- Unit testing frameworktest-quality-analysis- Detecting test smells