mutation-testing

Evaluate Python test suite quality using mutmut to introduce code mutations and verify tests catch them. Use for mutation testing, test quality assessment, mutant detection, and test effectiveness analysis.

⭐3stars🔀0forks📁View on GitHub🕐Updated Jan 8, 2026⬇️1downloads

When & Why to Use This Skill

This Claude skill enables automated mutation testing for Python projects using the mutmut library. It goes beyond traditional code coverage by intentionally injecting small bugs (mutants) into your source code to verify if your test suite can detect them. This process helps developers identify weak tests, uncover hidden logic gaps, and ensure the highest level of software reliability and test effectiveness.

Use Cases

  • Identifying 'survived' mutants to pinpoint specific areas where tests pass despite logic errors, indicating a need for more robust assertions.
  • Evaluating the true quality of a test suite by calculating a Mutation Score, providing a more meaningful metric than simple line coverage.
  • Strengthening critical business logic by uncovering missed boundary conditions, such as incorrect comparison operators or off-by-one errors.
  • Integrating mutation testing into CI/CD pipelines to automatically prevent the merging of code with insufficient test validation.
  • Refining unit tests to ensure they verify actual return values and state changes rather than just code execution paths.
namemutation-testing
descriptionEvaluate Python test suite quality using mutmut to introduce code mutations and verify tests catch them. Use for mutation testing, test quality assessment, mutant detection, and test effectiveness analysis.

Mutation Testing with mutmut

Mutation testing assesses test suite quality by introducing small changes (mutations) to source code and verifying that tests fail. If tests don't catch a mutation, it indicates gaps in test coverage or quality.

Key Concepts

  • Mutant: Modified version of code with small change
  • Killed: Test fails when mutation introduced (good)
  • Survived: Test passes despite mutation (test gap)
  • Mutation Score: Percentage of mutants killed

Quick Start

# Install
pip install mutmut

# Run mutation testing
mutmut run

# View results
mutmut results

# Inspect specific mutant
mutmut show 1

# Apply mutation to see the change
mutmut apply 1

# Reset applied mutations
mutmut apply 0

Configuration

# setup.cfg or pyproject.toml [tool.mutmut]
[mutmut]
paths_to_mutate=src/
backup=False
runner=python -m pytest -x
tests_dir=tests/
dict_synonyms=Struct, NamedStruct

Example: Weak vs Strong Tests

# src/validators.py
def is_valid_order(order: dict) -> bool:
    if not order:
        return False
    if not order.get("items"):
        return False
    if order.get("total", 0) <= 0:
        return False
    return True

def calculate_discount(total: float, tier: str) -> float:
    if tier == "gold":
        return total * 0.2
    elif tier == "silver":
        return total * 0.1
    return 0.0
# ❌ Weak tests - mutations survive
def test_order_basic():
    order = {"items": ["a"], "total": 10}
    assert is_valid_order(order) == True  # Only tests happy path

# ✅ Strong tests - kill mutations
def test_order_null():
    assert is_valid_order(None) == False
    assert is_valid_order({}) == False

def test_order_empty_items():
    assert is_valid_order({"items": [], "total": 10}) == False

def test_order_zero_total():
    assert is_valid_order({"items": ["a"], "total": 0}) == False

def test_order_negative_total():
    assert is_valid_order({"items": ["a"], "total": -5}) == False

def test_order_valid():
    assert is_valid_order({"items": ["a"], "total": 10}) == True

def test_discount_gold():
    assert calculate_discount(100, "gold") == 20.0

def test_discount_silver():
    assert calculate_discount(100, "silver") == 10.0

def test_discount_none():
    assert calculate_discount(100, "bronze") == 0.0

Common Mutation Operators

Original Mutations
> >=, <, ==
< <=, >, ==
== !=
and or
+ -
return True return False
return x return x + 1

Improving Mutation Score

When a mutant survives, add tests for:

  1. Boundary conditions: Test at exact threshold values
  2. Return values: Verify actual results, not just truthiness
  3. All branches: Cover each conditional path
  4. Edge cases: Empty, None, negative, zero
# Mutant survives: >= changed to >
def is_adult(age: int) -> bool:
    return age >= 18

# ❌ Weak: age=25 won't catch >= to > mutation
def test_adult_weak():
    assert is_adult(25) == True

# ✅ Strong: tests boundary
def test_adult_exactly_18():
    assert is_adult(18) == True

def test_not_adult_17():
    assert is_adult(17) == False

CI Integration

# .github/workflows/mutation.yml
name: Mutation Testing
on:
  pull_request:
    paths: ['src/**']

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - run: pip install mutmut pytest
      - run: mutmut run --CI
      - run: mutmut results

Best Practices

  • Target critical business logic, not all code
  • Aim for 80%+ mutation score on important modules
  • Review survived mutants to improve tests
  • Test boundaries and edge cases thoroughly
  • Verify actual values, not just code execution