verify-tests-fail-without-fix

dotnet's avatarfrom dotnet

Verifies UI tests catch the bug. Auto-detects mode based on git diff - if fix files exist, verifies FAIL without fix and PASS with fix. If only test files, verifies tests FAIL.

⭐23.1kstars🔀1.9kforks📁View on GitHub🕐Updated Jan 10, 2026
[Testing and QA]

When & Why to Use This Skill

This Claude skill automates the verification of UI tests to ensure they accurately detect bugs and validate fixes. By analyzing git diffs, it intelligently switches between 'failure-only' mode for new test creation and 'full verification' for bug fixes, ensuring that tests fail when the bug is present and pass once resolved. This prevents false positives in the testing pipeline and strengthens the reliability of Test-Driven Development (TDD) workflows.

Use Cases

  • Test-Driven Development (TDD): Verifying that a newly written UI test correctly fails to reproduce a bug before any fix code is implemented.
  • Pull Request Validation: Automatically ensuring that a bug fix is effectively covered by tests by checking the fail-then-pass transition during code reviews.
  • Regression Testing Integrity: Periodically validating that existing UI tests are still capable of catching the original bugs they were designed for by simulating a 'no-fix' state.
  • CI/CD Pipeline Automation: Integrating with automated workflows to block merges if a test doesn't demonstrate a clear failure without the accompanying fix files.
nameverify-tests-fail-without-fix
descriptionVerifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
authordotnet-maui
version"1.0"
compatibilityRequires git, PowerShell, and .NET SDK for building and running tests.

Verify Tests Fail Without Fix

Verifies UI tests actually catch the issue. Supports two workflow modes:

Mode 1: Verify Failure Only (Test Creation)

Use when creating tests before writing a fix:

  • Runs tests to verify they FAIL (proving they catch the bug)
  • No fix files required
  • Perfect for test-first development
# Auto-detect test filter from changed test files
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356"

Mode 2: Full Verification (Fix Validation)

Use when validating both tests and fix:

  1. Without fix - tests should FAIL (bug is present)
  2. With fix - tests should PASS (bug is fixed)
# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification

Note: -RequireFullVerification ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.

Requirements

Verify Failure Only Mode:

  • Test files in the PR (or working directory)

Full Verification Mode:

  • Test files in the PR
  • Fix files in the PR (non-test code changes)

The script auto-detects which mode to use based on whether fix files are present.

Expected Output

Verify Failure Only Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  Tests FAILED as expected!                                ║
║  This proves the tests correctly reproduce the bug.       ║
╚═══════════════════════════════════════════════════════════╝

Full Verification Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  - FAIL without fix (as expected)                         ║
║  - PASS with fix (as expected)                            ║
╚═══════════════════════════════════════════════════════════╝

What It Does

Verify Failure Only Mode (no fix files):

  1. Fetches base branch from origin (if available)
  2. Auto-detects test classes from changed test files
  3. Runs tests (should FAIL to prove they catch the bug)
  4. Reports result

Full Verification Mode (fix files detected):

  1. Fetches base branch from origin to ensure accurate diff
  2. Auto-detects fix files (non-test code) from git diff
  3. Auto-detects test classes from TestCases.Shared.Tests/*.cs
  4. Reverts fix files to base branch
  5. Runs tests (should FAIL without fix)
  6. Restores fix files
  7. Runs tests (should PASS with fix)
  8. Reports result

Troubleshooting

Problem Cause Solution
No fix files detected Base branch detection failed or no non-test files changed Use -FixFiles or -BaseBranch explicitly
Tests pass without fix Tests don't detect the bug Review test assertions, update test
Tests fail with fix Fix doesn't work or test is wrong Review fix implementation
App crashes Duplicate issue numbers, XAML error Check device logs
Element not found Wrong AutomationId, app crashed Verify IDs match

Optional Parameters

# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification

# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"

# Explicit fix files  
-FixFiles @("src/Core/src/File.cs")

# Explicit base branch
-BaseBranch "main"