retrospective-validation

yaleh 的头像作者 yaleh

使用历史数据在没有实时部署的情况下验证方法论有效性。当存在丰富的历史数据(100+实例)、方法论针对可观察的模式(错误预防、测试策略、性能优化)、模式匹配具有清晰的检测规则且实时部署具有高摩擦(CI/CD集成工作量、用户研究时间、部署风险)时使用。比前瞻性验证节省40-60%时间,节省60-80%成本。置信度计算模型提供统计严谨性。在错误恢复(1,336个错误,23.7%预防,0.79置信度)中验证。

📖 中文导读:使用历史数据在没有实时部署的情况下验证方法论有效性。当存在丰富的历史数据(100+实例)、方法论针对可观察的模式(错误预防、测试策略、性能优化)、模式匹配具有清晰的检测规则且实时部署具有高摩擦(CI/CD集成工作量、用户研究时间、部署风险)时使用。比前瞻性验证节省40-60%时间,节省60-80%成本。置信度计算模型提供统计严谨性。在错误恢复(1,336个错误,23.7%预防,0.79置信度)中验证。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: Retrospective Validation description: Validate methodology effectiveness using historical data without live deployment. Use when rich historical data exists (100+ instances), methodology targets observable patterns (error prevention, test strategy, performance optimization), pattern matching is feasible with clear detection rules, and live deployment has high friction (CI/CD integration effort, user study time, deployment risk). Enables 40-60% time reduction vs prospective validation, 60-80% cost reduction. Confidence calculation model provides statistical rigor. Validated in error recovery (1,336 errors, 23.7% prevention, 0.79 confidence). allowed-tools: Read, Grep, Glob, Bash

Retrospective Validation

Validate methodologies with historical data, not live deployment.

When you have 1,000 past errors, you don't need to wait for 1,000 future errors to prove your methodology works.


When to Use This Skill

Use this skill when:

  • 📊 Rich historical data: 100+ instances (errors, test failures, performance issues)
  • 🎯 Observable patterns: Methodology targets detectable issues
  • 🔍 Pattern matching feasible: Clear detection heuristics, measurable false positive rate
  • ⚡ High deployment friction: CI/CD integration costly, user studies time-consuming
  • 📈 Statistical rigor needed: Want confidence intervals, not just hunches
  • ⏰ Time constrained: Need validation in hours, not weeks

Don't use when:

  • ❌ Insufficient data (<50 instances)
  • ❌ Emergent effects (human behavior change, UX improvements)
  • ❌ Pattern matching unreliable (>20% false positive rate)
  • ❌ Low deployment friction (1-2 hour CI/CD integration)

Quick Start (30 minutes)

Step 1: Check Historical Data (5 min)

# Example: Error data for meta-cc
meta-cc query-tools --status error | jq '. | length'
# Output: 1336 errors ✅ (>100 threshold)

# Example: Test failures from CI logs
grep "FAILED" ci-logs/*.txt | wc -l
# Output: 427 failures ✅

Threshold: ≥100 instances for statistical confidence

Step 2: Define Detection Rule (10 min)

Tool: validate-path.sh
Prevents: "File not found" errors
Detection:
  - Error message matches: "no such file or directory"
  - OR "cannot read file"
  - OR "file does not exist"
Confidence: High (90%+) - deterministic check

Step 3: Apply Rule to Historical Data (10 min)

# Count matches
grep -E "(no such file|cannot read|does not exist)" errors.log | wc -l
# Output: 163 errors (12.2% of total)

# Sample manual validation (30 errors)
# True positives: 28/30 (93.3%)
# Adjusted: 163 * 0.933 = 152 preventable ✅

Step 4: Calculate Confidence (5 min)

Confidence = Data Quality × Accuracy × Logical Correctness
           = 0.85 × 0.933 × 1.0
           = 0.79 (High confidence)

Result: Tool would have prevented 152 errors with 79% confidence.


Four-Phase Process

Phase 1: Data Collection

1. Identify Data Sources

For Claude Code / meta-cc:

# Error history
meta-cc query-tools --status error

# User pain points
meta-cc query-user-messages --pattern "error|fail|broken"

# Error context
meta-cc query-context --error-signature "..."

For other projects:

  • Git history (commits, diffs, blame)
  • CI/CD logs (test failures, build errors)
  • Application logs (runtime errors)
  • Issue trackers (bug reports)

2. Quantify Baseline

Metrics needed:

  • Volume: Total instances (e.g., 1,336 errors)
  • Rate: Frequency (e.g., 5.78% error rate)
  • Distribution: Category breakdown (e.g., file-not-found: 12.2%)
  • Impact: Cost (e.g., MTTD: 15 min, MTTR: 30 min)

Phase 2: Pattern Definition

1. Create Detection Rules

For each tool/methodology:

what_it_prevents: Error type or failure mode
detection_rule: Pattern matching heuristic
confidence: Estimated accuracy (high/medium/low)

2. Define Success Criteria

prevention: Message matches AND tool would catch it
speedup: Tool faster than manual debugging
reliability: No false positives/negatives in sample

Phase 3: Validation Execution

1. Apply Rules to Historical Data

# Pseudo-code
for instance in historical_data:
  category = classify(instance)
  tool = find_applicable_tool(category)
  if would_have_prevented(tool, instance):
    count_prevented++

prevention_rate = count_prevented / total * 100

2. Sample Manual Validation

Sample size: 30 instances (95% confidence)
For each: "Would tool have prevented this?"
Calculate: True positive rate, False positive rate
Adjust: prevention_claim * true_positive_rate

Example (Bootstrap-003):

Sample: 30/317 claimed prevented
True positives: 28 (93.3%)
Adjusted: 317 * 0.933 = 296 errors
Confidence: High (93%+)

3. Measure Performance

# Tool time
time tool.sh < test_input
# Output: 0.05s

# Manual time (estimate from historical data)
# Average debug time: 15 min = 900s

# Speedup: 900 / 0.05 = 18,000x

Phase 4: Confidence Assessment

Confidence Formula:

Confidence = D × A × L

Where:
D = Data Quality (0.5-1.0)
A = Accuracy (True Positive Rate, 0.5-1.0)
L = Logical Correctness (0.5-1.0)

Data Quality (D):

  • 1.0: Complete, accurate, representative
  • 0.8-0.9: Minor gaps or biases
  • 0.6-0.7: Significant gaps
  • <0.6: Unreliable data

Accuracy (A):

  • 1.0: 100% true positive rate (verified)
  • 0.8-0.95: High (sample validation 80-95%)
  • 0.6-0.8: Medium (60-80%)
  • <0.6: Low (unreliable pattern matching)

Logical Correctness (L):

  • 1.0: Deterministic (tool directly addresses root cause)
  • 0.8-0.9: High correlation (strong evidence)
  • 0.6-0.7: Moderate correlation
  • <0.6: Weak or speculative

Example (Bootstrap-003):

D = 0.85 (Complete error logs, minor gaps in context)
A = 0.933 (93.3% true positive rate from sample)
L = 1.0 (File validation is deterministic)

Confidence = 0.85 × 0.933 × 1.0 = 0.79 (High)

Interpretation:

  • ≥0.75: High confidence (publishable)
  • 0.60-0.74: Medium confidence (needs caveats)
  • 0.45-0.59: Low confidence (suggestive, not conclusive)
  • <0.45: Insufficient confidence (need prospective validation)

Comparison: Retrospective vs Prospective

Aspect Retrospective Prospective
Time Hours-days Weeks-months
Cost Low (queries) High (deployment)
Risk Zero May introduce issues
Confidence 0.60-0.95 0.90-1.0
Data Historical New
Scope Full history Limited window
Bias Hindsight None

When to use each:

  • Retrospective: Fast validation, high data volume, observable patterns
  • Prospective: Behavioral effects, UX, emergent properties
  • Hybrid: Retrospective first, limited prospective for edge cases

Success Criteria

Retrospective validation succeeded when:

  1. Sufficient data: ≥100 instances analyzed
  2. High confidence: ≥0.75 overall confidence score
  3. Sample validated: ≥80% true positive rate
  4. Impact quantified: Prevention % or speedup measured
  5. Time savings: 40-60% faster than prospective validation

Bootstrap-003 Validation:

  • ✅ Data: 1,336 errors analyzed
  • ✅ Confidence: 0.79 (high)
  • ✅ Sample: 93.3% true positive rate
  • ✅ Impact: 23.7% error prevention
  • ✅ Time: 3 hours vs 2+ weeks (prospective)

Parent framework:

Complementary acceleration:


References

Core guide:

Examples:


Status: ✅ Validated | Bootstrap-003 | 0.79 confidence | 40-60% time reduction