evaluation

craigtkhill's avatarfrom craigtkhill

Guidelines for creating evaluation suites, including spec.md templates, rubric structures, and code-based validation patterns. Use this when creating new evals or updating existing ones.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 3, 2025
[Evaluation and Benchmarks]

When & Why to Use This Skill

This Claude skill provides a standardized framework for building and maintaining robust AI agent evaluation suites. It offers structured templates for specification files and reasoning rubrics, enabling developers to implement systematic validation through code-based checks, ground truth matching, and LLM-as-judge patterns to ensure agent reliability and performance accuracy.

Use Cases

  • Case 1: Creating a new evaluation suite for a specific AI agent feature using standardized spec.md templates to define requirements.
  • Case 2: Updating existing benchmarks with new validation patterns to track performance regressions and agent behavior changes.
  • Case 3: Implementing LLM-as-judge rubrics to objectively assess the quality of reasoning traces in complex agentic workflows.
  • Case 4: Standardizing evaluation identifiers and implementation types (Ground Truth, Code, LLM) across a development team for consistent QA reporting.
nameevaluation
descriptionUse when creating or updating agent evaluation suites. Defines eval structure, rubrics, and validation patterns.

Evaluation Skill

Guidelines for creating comprehensive evaluation suites.

When to Use This Skill

Use this skill when:

  • Creating a NEW evaluation suite for a feature
  • Updating an EXISTING evaluation suite
  • Understanding the evaluation framework patterns
  • Writing spec.md, rubric.md, or evaluation files

Evaluation Framework Overview

All evaluations in evals/ follow a consistent structure with both code-based and LLM-as-judge validations.

spec.md Template

Use this template for all spec.md files:

# [Feature Name] Evaluation Specification

## Requirements
Format: `[IS-EVAL-IMPLEMENTED] IDENTIFIER: example case`
- G = matches ground truth
- C = implemented via code
- L = implemented via LLM as judge using rubric
- O = not yet implemented

### [Category Name 1]
- [G] REQ-EVAL-XX-001: Description of first code-based requirement
- [C] REQ-EVAL-XX-002: Description of second code-based requirement

### [Category Name 2]
- [L] REQ-EVAL-XX-003: Description of LLM-judged requirement
- [O] REQ-EVAL-XX-004: Description of LLM-judged requirement

Template Rules:

  • Identifier Format: REQ-EVAL-XX-NNN
    • XX = 2-3 letter eval abbreviation (e.g., AG for action_generation, AS for action_scenarios)
    • NNN = Sequential 3-digit number starting at 001
  • Implementation Types:
    • [G] = Ground truth validation (matches expected output)
    • [C] = Code-based validation (deterministic checks)
    • [L] = LLM-as-judge validation (quality assessment)
    • [O] = Not yet implemented (planned for future)
  • Categories: Group related requirements logically

rubric.md Template

Use this template for all rubric.md files:

# [Feature Name] Reasoning Trace Rubric

## Format
`[PASS/FAIL] RUBRIC-ID: Criterion description`

## Based on: [Concrete example with specific values]

### [Category Name]
- [ ] RUB-XX-001: Specific, objective criterion
- [ ] RUB-XX-002: Another specific criterion

Template Rules:

  • Identifier Format: RUB-XX-NNN (matches spec.md abbreviation)
  • Categories: Organize criteria into logical groups
  • Criteria: Write concrete, objectively verifiable rules, not subjective assessments
  • Specificity: Reference actual values, fields, or behaviors that can be checked
  • Checkboxes: Use - [ ] format for LLM judge to mark pass/fail
  • Avoid subjective language: Do not use vague terms; state exactly what to verify