skillset-smoke-test

outfitter-dev's avatarfrom outfitter-dev

Run and evaluate the Skillset smoke test (aliases, sets, and skill invocation) and interpret its JSON reports. Use this when validating $skill/$set behavior or checking Claude Code/Codex headless runs.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 31, 2025
[Evaluation and Benchmarks]

When & Why to Use This Skill

The Skillset Smoke Test runner is a specialized developer tool designed to automate the validation and evaluation of Claude agent skills, including aliases, sets, and invocations. It streamlines the debugging process by interpreting structured JSON reports and ensuring that AI agent capabilities function correctly across different environments like Claude Code and Codex headless runs, significantly improving the reliability of agentic workflows.

Use Cases

  • Automated CI/CD Validation: Running smoke tests in CI environments to ensure new skill updates or configuration changes don't break existing aliases or sets.
  • Headless Environment Testing: Verifying the behavior and stability of Claude Code and Codex in automated, non-interactive, or headless execution environments.
  • Failure Triage and Debugging: Utilizing detailed JSON reports and evidence logs to pinpoint the exact cause of skill invocation failures during development.
  • Environment Consistency Checks: Ensuring that local development sandboxes and production-like CLI setups are correctly configured for consistent skill execution.
nameskillset-smoke-test
descriptionRun and evaluate the Skillset smoke test (aliases, sets, and skill invocation) and interpret its JSON reports. Use this when validating $skill/$set behavior or checking Claude Code/Codex headless runs.

Skillset Smoke Test Runner

Keep this skill self-contained. If you need more detail, read REFERENCE.md.

Prereqs

  • Ensure bun is available.
  • For full runs, ensure claude and codex CLIs are installed and authenticated.
  • Run from the repo root so the smoke test can find scripts/skillset-smoke.ts.

Run the smoke test

  • Prefer the repo script so results are structured and gitignored.
  • Quick smoke (hook-only):
    • bun run test:smoke:ci
  • Hook-only (CLI mode):
    • bun run test:smoke:cli
  • Full run (hook + Claude + Codex):
    • bun run test:smoke

Options

  • --tools hook,claude,codex to limit which tools run.
  • --hook-mode ci|cli to choose the hook path (repeatable or comma-separated).
  • --no-clean to preserve the sandbox workspace and XDG dirs.
  • --clean-all to wipe all smoke test artifacts (workspace + reports).
  • --strict to exit non-zero if any step fails.

Environment overrides

  • SKILLSET_SMOKE_CLAUDE_CMD (default: claude)
  • SKILLSET_SMOKE_CLAUDE_ARGS (extra args)
  • SKILLSET_SMOKE_CODEX_CMD (default: codex)
  • SKILLSET_SMOKE_CODEX_ARGS (extra args)

Interpret results

  • Reports live under .skillset-smoke/reports/<runId>/report.json.
  • Check steps[*].status and steps[*].details.evidence for sentinel hits.
  • When a step fails, inspect its stdoutPath/stderrPath for errors.
    • Hook steps are labeled hook-ci and hook-cli.
    • The smoke test cleans by default; re-run with --no-clean to keep state for debugging.

Triage checklist

Use this checklist in order when a run fails or evidence is missing:

  1. Open the report JSON and find the first steps[*].status !== "ok".
  2. If stdoutPath/stderrPath exist, read them for the error message.
  3. Confirm the workspace and XDG paths in the report (they must be under .skillset-smoke/).
  4. If a tool step is skipped, verify the CLI is installed and auth is valid.
  5. If hook evidence is missing for a set, confirm the set resolves in set-load and re-run --tools hook to isolate hook output issues.
  6. If sentinels are missing for skills, confirm the smoke test created fixtures in .skillset-smoke/workspace/.claude/skills.
  7. Re-run with --tools hook to isolate local resolution issues before retrying full runs.

Notes

  • Outputs are stored under .skillset-smoke/ and are gitignored.
  • If the .codex/skills symlink is missing, re-create it to mirror this skill.