agentic-kpi-tracking
Track and measure agentic coding KPIs for ZTE progression. Use when measuring workflow effectiveness, tracking Size/Attempts/Streak/Presence metrics, or assessing readiness for autonomous operation.
When & Why to Use This Skill
This Claude skill provides a robust framework for tracking and measuring agentic coding Key Performance Indicators (KPIs) to facilitate Zero-Touch Engineering (ZTE) progression. It enables developers and teams to quantitatively assess AI agent performance, workflow effectiveness, and readiness for autonomous operation by monitoring critical metrics such as success streaks, attempt counts, and code change magnitude.
Use Cases
- ZTE Readiness Assessment: Determine if an AI agent is stable enough for unsupervised operation by analyzing success streaks and average presence metrics.
- Workflow Efficiency Analysis: Identify technical bottlenecks in agentic processes by tracking the number of attempts required to reach a successful code patch.
- Performance Trend Monitoring: Visualize the evolution of agent capabilities over time, measuring its ability to handle increasing task complexity and larger plan sizes.
- Engineering Impact Reporting: Quantify the tangible contributions of AI agents to a codebase by monitoring total diff sizes, lines added/removed, and files modified per run.
| name | agentic-kpi-tracking |
|---|---|
| description | Track and measure agentic coding KPIs for ZTE progression. Use when measuring workflow effectiveness, tracking Size/Attempts/Streak/Presence metrics, or assessing readiness for autonomous operation. |
| allowed-tools | Read, Grep, Glob |
Agentic KPI Tracking Skill
Guide measurement and tracking of agentic coding KPIs to assess ZTE readiness.
When to Use
- Measuring agentic workflow effectiveness
- Tracking progress toward ZTE
- Analyzing success patterns
- Identifying improvement areas
Core KPIs
Summary Metrics
| Metric | Calculation | Target |
|---|---|---|
| Current Streak | Consecutive successes (Attempts <= 2) | Higher is better |
| Longest Streak | Best consecutive success run | Track improvement |
| Average Presence | Mean attempts across all runs | Target: 1 |
| Total Plan Size | Sum of all plan sizes | Track scaling |
| Total Diff Size | Sum of all changes (added + removed) | Track throughput |
Per-Run Metrics
| Metric | Source | Meaning |
|---|---|---|
| Attempts | Count of plan/patch runs | 1 = perfect, higher = retries |
| Plan Size | Lines in plan file | Task complexity |
| Diff Size | Lines added + removed | Change magnitude |
| Files Changed | Number of files modified | Change scope |
Calculation Methods
Attempts Count
Only count workflow restarts:
attempts_incrementing = ["adw_plan_iso", "adw_patch_iso"]
attempts = count(workflow in all_adws if workflow in attempts_incrementing)
Build/test/review don't increment - only full replans.
Streak Calculation
current_streak = 0
for run in reversed(runs):
if run.attempts <= 2:
current_streak += 1
else:
break
Diff Statistics
git diff origin/main --shortstat
# Output: X files changed, Y insertions(+), Z deletions(-)
KPI File Format
Store in app_docs/agentic_kpis.md or equivalent:
# Agentic KPIs
## Summary
| Metric | Value |
| --- | --- |
| Current Streak | 5 |
| Longest Streak | 12 |
| Average Presence | 1.3 |
| Total Plan Size | 450 lines |
| Total Diff Size | 2,340 lines |
## Detail
| Date | ADW ID | Issue | Class | Attempts | Plan Size | Diff +/- | Files |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 2024-01-15 | abc123 | #45 | /bug | 1 | 35 | +45/-12 | 3 |
| 2024-01-14 | def456 | #44 | /feature | 2 | 85 | +120/-30 | 8 |
Tracking Workflow
Step 1: Gather Current Run Data
From state or git:
- ADW ID
- Issue number
- Issue classification
- Plan file path
- All workflows run (for attempts)
Step 2: Calculate Metrics
attempts = count_attempts(all_adws)
plan_size = wc_lines(plan_file)
diff_stats = parse_git_diff()
Step 3: Update Detail Table
Add new row with current run data.
Step 4: Recalculate Summary
Update all summary metrics based on full detail table.
Step 5: Analyze Trends
- Is streak increasing?
- Is average presence decreasing?
- Are plan sizes growing (handling bigger tasks)?
ZTE Readiness Indicators
Based on KPIs, assess ZTE readiness:
| Indicator | Threshold | Status |
|---|---|---|
| Current Streak | >= 5 | Ready to try ZTE |
| Average Presence | <= 1.5 | Good efficiency |
| Recent Failures | 0 in last 10 | High confidence |
| Plan Size Trend | Increasing | Scaling up |
Key Memory References
- @agentic-kpis.md - KPI definitions from Lesson 002
- @zte-progression.md - How KPIs relate to ZTE levels
- @zte-confidence-building.md - Using KPIs for confidence
Output Format
Provide KPI update:
## KPI Update
**Run:** {adw_id}
**Issue:** #{issue_number} ({issue_class})
### This Run
- Attempts: 1
- Plan Size: 45 lines
- Diff: +67/-23 (4 files)
### Updated Summary
- Current Streak: 6 (was 5)
- Longest Streak: 12 (unchanged)
- Average Presence: 1.28 (improved)
### Analysis
[Trend observations and recommendations]
Anti-Patterns
- Gaming metrics (easy tasks only)
- Ignoring failures (not counting retries)
- Not tracking consistently
- Celebrating streaks over actual delivery
Version History
- v1.0.0 (2025-12-26): Initial release
Last Updated
Date: 2025-12-26 Model: claude-opus-4-5-20251101