prometheus
Prometheus ops skill for metrics instrumentation, scraping configuration, alerting (Alertmanager), dashboarding (Grafana), SLO/SLA monitoring, and troubleshooting missing/incorrect metrics. Use for tasks like designing alert rules, improving observability, and building production monitoring runbooks.
When & Why to Use This Skill
This Prometheus skill provides comprehensive support for metrics instrumentation, scraping configuration, and observability stack management. It enables SRE and DevOps teams to design actionable alerting rules via Alertmanager, build insightful Grafana dashboards, and implement SLO/SLA monitoring to ensure high system availability and performance. By focusing on golden signals and best practices for label cardinality, it helps maintain a scalable and reliable monitoring infrastructure.
Use Cases
- Observability Stack Setup: Designing and implementing a full-stack monitoring solution using Prometheus, Alertmanager, and Grafana for Kubernetes (kube-prometheus-stack) or standalone environments.
- SLO-Based Alerting: Creating sophisticated alert rules based on golden signals (latency, traffic, errors, saturation) and multi-burn-rate strategies to minimize alert fatigue and focus on critical user journeys.
- Metrics Instrumentation & Optimization: Guiding the implementation of Prometheus metrics in applications while avoiding high-cardinality pitfalls and ensuring correct use of histograms and counters.
- Performance Troubleshooting: Identifying and resolving issues related to missing metrics, scrape failures, relabeling errors, and resource-heavy PromQL queries in production environments.
- Incident Response & Runbooks: Developing production-ready runbooks that link specific monitoring symptoms to mitigation steps, improving MTTR (Mean Time To Recovery) through automated documentation.
| name | prometheus |
|---|---|
| description | Prometheus ops skill for metrics instrumentation, scraping configuration, alerting (Alertmanager), dashboarding (Grafana), SLO/SLA monitoring, and troubleshooting missing/incorrect metrics. Use for tasks like designing alert rules, improving observability, and building production monitoring runbooks. |
prometheus
Use this skill for Prometheus + Alertmanager + Grafana 监控体系建设与运维。
Defaults / assumptions to confirm
- Deployment: kube-prometheus-stack / standalone
- Alert routing: Alertmanager receivers (Slack/WeCom/PagerDuty)
- Metrics source: app exporters, node-exporter, kube-state-metrics
- Naming conventions and label cardinality constraints
Workflow
- Understand what to measure
- Identify golden signals: latency, traffic, errors, saturation.
- Map business KPIs and critical user journeys to technical indicators.
- Instrumentation guidance
- Prefer stable metric names and bounded label sets.
- Avoid high-cardinality labels (user_id, request_id, raw URLs).
- Use histograms for latency (p50/p95/p99 via
histogram_quantile).
- Scraping configuration
- Confirm scrape targets (ServiceMonitor/PodMonitor or static configs).
- Ensure relabeling rules are correct; set scrape intervals/timeouts appropriately.
- Alert design (practical)
- Alerts should be actionable and low-noise.
- Use multi-window multi-burn-rate for SLO alerts where applicable.
- Add
for:to avoid flapping; include runbook links in annotations.
- Dashboarding
- Provide per-service dashboards: RPS, p95 latency, error rate, resource usage.
- Add drill-down: by route group, instance, and dependency.
- Troubleshooting checklist
- Missing metrics: target down, wrong labels, scrape failures, RBAC/network issues.
- Wrong metrics: unit mismatch, counter resets, histogram buckets incorrect.
- High load: cardinality explosion, too frequent scrapes, heavy queries.
Outputs
- Metrics plan: required metrics, labels, and thresholds.
- Alert rules: PromQL + severity + routing + runbook.
- Grafana dashboard layout and key panels.
- Runbook: symptom → checks → mitigation → rollback.