tempo-troubleshoot
Help craft efficient Tempo trace queries, troubleshoot distributed traces, link logs and metrics (Loki/Mimir), and perform request-level root cause analysis.
When & Why to Use This Skill
The Tempo Distributed Tracing Troubleshooting skill is a specialized assistant designed to streamline observability workflows within the Grafana stack. It excels at crafting optimized TraceQL queries, debugging complex distributed systems, and performing request-level root cause analysis. By bridging the gap between traces, logs (Loki), and metrics (Mimir), this skill empowers DevOps and SRE teams to rapidly identify performance bottlenecks and resolve production incidents.
Use Cases
- Optimizing TraceQL queries: Generate efficient queries using service names, time ranges, and duration filters to locate specific spans without hitting cardinality limits.
- Latency Bottleneck Analysis: Identify slow microservices by analyzing distributed traces and pinpointing spans that exceed performance thresholds.
- Full-Stack Correlation: Link specific Trace IDs with Loki logs and Mimir metrics to provide a unified view of system behavior during troubleshooting.
- Error Root Cause Investigation: Perform deep-dives into failed requests by filtering for error statuses and inspecting span attributes to find the source of failure.
| name | tempo-troubleshoot |
|---|---|
| description | "Help craft efficient Tempo trace queries, troubleshoot distributed traces, link logs and metrics (Loki/Mimir), and perform request-level root cause analysis." |
Tempo Distributed Tracing Troubleshooting Skill
What this Skill does
Use for:
- TraceQL queries
- Distributed tracing debugging
- Correlating traces with logs & metrics
- Investigating latency, errors, bottlenecks
Best Practices
Always start with service + time range
Examples:
resource.service.name = "payments"span.status = "error"span.duration > 500ms"
Avoid high-cardinality attributes
UUIDs, user IDs → only secondary filters.
Use TraceQL for structure
Preferring simple comparisons & attribute filters.
Link with Loki / Mimir when relevant
- Loki:
|= "<trace_id>" - Prometheus: find metrics for same window
Example Queries
Slow frontend traces
{ resource.service.name = "frontend" } | span.duration > 1s
Payment errors
{ resource.service.name = "payments", span.status = "error", start >= now() - 15m }
Limitations
- Skill does not execute TraceQL