troubleshoot
Diagnose and fix cluster issues. Use when pods fail, deployments don't work, or services are unreachable.
When & Why to Use This Skill
This Claude skill provides a comprehensive framework for diagnosing and resolving Kubernetes cluster issues. It enables users to quickly identify problem pods, analyze logs, and troubleshoot common deployment errors such as ImagePullBackOff, CrashLoopBackOff, and resource-related pending states, ensuring high availability and rapid incident recovery.
Use Cases
- Detecting and investigating pods in non-running or failed states across all namespaces to minimize system downtime.
- Debugging container startup failures by analyzing real-time logs, previous container history, and environment variable configurations.
- Resolving image-related deployment issues by verifying registry authentication and ensuring correct image tags are pushed.
- Addressing resource bottlenecks and node-level constraints that prevent pods from being scheduled or cause pending states.
- Executing emergency recovery actions such as rolling out deployment restarts or performing forced pod deletions during critical service outages.
| name | troubleshoot |
|---|---|
| description | Diagnose and fix cluster issues. Use when pods fail, deployments don't work, or services are unreachable. |
Troubleshoot Cluster Issues
Diagnose and resolve common cluster problems.
Instructions
1. Identify Problem Pods
KUBECONFIG=/home/al/.kube/config
echo "=== Problem Pods ==="
kubectl get pods -A | grep -v Running | grep -v Completed
2. Check Pod Details
KUBECONFIG=/home/al/.kube/config
POD_NAME="k8s-monitor-xxx"
NAMESPACE="ai-agents"
# Describe pod
kubectl describe pod $POD_NAME -n $NAMESPACE
# Check logs
kubectl logs $POD_NAME -n $NAMESPACE --tail=50
# Check previous container logs (if crash loop)
kubectl logs $POD_NAME -n $NAMESPACE --previous --tail=50
3. Common Issues
ImagePullBackOff
- Image not pushed to registry
- Wrong image tag
- Registry authentication issue
# Check if image exists
docker images | grep k8s-monitor
# Push if missing
docker push registry.almckay.io/k8s-monitor:TAG
CrashLoopBackOff
- Application error on startup
- Missing environment variables
- Failed dependency connections
# Check logs for error
KUBECONFIG=/home/al/.kube/config kubectl logs $POD_NAME -n $NAMESPACE --tail=100
# Check env vars
kubectl get pod $POD_NAME -n $NAMESPACE -o jsonpath='{.spec.containers[0].env}'
Pending
- Insufficient resources
- Node selector mismatch
- PVC not bound
# Check events
KUBECONFIG=/home/al/.kube/config kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -20
# Check node resources
kubectl top nodes
4. Temporal Workflow Issues
# Check worker connectivity
KUBECONFIG=/home/al/.kube/config kubectl logs -n ai-agents -l app.kubernetes.io/name=k8s-monitor --tail=50 | grep -i temporal
# List running workflows
# (use tctl or Temporal UI)
5. Restart Deployment
KUBECONFIG=/home/al/.kube/config kubectl rollout restart deployment/$DEPLOYMENT -n $NAMESPACE
6. Force Pod Deletion
KUBECONFIG=/home/al/.kube/config kubectl delete pod $POD_NAME -n $NAMESPACE --force --grace-period=0
Escalation
If unable to resolve:
- Check node health:
kubectl get nodes - Check kubelet logs on node
- Review recent changes in git history
- Check Flux reconciliation status