troubleshoot

X-McKay 的头像作者 X-McKay

诊断和修复集群问题。用于 Pod 失败、部署不工作或服务不可达时。

📖 中文导读:诊断和修复集群问题。用于 Pod 失败、部署不工作或服务不可达时。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: troubleshoot description: Diagnose and fix cluster issues. Use when pods fail, deployments don't work, or services are unreachable.

Troubleshoot Cluster Issues

Diagnose and resolve common cluster problems.

Instructions

1. Identify Problem Pods

KUBECONFIG=/home/al/.kube/config

echo "=== Problem Pods ==="
kubectl get pods -A | grep -v Running | grep -v Completed

2. Check Pod Details

KUBECONFIG=/home/al/.kube/config
POD_NAME="k8s-monitor-xxx"
NAMESPACE="ai-agents"

# Describe pod
kubectl describe pod $POD_NAME -n $NAMESPACE

# Check logs
kubectl logs $POD_NAME -n $NAMESPACE --tail=50

# Check previous container logs (if crash loop)
kubectl logs $POD_NAME -n $NAMESPACE --previous --tail=50

3. Common Issues

ImagePullBackOff

  • Image not pushed to registry
  • Wrong image tag
  • Registry authentication issue
# Check if image exists
docker images | grep k8s-monitor

# Push if missing
docker push registry.almckay.io/k8s-monitor:TAG

CrashLoopBackOff

  • Application error on startup
  • Missing environment variables
  • Failed dependency connections
# Check logs for error
KUBECONFIG=/home/al/.kube/config kubectl logs $POD_NAME -n $NAMESPACE --tail=100

# Check env vars
kubectl get pod $POD_NAME -n $NAMESPACE -o jsonpath='{.spec.containers[0].env}'

Pending

  • Insufficient resources
  • Node selector mismatch
  • PVC not bound
# Check events
KUBECONFIG=/home/al/.kube/config kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -20

# Check node resources
kubectl top nodes

4. Temporal Workflow Issues

# Check worker connectivity
KUBECONFIG=/home/al/.kube/config kubectl logs -n ai-agents -l app.kubernetes.io/name=k8s-monitor --tail=50 | grep -i temporal

# List running workflows
# (use tctl or Temporal UI)

5. Restart Deployment

KUBECONFIG=/home/al/.kube/config kubectl rollout restart deployment/$DEPLOYMENT -n $NAMESPACE

6. Force Pod Deletion

KUBECONFIG=/home/al/.kube/config kubectl delete pod $POD_NAME -n $NAMESPACE --force --grace-period=0

Escalation

If unable to resolve:

  1. Check node health: kubectl get nodes
  2. Check kubelet logs on node
  3. Review recent changes in git history
  4. Check Flux reconciliation status