kubernetes-operations

在minikube上执行Kubernetes集群操作,包括可观察性(Grafana、Prometheus、Alertmanager、Loki、Tempo)、调试(kubectl debug、临时容器)和集群管理(ArgoCD)。在处理集群/清单、Kubernetes工作负载、Pods、Deployments、Operators、Controllers或集群组件时使用。

📖 中文导读:在minikube上执行Kubernetes集群操作,包括可观察性(Grafana、Prometheus、Alertmanager、Loki、Tempo)、调试(kubectl debug、临时容器)和集群管理(ArgoCD)。在处理集群/清单、Kubernetes工作负载、Pods、Deployments、Operators、Controllers或集群组件时使用。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: kubernetes-operations description: Kubernetes cluster operations on minikube including observability (Grafana, Prometheus, Alertmanager, Loki, Tempo), debugging (kubectl debug, ephemeral containers), and cluster management (ArgoCD). Use when working with cluster/manifests/, Kubernetes workloads, pods, deployments, operators, controllers, or cluster components. keywords: kubernetes, k8s, minikube, grafana, prometheus, loki, tempo, argocd, pod, クラスタ, 監視, kubectl, deployment

  • Access Grafana at grafana.minikube.127.0.0.1.nip.io
  • Use EphemeralContainer for in-container debugging
  • Disable ArgoCD selfHeal before manual changes, re-enable after

Debugging with Ephemeral Container

kubectl debug <pod-name> -n <namespace> \
  --profile=restricted \
  --image=ghcr.io/hippocampus-dev/hippocampus/ephemeral-container:main \
  --target=<container-name> \
  -- <command>

Note: Do not use -it flag when executing commands. It causes output streaming issues.

ArgoCD selfHeal Control

# Disable selfHeal
kubectl patch application <app-name> -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"selfHeal":false}}}'

# Re-enable selfHeal (after work is complete)
kubectl patch application <app-name> -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"selfHeal":true}}}'

Observability Signal Overview

Signal Backend Query Language Use Case
Traces Tempo TraceQL Request flow, latency analysis
Metrics Mimir (Prometheus) PromQL Resource usage, alerting
Logs Loki LogQL Error investigation, audit
Profiles Pyroscope Flamegraph UI CPU/memory hotspots
Probes Blackbox Exporter PromQL Endpoint reachability

Debugging Workflow

  1. Get query parameters - Check cluster/manifests/<app>/ for namespace, labels, OTEL_SERVICE_NAME
  2. Check dashboard - Open Grafana (kubernetes/namespace, kubernetes/workload, kubernetes/pod)
  3. Query by symptom - Use appropriate signal based on issue type
Symptom Signal Action
Errors in logs Loki → Tempo Extract traceid, trace in Tempo
Latency/5xx Tempo Search traces with status = error
Resource saturation Mimir Query CPU/memory metrics
High CPU/memory Pyroscope Check flamegraphs
Pod not starting kubectl kubectl get events -n <namespace>

Observability Stack Manifests

Component Path
Grafana cluster/manifests/grafana/
Tempo cluster/manifests/tempo/
Mimir cluster/manifests/mimir/
Loki cluster/manifests/loki/
Pyroscope cluster/manifests/pyroscope/
Prometheus cluster/manifests/prometheus/
Fluentd cluster/manifests/fluentd/
OpenTelemetry cluster/manifests/otel-agent/, cluster/manifests/otel-collector/

Reference

If writing observability queries: See Queries