debugging-omnistrate-deployments

omnistrate-oss's avatarfrom omnistrate-oss

Systematically debug failed Omnistrate instance deployments using a progressive workflow that identifies root causes efficiently while avoiding token limits. Applies to deployment failures, probe issues, and helm-based resources.

⭐0stars🔀0forks📁View on GitHub🕐Updated Nov 16, 2025
[Incident Response]

When & Why to Use This Skill

This Claude skill provides a systematic, progressive workflow for debugging failed Omnistrate instance deployments. It enables users to efficiently identify root causes for pod unhealthy statuses, probe failures, and Helm release errors while optimizing token usage through targeted tool calls and remote tunneling.

Use Cases

  • Troubleshooting instances stuck in FAILED or DEPLOYING states to pinpoint specific resource bottlenecks and unhealthy pods.
  • Diagnosing startup or readiness probe failures (e.g., HTTP 503) by analyzing container logs and application-level errors via remote tunneling.
  • Analyzing complex Helm-based resource deployments to verify release status, revision history, and extract application credentials.
  • Identifying infrastructure constraints such as VM allocation failures, persistent volume issues, or node taints that prevent successful deployment.
nameDebugging Omnistrate Deployments
descriptionSystematically debug failed Omnistrate instance deployments using a progressive workflow that identifies root causes efficiently while avoiding token limits. Applies to deployment failures, probe issues, and helm-based resources.

Debugging Omnistrate Deployments

When to Use This Skill

  • Instance deployments showing FAILED or DEPLOYING status
  • Resources with unhealthy pod statuses or deployment errors
  • Startup/readiness probe failures (HTTP 503, timeouts)
  • Helm releases with unclear deployment states
  • Need to identify root cause of deployment failures

IMPORTANT! DO NOT USE THE AWS CLI / AZURE CLI / GCLOUD CLI TO CONFIGURE ACCESS TO THE KUBERNETES CLUSTER AT ANY STEP. OMNISTRATE PROVIDES A REMOTE TUNNELING FEATURE THAT YOU CAN REFERENCE BELOW AND THESE STEPS STRICTLY WORK ONLY WITH THAT APPROACH.

Progressive Debugging Workflow

1. Get Deployment Status

Tool: mcp__omnistrate-platform__omnistrate-ctl_instance_describe Flags: --deployment-status --output json

Extract:

  • Overall instance status
  • Resources with deployment errors or unhealthy pod statuses
  • Focus subsequent analysis on problematic resources only

Key Benefit: Returns concise status, significantly reduces token usage vs full describe

2. Identify Workflows

Tool: mcp__omnistrate-platform__omnistrate-ctl_workflow_list Flags: --instance-id <id> --output json

Extract workflow IDs, types, and start/end times for failed deployments.

3. Analyze Workflow Events (Two-Phase)

Phase 1 - Summary (Always Start Here):

omctl workflow events <workflow-id> --service-id <id> --environment-id <id> --output json

Extract:

  • All resources with workflow step status (failed/in-progress/success)
  • Step duration analysis and event count patterns
  • Identify specific failed/stuck steps

Phase 2 - Detail (Only for Failed Steps):

omctl workflow events <workflow-id> --service-id <id> --environment-id <id> \
  --resource-key <name> --step-types <type> --detail --output json

Use parameters:

  • --resource-key: Target specific resource
  • --step-types: Filter to specific step (Bootstrap, Compute, Deployment, Network, Storage, Monitoring)
  • --detail: Include full event details (use sparingly)
  • --since/--until: Time-bound queries

Extract from detail view:

  • WorkflowStepDebug error messages
  • VM allocation failures and constraints
  • Pod scheduling issues
  • Container readiness failures

Pod Event Timeline: Create ASCII visualizations showing deployment progression:

HH:MM:SS ┬─── ✗ FailedScheduling
         │    pod/app-0: Insufficient memory
         │
HH:MM:SS ├─── ⚡ TriggeredScaleUp
         │    nodegroup-1: adding 2 nodes
         │
HH:MM:SS ├─── 📥 Pulling image:latest
         │    (duration: 2m15s)
         │
HH:MM:SS └─── ✅ Started
              3/3 pods Running

Symbols: ✗ failed, ✅ success, ⚡ autoscaler, 💾 storage, 📥 image, 🚀 runtime, ⚠️ warning

4. Application-Level Investigation using Omnistrate's Remote Tunneling feature

When:

  • Resource DEPLOYING with probe failures
  • Containers Running but not Ready
  • Response from the previous tool is too large
  • No conclusive evidence from previous steps

Tool: mcp__omnistrate-platform__omnistrate-ctl_deployment-cell_update-kubeconfig + kubectl

omctl deployment-cell update-kubeconfig <cell-id> --kubeconfig /tmp/kubeconfig
kubectl get pods -n <instance-id> --kubeconfig /tmp/kubeconfig
kubectl logs <pod-name> -c service -n <instance-id> --kubeconfig /tmp/kubeconfig --tail=50

Look for:

  • Database connection failures
  • Application syntax/runtime errors (Python SyntaxError, Java compilation errors)
  • Service dependency failures
  • Configuration issues

5. Helm-Specific Verification (ONLY FOR HELM TYPE RESOURCES)

When: Helm resources with conflicting status, need application credentials, deployment state unclear

Tool: Same kubeconfig setup with --role cluster-admin + helm

omctl deployment-cell update-kubeconfig <cell-id> --kubeconfig /tmp/kubeconfig --role cluster-admin
helm list -n <instance-id> --kubeconfig /tmp/kubeconfig
helm status <release-name> -n <instance-id> --kubeconfig /tmp/kubeconfig

Extract:

  • Release status (deployed/failed/pending)
  • Revision number and last deployed time
  • Application credentials from release notes
  • Pod health ratio (Running vs Failed)

Common Failure Patterns

Infrastructure Constraints

  • VM allocation failures with restrictive constraints (AZ + Network + Size)
  • PersistentVolumeClaim not found
  • Node taints/affinity issues

Container Lifecycle

  • Back-off restarting failed container
  • ProcessLiveness: UNHEALTHY
  • Image pull failures

Probe Failures

  • Startup/readiness probe HTTP 503
  • Database connectivity timeouts
  • Application syntax errors preventing startup
  • Service dependency unavailability

Resource Prioritization

  1. Core infrastructure: databases, message queues, storage
  2. Application services: web servers, APIs
  3. Support services: monitoring, logging

Response Management

  • Always use --output json
  • If token limit exceeded: add more specific filters, use smaller time windows, target specific resources
  • Provide analysis in template format (see Failure Analysis Template in OMNISTRATE_SRE_REFERENCE.md)

Reference

See OMNISTRATE_SRE_REFERENCE.md for:

  • Detailed tool parameter documentation
  • Complete failure analysis template
  • Extended examples
  • Tool alternatives table