Caddy Server Management Skill

Version: 1.0.0 Author: Codex CLI Purpose: Comprehensive Caddy web server management across multiple environments

Overview and Purpose

This skill provides enterprise-grade Caddy server management across local development and production environments. It handles:

• Multi-server operations: Manage Caddy on local (macOS), bigbox (production), and registry (production) servers
• Zero-downtime deployments: Graceful reloads with automatic validation
• Secret management: Integrated 1Password support for secure credential handling
• Custom builds: Build Caddy with required plugins (security, cloudflare, cgi, tlsredis)
• Systemd management: Proper service lifecycle management
• Log analysis: Parse and analyze JSON logs for troubleshooting
• Pattern library: Ready-to-use configurations for common use cases

Critical Mandates:

• Zero-Downtime: Always use graceful reloads, never kill processes
• Validation-First: Mandatory syntax validation before any deployment
• Secret Safety: Never display plaintext secrets, always use 1Password
• Backup-First: Always create timestamped backups before modifications

Prerequisites

Required

1. Server Access

   • Local: macOS with mdfind available
   • bigbox: SSH access configured (ssh bigbox)
   • registry: SSH access configured (ssh registry)

2. Caddy Installation

   # Check Caddy on servers
   ssh bigbox "which caddy && caddy version"
   ssh registry "which caddy && caddy version"
   

3. 1Password CLI (for secret management)

   # Verify 1Password skill is available
   which op && op --version
   # See: ~/.codex/skills/1password-secrets/SKILL.md
   

4. Systemd (on production servers)

   # Verify systemd available
   ssh bigbox "systemctl --version"
   ssh registry "systemctl --version"
   

Optional

• xcaddy: For building custom Caddy binaries
• jq: For JSON log parsing (usually available)
• Go toolchain: Required for xcaddy builds

Server Configuration

This skill operates across 3 servers with different characteristics:

Local (Development)

• Platform: macOS
• Search: mdfind (fast Spotlight search)
• Caddy: Not running (development configs only)
• Use Cases: Testing configs, pattern development

bigbox (Production - Migration Required)

• Platform: Linux
• Caddy Version: v2.6.4 (custom build)
• Binary: /home/alex/caddy_terraphim/caddy
• Active Config: /home/alex/caddy_terraphim/conf/Caddyfile_auth
• Status: Currently running in tmux (needs systemd migration)
• Plugins: caddy-security, caddy-tlsredis, cloudflare, cgi
• ⚠️ Action Required: Migrate to systemd + 1Password

registry (Production - Best Practice)

• Platform: Linux
• Caddy Version: v2.10.2 (standard)
• Binary: /usr/bin/caddy
• Active Config: /etc/caddy/Caddyfile
• Status: Properly managed via systemd
• ✓ Reference: Clean 41-line config with snippets

Server configuration details: ~/.codex/skills/caddy/servers.json

---

Workflow 1: Discover Caddyfiles

Purpose: Find all Caddyfiles across servers quickly (<5 seconds).

When to Use

• Auditing Caddyfile locations
• Before making configuration changes
• Identifying active vs backup configs
• Searching for specific configuration patterns

How It Works

1. User specifies server(s): local, bigbox, registry, or all
2. Skill loads server configuration from servers.json
3. Skill executes fast search command (mdfind locally, find remotely)
4. Skill filters out backup files (.backup., *.bak)
5. Skill identifies active configuration (from systemd service)
6. Skill displays sorted results (active first, then by modification time)

Usage Patterns

Pattern 1: Find on Single Server

User: "Find all Caddyfiles on bigbox"

Assistant Process:
1. Load bigbox config from servers.json
2. Execute via SSH: find /home/alex/caddy_terraphim /home/alex/infrastructure -name 'Caddyfile*' ! -name '*.backup.*' -type f
3. Identify active config from systemd service
4. Sort results (active first)
5. Display with metadata

Example Output:
Found 14 Caddyfiles on bigbox:

ACTIVE:
  ✓ /home/alex/caddy_terraphim/conf/Caddyfile_auth (324 lines, modified 2 days ago)
    Service: caddy-terraphim (currently tmux-based)
    Status: Running (PID 24566, 35+ days uptime)

INACTIVE:
  /home/alex/infrastructure/atomic-server-turso/Caddyfile.enhanced (112 lines)
  /home/alex/infrastructure/atomic-server-turso/Caddyfile.production (86 lines)
  /home/alex/caddy_terraphim/conf/Caddyfile_auth2 (63 lines)
  ... (10 more files)

💡 Tip: Use 'Validate' workflow to check syntax before deployment

Pattern 2: Find on All Servers

User: "Find all Caddyfiles across all servers"

Assistant Process:
1. Load all server configs
2. Execute searches in parallel (local + SSH to bigbox + SSH to registry)
3. Aggregate results by server
4. Mark active configs
5. Display grouped by server

Example Output:
Found 28 Caddyfiles across 3 servers:

LOCAL (14 files):
  /Users/alex/projects/zestic-ai/charm/charm-impact/infrastructure/Caddyfile.production (167 lines)
  /Users/alex/projects/zestic-ai/charm/charm-impact/infrastructure/Caddyfile_docker (31 lines)
  ... (12 more)

BIGBOX (14 files):
  ✓ ACTIVE: /home/alex/caddy_terraphim/conf/Caddyfile_auth (324 lines)
  /home/alex/infrastructure/atomic-server-turso/Caddyfile.enhanced (112 lines)
  ... (12 more)

REGISTRY (1 file):
  ✓ ACTIVE: /etc/caddy/Caddyfile (41 lines)

⚠️  bigbox: Migration to systemd + 1Password recommended
✓  registry: Properly configured (reference implementation)

Pattern 3: Search with Pattern

User: "Find Caddyfiles with 'auth' in the name on bigbox"

Assistant Process:
1. Load bigbox config
2. Add pattern filter: -name '*auth*'
3. Execute search
4. Display results

Example Output:
Found 3 Caddyfiles matching 'auth' on bigbox:
  ✓ /home/alex/caddy_terraphim/conf/Caddyfile_auth (ACTIVE, 324 lines)
  /home/alex/caddy_terraphim/conf/Caddyfile_auth2 (63 lines)
  /home/alex/caddy_terraphim/conf/Caddyfile_localauth (123 lines)

Implementation Details

Step 1: Load Server Configuration

# Read servers.json
cat ~/.codex/skills/caddy/servers.json | jq -r '.servers.bigbox'

Step 2: Execute Search

For local (macOS):

mdfind -name Caddyfile | grep -v ".backup\." | head -20

For remote servers (Linux):

ssh bigbox "find /home/alex/caddy_terraphim /home/alex/infrastructure \
  -name 'Caddyfile*' ! -name '*.backup.*' ! -name '*~' -type f"

Step 3: Identify Active Configuration

# Check systemd service for active config
ssh bigbox "systemctl show caddy-terraphim --property=ExecStart"
# Parse output to extract config path

Step 4: Get File Metadata

# For each file, get line count and modification time
ssh bigbox "wc -l /path/to/Caddyfile; stat -c %y /path/to/Caddyfile"

Step 5: Sort and Display

• Active configs first
• Then by modification time (newest first)
• Show metadata: lines, modification date, status

Performance Optimization

• Local search: mdfind is instant (Spotlight index)
• Remote search: find with path limits (<5 seconds)
• Parallel execution: Search all servers concurrently
• Caching: Results cached for 60 seconds (optional)

Error Handling

SSH Connection Failed:

✗ Error: Cannot connect to bigbox
  Check: ssh bigbox "echo test"
  Solution: Verify SSH config and connectivity

No Caddyfiles Found:

ℹ No Caddyfiles found on local
  Searched paths: /Users/alex/projects/
  Tip: Caddyfiles may be in different locations

Permission Denied:

✗ Error: Permission denied accessing /etc/caddy/Caddyfile
  Solution: Add sudo to remote command or adjust permissions

Security Guarantees

✓ Read-only operations (no file modifications) ✓ SSH uses existing configuration (no password prompts) ✓ No secrets displayed (even if found in configs) ✓ Respects file permissions (won't force access)

---

Workflow 2: Validate Configuration

Purpose: Validate Caddyfile syntax before deployment to prevent downtime.

When to Use

• Before deploying configuration changes
• After editing Caddyfiles
• When troubleshooting syntax errors
• As part of CI/CD pipeline

How It Works

1. User provides Caddyfile path and server
2. Skill reads server configuration
3. Skill executes caddy validate --config <path>
4. Skill parses validation output for errors
5. Skill checks for environment variables (coordinates with 1Password skill if needed)
6. Skill reports results with clear error messages

Usage Patterns

Pattern 1: Validate Single File

User: "Validate /home/alex/caddy_terraphim/conf/Caddyfile_auth on bigbox"

Assistant Process:
1. Load bigbox server config
2. Check Caddy binary exists
3. Execute: ssh bigbox "/home/alex/caddy_terraphim/caddy validate --config /home/alex/caddy_terraphim/conf/Caddyfile_auth"
4. Parse output
5. Check for undefined environment variables
6. Report results

Example Output (Valid):
✓ Configuration is valid
  File: /home/alex/caddy_terraphim/conf/Caddyfile_auth
  Lines: 324
  Binary: /home/alex/caddy_terraphim/caddy (v2.6.4)
  Validation time: 0.8s

Environment variables detected:
  ⚠️  GITHUB_CLIENT_ID (defined in caddy_complete.env)
  ⚠️  GITHUB_CLIENT_SECRET (plaintext in caddy_complete.env)
  ⚠️  JWT_SHARED_KEY (plaintext in caddy_complete.env)

Recommendation: Migrate secrets to 1Password
  See: Workflow "Migrate Secrets" or 1Password skill

Pattern 2: Validate with Errors

Example Output (Invalid):
✗ Configuration has errors

Error 1 (Line 45):
  /home/alex/caddy_terraphim/conf/Caddyfile_auth:45
  parsing Caddyfile tokens: missing closing brace

  Context:
  43: reverse_proxy localhost:3000 {
  44:     header_up X-Real-IP {remote_host}
  45:     # Missing closing brace here
  46:
  47: handle /api {

Fix: Add closing brace after line 44

Error 2 (Line 102):
  Unrecognized directive: 'auth_portal'
  Did you mean: 'authentication portal' ?

Recommendation: Fix these 2 errors before deployment

Pattern 3: Validate Before Deployment

User: "Validate the active Caddyfile on bigbox before I deploy changes"

Assistant Process:
1. Identify active config (Workflow 1)
2. Validate current active config
3. If user modified a file, validate that too
4. Compare and report

Example Output:
✓ Current active config is valid
  File: /home/alex/caddy_terraphim/conf/Caddyfile_auth
  Status: Currently running, no errors in logs

✓ Modified config is valid
  File: /home/alex/caddy_terraphim/conf/Caddyfile_auth
  Changes: 15 lines modified (comparing to last backup)

Validation Summary:
  - Syntax: ✓ Valid
  - Directives: ✓ All recognized
  - Env vars: ⚠️  3 plaintext secrets detected
  - Conflicts: ✓ No port conflicts

Ready for deployment: Yes
Recommendation: Deploy with 'Workflow 4: Deploy Changes'

Implementation Details

Step 1: Check Caddy Binary

# Verify Caddy is available
ssh bigbox "which /home/alex/caddy_terraphim/caddy && /home/alex/caddy_terraphim/caddy version"

Step 2: Execute Validation

# Validate Caddyfile
ssh bigbox "/home/alex/caddy_terraphim/caddy validate --config /path/to/Caddyfile --adapter caddyfile"

Step 3: Parse Output

# Validation successful if exit code 0
# Parse stderr for error messages
# Extract line numbers and error types

Step 4: Check Environment Variables

# Scan Caddyfile for {env.VAR_NAME} or {$VAR_NAME} syntax
grep -oP '\{(env\.)?\$?[A-Z_]+\}' Caddyfile

# Check if variables defined in env file or 1Password template
cat caddy_complete.env | grep "VAR_NAME="

Step 5: Detect Secrets

# Coordinate with 1Password skill to detect plaintext secrets
# See: ~/.codex/skills/1password-secrets/SKILL.md Workflow 1

Validation Checks

Syntax Validation:

• Valid Caddyfile format
• Matching braces and brackets
• Recognized directives
• Valid matcher syntax
• Proper nesting

Semantic Validation:

• No duplicate site blocks
• No conflicting matchers
• Valid port bindings (not in use)
• Valid upstream addresses
• Proper TLS configuration

Environment Validation:

• All referenced variables defined
• No plaintext secrets (warn if found)
• Env file readable if specified

Security Validation:

• Secrets in 1Password (recommended)
• No exposed credentials
• Proper TLS configuration
• Security headers present (recommended)

Error Messages

Syntax Errors:

Line 45: Missing closing brace
  Expected '}' to close block started at line 43

  43: reverse_proxy localhost:3000 {
  44:     header_up X-Real-IP {remote_host}
  45:     # Missing '}'

Directive Errors:

Line 102: Unrecognized directive 'auth_portal'
  Available in module: http.handlers.authentication
  Suggestion: authentication portal myportal
  Documentation: https://caddyserver.com/docs/...

Environment Errors:

Line 15: Undefined environment variable 'API_KEY'
  Referenced as: {env.API_KEY}
  Solution 1: Define in caddy_complete.env
  Solution 2: Migrate to 1Password: op://Vault/Item/api_key

Safety Guarantees

✓ Read-only validation (no file changes) ✓ No service disruption (validation only) ✓ No secret exposure (secrets masked in output) ✓ Detailed error reporting (line numbers, context)

---

Workflow 3: Format Caddyfile

Purpose: Format Caddyfile for consistent style and readability.

When to Use

• After manual edits
• Before committing to version control
• When adopting team style standards
• After merging configuration changes

How It Works

1. User provides Caddyfile path and server
2. Skill creates timestamped backup
3. Skill executes caddy fmt --overwrite <path>
4. Skill validates formatted output
5. Skill shows diff (before/after) if requested
6. Skill provides rollback instructions

Usage Pattern

User: "Format /home/alex/caddy_terraphim/conf/Caddyfile_auth on bigbox"

Assistant Process:
1. Load bigbox config
2. Create backup: Caddyfile_auth.backup.20251229_143000
3. Execute: caddy fmt --overwrite Caddyfile_auth
4. Validate formatted file
5. Show summary

Example Output:
✓ Caddyfile formatted successfully
  File: /home/alex/caddy_terraphim/conf/Caddyfile_auth
  Backup: /home/alex/caddy_terraphim/conf/Caddyfile_auth.backup.20251229_143000
  Changes: Indentation normalized, spacing adjusted

Validation: ✓ Formatted file is valid

To rollback:
  ssh bigbox "cp /home/alex/caddy_terraphim/conf/Caddyfile_auth.backup.20251229_143000 \
              /home/alex/caddy_terraphim/conf/Caddyfile_auth"

Recommendation: Review changes before deployment

Implementation Details

Step 1: Create Backup

# Timestamped backup
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
cp Caddyfile_auth Caddyfile_auth.backup.$TIMESTAMP

Step 2: Format

# Execute format with overwrite
caddy fmt --overwrite /path/to/Caddyfile

Step 3: Validate

# Ensure formatting didn't break syntax
caddy validate --config /path/to/Caddyfile

Step 4: Show Diff (Optional)

# Compare before/after
diff -u Caddyfile_auth.backup.$TIMESTAMP Caddyfile_auth

Safety Guarantees

✓ Always creates backup before formatting ✓ Validates after formatting to ensure no breakage ✓ Provides rollback instructions immediately ✓ Never formats active config without confirmation

---

Workflow 4: Deploy Changes (Graceful Reload)

Purpose: Deploy Caddyfile changes with zero downtime using graceful reload.

When to Use

• After validating configuration changes
• Deploying new site configurations
• Updating TLS settings
• Modifying reverse proxy rules

How It Works

1. Pre-deployment checks (validation, service status, secret detection)
2. Create backup of current config
3. Detect deployment method (systemd reload or caddy reload API)
4. Execute graceful reload
5. Post-deployment verification (service status, logs, endpoint tests)
6. Automatic rollback on failure

Usage Pattern

User: "Deploy Caddyfile changes to bigbox"

Assistant Process:
1. Pre-deployment Checks:
   ✓ Validate configuration
   ✓ Check service is running
   ⚠️  Detect 3 plaintext secrets (recommend 1Password migration)
   ✓ Create backup

2. Deployment:
   Method: systemd reload (zero-downtime)
   Command: ssh bigbox "systemctl reload caddy-terraphim"
   Status: ✓ Reload initiated

3. Verification:
   ✓ Service status: active (running)
   ✓ No errors in logs (last 50 lines)
   ✓ Test endpoint: https://alpha.truthforge.terraphim.cloud/health → 200 OK
   ✓ Requests dropped during reload: 0

Deployment Summary:
  - Status: ✓ Success
  - Duration: 2.3 seconds
  - Downtime: 0 seconds (graceful reload)
  - Backup: Caddyfile_auth.backup.20251229_143500

⚠️  Recommendation: Migrate secrets to 1Password for enhanced security
    See: 1Password skill or Workflow "Migrate Secrets"

Pre-Deployment Checks

Check 1: Validate Configuration

# Must pass validation
caddy validate --config /path/to/Caddyfile

Check 2: Service Running

# Verify Caddy is running
systemctl is-active caddy-terraphim
# Or check process for tmux-based
ps aux | grep caddy | grep -v grep

Check 3: Detect Secrets

# Scan for plaintext secrets (coordinate with 1Password skill)
# Warn user but don't block deployment

Check 4: Create Backup

# Always backup current active config
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
cp /path/to/Caddyfile /path/to/Caddyfile.backup.$TIMESTAMP

Deployment Methods

Method 1: Systemd Reload (Recommended)

# Graceful reload via systemd
systemctl reload caddy-terraphim

# Benefits:
# - Zero downtime (in-place reload)
# - No dropped connections
# - Automatic rollback on failure
# - Logs in journalctl

Method 2: Caddy Reload API

# Direct reload via Caddy admin API
curl -X POST http://localhost:2019/load \
  -H "Content-Type: application/json" \
  -d @config.json

# Benefits:
# - Finer control
# - Can preview config before apply
# - Works without systemd

Method 3: Signal Reload

# Send USR1 signal (for tmux-based or non-systemd)
kill -USR1 $(cat /var/run/caddy.pid)

# Note: Less reliable, use systemd if possible

Post-Deployment Verification

Step 1: Check Service Status

systemctl status caddy-terraphim --no-pager
# Verify: active (running), no recent errors

Step 2: Check Recent Logs

# Last 50 log lines for errors
journalctl -u caddy-terraphim -n 50 --no-pager | grep -i error

Step 3: Test Endpoints

# Test configured domains
curl -f https://alpha.truthforge.terraphim.cloud/health
curl -f https://ci.terraphim.cloud/health

# Verify: 200 OK responses

Step 4: Check Dropped Requests

# Parse logs for any dropped connections during reload
# Caddy's graceful reload should have 0 drops

Rollback on Failure

If deployment fails (validation error, service won't start, endpoints down):

# Automatic rollback process:
1. Restore backup:
   cp Caddyfile.backup.20251229_143500 Caddyfile

2. Reload with backup config:
   systemctl reload caddy-terraphim

3. Verify rollback successful:
   systemctl status caddy-terraphim
   curl -f https://domain.com/health

4. Report failure details to user with logs

Safety Guarantees

✓ Zero-downtime reloads (graceful, no dropped requests) ✓ Always validates before deployment ✓ Always creates backup before changes ✓ Automatic rollback on failure ✓ Post-deployment verification confirms success

---

Workflow 5: Check Status

Purpose: Check Caddy service health and current configuration.

Usage Pattern

User: "Check Caddy status on bigbox"

Assistant Process:
1. Query systemd service status
2. Check process details (PID, uptime, memory)
3. Verify listening ports
4. Check recent log entries
5. Test configured endpoints

Example Output:
Caddy Status on bigbox:

Service: caddy-terraphim
  Status: ✓ active (running)
  PID: 24566
  Uptime: 35 days 14 hours
  Memory: 27.6 MB
  User: caddy

Configuration:
  Active: /home/alex/caddy_terraphim/conf/Caddyfile_auth (324 lines)
  Loaded: 3 hours ago
  Last reload: 2 hours ago (successful)

Listening Ports:
  ✓ :80 (HTTP)
  ✓ :443 (HTTPS)

Recent Activity (last 10 minutes):
  - 245 requests processed
  - 0 errors
  - Average response time: 45ms

Domains Configured:
  ✓ alpha.truthforge.terraphim.cloud (responding)
  ✓ ci.terraphim.cloud (responding)
  ✓ logs.terraphim.cloud (responding)

Health: ✓ All systems operational

⚠️  Note: Service running in tmux, migration to systemd recommended

Implementation

# Service status
systemctl status caddy-terraphim

# Process details
ps aux | grep caddy | grep -v grep

# Listening ports
ss -tulpn | grep caddy

# Recent logs
journalctl -u caddy-terraphim --since "10 minutes ago"

# Test endpoints
curl -sf https://domain.com/health

---

Integration with 1Password Skill

This skill coordinates with the 1Password Secret Management Skill for secure credential handling.

Secret Detection

When scanning Caddyfiles, this skill detects:

• Environment variable references: {env.VAR_NAME}
• Plaintext secrets in env files
• Basic auth credentials
• API keys and tokens

Then coordinates with 1Password skill (Workflow 1: Detect Secrets) to:

• Identify secret types
• Calculate confidence scores
• Recommend migration to op:// references

Secret Migration

For bigbox migration, this skill will:

1. Detect plaintext secrets in caddy_complete.env
2. Use 1Password skill (Workflow 2: Generate Template) to create .env.template
3. Guide user to create 1Password items
4. Update systemd service to use op run --no-masking
5. Verify migration successful

See: ~/.codex/skills/1password-secrets/SKILL.md for detailed secret management workflows.

---

Quick Reference

Common Commands

# Discovery
"Find all Caddyfiles on bigbox"
"Find Caddyfiles with 'auth' in name on all servers"

# Validation
"Validate /path/to/Caddyfile on bigbox"
"Validate the active Caddyfile before deployment"

# Format
"Format /path/to/Caddyfile on registry"

# Deploy
"Deploy Caddyfile changes to bigbox"

# Status
"Check Caddy status on registry"

Server Names

• local: Local development machine (macOS)
• bigbox: Production server (migration needed)
• registry: Production server (reference implementation)
• all: All servers

File Locations

• Local: /Users/alex/projects/*/infrastructure/Caddyfile*
• bigbox: /home/alex/caddy_terraphim/conf/Caddyfile_auth (active)
• registry: /etc/caddy/Caddyfile (active)

---

Related Skills

• 1Password Secret Management: ~/.codex/skills/1password-secrets/SKILL.md
• Systemd Management: For service lifecycle operations
• SSH Management: For remote server access

---

Version History:

• 1.0.0 (2025-12-29): Initial release with core workflows (Discovery, Validation, Format, Deploy, Status)

Maintainer: Codex CLI License: MIT Documentation: This file Server Configuration: ~/.codex/skills/caddy/servers.json Pattern Library: ~/.codex/skills/caddy/patterns/ (coming soon)

---

Next Workflows Coming Soon:

• Workflow 6: Build Custom Caddy (xcaddy with plugins)
• Workflow 7: Analyze Logs (JSON parsing, error detection)
• Workflow 8: Extract Snippets (DRY principle, pattern identification)
• Workflow 9: Migrate to Systemd (bigbox migration with 1Password)
• Workflow 10: Pattern Library (reverse proxy, auth, WebSocket, security headers)