emergency-ulimit-mitigation
Emergency steps to mitigate ulimit-related crashes by increasing file descriptor limits and restarting services.
When & Why to Use This Skill
This Claude skill provides a critical emergency runbook for mitigating system crashes caused by ulimit restrictions and file descriptor exhaustion. It offers immediate commands to increase system limits, monitor resource usage in real-time, and restart essential services like Docker and Traefik, ensuring rapid recovery and improved system stability during high-load incidents.
Use Cases
- Scenario 1: Rapidly resolving 'Too many open files' errors on production Linux servers to prevent or recover from service downtime.
- Scenario 2: Emergency recovery of containerized environments and reverse proxies (e.g., Docker, Traefik) after resource exhaustion leads to service failure.
- Scenario 3: Real-time monitoring and auditing of file descriptor usage to identify leaking processes or capacity bottlenecks before they cause a crash.
- Scenario 4: Deploying automated cleanup scripts and persistent ulimit configurations to harden system infrastructure against future resource-related instability.
| name | emergency-ulimit-mitigation |
|---|---|
| description | Emergency steps to mitigate ulimit-related crashes by increasing file descriptor limits and restarting services. |
qqu='qq "cat ~/d/AI/Prompts/Commands/fix-ulimit-exceeded.md"'# Emergency ulimit Mitigation
Tags: AI/Prompt, ulimit, filedescriptors, crashing, toomanyfilesopen, Stability, Maintenance
Modified: 2025-10-27T08:55:02-04:00
Immediate Emergency Fix
# Instant relief - run immediately when services are crashing
ulimit -n 65536
echo 2097152 | sudo tee /proc/sys/fs/file-max
Emergency Service Recovery
# Restart crashed services
sudo systemctl restart docker traefik
sudo docker system prune -f
Check Current Status
# See what's using FDs
lsof | wc -l
cat /proc/sys/fs/file-nr | awk '{printf "FDs: %d/%d (%.1f%%)\n", $1, $3, ($1/$3)*100}'
Deploy Full Solution
sudo ./scripts/set-ulimits.sh
sudo systemctl enable --now fd-monitor.timer
Emergency Cleanup
./scripts/fd-cleanup.sh