system-configuration-analysis
Analyze system configuration data from sosreport archives, extracting OS details, installed packages, systemd service status, SELinux/AppArmor policies, and kernel parameters from the sosreport directory structure to diagnose configuration-related system issues
When & Why to Use This Skill
This Claude skill automates the comprehensive analysis of Linux system configuration data extracted from sosreport archives. It streamlines the troubleshooting process by automatically parsing OS details, package versions, systemd service statuses, and security policies (SELinux/AppArmor). By identifying misconfigurations, failed services, and kernel parameter anomalies, it enables SREs and System Administrators to rapidly perform root cause analysis and resolve complex system-level incidents.
Use Cases
- Root Cause Analysis: Quickly diagnose why a Linux server is failing by analyzing service states and error logs within a sosreport archive.
- Security Policy Troubleshooting: Identify and resolve application blocks caused by SELinux or AppArmor denials by inspecting audit logs and enforcement modes.
- System Health Auditing: Verify package versions, kernel parameters, and system uptime to ensure servers meet compliance standards and are running stable software.
- Service Failure Investigation: Extract specific failure reasons for systemd units and correlate them with system-wide configuration changes or resource limits.
| name | System Configuration Analysis |
|---|---|
| description | Analyze system configuration data from sosreport archives, extracting OS details, installed packages, systemd service status, SELinux/AppArmor policies, and kernel parameters from the sosreport directory structure to diagnose configuration-related system issues |
System Configuration Analysis Skill
This skill provides detailed guidance for analyzing system configuration from sosreport archives, including OS information, installed packages, systemd services, and SELinux/AppArmor settings.
When to Use This Skill
Use this skill when:
- Analyzing the
/sosreport:analyzecommand's system configuration phase - Investigating service failures or misconfigurations
- Verifying package versions and updates
- Checking security policy settings (SELinux/AppArmor)
- Understanding system state and configuration
Prerequisites
- Sosreport archive must be extracted to a working directory
- Path to the sosreport root directory must be known
- Understanding of Linux system administration
Key Configuration Data Locations in Sosreport
System Information:
uname- Kernel versionetc/os-release- OS distribution and versionuptime- System uptimeproc/uptime- Uptime in secondssos_commands/release/- Release information
Package Information:
installed-rpms- RPM packages (RHEL/Fedora/CentOS)installed-debs- DEB packages (Debian/Ubuntu)sos_commands/yum/- Yum/DNF informationsos_commands/rpm/- RPM database queries
Service Status:
sos_commands/systemd/systemctl_list-units- All unitssos_commands/systemd/systemctl_list-units_--failed- Failed unitssos_commands/systemd/systemctl_status_--all- Detailed service statussos_commands/systemd/systemctl_list-unit-files- Unit files
SELinux:
sos_commands/selinux/sestatus- SELinux statussos_commands/selinux/getenforce- Current enforcement modesos_commands/selinux/selinux-policy- Policy informationvar/log/audit/audit.log- SELinux denials
AppArmor (if applicable):
sos_commands/apparmor/- AppArmor configurationetc/apparmor.d/- AppArmor profiles
System Configuration Files:
etc/- System-wide configurationetc/sysctl.conforetc/sysctl.d/- Kernel parametersetc/security/limits.conf- Resource limits
Implementation Steps
Step 1: Analyze System Information
Check OS version and distribution:
if [ -f etc/os-release ]; then cat etc/os-release fiGet kernel version:
if [ -f uname ]; then cat uname elif [ -f proc/version ]; then cat proc/version fiCheck system uptime:
if [ -f uptime ]; then cat uptime elif [ -f proc/uptime ]; then # Parse uptime from proc/uptime (seconds) awk '{printf "%.2f days\n", $1/86400}' proc/uptime fiExtract key system details:
- OS name and version
- Kernel version
- System architecture (x86_64, aarch64, etc.)
- Uptime (days)
Check for outdated kernel or OS:
- Compare kernel version with current stable
- Note if system hasn't been rebooted in a very long time (>365 days)
- Identify if OS version is EOL
Step 2: Analyze Installed Packages
List installed packages:
# For RPM-based systems if [ -f installed-rpms ]; then cat installed-rpms fi # For DEB-based systems if [ -f installed-debs ]; then cat installed-debs fiExtract key package versions:
# Important system packages grep -E "^(kernel|systemd|glibc|openssh|openssl)" installed-rpms 2>/dev/null # Or use awk to parse package name and version awk '{print $1}' installed-rpms | head -20Check for known problematic versions:
- Security vulnerabilities (if known CVEs)
- Buggy package versions
- Compatibility issues
Identify package manager issues:
# Check yum/dnf logs for errors if [ -d sos_commands/yum ]; then grep -i "error\|fail" sos_commands/yum/* 2>/dev/null fiCount packages and categorize:
- Total packages installed
- Key package versions (kernel, systemd, glibc, etc.)
- Recently updated packages (if timestamps available)
Step 3: Analyze Service Status
List all systemd units:
if [ -f sos_commands/systemd/systemctl_list-units ]; then cat sos_commands/systemd/systemctl_list-units fiIdentify failed services:
if [ -f sos_commands/systemd/systemctl_list-units_--failed ]; then cat sos_commands/systemd/systemctl_list-units_--failed elif [ -f sos_commands/systemd/systemctl_list-units ]; then grep "failed" sos_commands/systemd/systemctl_list-units fiCheck service details:
# Parse detailed status for failed services if [ -f sos_commands/systemd/systemctl_status_--all ]; then # Extract service names and their status grep -E "●|Active:" sos_commands/systemd/systemctl_status_--all | head -50 fiCount services by state:
# Count running, failed, inactive services if [ -f sos_commands/systemd/systemctl_list-units ]; then awk '{print $4}' sos_commands/systemd/systemctl_list-units | sort | uniq -c fiIdentify critical service failures:
- System services (systemd-*, dbus, NetworkManager)
- Application services (httpd, nginx, postgresql, etc.)
- Custom services
Extract failure reasons from logs:
# For each failed service, find related log entries grep -i "failed to start\|service.*failed" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20
Step 4: Analyze SELinux Configuration
Check SELinux status:
if [ -f sos_commands/selinux/sestatus ]; then cat sos_commands/selinux/sestatus fiGet SELinux mode:
if [ -f sos_commands/selinux/getenforce ]; then cat sos_commands/selinux/getenforce fiCheck for SELinux denials:
# Look for AVC denials in audit log if [ -f var/log/audit/audit.log ]; then grep "avc.*denied" var/log/audit/audit.log | head -50 fi # Or in journald logs grep -i "selinux.*denied\|avc.*denied" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20Parse denial information:
- Extract denied operations (read, write, execute, etc.)
- Identify source and target contexts
- Note which services are affected
Check for SELinux booleans:
if [ -f sos_commands/selinux/getsebool_-a ]; then cat sos_commands/selinux/getsebool_-a fiIdentify SELinux issues:
- SELinux in permissive mode (may hide errors)
- SELinux disabled (security concern)
- Frequent AVC denials (policy may need adjustment)
- Context mismatches
Step 5: Check System Configuration
Review kernel parameters:
# Check sysctl settings if [ -f sos_commands/kernel/sysctl_-a ]; then cat sos_commands/kernel/sysctl_-a elif [ -d etc/sysctl.d ]; then cat etc/sysctl.d/*.conf 2>/dev/null fiCheck resource limits:
if [ -f etc/security/limits.conf ]; then grep -v "^#\|^$" etc/security/limits.conf fi # Check limits.d directory if [ -d etc/security/limits.d ]; then cat etc/security/limits.d/*.conf 2>/dev/null fiReview boot parameters:
if [ -f sos_commands/boot/grub2-editenv_list ]; then cat sos_commands/boot/grub2-editenv_list elif [ -f proc/cmdline ]; then cat proc/cmdline fiCheck systemd configuration:
# Look for systemd configuration overrides if [ -d etc/systemd/system ]; then find etc/systemd/system -name "*.conf" 2>/dev/null fi
Step 6: Generate System Configuration Summary
Create a structured summary with the following sections:
System Information:
- OS name and version
- Kernel version
- Architecture
- System uptime
- Last boot time
Package Summary:
- Total packages installed
- Key package versions (kernel, systemd, glibc, openssl, openssh)
- Known problematic packages (if any)
- Package manager issues
Service Status:
- Total services
- Running services count
- Failed services count
- List of failed services with reasons
- Critical service status
SELinux/AppArmor:
- SELinux status (enabled/disabled)
- SELinux mode (enforcing/permissive)
- Denial count
- Top denied operations
- Policy recommendations
Configuration Issues:
- Kernel parameter anomalies
- Resource limit issues
- Boot parameter problems
- Configuration file errors
Error Handling
Missing configuration files:
- Different distributions have different file locations
- Some files may not be collected based on sosreport options
- Document missing data in summary
Package manager variations:
- Handle both RPM and DEB systems
- Account for different package naming conventions
- Support multiple package managers (yum, dnf, apt)
SELinux vs AppArmor:
- Check which MAC system is in use
- Analyze accordingly
- Note if both or neither are present
Systemd vs init:
- Older systems may use init instead of systemd
- Check for both service management systems
- Adapt analysis based on what's present
Output Format
The system configuration analysis should produce:
SYSTEM CONFIGURATION SUMMARY
============================
SYSTEM INFORMATION
------------------
OS: {os_name} {os_version}
Kernel: {kernel_version}
Architecture: {arch}
Uptime: {uptime_days} days ({last_boot_time})
Status: {OK|WARNING|CRITICAL}
Notes:
- {system_info_note}
INSTALLED PACKAGES
------------------
Total Packages: {count}
Key Package Versions:
kernel: {version}
systemd: {version}
glibc: {version}
openssl: {version}
openssh-server: {version}
Status: {OK|WARNING|CRITICAL}
Issues:
- {package_issue_description}
SYSTEMD SERVICES
----------------
Total Units: {total}
Active: {active_count}
Failed: {failed_count}
Inactive: {inactive_count}
Failed Services:
● {service_name}.service - {description}
Reason: {failure_reason}
Last Failed: {timestamp}
● {service_name}.service - {description}
Reason: {failure_reason}
Last Failed: {timestamp}
Status: {OK|WARNING|CRITICAL}
Recommendations:
- {service_recommendation}
SELINUX
-------
Status: {enabled|disabled}
Mode: {enforcing|permissive|disabled}
Policy: {policy_name}
AVC Denials: {count} denials found
Top Denied Operations:
[{count}x] {operation} on {target} by {source}
[{count}x] {operation} on {target} by {source}
SELinux Booleans: {count} custom settings
Status: {OK|WARNING|CRITICAL}
Issues:
- {selinux_issue_description}
Recommendations:
- {selinux_recommendation}
KERNEL PARAMETERS
-----------------
Key sysctl Settings:
vm.swappiness: {value}
net.ipv4.ip_forward: {value}
kernel.panic: {value}
Custom Parameters: {count} custom settings found
Status: {OK|WARNING|CRITICAL}
Notes:
- {kernel_param_note}
RESOURCE LIMITS
---------------
Custom Limits Found: {count}
{user_or_group} {type} {item} {value}
Status: {OK|WARNING}
Notes:
- {limits_note}
CRITICAL CONFIGURATION ISSUES
-----------------------------
{severity}: {issue_description}
Evidence: {file_path}
Impact: {impact_description}
Recommendation: {remediation_action}
RECOMMENDATIONS
---------------
1. {actionable_recommendation}
2. {actionable_recommendation}
DATA SOURCES
------------
- OS Info: {sosreport_path}/etc/os-release
- Kernel: {sosreport_path}/uname
- Packages: {sosreport_path}/installed-rpms
- Services: {sosreport_path}/sos_commands/systemd/systemctl_list-units
- SELinux: {sosreport_path}/sos_commands/selinux/sestatus
- Audit Log: {sosreport_path}/var/log/audit/audit.log
Examples
Example 1: Failed Service Analysis
# List failed services
$ cat sos_commands/systemd/systemctl_list-units_--failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● httpd.service loaded failed failed Apache Web Server
● postgresql.service loaded failed failed PostgreSQL database
# Find failure reason in logs
$ grep "httpd.service" sos_commands/logs/journalctl_--no-pager | grep -i "failed\|error"
Jan 15 10:23:45 server systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 10:23:45 server systemd[1]: httpd.service: Failed with result 'exit-code'
Jan 15 10:23:45 server httpd[12345]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
# Interpretation: httpd failed because port 80 is already in use
Example 2: SELinux Denial Analysis
# Check for AVC denials
$ grep "avc.*denied" var/log/audit/audit.log | head -5
type=AVC msg=audit(1705320245.123:456): avc: denied { write } for pid=1234 comm="httpd" name="index.html" dev="sda1" ino=789012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0
# Interpretation:
# - httpd (web server) was denied write access
# - Target file: index.html with context user_home_t
# - Issue: Web server trying to write to user home directory
# - Solution: Fix file context or move file to proper location
Example 3: Package Version Check
# Check for specific package versions
$ grep "^openssl" installed-rpms
openssl-1.1.1k-7.el8_6.x86_64
openssl-libs-1.1.1k-7.el8_6.x86_64
$ grep "^kernel" installed-rpms
kernel-4.18.0-425.el8.x86_64
kernel-4.18.0-477.el8.x86_64
kernel-core-4.18.0-425.el8.x86_64
kernel-core-4.18.0-477.el8.x86_64
# Interpretation:
# - OpenSSL version 1.1.1k (check for known CVEs)
# - Multiple kernels installed (good for rollback)
# - Current kernel is 4.18.0-477 (from uname)
Tips for Effective Analysis
- Check service dependencies: Failed service may be due to dependency failure
- Correlate with logs: Service failures often have detailed errors in logs
- Verify configurations: Check service config files for syntax errors
- Consider timing: When did service fail? Correlate with system events
- SELinux context matters: File contexts must match policy expectations
- Package versions: Compare with known good/bad versions
- Uptime significance: Very long uptime may mean missed security updates
Common Configuration Patterns and Issues
- Service dependency failure: ServiceB fails because ServiceA is not running
- Port conflict: Service fails to bind - port already in use
- Permission denied: Service can't access required files/directories
- SELinux blocking: Service denied access by SELinux policy
- Missing dependencies: Required package not installed
- Configuration error: Syntax error in config file
- Resource limits: Service hits ulimit (open files, processes, etc.)
- Outdated kernel: Running kernel doesn't match installed packages
Configuration Issue Severity Classification
| Issue Type | Severity | Impact |
|---|---|---|
| Critical service failed | High | Core functionality unavailable |
| Optional service failed | Low | Non-essential feature unavailable |
| SELinux in permissive | Warning | Reduced security, hiding issues |
| SELinux disabled | Critical | No mandatory access control |
| Kernel very outdated | High | Missing security fixes |
| EOL OS version | Critical | No security updates |
| Many AVC denials | Warning | Policy may need tuning |
See Also
- Logs Analysis Skill: For detailed service failure log analysis
- Resource Analysis Skill: For resource limit issues
- Network Analysis Skill: For network service configuration