This Claude skill provides a comprehensive toolkit for mastering production-grade shell utilities, including jq, xargs, and GNU parallel. It enables users to build high-performance data pipelines, automate complex system tasks, and perform advanced JSON manipulation directly from the command line, significantly enhancing DevOps and backend development workflows.

When should I use shell-tools?

shell-tools is useful in the following scenarios: • API Data Transformation: Efficiently parsing, filtering, and converting complex JSON responses from REST APIs into structured formats like CSV or TSV using jq. • High-Performance Batch Processing: Utilizing GNU parallel to distribute CPU-intensive tasks, such as log compression or image resizing, across multiple processor cores to reduce execution time. • Safe File Operations: Handling large-scale file deletions or modifications using xargs with null-terminators to prevent 'argument list too long' errors and ensure filename safety. • Automated Log Analysis: Constructing robust pipelines with sort, uniq, and grep to extract actionable insights, count occurrences, and identify anomalies within massive system log files. • Data Migration and Cleaning: Streamlining the process of reformatting and migrating data between different systems using text transformation tools like tr, cut, and paste.

name	shell-tools
description	Production-grade shell tools - jq, xargs, parallel, pipelines
sasmp_version	"1.3.0"
bonded_agent	07-shell-tools
bond_type	PRIMARY_BOND
version	"2.0.0"
difficulty	advanced
estimated_time	"6-8 hours"

Shell Tools Skill

Master jq, xargs, GNU parallel, and advanced pipelines

Learning Objectives

After completing this skill, you will be able to:

Process JSON with jq
Use xargs for argument handling
Parallelize tasks with GNU parallel
Build efficient data pipelines
Use utility commands effectively

Prerequisites

Strong Bash fundamentals
Text processing basics
Understanding of pipes

Core Concepts

1. jq Essentials

# Basic queries
jq '.' file.json              # Pretty print
jq '.key' file.json           # Get key
jq '.array[0]' file.json      # First element
jq '.nested.key' file.json    # Nested

# Filtering
jq '.[] | select(.active)'    # Filter
jq '.[] | select(.count > 10)'

# Transform
jq '.[] | {id, name}'         # Select fields
jq 'map(.price * .qty)'       # Calculate
jq -r '.[] | @csv'            # To CSV

# From variables
jq -n --arg x "$VAR" '{value: $x}'

2. Xargs

# Basic usage
echo "a b c" | xargs echo

# Safe with spaces
find . -print0 | xargs -0 rm

# Limit arguments
cat list | xargs -n 1 process
cat list | xargs -n 10 process

# Parallel
cat list | xargs -P 4 -n 1 process

# Placeholder
cat urls | xargs -I {} curl {}

3. GNU Parallel

# Basic
parallel echo ::: a b c

# From file
parallel process :::: list.txt

# With options
parallel -j 4 process ::: *.txt
parallel --progress process ::: *.txt

# Complex
parallel -j 4 --delay 0.5 \
    'curl -s {} | jq .name' :::: urls.txt

4. Pipeline Utilities

# Sort and unique
sort file.txt
sort -n file.txt          # Numeric
sort -u file.txt          # Unique
sort file | uniq -c       # Count

# Cut and paste
cut -d',' -f1,3 file.csv
paste file1.txt file2.txt

# Transform
tr 'a-z' 'A-Z' < file
tr -d '\r' < dos.txt > unix.txt

Common Patterns

API Data Pipeline

curl -s 'https://api.example.com/users' |
    jq -r '.[] | select(.active) | [.id, .email] | @csv' |
    sort -t',' -k2 |
    head -20

Parallel Processing

# Compress all logs in parallel
find . -name "*.log" |
    parallel -j 4 gzip

# Batch API calls with rate limit
cat ids.txt |
    parallel -j 5 --delay 0.2 \
        'curl -s "https://api.example.com/item/{}"'

Data Transformation

# JSON to formatted output
cat data.json |
    jq -r '.items[] | "\(.id)\t\(.name)\t\(.price)"' |
    column -t

Anti-Patterns

Don't	Do	Why
Parse JSON with grep	Use jq	Proper parsing
Sequential when parallel	Use parallel	Speed
`cat \| xargs`	`xargs < file`	Efficiency

Practice Exercises

JSON Processor: Transform API response
Batch Processor: Parallel file processing
Log Analyzer: Complex log pipeline
Data Migrator: Transform and load data

Troubleshooting

Common Errors

Error	Cause	Fix
`jq: error`	Invalid JSON	Validate with `jq .`
`xargs: arg too long`	Too many args	Use `-n`
`parallel: not found`	Not installed	`apt install parallel`

Debug Techniques

# Validate JSON
jq '.' < input.json

# Debug pipeline
command1 | tee /dev/stderr | command2

# Test jq filter
echo '{"a":1}' | jq '.a'

Performance Tips

# Faster sorting
LC_ALL=C sort file.txt

# Parallel for CPU-bound
parallel -j $(nproc) process ::: *.txt

# Stream large files
jq -c '.[]' large.json | while read -r line; do
    # process line
done

shell-tools

When & Why to Use This Skill

Use Cases