This Claude skill provides comprehensive guidance on designing and optimizing LLM inference infrastructure. It covers industry-standard serving frameworks like vLLM and TensorRT-LLM, advanced quantization techniques (AWQ, GPTQ), and high-performance patterns such as continuous batching and PagedAttention to minimize latency and maximize throughput in production environments.
Use Cases
Architecting high-throughput LLM serving systems using frameworks like vLLM, TGI, or TensorRT-LLM based on specific hardware and latency requirements.
Implementing quantization strategies (INT8, INT4, AWQ) to reduce GPU memory footprint and lower operational costs without significant quality loss.
Optimizing inference performance for high-concurrency applications through continuous batching and efficient KV cache management using PagedAttention.
Designing real-time interactive AI applications by implementing streaming response patterns via Server-Sent Events (SSE) or WebSockets.
Scaling large-scale model deployments using model parallelism techniques like Tensor Parallelism and Pipeline Parallelism for multi-GPU clusters.
name
llm-serving-patterns
description
LLM inference infrastructure, serving frameworks (vLLM, TGI, TensorRT-LLM), quantization techniques, batching strategies, and streaming response patterns. Use when designing LLM serving infrastructure, optimizing inference latency, or scaling LLM deployments.
allowed-tools
Read, Glob, Grep
LLM Serving Patterns
When to Use This Skill
Use this skill when:
Designing LLM inference infrastructure
Choosing between serving frameworks (vLLM, TGI, TensorRT-LLM)
Implementing quantization for production deployment
PagedAttention, high throughput, continuous batching
General LLM serving, high concurrency
Python-native, active community
TGI (Text Generation Inference)
Production-ready, Hugging Face integration
Enterprise deployment, HF models
Rust backend, Docker-first
TensorRT-LLM
NVIDIA optimization, lowest latency
NVIDIA GPUs, latency-critical
NVIDIA-only, complex setup
Triton Inference Server
Multi-model, multi-framework
Heterogeneous model serving
Enterprise complexity
Ollama
Simple local deployment
Development, edge deployment
Limited scaling features
llama.cpp
CPU inference, quantization
Resource-constrained, edge
C++ integration required
Framework Selection Decision Tree
Need lowest latency on NVIDIA GPUs?
├── Yes → TensorRT-LLM
└── No
└── Need high throughput with many concurrent users?
├── Yes → vLLM (PagedAttention)
└── No
└── Need enterprise features + HF integration?
├── Yes → TGI
└── No
└── Simple local/edge deployment?
├── Yes → Ollama or llama.cpp
└── No → vLLM (general purpose)
Quantization Techniques
Precision Levels
Precision
Bits
Memory Reduction
Quality Impact
Use Case
FP32
32
Baseline
None
Training, reference
FP16/BF16
16
2x
Minimal
Standard serving
INT8
8
4x
Low
Production serving
INT4
4
8x
Moderate
Resource-constrained
INT2
2
16x
Significant
Experimental
Quantization Methods
Method
Description
Quality
Speed
PTQ (Post-Training Quantization)
Quantize after training, no retraining
Good
Fast to apply
QAT (Quantization-Aware Training)
Simulate quantization during training
Better
Requires training
GPTQ
One-shot weight quantization
Very good
Moderate
AWQ (Activation-aware Weight Quantization)
Preserves salient weights
Excellent
Moderate
GGUF/GGML
llama.cpp format, CPU-optimized
Good
Very fast inference
SmoothQuant
Migrates difficulty to weights
Excellent
Moderate
Quantization Selection
Quality vs. Efficiency Trade-off:
Quality ────────────────────────────────────────────▶ Efficiency
│ │
│ FP32 FP16 INT8+AWQ INT8+GPTQ INT4 INT2 │
│ ○───────○────────○──────────○──────────○──────○ │
│ │ │ │ │ │ │ │
│ Best Great Good Good Fair Poor │
│ │
Batching Strategies
Static Batching
Request 1: [tokens: 100] ─┐
Request 2: [tokens: 50] ─┼──▶ [Batch: pad to 100] ──▶ Process ──▶ All complete
Request 3: [tokens: 80] ─┘
Problem: Short requests wait for long ones (head-of-line blocking)
Attention: Q × K^T × V
For each token generated:
• Must recompute attention with ALL previous tokens
• K and V tensors grow with sequence length
• Memory: O(batch_size × seq_len × num_layers × hidden_dim)
Example (70B model, 4K context):
• KV cache per request: ~8GB
• 10 concurrent requests: ~80GB GPU memory