web-fetch
Fetches web content with intelligent content extraction, converting HTML to clean markdown. Use for documentation, articles, and reference pages http/https URLs.
When & Why to Use This Skill
The web-fetch skill is a high-performance tool designed for intelligent web content extraction, transforming complex HTML into clean, structured Markdown. By utilizing precise CSS selectors and automated content detection, it effectively strips away boilerplate elements such as navigation bars, footers, and advertisements. This ensures that Claude receives high-quality, noise-free text, making it an essential utility for documentation retrieval, technical research, and automated data gathering.
Use Cases
- Technical Documentation Retrieval: Seamlessly fetch and convert developer docs from platforms like MDN, GitHub, or Anthropic into clean Markdown for immediate coding assistance.
- Article and Blog Extraction: Extract the core content from news articles and blog posts, removing distracting UI elements to facilitate accurate summarization and analysis.
- Knowledge Base Integration: Convert web-based reference pages into a standardized Markdown format for inclusion in internal knowledge bases or RAG (Retrieval-Augmented Generation) systems.
- Automated Web Research: Use site-specific patterns or universal fallback scripts to gather high-fidelity text data from any public URL for competitive analysis or market research.
| name | web-fetch |
|---|---|
| description | Fetches web content with intelligent content extraction, converting HTML to clean markdown. Use for documentation, articles, and reference pages http/https URLs. |
Web Content Fetching
Fetch web content using curl | html2markdown with CSS selectors for clean, complete markdown output.
Quick Usage (Known Sites)
Use site-specific selectors for best results:
# Anthropic docs
curl -s "<url>" | html2markdown --include-selector "#content-container"
# MDN Web Docs
curl -s "<url>" | html2markdown --include-selector "article"
# GitHub docs
curl -s "<url>" | html2markdown --include-selector "article" --exclude-selector "nav,.sidebar"
# Generic article pages
curl -s "<url>" | html2markdown --include-selector "article,main,[role=main]" --exclude-selector "nav,header,footer"
Site Patterns
| Site | Include Selector | Exclude Selector |
|---|---|---|
| platform.claude.com | #content-container |
- |
| docs.anthropic.com | #content-container |
- |
| developer.mozilla.org | article |
- |
| github.com (docs) | article |
nav,.sidebar |
| Generic | article,main |
nav,header,footer,script,style |
Universal Fallback (Unknown Sites)
For sites without known patterns, use the Bun script which auto-detects content:
bun ~/.claude/skills/web-fetch/fetch.ts "<url>"
Setup (one-time)
cd ~/.claude/skills/web-fetch && bun install
Finding the Right Selector
When a site isn't in the patterns list:
# Check what content containers exist
curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selector
curl -s "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line count
curl -s "<url>" | html2markdown --include-selector "<selector>" | wc -l
Options Reference
--include-selector "CSS" # Only include matching elements
--exclude-selector "CSS" # Remove matching elements
--domain "https://..." # Convert relative links to absolute
Comparison
| Method | Anthropic Docs | Code Blocks | Complexity |
|---|---|---|---|
| Full page | 602 lines | Yes | Noisy |
--include-selector "#content-container" |
385 lines | Yes | Clean |
| Bun script (universal) | 383 lines | Yes | Clean |
Troubleshooting
Wrong content selected: The site may have multiple articles. Inspect the HTML:
curl -s "<url>" | grep -o '<article[^>]*>'
Empty output: The selector doesn't match. Try broader selectors like main or body.
Missing code blocks: Check if the site uses non-standard code formatting.
Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.