crawl4ai

alexjx's avatarfrom alexjx

Web crawler that converts URLs to clean markdown for LLM consumption. Use when user wants to crawl a webpage, extract web content, convert URL to markdown, or scrape website content for AI processing.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 23, 2025
[Web Automation]

When & Why to Use This Skill

The Crawl4AI Claude skill is a high-performance web crawler designed to convert complex URLs into clean, LLM-friendly markdown. By stripping away HTML noise and focusing on essential content, it enables AI agents to efficiently process, analyze, and synthesize web data for research and automation tasks.

Use Cases

  • AI-Driven Web Research: Automatically fetch and convert multiple web pages into structured markdown to provide deep context for research reports.
  • RAG Data Preparation: Extract clean, high-quality text from documentation or blogs to populate vector databases for Retrieval-Augmented Generation (RAG) systems.
  • Automated Content Scraping: Streamline the collection of web-based information for AI processing without manual copy-pasting, supporting images and link extraction.
  • Competitive Analysis: Quickly gather and format content from competitor websites to perform automated SWOT analysis or market comparisons.
namecrawl4ai
descriptionWeb crawler that converts URLs to clean markdown. Use when user wants to fetch a webpage, extract web content, convert URL to markdown, or scrape website content.

Crawl4AI Web Crawler

Converts web pages to clean, LLM-friendly markdown using crawl4ai.

Requirements

Setup

Before first use, run the setup script:

skills/crawl4ai/scripts/setup.sh

This will:

  1. Check for uv installation
  2. Run uv sync to create .venv/ and install dependencies
  3. Run crawl4ai-setup to install playwright browsers
  4. Run crawl4ai-doctor to verify installation

Usage

To crawl a URL and get markdown (from the skill directory):

cd skills/crawl4ai
uv run python scripts/crawl.py "https://example.com"

Options:

  • --output FILE - Save markdown to file instead of stdout
  • --include-links - Include hyperlinks in markdown output
  • --include-images - Include image references
  • --no-headless - Show browser window (default: headless)
  • --timeout SECONDS - Page load timeout (default: 30)

Examples

Basic crawl

cd skills/crawl4ai
uv run python scripts/crawl.py "https://docs.python.org/3/"

Save to file

uv run python scripts/crawl.py "https://example.com" --output content.md

With all options

uv run python scripts/crawl.py "https://example.com" \
  --include-links \
  --include-images \
  --timeout 60 \
  --output result.md

Troubleshooting

If crawling fails:

  1. Run uv run crawl4ai-doctor to diagnose
  2. Ensure playwright browsers are installed: uv run python -m playwright install chromium
  3. Check the URL is accessible and not blocked by robots.txt