crawl-to-markdown
Use when web.run fails to parse a webpage and you need raw Markdown via Crawl4AI.
When & Why to Use This Skill
The crawl-to-markdown skill is a robust web scraping utility that leverages Crawl4AI to convert webpages into clean, raw Markdown. It serves as a critical fallback for standard web parsing tools, ensuring that Claude can access and process content from complex or dynamic websites that traditional methods fail to capture.
Use Cases
- Reliable Fallback Scraping: Use this skill when built-in web tools fail to return usable content, ensuring uninterrupted data retrieval for agentic workflows.
- LLM Context Enrichment: Convert multiple URLs into structured Markdown to provide high-quality, readable context for Claude to perform deep analysis or summarization.
- Data Extraction from Dynamic Sites: Retrieve raw text from JavaScript-heavy or complex websites that typically resist standard HTML parsing.
- Batch Research Processing: Efficiently crawl a list of multiple URLs simultaneously to gather information for market research, academic study, or knowledge base construction.
| name | crawl-to-markdown |
|---|---|
| description | Use when web.run fails to parse a webpage and you need raw Markdown via Crawl4AI. |
Use this skill only after the built-in web.run tool fails to return usable content for a page.
What this skill does
- Crawls one or more URLs with Crawl4AI.
- Returns raw Markdown for each URL.
How to run
This script uses uv inline metadata. If uv is available just run:
uv run scripts/crawl_to_markdown.py https://example.com
Multiple URLs:
uv run scripts/crawl_to_markdown.py https://example.com https://example.org
Or pipe a newline-delimited list:
printf "https://example.com\nhttps://example.org\n" | uv run scripts/crawl_to_markdown.py
Troubleshooting
Crawl succeeds but output is blank
Some sites don't match the default --selector and can return empty/whitespace Markdown.
The script will retry without CSS selection automatically, but you can also force it explicitly:
uv run scripts/crawl_to_markdown.py --selector '' https://example.com
uv cache permission errors
If uv fails accessing its cache directory (e.g. ~/.cache/uv), either point it at a
writable cache directory:
uv --cache-dir .uv-cache run scripts/crawl_to_markdown.py https://example.com
Or avoid the cache entirely:
uv --no-cache run scripts/crawl_to_markdown.py https://example.com
Output format
For each URL, the script prints a header and raw Markdown content:
URL: https://example.com
<raw markdown>
---
If a crawl fails, the output includes an error block for that URL.