crawl-to-markdown

当 web.run 无法解析网页且需要通过 Crawl4AI 获取原始 Markdown 时使用。

📖 中文导读:当 web.run 无法解析网页且需要通过 Crawl4AI 获取原始 Markdown 时使用。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: crawl-to-markdown description: Use when web.run fails to parse a webpage and you need raw Markdown via Crawl4AI.

Use this skill only after the built-in web.run tool fails to return usable content for a page.

What this skill does

  • Crawls one or more URLs with Crawl4AI.
  • Returns raw Markdown for each URL.

How to run

This script uses uv inline metadata. If uv is available just run:

uv run scripts/crawl_to_markdown.py https://example.com

Multiple URLs:

uv run scripts/crawl_to_markdown.py https://example.com https://example.org

Or pipe a newline-delimited list:

printf "https://example.com\nhttps://example.org\n" | uv run scripts/crawl_to_markdown.py

Troubleshooting

Crawl succeeds but output is blank

Some sites don't match the default --selector and can return empty/whitespace Markdown. The script will retry without CSS selection automatically, but you can also force it explicitly:

uv run scripts/crawl_to_markdown.py --selector '' https://example.com

uv cache permission errors

If uv fails accessing its cache directory (e.g. ~/.cache/uv), either point it at a writable cache directory:

uv --cache-dir .uv-cache run scripts/crawl_to_markdown.py https://example.com

Or avoid the cache entirely:

uv --no-cache run scripts/crawl_to_markdown.py https://example.com

Output format

For each URL, the script prints a header and raw Markdown content:

URL: https://example.com
<raw markdown>
---

If a crawl fails, the output includes an error block for that URL.