crawl4ai

tao3k 的头像作者 tao3k

使用 Crawl4AI 进行网络爬行和数据提取的完整工具包。当用户需要抓取网站、提取结构化数据、处理 JavaScript 密集型页面、爬行多个 URL 或构建自动化 Web 数据管道时,应该使用此技能。包括优化的提取模式和模式生成,以实现高效、无法学硕士的提取。

📖 中文导读:使用 Crawl4AI 进行网络爬行和数据提取的完整工具包。当用户需要抓取网站、提取结构化数据、处理 JavaScript 密集型页面、爬行多个 URL 或构建自动化 Web 数据管道时,应该使用此技能。包括优化的提取模式和模式生成,以实现高效、无法学硕士的提取。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: crawl4ai version: 0.1.0 description: High-performance web crawler skill using Sidecar Execution Pattern author: Omni Team routing_keywords: [crawl, scrape, web, fetch, scrape] execution_mode: subprocess intents: - "Crawl a webpage and extract its content" - "Fetch website content as markdown" - "Scrape web pages for information"

Crawl4ai Skill

High-performance web crawler using the Sidecar Execution Pattern for dependency isolation.

Architecture

This skill demonstrates Sidecar Execution Pattern:

  • scripts/__init__.py: Lightweight interface (loaded by main agent via @skill_command)
  • scripts/engine.py: Actual crawler implementation (runs in isolated uv environment)
  • pyproject.toml: Skill-specific dependencies (crawl4ai, fire, pydantic)

The main agent never imports crawl4ai directly. Instead, it uses common.isolation.run_skill_command() to execute the crawler in a subprocess with its own isolated environment.

Commands

crawl_webpage

Crawl a webpage and extract its content as markdown.

@omni("skill.run crawl4ai.crawl_webpage url='https://example.com'")

Parameters:

  • url (str, required): Target URL to crawl
  • fit_markdown (bool, default: True): Clean and simplify the markdown

Returns:

{
    "success": True,
    "url": "https://example.com",
    "markdown": "# Example\n\nContent...",
    "metadata": {"title": "Example", "description": "..."},
    "error": None
}

check_crawler_ready

Check if the crawler skill is properly configured.

@omni("skill.run crawl4ai.check_crawler_ready")

Setup

Dependencies are automatically managed by uv:

cd assets/skills/crawl4ai
uv sync  # Installs crawl4ai and its dependencies

Usage Example

# In conversation with Claude
Please crawl https://github.com and give me the main content as markdown.

# Claude will invoke:
# @omni("skill.run crawl4ai.crawl_webpage url='https://github.com'")

Why Sidecar Pattern?

  1. Zero Pollution: Main agent doesn't need crawl4ai, playwright, or other heavy deps
  2. Version Isolation: Each skill can use different versions of the same library
  3. Hot Swappable: Skills can be added/removed without restarting the agent
  4. Security: Compromised skill code has limited blast radius