firecrawl-api

s-hiraoku's avatarfrom s-hiraoku

This skill enables web scraping and content extraction using Firecrawl API directly via curl. Use when scraping web pages, crawling websites, or extracting structured data. MCP server not required.

⭐6stars🔀4forks📁View on GitHub🕐Updated Jan 3, 2026
[Web Automation]

When & Why to Use This Skill

This Claude skill integrates the Firecrawl API to provide advanced web scraping, website crawling, and structured content extraction capabilities directly via curl. It excels at converting complex web pages into LLM-ready markdown, mapping entire website architectures, and performing targeted web searches, making it a powerful tool for automated data gathering without the need for an MCP server.

Use Cases

  • Documentation Harvesting: Automatically crawl and convert entire technical documentation sites into markdown format to build local knowledge bases or RAG (Retrieval-Augmented Generation) systems.
  • Market Intelligence & Monitoring: Scrape competitor websites to extract structured data on product pricing, features, and updates for automated market analysis.
  • SEO & Site Mapping: Use the mapping endpoint to discover all URLs within a domain and extract metadata to perform comprehensive content audits or identify indexing issues.
  • Enhanced Web Research: Perform real-time web searches and immediately scrape the main content of the top results to provide grounded, up-to-date answers to complex user queries.
namefirecrawl-api
descriptionThis skill enables web scraping and content extraction using Firecrawl API directly via curl. Use when scraping web pages, crawling websites, or extracting structured data. MCP server not required.

Firecrawl API (Direct)

Overview

Firecrawl provides powerful web scraping and content extraction. This skill uses the API directly via curl - no MCP server required.

API Key

Environment variable: FIRECRAWL_API_KEY

Scrape Single Page

Extract content from a single URL.

Endpoint

POST https://api.firecrawl.dev/v1/scrape

Usage

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"]
  }'

Parameters

  • url (required): URL to scrape
  • formats (optional): Output formats - markdown, html, rawHtml, links
  • onlyMainContent (optional): Extract main content only (default: true)
  • waitFor (optional): Wait time in ms for dynamic content

Example with Options

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com/api",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 1000
  }' | jq '.data.markdown'

Map Website URLs

Discover all URLs on a website.

Endpoint

POST https://api.firecrawl.dev/v1/map

Usage

curl -s -X POST "https://api.firecrawl.dev/v1/map" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }' | jq '.links[]'

Search Web

Search the web and optionally scrape results.

Endpoint

POST https://api.firecrawl.dev/v1/search

Usage

curl -s -X POST "https://api.firecrawl.dev/v1/search" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "xterm.js WebGL performance",
    "limit": 5
  }'

Crawl Website

Crawl multiple pages from a website.

Endpoint

POST https://api.firecrawl.dev/v1/crawl

Usage

curl -s -X POST "https://api.firecrawl.dev/v1/crawl" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 10,
    "maxDepth": 2
  }'

This returns a job ID. Check status with:

curl -s "https://api.firecrawl.dev/v1/crawl/JOB_ID" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"

Common Workflows

Scrape Documentation

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://xtermjs.org/docs/",
    "formats": ["markdown"]
  }' | jq '.data.markdown'

Extract Links from Page

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["links"]
  }' | jq '.data.links[]'

Response Processing

# Get markdown content
| jq '.data.markdown'

# Get metadata
| jq '.data.metadata'

# Get title and description
| jq '.data.metadata | {title, description}'

Notes

  • API key required (set FIRECRAWL_API_KEY)
  • Credits consumed per request
  • Use waitFor for JavaScript-heavy pages
  • Crawl jobs are async - poll for completion