apify-scraper-builder
Build Apify Actors (web scrapers) using Node.js and Crawlee. Use when creating new scrapers, defining input schemas, configuring Dockerfiles, or deploying to Apify. Triggers include apify, actor, scraper, crawlee, web scraping, data extraction.
When & Why to Use This Skill
The Apify Scraper Builder skill streamlines the development of production-ready web scrapers (Actors) on the Apify platform. It provides expert guidance on using Node.js and the Crawlee library to build, validate, and deploy scrapers, offering a structured workflow for selecting the optimal crawler type—Cheerio for speed, or Playwright/Puppeteer for JavaScript-heavy sites—to ensure efficient and scalable data extraction.
Use Cases
- E-commerce Intelligence: Automate the collection of product prices, stock levels, and customer reviews from retail websites for competitive benchmarking.
- Dynamic Web Scraping: Build scrapers that interact with modern JavaScript frameworks (React, Vue, etc.) to extract data from single-page applications using Playwright or Puppeteer.
- Market Research & Lead Gen: Create custom Actors to aggregate business information, social media trends, or news data into structured datasets for analysis.
- Content Aggregation: Develop automated pipelines to pull and normalize content from multiple web sources for use in CMS or AI training models.
| name | apify-scraper-builder |
|---|---|
| description | Build Apify Actors (web scrapers) using Node.js and Crawlee. Use when creating new scrapers, defining input schemas, configuring Dockerfiles, or deploying to Apify. Triggers include apify, actor, scraper, crawlee, web scraping, data extraction. |
Apify Scraper Builder
Build production-ready Apify Actors using Node.js/TypeScript and Crawlee.
Crawler Type Decision Tree
| Scenario | Crawler | Why |
|---|---|---|
| Static HTML, no JavaScript | CheerioCrawler | Fastest, lowest memory |
| JavaScript-rendered content | PlaywrightCrawler | Modern, cross-browser |
| Legacy sites, specific Chrome behavior | PuppeteerCrawler | Chrome-specific features |
| Need to handle both static and JS | PlaywrightCrawler | More versatile |
| High-volume scraping (1000s pages) | CheerioCrawler | Best performance |
Actor Creation Workflow
Step 1: Initialize Project
python scripts/init_actor.py my-scraper --type cheerio
Or manually create structure:
my-scraper/
├── .actor/
│ ├── actor.json # REQUIRED
│ ├── input_schema.json # Recommended
│ └── Dockerfile # REQUIRED
├── src/
│ └── main.ts # Entry point
├── package.json
└── tsconfig.json
Step 2: Configure actor.json
{
"actorSpecification": 1,
"name": "my-scraper",
"version": "0.0",
"buildTag": "latest",
"input": "./input_schema.json",
"dockerfile": "./Dockerfile"
}
Step 3: Define Input Schema
python scripts/generate_input_schema.py "Scrape product pages with URLs, max items limit, and proxy support"
Or use templates from references/input-schema-guide.md
Step 4: Implement Crawler
Use patterns from references/crawlee-patterns.md
Step 5: Validate Configuration
python scripts/validate_actor.py /path/to/actor
Step 6: Deploy
apify login
apify push
Project Structure
Required Files
.actor/actor.json
{
"actorSpecification": 1,
"name": "my-scraper",
"version": "0.0",
"buildTag": "latest",
"minMemoryMbytes": 256,
"maxMemoryMbytes": 4096,
"dockerfile": "./Dockerfile",
"input": "./input_schema.json",
"storages": {
"dataset": "./dataset_schema.json"
}
}
.actor/Dockerfile (Node.js)
FROM apify/actor-node:20
COPY package*.json ./
RUN npm --quiet set progress=false \
&& npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
&& npm list || true \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
COPY . ./
CMD npm start
package.json
{
"name": "my-scraper",
"version": "0.0.1",
"type": "module",
"main": "dist/main.js",
"scripts": {
"start": "node dist/main.js",
"build": "tsc"
},
"dependencies": {
"apify": "^3.0.0",
"crawlee": "^3.0.0"
},
"devDependencies": {
"typescript": "^5.0.0"
}
}
Input Schema Editors
| Editor | Use Case | Example |
|---|---|---|
textfield |
Single-line text | Name, URL |
textarea |
Multi-line text | CSS selectors, notes |
requestListSources |
URL list with labels | Start URLs |
proxy |
Proxy configuration | Apify Proxy settings |
json |
JSON object/array | Custom configuration |
select |
Dropdown options | Country, category |
checkbox |
Boolean toggle | Debug mode |
number |
Integer/float | Max items, delay |
datepicker |
Date selection | Date range filter |
Common Input Schema Pattern
{
"title": "Scraper Input",
"type": "object",
"schemaVersion": 1,
"properties": {
"startUrls": {
"title": "Start URLs",
"type": "array",
"description": "URLs to start scraping from",
"editor": "requestListSources",
"prefill": [{"url": "https://example.com"}]
},
"maxItems": {
"title": "Max Items",
"type": "integer",
"description": "Maximum number of items to scrape",
"default": 100,
"minimum": 1
},
"proxyConfig": {
"title": "Proxy Configuration",
"type": "object",
"description": "Proxy settings for the scraper",
"editor": "proxy",
"default": {"useApifyProxy": true}
}
},
"required": ["startUrls"]
}
Crawlee Patterns
CheerioCrawler (Fast HTML Parsing)
import { Actor } from 'apify';
import { CheerioCrawler, Dataset } from 'crawlee';
await Actor.init();
const input = await Actor.getInput<{
startUrls: { url: string }[];
maxItems: number;
}>();
const crawler = new CheerioCrawler({
maxRequestsPerCrawl: input?.maxItems || 100,
async requestHandler({ request, $, enqueueLinks }) {
const title = $('h1').text().trim();
const price = $('.price').text().trim();
await Dataset.pushData({
url: request.url,
title,
price,
});
// Enqueue pagination links
await enqueueLinks({
selector: 'a.next-page',
});
},
});
await crawler.run(input?.startUrls?.map(u => u.url) || []);
await Actor.exit();
PlaywrightCrawler (JavaScript Rendering)
import { Actor } from 'apify';
import { PlaywrightCrawler, Dataset } from 'crawlee';
await Actor.init();
const input = await Actor.getInput<{
startUrls: { url: string }[];
maxItems: number;
}>();
const proxyConfiguration = await Actor.createProxyConfiguration(
input?.proxyConfig
);
const crawler = new PlaywrightCrawler({
proxyConfiguration,
maxRequestsPerCrawl: input?.maxItems || 100,
async requestHandler({ page, request, enqueueLinks }) {
// Wait for dynamic content
await page.waitForSelector('.product-list');
const products = await page.$$eval('.product', items =>
items.map(item => ({
title: item.querySelector('h2')?.textContent?.trim(),
price: item.querySelector('.price')?.textContent?.trim(),
}))
);
for (const product of products) {
await Dataset.pushData({
url: request.url,
...product,
});
}
await enqueueLinks({
selector: 'a.pagination',
});
},
});
await crawler.run(input?.startUrls?.map(u => u.url) || []);
await Actor.exit();
PuppeteerCrawler (Chrome-specific)
import { Actor } from 'apify';
import { PuppeteerCrawler, Dataset } from 'crawlee';
await Actor.init();
const input = await Actor.getInput<{
startUrls: { url: string }[];
}>();
const crawler = new PuppeteerCrawler({
launchContext: {
launchOptions: {
headless: true,
},
},
async requestHandler({ page, request }) {
await page.waitForSelector('.content');
const data = await page.evaluate(() => ({
title: document.querySelector('h1')?.textContent,
content: document.querySelector('.content')?.innerHTML,
}));
await Dataset.pushData({
url: request.url,
...data,
});
},
});
await crawler.run(input?.startUrls?.map(u => u.url) || []);
await Actor.exit();
Scripts
Initialize New Actor
python scripts/init_actor.py <name> --type <cheerio|playwright|puppeteer> [--path <dir>]
Validate Actor Configuration
python scripts/validate_actor.py <actor-path>
Generate Input Schema
python scripts/generate_input_schema.py "<description>" [--output <path>]
Deployment Commands
# Install Apify CLI
npm install -g @apify/cli
# Login to Apify
apify login
# Create new Actor from template (interactive)
apify create my-actor
# Run Actor locally
apify run --purge
# Push to Apify platform
apify push
# Build Actor remotely
apify actors build
# Call Actor remotely
apify actors call <actor-id>
# Pull Actor code from Apify
apify actors pull <actor-id>
Validation Checklist
Before Building
- Correct crawler type selected for target site
- Input schema defines all required parameters
- Dependencies in package.json are correct
Configuration
- actor.json has actorSpecification: 1
- actor.json has valid name and version
- Dockerfile uses correct Node.js base image
- Input schema editors match field types
Code Quality
- Error handling for network failures
- Proxy configuration used for production
- Rate limiting/delays configured
- Data validation before pushData
Pre-Deployment
-
apify run --purgesucceeds locally - Output data structure is correct
- Memory limits are appropriate
References
| Topic | File |
|---|---|
| actor.json Specification | references/actor-json-spec.md |
| Input Schema Editors | references/input-schema-guide.md |
| Crawlee Patterns | references/crawlee-patterns.md |
Templates
| Template | Description | Path |
|---|---|---|
| Cheerio | Fast HTML scraping | templates/crawlee-cheerio/ |
| Playwright | JS-rendered content | templates/crawlee-playwright/ |
| Puppeteer | Chrome-specific | templates/crawlee-puppeteer/ |