firecrawl-webhooks-events
Implement FireCrawl webhook signature validation and event handling.Use when setting up webhook endpoints, implementing signature verification,or handling FireCrawl event notifications securely.Trigger with phrases like "firecrawl webhook", "firecrawl events","firecrawl webhook signature", "handle firecrawl events", "firecrawl notifications".
When & Why to Use This Skill
This Claude skill enables developers to securely implement and manage FireCrawl webhook endpoints. It provides a comprehensive framework for HMAC SHA256 signature validation, replay attack protection, and idempotent event handling. By automating the setup of secure listeners, it ensures that web crawling data is integrated into your application with enterprise-grade security and reliability, specifically optimized for Express.js and Node.js environments.
Use Cases
- Secure Data Synchronization: Automatically update your internal database or search index when a FireCrawl scraping job finishes, using signature verification to ensure the data is authentic.
- Real-time Workflow Triggers: Initiate downstream processes, such as AI summarization or lead scoring, immediately upon receiving 'resource.created' or 'resource.updated' events from FireCrawl.
- Replay Attack Prevention: Protect your infrastructure by implementing timestamp-based validation to reject old or intercepted webhook notifications.
- Idempotent Event Processing: Use the included Redis-based patterns to prevent duplicate processing of the same crawl event, ensuring system stability during high-volume operations.
- Local Development & Debugging: Quickly set up a local testing environment using ngrok and the FireCrawl CLI to simulate and validate webhook payloads before production deployment.
| name | firecrawl-webhooks-events |
|---|---|
| description | | |
| allowed-tools | Read, Write, Edit, Bash(curl:*) |
| version | 1.0.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
FireCrawl Webhooks & Events
Overview
Securely handle FireCrawl webhooks with signature validation and replay protection.
Prerequisites
- FireCrawl webhook secret configured
- HTTPS endpoint accessible from internet
- Understanding of cryptographic signatures
- Redis or database for idempotency (optional)
Webhook Endpoint Setup
Express.js
import express from 'express';
import crypto from 'crypto';
const app = express();
// IMPORTANT: Raw body needed for signature verification
app.post('/webhooks/firecrawl',
express.raw({ type: 'application/json' }),
async (req, res) => {
const signature = req.headers['x-firecrawl-signature'] as string;
const timestamp = req.headers['x-firecrawl-timestamp'] as string;
if (!verifyFireCrawlSignature(req.body, signature, timestamp)) {
return res.status(401).json({ error: 'Invalid signature' });
}
const event = JSON.parse(req.body.toString());
await handleFireCrawlEvent(event);
res.status(200).json({ received: true });
}
);
Signature Verification
function verifyFireCrawlSignature(
payload: Buffer,
signature: string,
timestamp: string
): boolean {
const secret = process.env.FIRECRAWL_WEBHOOK_SECRET!;
// Reject old timestamps (replay attack protection)
const timestampAge = Date.now() - parseInt(timestamp) * 1000;
if (timestampAge > 300000) { // 5 minutes
console.error('Webhook timestamp too old');
return false;
}
// Compute expected signature
const signedPayload = `${timestamp}.${payload.toString()}`;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
// Timing-safe comparison
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
Event Handler Pattern
type FireCrawlEventType = 'resource.created' | 'resource.updated' | 'resource.deleted';
interface FireCrawlEvent {
id: string;
type: FireCrawlEventType;
data: Record<string, any>;
created: string;
}
const eventHandlers: Record<FireCrawlEventType, (data: any) => Promise<void>> = {
'resource.created': async (data) => { /* handle */ },
'resource.updated': async (data) => { /* handle */ },
'resource.deleted': async (data) => { /* handle */ }
};
async function handleFireCrawlEvent(event: FireCrawlEvent): Promise<void> {
const handler = eventHandlers[event.type];
if (!handler) {
console.log(`Unhandled event type: ${event.type}`);
return;
}
try {
await handler(event.data);
console.log(`Processed ${event.type}: ${event.id}`);
} catch (error) {
console.error(`Failed to process ${event.type}: ${event.id}`, error);
throw error; // Rethrow to trigger retry
}
}
Idempotency Handling
import { Redis } from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);
async function isEventProcessed(eventId: string): Promise<boolean> {
const key = `firecrawl:event:${eventId}`;
const exists = await redis.exists(key);
return exists === 1;
}
async function markEventProcessed(eventId: string): Promise<void> {
const key = `firecrawl:event:${eventId}`;
await redis.set(key, '1', 'EX', 86400 * 7); // 7 days TTL
}
Webhook Testing
# Use FireCrawl CLI to send test events
firecrawl webhooks trigger resource.created --url http://localhost:3000/webhooks/firecrawl
# Or use webhook.site for debugging
curl -X POST https://webhook.site/your-uuid \
-H "Content-Type: application/json" \
-d '{"type": "resource.created", "data": {}}'
Instructions
Step 1: Register Webhook Endpoint
Configure your webhook URL in the FireCrawl dashboard.
Step 2: Implement Signature Verification
Use the signature verification code to validate incoming webhooks.
Step 3: Handle Events
Implement handlers for each event type your application needs.
Step 4: Add Idempotency
Prevent duplicate processing with event ID tracking.
Output
- Secure webhook endpoint
- Signature validation enabled
- Event handlers implemented
- Replay attack protection active
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Invalid signature | Wrong secret | Verify webhook secret |
| Timestamp rejected | Clock drift | Check server time sync |
| Duplicate events | Missing idempotency | Implement event ID tracking |
| Handler timeout | Slow processing | Use async queue |
Examples
Testing Webhooks Locally
# Use ngrok to expose local server
ngrok http 3000
# Send test webhook
curl -X POST https://your-ngrok-url/webhooks/firecrawl \
-H "Content-Type: application/json" \
-d '{"type": "test", "data": {}}'
Resources
Next Steps
For performance optimization, see firecrawl-performance-tuning.