web-scraping
Extract structured data from web pages using CSS selectors and XPath
When & Why to Use This Skill
This Claude skill enables automated web scraping to extract structured data from any website using CSS selectors and XPath. It simplifies complex data collection by converting raw HTML into organized JSON or CSV formats, supporting advanced features like pagination and regex pattern matching for efficient information gathering.
Use Cases
- E-commerce Monitoring: Automatically track competitor product prices, descriptions, and stock status to inform dynamic pricing strategies.
- Market Research: Aggregate news headlines, blog posts, or social proof from various websites to identify emerging industry trends and consumer sentiment.
- Lead Generation: Systematically extract business contact information and company profiles from professional directories or industry-specific listings.
- Content Aggregation: Collect and format data from multiple web sources to build comprehensive databases for research or internal reporting.
| name | web-scraping |
|---|---|
| description | Extract structured data from web pages using CSS selectors and XPath |
Web Scraping
Extract structured data from web pages.
Capabilities
- Fetch HTML content from URLs
- Parse and extract specific elements (tables, lists, text)
- Handle pagination
- Output in JSON or CSV format
Supported Selectors
- CSS selectors:
.class,#id,tag - XPath expressions
- Text patterns (regex)
Rate Limiting
Always respect robots.txt and implement delays between requests. Default delay: 1 second between requests.
Example
Scrape product names and prices from example.com/products
Output as JSON with fields: name, price, url