context-scrapers
Web scraping spiders (Scrapy), parsers, and source management.
When & Why to Use This Skill
This Claude skill provides a comprehensive infrastructure for automated web scraping and data extraction using the Scrapy framework. It is specifically designed to handle complex data acquisition from municipal sources such as Legistar and Municode, facilitating efficient parsing, source management, and database integration for structured data harvesting.
Use Cases
- Automating the retrieval of city council meeting records, agendas, and legislative data from Legistar portals.
- Extracting and organizing municipal codes and legal documents from Municode for policy analysis or regulatory compliance research.
- Developing and maintaining custom Scrapy spiders with integrated pipelines for seamless data storage and verification.
- Debugging and optimizing web ingestion workflows to ensure reliable data collection from dynamic and complex web structures.
| name | context-scrapers |
|---|---|
| description | Web scraping spiders (Scrapy), parsers, and source management. |
| tags | [backend, data, scrapers] |
Scrapers Context
Overview
Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).
Active Files
Spiders
backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py- Legistar meeting scraperbackend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py- Municode scraper
Configuration
backend/affordabot_scraper/affordabot_scraper/pipelines.py- Item pipelines (DB storage)backend/affordabot_scraper/affordabot_scraper/settings.py- Scrapy settings
Verification Scripts
scripts/verify_raw_scrapes.py- Check raw_scrapes tablescripts/verify_municode_discovery.py- Verify discovery logic
Usage
Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.