context-scrapers
网页爬虫(Scrapy)、解析器和源代码管理。
📖 中文导读:网页爬虫(Scrapy)、解析器和源代码管理。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。
name: context-scrapers description: Web scraping spiders (Scrapy), parsers, and source management. tags: [backend, data, scrapers]
Scrapers Context
Overview
Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).
Active Files
Spiders
backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py- Legistar meeting scraperbackend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py- Municode scraper
Configuration
backend/affordabot_scraper/affordabot_scraper/pipelines.py- Item pipelines (DB storage)backend/affordabot_scraper/affordabot_scraper/settings.py- Scrapy settings
Verification Scripts
scripts/verify_raw_scrapes.py- Check raw_scrapes tablescripts/verify_municode_discovery.py- Verify discovery logic
Usage
Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.