context-scrapers

stars-end 的头像作者 stars-end

网页爬虫(Scrapy)、解析器和源代码管理。

📖 中文导读:网页爬虫(Scrapy)、解析器和源代码管理。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: context-scrapers description: Web scraping spiders (Scrapy), parsers, and source management. tags: [backend, data, scrapers]

Scrapers Context

Overview

Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).

Active Files

Spiders

  • backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py - Legistar meeting scraper
  • backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py - Municode scraper

Configuration

  • backend/affordabot_scraper/affordabot_scraper/pipelines.py - Item pipelines (DB storage)
  • backend/affordabot_scraper/affordabot_scraper/settings.py - Scrapy settings

Verification Scripts

  • scripts/verify_raw_scrapes.py - Check raw_scrapes table
  • scripts/verify_municode_discovery.py - Verify discovery logic

Usage

Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.