context-scrapers

Web scraping spiders (Scrapy), parsers, and source management.

⭐1stars🔀0forks📁View on GitHub🕐Updated Jan 2, 2026⬇️1downloads

When & Why to Use This Skill

This Claude skill provides a comprehensive infrastructure for automated web scraping and data extraction using the Scrapy framework. It is specifically designed to handle complex data acquisition from municipal sources such as Legistar and Municode, facilitating efficient parsing, source management, and database integration for structured data harvesting.

Use Cases

  • Automating the retrieval of city council meeting records, agendas, and legislative data from Legistar portals.
  • Extracting and organizing municipal codes and legal documents from Municode for policy analysis or regulatory compliance research.
  • Developing and maintaining custom Scrapy spiders with integrated pipelines for seamless data storage and verification.
  • Debugging and optimizing web ingestion workflows to ensure reliable data collection from dynamic and complex web structures.
namecontext-scrapers
descriptionWeb scraping spiders (Scrapy), parsers, and source management.
tags[backend, data, scrapers]

Scrapers Context

Overview

Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).

Active Files

Spiders

  • backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py - Legistar meeting scraper
  • backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py - Municode scraper

Configuration

  • backend/affordabot_scraper/affordabot_scraper/pipelines.py - Item pipelines (DB storage)
  • backend/affordabot_scraper/affordabot_scraper/settings.py - Scrapy settings

Verification Scripts

  • scripts/verify_raw_scrapes.py - Check raw_scrapes table
  • scripts/verify_municode_discovery.py - Verify discovery logic

Usage

Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.