doc-scraper

Generic web scraper for extracting and organizing Snowflake documentation with intelligent caching and configurable spider depth. Scrapes any section of docs.snowflake.com controlled by --base-path.

⭐23stars🔀6forks📁View on GitHub🕐Updated Dec 28, 2025⬇️2downloads

When & Why to Use This Skill

The Snowflake Documentation Scraper is a specialized tool designed to automate the extraction and organization of Snowflake's technical documentation into structured Markdown format. It solves the problem of manual information gathering by providing a configurable, automated way to build local knowledge bases. With features like intelligent SQLite caching (7-day expiration), adjustable spider depth, and targeted path scraping, it enables developers to maintain up-to-date, offline-accessible documentation that is perfectly formatted for personal reference or as high-quality context for LLM-powered development workflows.

Use Cases

  • Offline Knowledge Base Creation: Automatically download and convert specific Snowflake documentation sections into Markdown for fast, offline searching and internal team wikis.
  • RAG and AI Context Augmentation: Scrape the latest SQL references or migration guides to provide structured, up-to-date technical context for Retrieval-Augmented Generation (RAG) systems and AI coding assistants.
  • Documentation Migration & Archiving: Efficiently extract content from docs.snowflake.com to transition technical guides into internal project management tools or version-controlled repositories.
  • Targeted Technical Research: Use the configurable spider depth and base-path filters to quickly gather all related sub-pages for a specific Snowflake feature, such as 'migrations' or 'sql-reference', without manual browsing.
namedoc-scraper

Snowflake Documentation Scraper

Scrapes docs.snowflake.com sections to Markdown with SQLite caching (7-day expiration).

Usage

First time setup (auto-installs uv and doc-scraper):

python3 .claude/skills/doc-scraper/scripts/doc_scraper.py

Subsequent runs:

doc-scraper --output-dir=./snowflake-docs
doc-scraper --output-dir=./snowflake-docs --base-path="/en/sql-reference/"
doc-scraper --output-dir=./snowflake-docs --spider-depth=2

Command Options

Option Default Description
--output-dir Required Output directory for scraped docs
--base-path /en/migrations/ URL section to scrape
--spider-depth 1 Link depth: 0=seeds, 1=+links, 2=+2nd
--limit None Cap URLs (for testing)
--dry-run - Preview without writing

Output

output-dir/
├── SKILL.md              # Auto-generated index
├── scraper_config.yaml   # Editable config (auto-created)
├── .cache/               # SQLite cache (auto-managed)
└── en/migrations/*.md    # Scraped pages with frontmatter

Configuration

Auto-created at {output-dir}/scraper_config.yaml:

rate_limiting:
  max_concurrent_threads: 4
spider:
  max_pages: 1000
  allowed_paths: ["/en/"]
scraped_pages:
  expiration_days: 7

Troubleshooting

Issue Solution
Too many pages Lower --spider-depth or edit config
Missing pages Increase --spider-depth
Cache corruption Delete {output-dir}/.cache/ (rare)