fetch-url
渲染网页 URL,去噪提取正文并输出为 Markdown(默认)或其他格式/原始 HTML,以减少 Token。
When & Why to Use This Skill
The fetch-url skill is a high-performance web content extraction tool designed to optimize LLM workflows. By utilizing Playwright to render dynamic JavaScript-heavy websites and Trafilatura for denoising, it extracts the core text and converts it into clean Markdown, HTML, or JSON. This process significantly reduces token usage and eliminates 'noise' like ads and navigation menus, ensuring that Claude receives only the most relevant information for processing.
Use Cases
- Web Research & Summarization: Automatically fetch and convert long-form articles or documentation into Markdown for efficient reading and analysis by the AI.
- Token-Efficient Data Extraction: Reduce input costs and context window pressure by stripping away unnecessary HTML tags and boilerplate code before feeding data to the model.
- Automated Content Monitoring: Scrape specific web pages periodically to track updates, news, or changes in a structured format like CSV or JSON.
- Cross-Format Web Conversion: Seamlessly transform live web pages into various formats (XML, TXT, XML-TEI) for integration into different data pipelines or knowledge bases.
| name | fetch-url |
|---|---|
| description | 渲染网页 URL,去噪提取正文并输出为 Markdown(默认)或其他格式/原始 HTML,以减少 Token。 |
在当前文件所在目录运行:./scripts/fetch_url.py URL(仅支持 http / https)。
说明:必须直接当作可执行文件执行。
默认自动探测本地 Chromium 系浏览器路径;未探测到时需安装 Playwright 浏览器:
uv run playwright install chromium
参数:
--output:将输出写入文件(默认 stdout)。--timeout-ms:Playwright 导航超时(毫秒,默认 60000)。--browser-path:指定本地 Chromium 系浏览器路径(默认自动探测)。--output-format:输出格式(默认markdown),支持csv、html、json、markdown、raw-html、txt、xml、xmltei;raw-html直接输出渲染后的 HTML(不经 trafilatura)。
示例:
./scripts/fetch_url.py https://example.com --output ./page.md --timeout-ms 60000
Reference:scripts/fetch_url.py