web-resource-checker

naporin0624's avatarfrom naporin0624

Validates essential web resource files (sitemap.xml, robots.txt, llms.txt, security.txt) for compliance with their specifications. Use when user asks about "sitemap validation", "robots.txt check", "llms.txt", "security.txt", "RFC 9116", "RFC 9309", "web resource audit", "サイトマップ", "セキュリティ", or wants to verify crawler/LLM accessibility files.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 26, 2025
[SEO and Content Briefs]

When & Why to Use This Skill

The Web Resource Checker is a specialized tool for validating and auditing essential website configuration files such as sitemap.xml, robots.txt, llms.txt, and security.txt. It helps web developers and SEO specialists ensure full compliance with industry standards like RFC 9116 and RFC 9309, optimizing site discoverability for search engines and accessibility for AI models.

Use Cases

  • SEO Optimization: Validate sitemap.xml and robots.txt to ensure search engine crawlers can efficiently index your site while respecting access restrictions.
  • AI & LLM Accessibility: Verify llms.txt and llms-full.txt files to provide optimized, structured content for AI agents and Large Language Models.
  • Security Compliance: Audit security.txt files for RFC 9116 compliance to establish a professional channel for vulnerability reporting.
  • Technical Web Audit: Check for the presence and correctness of ads.txt and humans.txt to maintain industry transparency and team attribution standards.
nameweb-resource-checker
descriptionValidates essential web resource files (sitemap.xml, robots.txt, llms.txt, security.txt) for compliance with their specifications. Use when user asks about "sitemap validation", "robots.txt check", "llms.txt", "security.txt", "RFC 9116", "RFC 9309", "web resource audit", "サイトマップ", "セキュリティ", or wants to verify crawler/LLM accessibility files.

Web Resource Checker

Validates essential web resource files for SEO, security, and LLM accessibility compliance.

Supported Files

File Purpose Specification
sitemap.xml URL listing for crawlers sitemaps.org
robots.txt Crawler access control RFC 9309
llms.txt LLM site overview llmstxt.org
llms-full.txt Complete LLM docs llmstxt.org
security.txt Vulnerability disclosure RFC 9116
humans.txt Team credits humanstxt.org
ads.txt Ad transparency IAB Tech Lab

Lookup Workflow

  1. Identify the query type:

    • File specification lookup (e.g., "sitemap.xml format", "security.txt fields")
    • Validation request (e.g., "check my sitemap", "validate robots.txt")

  2. For specification lookup:

  3. For validation:

    • Run the appropriate validator script
    • Return issues found with severity and fix suggestions

Usage

Validate All Resources

# Check local directory
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker ./public

# Check live site
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com

# JSON output
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com --json

# Check specific files only
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com --only=sitemap,robots

# With custom timeout (in milliseconds)
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com --timeout=30000

Lookup Specifications

# Get sitemap.xml specification
cat ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker/web-resources-index.json | jq '.files["sitemap.xml"]'

# Get security.txt required fields
cat ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker/web-resources-index.json | jq '.files["security.txt"].requiredFields'

# Get llms.txt example
cat ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker/web-resources-index.json | jq -r '.files["llms.txt"].example'

Response Format

For Specification Lookup

### [File Name]

**Summary**: [1-2 sentence explanation]

**Required Elements**:
- [Element 1]
- [Element 2]

**Common Issues**:
- [Issue description]

**Official Specification**: [URL]

For Validation Results

### [File Name] - [Status]

**Issues Found**: [count]

1. **[Issue Title]** ([severity])
   - Problem: [description]
   - Fix: [suggestion]

2. ...

Quick Reference

sitemap.xml

  • Location: /sitemap.xml
  • Max URLs: 50,000 per file
  • Max Size: 50MB uncompressed
  • Required: <urlset>, <url>, <loc>
  • Recommended: <lastmod>, <changefreq>, <priority>

robots.txt

  • Location: /robots.txt (site root only)
  • Common Directives: User-agent, Disallow, Allow, Sitemap
  • Case: Directives are case-insensitive
  • Tip: Always include Sitemap: directive

llms.txt (llmstxt.org)

  • Location: /llms.txt
  • Format: Markdown
  • Required: H1 title (# Site Name)
  • Recommended: Blockquote summary, H2 sections with links

llms-full.txt

  • Location: /llms-full.txt
  • Purpose: Complete documentation for LLM context
  • Size: Should fit in ~100k tokens

security.txt (RFC 9116)

  • Location: /.well-known/security.txt (preferred) or /security.txt
  • Required Fields: Contact, Expires
  • Expires Format: ISO 8601 (e.g., 2025-12-31T23:59:59.000Z)
  • HTTPS: Must be served over HTTPS

Common Patterns

Complete sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2024-01-15</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2024-01-10</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

Complete robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Complete llms.txt

# Example Site

> A comprehensive platform for developers to learn and build modern applications.

## Documentation

- [Getting Started](https://example.com/docs/start): Quick start guide for new users
- [API Reference](https://example.com/docs/api): Complete API documentation
- [Tutorials](https://example.com/docs/tutorials): Step-by-step guides

## Optional

- [Blog](https://example.com/blog): Latest updates and articles
- [Changelog](https://example.com/changelog): Version history

Complete security.txt

Contact: mailto:security@example.com
Contact: https://example.com/security-contact
Expires: 2025-12-31T23:59:59.000Z
Encryption: https://example.com/.well-known/pgp-key.txt
Acknowledgments: https://example.com/security/thanks
Preferred-Languages: en, ja
Canonical: https://example.com/.well-known/security.txt
Policy: https://example.com/security-policy
Hiring: https://example.com/careers/security

External Resources