web-cli

forest's avatarfrom forest

Portable web scraper CLI for LLMs with JavaScript rendering, form submission, and session persistence.USE WHEN user needs to scrape a URL, fill forms, take screenshots, or interact with web pagesrequiring JavaScript execution. Supports Phoenix LiveView applications natively.

⭐4stars🔀0forks📁View on GitHub🕐Updated Jan 3, 2026⬇️1downloads

When & Why to Use This Skill

The Web CLI skill is a powerful, portable web scraping and automation tool designed for LLMs to interact with modern web environments. It excels at handling dynamic content through JavaScript rendering, managing complex form submissions, and maintaining session persistence across requests. By supporting advanced features like Phoenix LiveView integration and screenshot capture, it enables AI agents to navigate, extract data from, and interact with sophisticated Single Page Applications (SPAs) and authenticated web portals that traditional static scrapers cannot reach.

Use Cases

  • Dynamic Data Extraction: Scraping content from modern web frameworks like React, Vue, or Angular where JavaScript execution is required to render the page.
  • Automated Form Submission: Filling out and submitting web forms, including login sequences and multi-step data entry workflows.
  • Session-Based Research: Maintaining authenticated states using session profiles to access and monitor private dashboards or protected user data.
  • Visual Monitoring: Capturing screenshots of specific web page states after JavaScript interactions to verify UI changes or archive visual data.
  • LiveView Interaction: Seamlessly interacting with Phoenix LiveView applications, handling real-time state updates and form loading states natively.
  • Custom Web Interactions: Executing specific JavaScript snippets to trigger button clicks, scroll events, or UI transitions before extracting data.
nameweb-cli
description|

Web CLI Skill

When to Activate This Skill

Direct Web Scraping

  • "scrape this URL with JavaScript"
  • "fetch this page and render it"
  • "get the page after JavaScript loads"
  • "convert this page to markdown"

Form Submission

  • "fill out this form"
  • "submit login credentials to this page"
  • "enter values into form fields"
  • "automate form submission"

Screenshot Capture

  • "take a screenshot of this page"
  • "capture this webpage as an image"
  • "screenshot after JavaScript loads"

Session/Authentication

  • "use my logged-in session"
  • "maintain session across requests"
  • "use a specific browser profile"

LiveView/SPA Support

  • "scrape this Phoenix LiveView app"
  • "interact with this dynamic page"
  • "wait for JavaScript to load"

Tool: web

web <url> [options]

Core Options

Option Description
--raw Output raw HTML instead of markdown
--truncate-after <n> Truncate after n characters (default: 100000)
--screenshot <path> Save screenshot to filepath
--profile <name> Use named session profile (default: "default")
--js <code> Execute JavaScript after page loads

Form Submission Options

Option Description
--form <id> Target form by ID
--input <name> Field name to fill (repeatable)
--value <value> Value for preceding --input
--after-submit <url> URL to load after form submission

Key Features

JavaScript Rendering

Unlike simple HTTP fetches, web renders JavaScript. Essential for:

  • Single Page Applications (SPAs)
  • Phoenix LiveView pages
  • React/Vue/Angular apps
  • Pages with dynamic content loading

Phoenix LiveView Support

Automatically handles:

  • .phx-connected state waiting
  • Form submissions with loading states
  • State management between interactions

Session Persistence

The --profile option maintains cookies and session state across requests:

# Login once
web localhost:4000/login --form login_form \
  --input email --value user@example.com \
  --input password --value secret \
  --profile myapp

# Subsequent requests use same session
web localhost:4000/dashboard --profile myapp

Usage Examples

Basic Page Fetch

web https://example.com

Screenshot with Truncated Output

web https://example.com --screenshot page.png --truncate-after 5000

Form Login

web localhost:4000/login \
  --form login_form \
  --input email --value test@example.com \
  --input password --value secret

Execute JavaScript

web https://example.com --js "document.querySelector('button').click()"
web localhost:4000/login \
  --form login_form \
  --input email --value admin@test.com \
  --input password --value password123 \
  --after-submit localhost:4000/admin/dashboard

When to Use web vs Other Tools

Scenario Tool
Simple static page WebFetch
JavaScript-rendered content web
Form submission needed web
Screenshot required web
Session persistence needed web
Bot detection bypass needed brightdata skill
Phoenix LiveView app web

Integration with Other Skills

  • brightdata: If web fails due to bot detection, escalate to brightdata skill
  • WebFetch: Use for simple static pages (faster, no JS overhead)

Common Patterns

Authenticated Scraping Session

# 1. Login and establish session
web localhost:4000/login \
  --form login_form \
  --input email --value user@example.com \
  --input password --value secret \
  --profile app_session

# 2. Access protected pages
web localhost:4000/protected/data --profile app_session

Capture State After Interaction

# Click a button and capture result
web https://app.example.com \
  --js "document.querySelector('#load-more').click(); await new Promise(r => setTimeout(r, 2000));" \
  --screenshot after-click.png

LiveView Form Submission

# Phoenix LiveView forms work seamlessly
web localhost:4000/users/new \
  --form user_form \
  --input "user[name]" --value "John Doe" \
  --input "user[email]" --value "john@example.com"

Last Updated: 2025-01-02