← Back to Skills Marketplace
Web Scraper
by
Guilherme Favaron
· GitHub ↗
· v0.1.1
· MIT-0
11938
Downloads
3
Stars
0
Active Installs
2
Versions
Install in OpenClaw
/install web-scraper
Description
Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and...
Usage Guidance
Install only if you are comfortable supervising its use: avoid using it to reveal or collect paywalled content, do not enable the OpenRouter entity-extraction stage for private/authenticated/sensitive pages without explicit approval, and prefer pinned dependencies in an isolated environment.
Capability Analysis
Type: OpenClaw Skill
Name: web-scraper
Version: 0.1.1
The 'web-scraper' skill is a well-structured and documented tool for multi-stage web content extraction and analysis. It follows industry best practices such as rate limiting, robots.txt compliance, and a cascade approach to resource usage, while explicitly instructing the agent to avoid sensitive files like .env and to handle API keys securely via environment variables (SKILL.md, claw.json).
Capability Assessment
Credentials
Filesystem and network permissions are proportionate for creating scraper scripts, fetching pages, and writing outputs; the OpenRouter stage is disclosed and optional, but scraped content may leave the local environment when enabled.
Install Mechanism
The package is a Markdown skill with no bundled executable scripts, but it recommends unpinned pip packages and Playwright browser installation, which is normal for this tooling but not reproducible.
Instruction Scope
The skill has useful planning, rate-limit, robots.txt, and credential-scope rules, but its paywall instructions are internally under-scoped: it forbids hard-paywall bypass while permitting DOM manipulation for soft paywalls.
Persistence & Privilege
No evidence of background services, scheduled tasks, tool hijacking, credential-file reads, persistent memory writes, or privilege escalation was found; checkpoint/output persistence is ordinary scraper output.
Purpose & Capability
The scraping, parsing, Playwright rendering, metadata extraction, local output files, and optional LLM entity extraction generally match the stated web-scraper purpose, but the soft-paywall guidance explicitly tells the agent to remove overlays and reveal hidden paragraphs, which goes beyond ordinary content extraction.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install web-scraper - After installation, invoke the skill by name or use
/web-scraper - Provide required inputs per the skill's parameter spec and get structured output
Version History
v0.1.1
web-scraper 0.1.1
- Added a CHANGELOG.md file for better tracking of updates.
- Clarified credential handling: the skill itself never makes direct API calls, and only generated scripts reference `OPENROUTER_API_KEY` if LLM extraction is used.
- Updated environment survey instructions to check for presence of `OPENROUTER_API_KEY` (for generated code), not the value.
- Improved documentation for credential scope and clarified planning protocol steps.
v0.1.0
web-scraper 0.1.0 — Initial release
- Introduces a web scraping and content comprehension agent with multi-strategy extraction and cascade fallback.
- Implements a mandatory planning protocol for all scraping requests, emphasizing environment and target analysis before action.
- Features a 5-stage pipeline: News/Article detection, multi-strategy extraction, cleaning/normalization, structured metadata extraction, and optional entity extraction using LLMs.
- Ensures safe credential handling by referencing `OPENROUTER_API_KEY` only in template code.
- Outputs structured JSON files with detailed content and quality metadata.
Metadata
Frequently Asked Questions
What is Web Scraper?
Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and... It is an AI Agent Skill for Claude Code / OpenClaw, with 11938 downloads so far.
How do I install Web Scraper?
Run "/install web-scraper" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Web Scraper free?
Yes, Web Scraper is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Web Scraper support?
Web Scraper is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Web Scraper?
It is built and maintained by Guilherme Favaron (@guifav); the current version is v0.1.1.
More Skills