← 返回 Skills 市场
Web Scraper
作者
Guilherme Favaron
· GitHub ↗
· v0.1.1
· MIT-0
11938
总下载
3
收藏
0
当前安装
2
版本数
在 OpenClaw 中安装
/install web-scraper
功能描述
Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and...
安全使用建议
Install only if you are comfortable supervising its use: avoid using it to reveal or collect paywalled content, do not enable the OpenRouter entity-extraction stage for private/authenticated/sensitive pages without explicit approval, and prefer pinned dependencies in an isolated environment.
功能分析
Type: OpenClaw Skill
Name: web-scraper
Version: 0.1.1
The 'web-scraper' skill is a well-structured and documented tool for multi-stage web content extraction and analysis. It follows industry best practices such as rate limiting, robots.txt compliance, and a cascade approach to resource usage, while explicitly instructing the agent to avoid sensitive files like .env and to handle API keys securely via environment variables (SKILL.md, claw.json).
能力评估
Credentials
Filesystem and network permissions are proportionate for creating scraper scripts, fetching pages, and writing outputs; the OpenRouter stage is disclosed and optional, but scraped content may leave the local environment when enabled.
Install Mechanism
The package is a Markdown skill with no bundled executable scripts, but it recommends unpinned pip packages and Playwright browser installation, which is normal for this tooling but not reproducible.
Instruction Scope
The skill has useful planning, rate-limit, robots.txt, and credential-scope rules, but its paywall instructions are internally under-scoped: it forbids hard-paywall bypass while permitting DOM manipulation for soft paywalls.
Persistence & Privilege
No evidence of background services, scheduled tasks, tool hijacking, credential-file reads, persistent memory writes, or privilege escalation was found; checkpoint/output persistence is ordinary scraper output.
Purpose & Capability
The scraping, parsing, Playwright rendering, metadata extraction, local output files, and optional LLM entity extraction generally match the stated web-scraper purpose, but the soft-paywall guidance explicitly tells the agent to remove overlays and reveal hidden paragraphs, which goes beyond ordinary content extraction.
如何使用
- 确保已安装 OpenClaw（本地或 Docker 部署）
- 在对话框中输入安装命令：
/install web-scraper - 安装完成后，直接呼叫该 Skill 的名称或使用
/web-scraper触发 - 根据 Skill 的参数说明提供必要输入，即可获得结构化输出
版本历史
v0.1.1
web-scraper 0.1.1
- Added a CHANGELOG.md file for better tracking of updates.
- Clarified credential handling: the skill itself never makes direct API calls, and only generated scripts reference `OPENROUTER_API_KEY` if LLM extraction is used.
- Updated environment survey instructions to check for presence of `OPENROUTER_API_KEY` (for generated code), not the value.
- Improved documentation for credential scope and clarified planning protocol steps.
v0.1.0
web-scraper 0.1.0 — Initial release
- Introduces a web scraping and content comprehension agent with multi-strategy extraction and cascade fallback.
- Implements a mandatory planning protocol for all scraping requests, emphasizing environment and target analysis before action.
- Features a 5-stage pipeline: News/Article detection, multi-strategy extraction, cleaning/normalization, structured metadata extraction, and optional entity extraction using LLMs.
- Ensures safe credential handling by referencing `OPENROUTER_API_KEY` only in template code.
- Outputs structured JSON files with detailed content and quality metadata.
元数据
常见问题
Web Scraper 是什么？
Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and... 它是一个面向 Claude Code / OpenClaw 的 AI Agent Skill 插件，目前累计下载 11938 次。
如何安装 Web Scraper？
在 OpenClaw 或 Claude Code 对话框中运行命令「/install web-scraper」即可一键安装，无需额外配置。
Web Scraper 是免费的吗？
是的，Web Scraper 完全免费，采用 MIT-0 许可证，可自由下载、安装和使用。
Web Scraper 支持哪些平台？
Web Scraper 跨平台运行，可在任意部署了 OpenClaw / Claude Code 的环境中使用（cross-platform）。
谁开发了 Web Scraper？
由 Guilherme Favaron（@guifav）开发并维护，当前版本 v0.1.1。
推荐 Skills