← Back to Skills Marketplace
11159
Downloads
9
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install scrape
Description
Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling.
Usage Guidance
Review before installing or using the sample code. The guidance is not deceptive and does not install anything, but users should change the robots.txt check to fail closed or ask before proceeding on errors, and add URL/redirect/IP validation before using the fetcher in any shared tool, server, or workflow that accepts user-supplied URLs.
Capability Analysis
Type: OpenClaw Skill
Name: scrape
Version: 1.0.0
The skill bundle is designed for ethical and legal web scraping, providing comprehensive guidelines in `SKILL.md` for compliance (robots.txt, ToS, GDPR/CCPA, rate limiting) and implementing these safeguards in `code.md`. The code uses standard Python libraries to perform web requests, handle rate limits, and log activity, all aligned with the stated purpose. There is no evidence of malicious intent, data exfiltration to unauthorized parties, persistence mechanisms, or prompt injection attempts designed to subvert the agent's purpose; instead, the instructions actively promote responsible behavior.
Capability Assessment
Credentials
Network fetching arbitrary URLs is expected for scraping, but the sample fetcher lacks URL, redirect, hostname, IP-range, timeout, and response-size controls, making it risky if reused in a service or with untrusted targets.
Install Mechanism
The artifact contains Markdown guidance and sample code only, with no executable install script, dependency installation, hooks, or package metadata requiring elevated authority.
Instruction Scope
The instructions tell users to check robots.txt, terms, authentication boundaries, and PII handling; the scanner's autonomous-decision concern is not supported by the artifact text.
Persistence & Privilege
No persistence, background workers, privilege escalation, credential harvesting, memory writes, or local profile/session access were found.
Purpose & Capability
The stated purpose is legal, robots.txt-compliant scraping, but code.md returns allowed on any robots.txt retrieval or parsing exception, which undermines that purpose.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install scrape - After installation, invoke the skill by name or use
/scrape - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.0
Initial release
Metadata
Frequently Asked Questions
What is Scrape?
Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. It is an AI Agent Skill for Claude Code / OpenClaw, with 11159 downloads so far.
How do I install Scrape?
Run "/install scrape" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Scrape free?
Yes, Scrape is completely free (open-source). You can download, install and use it at no cost.
Which platforms does Scrape support?
Scrape is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Scrape?
It is built and maintained by Iván (@ivangdavila); the current version is v1.0.0.
More Skills