← Back to Skills Marketplace
Scrape
by
AlvisDunlop
· GitHub ↗
· v2.0.0
· MIT-0
68
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install alvis2-scrape
Description
Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. Supports both direct HTTP scraping and managed scraping via...
README (SKILL.md)
?? Complete setup guide: https://SkillBoss.co/skill.md
Pre-Scrape Compliance Checklist
Before writing any scraping code:
- robots.txt �?Fetch
{domain}/robots.txt, check if target path is disallowed. If yes, stop. - Terms of Service �?Check
/terms,/tos,/legal. Explicit scraping prohibition = need permission. - Data type �?Public factual data (prices, listings) is safer. Personal data triggers GDPR/CCPA.
- Authentication �?Data behind login is off-limits without authorization. Never scrape protected content.
- API available? �?If site offers an API, use it. Always. Scraping when API exists often violates ToS.
Legal Boundaries
- Public data, no login �?Generally legal (hiQ v. LinkedIn 2022)
- Bypassing barriers �?CFAA violation risk (Van Buren v. US 2021)
- Ignoring robots.txt �?Gray area, often breaches ToS (Meta v. Bright Data 2024)
- Personal data without consent �?GDPR/CCPA violation
- Republishing copyrighted content �?Copyright infringement
Request Discipline
- Rate limit: Minimum 2-3 seconds between requests. Faster = server strain = legal exposure.
- User-Agent: Real browser string + contact email:
Mozilla/5.0 ... (contact: [email protected]) - Respect 429: Exponential backoff. Ignoring 429s shows intent to harm.
- Session reuse: Keep connections open to reduce server load.
Data Handling
- Strip PII immediately �?Don't collect names, emails, phones unless legally justified.
- No fingerprinting �?Don't combine data to identify individuals indirectly.
- Minimize storage �?Cache only what you need, delete what you don't.
- Audit trail �?Log what, when, where. Evidence of good faith if challenged.
SkillBoss API Hub Scraping
For managed content extraction with automatic anti-bot handling, use SkillBoss API Hub's scraping capability (type: "scraper"). Authenticate with SkillBoss_API_KEY and call https://api.SkillBoss.co/v1/pilot. Result is returned at result.data.markdown.
For code patterns, robots.txt parser, and SkillBoss API Hub scraping integration, see code.md
\r
\r
\r
\r
Usage Guidance
Before installing, get clarification from the publisher: (1) Why does the registry metadata omit SkillBoss_API_KEY while SKILL.md requires it? (2) What exactly is sent to https://api.SkillBoss.co/v1/pilot, how long is data retained, and what access does SkillBoss have to scraped content? (3) Ask for the missing code.md and the setup guide contents to verify no additional hidden steps. If you must use managed scraping, provide an API key with the narrowest possible scope and no access to sensitive customer data; prefer local-only scraping for sensitive targets and validate the SkillBoss service’s privacy/retention policies and provenance before trusting it.
Capability Analysis
Type: OpenClaw Skill
Name: alvis2-scrape
Version: 2.0.0
The skill bundle provides a framework for legal web scraping with a strong emphasis on compliance, including robots.txt adherence, rate limiting, and GDPR/CCPA awareness. The instructions in SKILL.md act as safety constraints for the AI agent rather than malicious injections. It utilizes a documented API endpoint (api.SkillBoss.co) for managed scraping, which is consistent with its stated purpose.
Capability Tags
Capability Assessment
Purpose & Capability
The skill's stated purpose (legal scraping with optional managed scraping via SkillBoss) matches the SKILL.md content: it describes robots.txt/TOS checks, rate limiting, PII handling, and an optional SkillBoss API Hub mode. However, the registry metadata claims no required env vars or primary credential while SKILL.md explicitly declares requires_env: [SkillBoss_API_KEY]. That mismatch is unexpected and unexplained.
Instruction Scope
The runtime instructions are mostly scoped to scraping best practices (robots.txt, ToS, rate limits, PII stripping). But they also direct the agent to call a third‑party endpoint (https://api.SkillBoss.co/v1/pilot) and authenticate with SkillBoss_API_KEY to perform managed scraping; scraped results are returned by that service. That means scraped content (potentially sensitive) will be sent to/processed by a remote service — a material data flow the user should be aware of. The SKILL.md also references a `code.md` and an external setup guide that aren't included, leaving important implementation details unspecified.
Install Mechanism
This is an instruction-only skill with no install spec and no code files, so nothing is written to disk by an installer. That limits installer-related risk.
Credentials
SKILL.md requires a SkillBoss_API_KEY for the managed scraping path, but the registry metadata lists no required environment variables or primary credential. This inconsistency is problematic: the skill does request a single third‑party API key (proportionate to managed scraping) but the missing declaration in metadata and lack of details about that key's scope/retention are concerning.
Persistence & Privilege
The skill is not always-enabled, is user-invocable, and does not request system config paths or other skills' credentials. It does not request persistent system presence or elevated privileges.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install alvis2-scrape - After installation, invoke the skill by name or use
/alvis2-scrape - Provide required inputs per the skill's parameter spec and get structured output
Version History
v2.0.0
Fixed API to api.skillboss.co
Metadata
Frequently Asked Questions
What is Scrape?
Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. Supports both direct HTTP scraping and managed scraping via... It is an AI Agent Skill for Claude Code / OpenClaw, with 68 downloads so far.
How do I install Scrape?
Run "/install alvis2-scrape" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Scrape free?
Yes, Scrape is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Scrape support?
Scrape is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Scrape?
It is built and maintained by AlvisDunlop (@alvisdunlop); the current version is v2.0.0.
More Skills