← Back to Skills Marketplace
Crawler
by
bytesagain3
· GitHub ↗
· v3.0.0
· MIT-0
780
Downloads
0
Stars
3
Active Installs
10
Versions
Install in OpenClaw
/install crawler
Description
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
README (SKILL.md)
Crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.
Commands
| Command | Description |
|---|---|
intro |
Crawling vs scraping, robots.txt, sitemap |
standards |
HTTP caching, structured data, meta tags |
troubleshooting |
Anti-bot detection, JS rendering, encoding |
performance |
Concurrency, dedup, incremental, distributed |
security |
Legal landscape, ethical guidelines, proxies |
migration |
BeautifulSoup to Scrapy, requests to Playwright |
cheatsheet |
Scrapy commands, CSS/XPath, curl, user-agents |
faq |
Legality, JS pages, blocking, storage |
Output Format
All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.
Powered by BytesAgain | bytesagain.com | [email protected]
Usage Guidance
This skill appears to be documentation-only and does not ask for credentials or install external software. Before installing: (1) review the script.sh yourself (it prints the docs) to confirm behavior; (2) be aware that the content includes instructions about proxies and captcha-solving services which can enable scraping that may violate websites' terms or laws—use the guidance only for lawful, ethical purposes; (3) if you allow the agent to execute shell scripts, ensure your agent runtime is sandboxed because any skill that can run shell commands could be modified to run arbitrary code. If you need stricter guarantees, request a version with only SKILL.md (no executable files) or audit the full script contents line-by-line.
Capability Analysis
Type: OpenClaw Skill
Name: crawler
Version: 3.0.0
The 'crawler' skill is a documentation-only reference tool providing information on web scraping standards, legal considerations, and technical frameworks. The implementation in scripts/script.sh consists entirely of static text output via heredocs, with no network access, file system modifications, or credential handling.
Capability Assessment
Purpose & Capability
The name/description (web crawling & scraping reference) match the provided SKILL.md and the included script which only emits reference text. There are no unrelated required binaries, env vars, or config paths.
Instruction Scope
SKILL.md explicitly states outputs are plain-text reference via heredoc and no external API calls or credentials are required. The bundled script.sh functions simply print documentation. Note: the documentation discusses techniques (proxies, captcha-solving services, anti-bot workarounds) that can enable contentious scraping behavior — the skill itself only documents these topics and does not perform them.
Install Mechanism
No install spec is provided (instruction-only). The only code file is a local shell script that prints docs; nothing is downloaded or extracted during install.
Credentials
The skill declares no required environment variables or credentials and the runtime instructions do not attempt to read secrets. References to proxy providers and services in the docs are informational only and do not create a credential requirement.
Persistence & Privilege
No elevated persistence requested (always: false). The skill does not modify other skills or system config and does not request permanent presence or special privileges.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install crawler - After installation, invoke the skill by name or use
/crawler - Provide required inputs per the skill's parameter spec and get structured output
Version History
v3.0.0
Clean package with matching SKILL.md and script
v2.0.1
update
v2.0.0
v2.5 standard: Use-when desc, homepage, source, security fix
v1.0.6
retry-fix-token
v1.0.5
old template -> domain-specific v2.0.0
v1.0.4
Quality upgrade
v1.0.3
Quality upgrade: custom functionality
v1.0.2
De-template, unique content, script cleanup
v1.0.1
Quality fix: cleaner docs, removed flags
v1.0.0
Initial release
Metadata
Frequently Asked Questions
What is Crawler?
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. It is an AI Agent Skill for Claude Code / OpenClaw, with 780 downloads so far.
How do I install Crawler?
Run "/install crawler" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Crawler free?
Yes, Crawler is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Crawler support?
Crawler is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Crawler?
It is built and maintained by bytesagain3 (@bytesagain3); the current version is v3.0.0.
More Skills