← 返回 Skills 市场
Crawler
作者
bytesagain3
· GitHub ↗
· v3.0.0
· MIT-0
780
总下载
0
收藏
3
当前安装
10
版本数
在 OpenClaw 中安装
/install crawler
功能描述
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
使用说明 (SKILL.md)
Crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.
Commands
| Command | Description |
|---|---|
intro |
Crawling vs scraping, robots.txt, sitemap |
standards |
HTTP caching, structured data, meta tags |
troubleshooting |
Anti-bot detection, JS rendering, encoding |
performance |
Concurrency, dedup, incremental, distributed |
security |
Legal landscape, ethical guidelines, proxies |
migration |
BeautifulSoup to Scrapy, requests to Playwright |
cheatsheet |
Scrapy commands, CSS/XPath, curl, user-agents |
faq |
Legality, JS pages, blocking, storage |
Output Format
All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.
Powered by BytesAgain | bytesagain.com | [email protected]
安全使用建议
This skill appears to be documentation-only and does not ask for credentials or install external software. Before installing: (1) review the script.sh yourself (it prints the docs) to confirm behavior; (2) be aware that the content includes instructions about proxies and captcha-solving services which can enable scraping that may violate websites' terms or laws—use the guidance only for lawful, ethical purposes; (3) if you allow the agent to execute shell scripts, ensure your agent runtime is sandboxed because any skill that can run shell commands could be modified to run arbitrary code. If you need stricter guarantees, request a version with only SKILL.md (no executable files) or audit the full script contents line-by-line.
功能分析
Type: OpenClaw Skill
Name: crawler
Version: 3.0.0
The 'crawler' skill is a documentation-only reference tool providing information on web scraping standards, legal considerations, and technical frameworks. The implementation in scripts/script.sh consists entirely of static text output via heredocs, with no network access, file system modifications, or credential handling.
能力评估
Purpose & Capability
The name/description (web crawling & scraping reference) match the provided SKILL.md and the included script which only emits reference text. There are no unrelated required binaries, env vars, or config paths.
Instruction Scope
SKILL.md explicitly states outputs are plain-text reference via heredoc and no external API calls or credentials are required. The bundled script.sh functions simply print documentation. Note: the documentation discusses techniques (proxies, captcha-solving services, anti-bot workarounds) that can enable contentious scraping behavior — the skill itself only documents these topics and does not perform them.
Install Mechanism
No install spec is provided (instruction-only). The only code file is a local shell script that prints docs; nothing is downloaded or extracted during install.
Credentials
The skill declares no required environment variables or credentials and the runtime instructions do not attempt to read secrets. References to proxy providers and services in the docs are informational only and do not create a credential requirement.
Persistence & Privilege
No elevated persistence requested (always: false). The skill does not modify other skills or system config and does not request permanent presence or special privileges.
如何使用
- 确保已安装 OpenClaw（本地或 Docker 部署）
- 在对话框中输入安装命令：
/install crawler - 安装完成后，直接呼叫该 Skill 的名称或使用
/crawler触发 - 根据 Skill 的参数说明提供必要输入，即可获得结构化输出
版本历史
v3.0.0
Clean package with matching SKILL.md and script
v2.0.1
update
v2.0.0
v2.5 standard: Use-when desc, homepage, source, security fix
v1.0.6
retry-fix-token
v1.0.5
old template -> domain-specific v2.0.0
v1.0.4
Quality upgrade
v1.0.3
Quality upgrade: custom functionality
v1.0.2
De-template, unique content, script cleanup
v1.0.1
Quality fix: cleaner docs, removed flags
v1.0.0
Initial release
元数据
常见问题
Crawler 是什么？
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. 它是一个面向 Claude Code / OpenClaw 的 AI Agent Skill 插件，目前累计下载 780 次。
如何安装 Crawler？
在 OpenClaw 或 Claude Code 对话框中运行命令「/install crawler」即可一键安装，无需额外配置。
Crawler 是免费的吗？
是的，Crawler 完全免费，采用 MIT-0 许可证，可自由下载、安装和使用。
Crawler 支持哪些平台？
Crawler 跨平台运行，可在任意部署了 OpenClaw / Claude Code 的环境中使用（cross-platform）。
谁开发了 Crawler？
由 bytesagain3（@bytesagain3）开发并维护，当前版本 v3.0.0。
推荐 Skills