← Back to Skills Marketplace
qthans

Anycrawl

by QThans · GitHub ↗ · v1.0.1 · MIT-0
cross-platform ✓ Security Clean
259
Downloads
1
Stars
0
Active Installs
2
Versions
Install in OpenClaw
/install anycrawl-cli
Description
Web scraping, search, crawling, and site mapping via the AnyCrawl CLI. Use when the user wants to search the web, scrape a page, find URLs on a site, or bulk...
README (SKILL.md)

AnyCrawl CLI

Web scraping, search, and crawling CLI. Returns clean markdown optimized for LLM context windows. Default engine: playwright.

Run anycrawl --help or anycrawl \x3Ccommand> --help for full option details.

Prerequisites

Must be installed and authenticated. Run anycrawl login or set ANYCRAWL_API_KEY.

If not ready, see rules/install.md. For output handling guidelines, see rules/security.md.

Commands

  • Search - No specific URL yet. Find pages, answer questions. Use --scrape to get full page content with results.
  • Scrape - Have a URL. Extract its content directly.
  • Map - Need to locate a specific page on a site. Discover URLs, then scrape the ones you need.
  • Crawl - Need bulk content from a site or section. Use crawl directly — no need for map first.
Need Command When
Find pages on a topic search No specific URL yet
Get a page's content scrape Have a URL
Find URLs within a site map Need to locate a specific subpage
Bulk extract a site section crawl Need many pages (e.g., all /docs/)

For detailed command reference, run anycrawl \x3Ccommand> --help (e.g., anycrawl search, anycrawl scrape).

Avoid redundant fetches: search --scrape already fetches full page content. Don't re-scrape those URLs. Check .anycrawl/ for existing data before fetching again.

Output & Organization

Write results to .anycrawl/ with -o. Add .anycrawl/ to .gitignore. Always quote URLs in shell commands. Never read entire output files at once — use grep, head, or incremental reads.

Documentation

Usage Guidance
This skill is an instruction-only wrapper for the AnyCrawl CLI and appears internally consistent with that purpose. Before installing or using it: 1) Verify the upstream npm package (anycrawl-cli) and its source/release provenance; prefer using npx if you don't want a global install. 2) Be aware the CLI fetches arbitrary third‑party web content and will store an API key under your user config directory — use a scoped/rotatable key and inspect where it is stored. 3) The SKILL.md references ANYCRAWL_API_KEY but the skill metadata doesn't declare it; ask the author to declare required env vars explicitly. 4) Follow the included security advice (write output to .anycrawl/, add to .gitignore, and read files incrementally) to reduce prompt‑injection and data exposure risk.
Capability Analysis
Type: OpenClaw Skill Name: anycrawl-cli Version: 1.0.1 The anycrawl-cli skill bundle is a well-structured integration for a web scraping service. It follows security best practices by explicitly instructing the agent to quote URLs to prevent command injection and recommending file-based isolation and incremental reading to mitigate indirect prompt injection from untrusted web content (rules/security.md). The installation steps (rules/install.md) and tool permissions (SKILL.md) are standard for CLI utilities and are appropriately scoped to the 'anycrawl' command.
Capability Assessment
✓ Purpose & Capability
The name/description match the instructions: the skill expects the AnyCrawl CLI and explains search/scrape/map/crawl workflows. Nothing requested or documented is unrelated to web crawling/scraping.
✓ Instruction Scope
SKILL.md only instructs use of the anycrawl CLI and contains safe-sounding handling guidance (write outputs to .anycrawl/, incremental reads, URL quoting, avoid returning fetched pages directly). It does not instruct reading unrelated system files or exfiltrating data.
ℹ Install Mechanism
There is no formal install spec (instruction-only), and install guidance points to the npm package anycrawl-cli and npx usage — a standard approach. Users should still verify the npm package and upstream docs before installing.
⚠ Credentials
SKILL.md and rules/install.md reference an ANYCRAWL_API_KEY and describe persistent credentials in ~/.config or %APPDATA%, but the skill metadata lists no required environment variables or primary credential. The skill should declare ANYCRAWL_API_KEY (or be explicit it is optional).
✓ Persistence & Privilege
The skill is not forced-always, is user-invocable, and does not request persistent platform privileges or modify other skills' configs. Autonomous invocation is allowed by default but not unusual here.
How to Use
  1. Make sure OpenClaw is installed (local or Docker)
  2. Run the install command in chat: /install anycrawl-cli
  3. After installation, invoke the skill by name or use /anycrawl-cli
  4. Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.1
- Clarified and simplified workflow section, now titled "Commands". - Improved explanations for command scenarios and when to use each command. - Updated the command reference table for clarity. - Added tips to avoid redundant fetches and directions to check existing data. - Retained full prerequisite, output, and documentation sections.
v1.0.0
anycrawl-cli 1.0.0 initial release - Introduces a CLI for web scraping, search, crawling, and site mapping using AnyCrawl. - Supports commands for searching, scraping, mapping site URLs, and bulk crawling site sections. - Outputs clean, LLM-optimized markdown for easy downstream processing. - Requires installation and authentication via login or API key. - Organizes results in the .anycrawl/ directory and provides guidance to avoid redundant fetches.
Metadata
Slug anycrawl-cli
Version 1.0.1
License MIT-0
All-time Installs 0
Active Installs 0
Total Versions 2
Frequently Asked Questions

What is Anycrawl?

Web scraping, search, crawling, and site mapping via the AnyCrawl CLI. Use when the user wants to search the web, scrape a page, find URLs on a site, or bulk... It is an AI Agent Skill for Claude Code / OpenClaw, with 259 downloads so far.

How do I install Anycrawl?

Run "/install anycrawl-cli" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.

Is Anycrawl free?

Yes, Anycrawl is completely free, licensed under MIT-0. You can download, install and use it at no cost.

Which platforms does Anycrawl support?

Anycrawl is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).

Who created Anycrawl?

It is built and maintained by QThans (@qthans); the current version is v1.0.1.

More Skills
self-improving agent
Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Clau...
⬇ 463283 · by pskoett
Skill Vetter
Security-first skill vetting for AI agents. Use before installing any skill from ClawdHub, GitHub, or other sources. Checks for red flags, permission scope, and suspicious patterns.
⬇ 259410 · by spclaudehome
Polymarket
Query Polymarket prediction markets. Check odds, find trending markets, search events, track price movements.
⬇ 232503 · by joelchance
Self-Improving + Proactive Agent
Self-reflection + Self-criticism + Self-learning + Self-organizing memory. Agent evaluates its own work, catches mistakes, and improves permanently. Use when...
⬇ 200423 · by ivangdavila
Github
Interact with GitHub using the `gh` CLI. Use `gh issue`, `gh pr`, `gh run`, and `gh api` for issues, PRs, CI runs, and advanced queries.
⬇ 191113 · by steipete
ontology
Typed knowledge graph for structured agent memory and composable skills. Use when creating/querying entities (Person, Project, Task, Event, Document), linkin...
⬇ 190156 · by oswalpalash

💬 Comments