← Back to Skills Marketplace
Tavily Crawl
by
Evan Rimer
· GitHub ↗
· v1.0.0
· MIT-0
297
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install tavily-crwal
Description
Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or...
Usage Guidance
Before installing or running this skill consider: 1) The script will attempt to read OAuth tokens from ~/.mcp-auth and will automatically use any valid Tavily token it finds — if you have other sensitive tokens in that directory, be cautious. 2) It runs 'npx -y mcp-remote ...' which downloads and executes an npm package at runtime (silently). If you don't trust the package or tavily.com, run the script in a sandbox/VM and audit the mcp-remote package first. 3) The script requires common CLI tools (jq, curl, npx, base64, find, date) though the skill metadata omits these — make sure those are present and you understand what they do. 4) If you prefer explicit control, set TAVILY_API_KEY in your environment before running (the SKILL.md suggests ~/.claude/settings.json but the script uses the env var or ~/.mcp-auth). 5) If you have sensitive files or tokens in your home directory, inspect ~/.mcp-auth and remove or isolate them before running. If these caveats are acceptable and you trust Tavily and the npm package used, the skill is coherent with its stated purpose; otherwise treat it as risky and run only in an isolated environment.
Capability Analysis
Type: OpenClaw Skill
Name: tavily-crwal
Version: 1.0.0
The skill provides a legitimate interface for the Tavily Crawl API, allowing users to extract website content into markdown files. The script `scripts/crawl.sh` implements a standard authentication flow that searches for existing Tavily tokens in `~/.mcp-auth/` or initiates an OAuth process via the official `mcp-remote` tool. The logic is transparent, matches the documentation in `SKILL.md`, and contains security checks to ensure only valid Tavily tokens are utilized.
Capability Assessment
Purpose & Capability
The declared purpose (crawl websites and save markdown) matches the script's behavior: it builds a JSON-RPC request to Tavily's MCP and writes results to files. However, metadata claims no required binaries or env vars while the script depends on external tools (jq, curl, npx, find, base64, date). The SKILL.md and script also reference different endpoints (api.tavily.com vs mcp.tavily.com) and the SKILL.md suggests placing an API key in ~/.claude/settings.json but the script only reads TAVILY_API_KEY from environment or tokens under ~/.mcp-auth — an inconsistency.
Instruction Scope
The script searches your home directory (~/.mcp-auth) for *_tokens.json and will decode/read JWT payloads there, then uses any valid Tavily token it finds. That is within the stated Tavily OAuth convenience flow, but it reads files from your user profile and may surface tokens automatically. The script also invokes npx to run 'mcp-remote' (backgrounded and silenced), which executes remote code outside the explicit skill scope. SKILL.md states 'No manual setup required' — true functionally, but the opaque npx invocation and home-directory token reads widen data access beyond 'just provide a URL'.
Install Mechanism
There is no install spec in the registry (instruction-only), but the script uses 'npx -y mcp-remote ...' at runtime. npx will fetch and execute code from the npm registry on-demand, which is effectively downloading and running third-party code without an explicit install step or review. That runtime fetch is a higher-risk operation compared to purely local scripts.
Credentials
The skill declares no required env vars, but the script expects TAVILY_API_KEY (or an OAuth token found in ~/.mcp-auth). Looking for tokens in ~/.mcp-auth is explainable for an OAuth convenience flow; the script also enforces an issuer check in JWTs (iss == 'https://mcp.tavily.com/'), which limits false positives. Still, the documentation's suggested location (~/.claude/settings.json) is not read by the script, and the script will instead search your auth cache and environment — this mismatch should be clarified before use.
Persistence & Privilege
The skill is not force-included (always:false) and does not change other skills or system-wide settings. It does spawn a temporary background npx process for OAuth flow, but it does not persistently install software or modify other agent configs.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install tavily-crwal - After installation, invoke the skill by name or use
/tavily-crwal - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.0
Initial release of the crawl skill for extracting website content as local markdown files.
- Crawl entire websites or specific sections with flexible depth, breadth, and content filters.
- Supports OAuth authentication (automatic on first run) and API key use.
- Save crawled pages as individual markdown files for offline use or analysis.
- Supports focused crawling via path filters, semantic instructions, and chunked results for agentic workflows.
- Includes comprehensive CLI and API documentation, with usage examples and performance tips.
Metadata
Frequently Asked Questions
What is Tavily Crawl?
Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or... It is an AI Agent Skill for Claude Code / OpenClaw, with 297 downloads so far.
How do I install Tavily Crawl?
Run "/install tavily-crwal" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Tavily Crawl free?
Yes, Tavily Crawl is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Tavily Crawl support?
Tavily Crawl is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Tavily Crawl?
It is built and maintained by Evan Rimer (@evanydl); the current version is v1.0.0.
More Skills