← Back to Skills Marketplace
robots-txt
by
Kostja Zhang
· GitHub ↗
· v1.1.1
· MIT-0
92
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install robots-txt
Description
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawler...
Usage Guidance
This skill is internally consistent and low-risk: it only contains prose instructions and needs no installs or credentials. Before installing, note that at runtime it may read project-context files (.claude/project-context.md or .cursor/project-context.md) if present to learn the site URL and indexing goals — remove or sanitize those files if they contain sensitive data you don't want shared. If you plan to audit a live site, the skill does not gain host credentials or make changes itself; provide robots.txt contents or site details explicitly when asked rather than granting broad access to production systems. If you prefer to avoid any automatic local-file reads, disable autonomous invocation for the skill or remove project-context files from the agent environment.
Capability Analysis
Type: OpenClaw Skill
Name: robots-txt
Version: 1.1.1
The skill bundle provides standard SEO guidance and instructions for an AI agent to assist users with robots.txt configuration and auditing. It includes technical best practices, crawler-specific strategies (e.g., GPTBot, ClaudeBot), and references to related SEO concepts like indexing and sitemaps. No malicious code, data exfiltration attempts, or harmful prompt injection instructions were found in SKILL.md or _meta.json.
Capability Assessment
Purpose & Capability
Name/description (robots.txt auditing/config) match the instructions: generate recommended files, audit for accidental blocks, and give per-user-agent rules. It does not request unrelated credentials or tooling.
Instruction Scope
Instructions are narrowly scoped to robots.txt guidance and auditing. They do tell the agent to read .claude/project-context.md or .cursor/project-context.md (if present) to obtain site URL/indexing goals — reasonable for context but the agent will read local project files, so users should be aware that any content in those files will be consumed.
Install Mechanism
No install spec and no code files — instruction-only skill. Nothing is written to disk or downloaded during install.
Credentials
The skill requires no environment variables, secrets, or external credentials. The requested access (local project-context files) is proportionate to the stated purpose.
Persistence & Privilege
always is false and the skill does not request elevated/system-wide privileges. Autonomous invocation is allowed (platform default) but not itself a problem here.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install robots-txt - After installation, invoke the skill by name or use
/robots-txt - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.1.1
Automated batch sync
Metadata
Frequently Asked Questions
What is robots-txt?
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawler... It is an AI Agent Skill for Claude Code / OpenClaw, with 92 downloads so far.
How do I install robots-txt?
Run "/install robots-txt" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is robots-txt free?
Yes, robots-txt is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does robots-txt support?
robots-txt is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created robots-txt?
It is built and maintained by Kostja Zhang (@kostja94); the current version is v1.1.1.
More Skills