← Back to Skills Marketplace
Robots Ai
by
sharozdawa
· GitHub ↗
· v1.0.0
· MIT-0
140
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install robotstxt-ai
Description
Analyze and generate robots.txt files with AI crawler awareness. Detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked o...
Usage Guidance
This skill appears coherent and safe in intent. Before installing, note that: (1) it will fetch robots.txt from arbitrary URLs you provide — ensure you are comfortable with the agent making outbound requests to those sites and with any logs that may record those URLs; (2) generated recommendations (e.g., 'block all AI' or blocking Googlebot) can have major consequences like removing your site from search/indexing — review generated robots.txt carefully before applying it to production; (3) verify the bot list if you rely on it for compliance, since crawler names and behaviors can change. If you plan to use it on private/internal sites, ensure the agent has appropriate network access and that no sensitive URLs are shared accidentally.
Capability Analysis
Type: OpenClaw Skill
Name: robotstxt-ai
Version: 1.0.0
The skill bundle is a set of instructions for an AI agent to analyze and generate robots.txt files with a focus on AI crawlers. It contains a database of AI user-agents and logical steps for auditing website permissions, with no executable code, network exfiltration, or malicious prompt injection identified in SKILL.md or _meta.json.
Capability Assessment
Purpose & Capability
Name/description (robots.txt analysis/generation with AI-crawler awareness) match the SKILL.md instructions. The skill requires no binaries, credentials, or config paths — appropriate for an instruction-only robots.txt helper.
Instruction Scope
SKILL.md explicitly tells the agent to fetch /robots.txt from arbitrary websites, parse User-agent/Allow/Disallow rules, and produce suggested robots.txt content. That behavior is expected for this purpose, but it implies the agent will perform network fetches of target sites (and may reveal provided site URLs to any runtime network logs). The instructions do not request unrelated files, credentials, or external endpoints beyond the target site's robots.txt.
Install Mechanism
No install spec and no code files are present — lowest-risk instruction-only skill. Nothing is downloaded or written to disk by an installer.
Credentials
The skill declares no required environment variables or credentials. No disproportionate access to secrets or unrelated services is requested.
Persistence & Privilege
always is false and the skill does not request special persistent privileges or modifications to other skills/configs. Autonomous invocation is allowed by default but not excessive here.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install robotstxt-ai - After installation, invoke the skill by name or use
/robotstxt-ai - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.0
Initial release with AI-focused robots.txt analysis and generation.
- Analyze robots.txt files to detect which AI bots (e.g., GPTBot, ClaudeBot, PerplexityBot) are allowed or blocked.
- Generate custom robots.txt files with easy AI bot toggling and explanatory comments.
- Audit existing robots.txt for completeness, identifying issues and suggesting improvements.
- List 20+ known AI crawlers with details on user-agents, company, and crawler type.
- Output always includes clear formatting and explanations for each section.
Metadata
Frequently Asked Questions
What is Robots Ai?
Analyze and generate robots.txt files with AI crawler awareness. Detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked o... It is an AI Agent Skill for Claude Code / OpenClaw, with 140 downloads so far.
How do I install Robots Ai?
Run "/install robotstxt-ai" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Robots Ai free?
Yes, Robots Ai is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Robots Ai support?
Robots Ai is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Robots Ai?
It is built and maintained by sharozdawa (@sharozdawa); the current version is v1.0.0.
More Skills