← Back to Skills Marketplace
johnnywang2001

Robots.txt Generator

by John Wang · GitHub ↗ · v1.0.0 · MIT-0
cross-platform ⚠ suspicious
252
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install robots-txt-gen
Description
Generate, validate, and analyze robots.txt files for websites. Use when creating robots.txt from scratch, validating existing robots.txt syntax, checking if...
Usage Guidance
This skill appears to do only what it says: generate, validate, and test robots.txt files. Things to keep in mind before installing or running it: - The tool may fetch remote robots.txt files over HTTP(S) when you use validate --url; only point it at sites you trust or expect to inspect. - It will read any local file path you pass to --file, so avoid pointing it at unrelated sensitive files (it expects robots.txt content but will read whatever path you give it). - There are no credentials requested and no installer downloads, which reduces risk. If you plan to run the script on sensitive systems, review the full script in your environment (the provided excerpt looked consistent, but always verify the complete file shipped with the skill). - If you want extra caution, run the script in a sandboxed environment or container first.
Capability Analysis
Type: OpenClaw Skill Name: robots-txt-gen Version: 1.0.0 The skill provides tools for managing robots.txt files but contains functional vulnerabilities. Specifically, scripts/robots_txt_gen.py implements a remote validation feature using urllib.request.urlopen that is susceptible to Server-Side Request Forgery (SSRF) due to a lack of URL sanitization. Additionally, the generate command allows writing to arbitrary file paths via the --output flag. While these features align with the stated purpose, the absence of input validation on URLs and file paths presents a security risk that could be exploited via prompt injection.
Capability Assessment
Purpose & Capability
Name/description, SKILL.md examples, and the included Python script all implement robots.txt generation, validation, and testing. There are no environment variables, unrelated binaries, or config paths requested that would be out of scope for this functionality.
Instruction Scope
Runtime instructions tell the agent to run the bundled Python script with file or URL inputs. The script reads local files and can fetch remote robots.txt via HTTP(S) (used by the validate --url flow), both of which are expected for this tool. It does not instruct the agent to read unrelated system files, credentials, or transmit data to unexpected endpoints.
Install Mechanism
This is instruction-only / script-only with no install spec. No downloads or package installs are requested, so nothing is written to disk by an installer beyond running the included script — low install risk.
Credentials
The skill requires no environment variables, credentials, or config paths. The script performs network fetches of user-specified URLs and reads user-specified local files, which is proportional to its stated purpose.
Persistence & Privilege
The skill does not request persistent or elevated privileges and always:false. It does not modify other skills or global agent configuration based on the provided materials.
How to Use
  1. Make sure OpenClaw is installed (local or Docker)
  2. Run the install command in chat: /install robots-txt-gen
  3. After installation, invoke the skill by name or use /robots-txt-gen
  4. Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.0
- Initial release of robots-txt-gen. - Generate, validate, and analyze robots.txt files via command line. - Supports platform presets for WordPress, Next.js, Django, Rails, Laravel, static sites, SPAs, and ecommerce. - Validate both local and remote robots.txt files for syntax and warnings. - Test if URLs are allowed or blocked for specific user-agents. - Optionally generate rules to block known AI crawlers with the --block-ai flag.
Metadata
Slug robots-txt-gen
Version 1.0.0
License MIT-0
All-time Installs 0
Active Installs 0
Total Versions 1
Frequently Asked Questions

What is Robots.txt Generator?

Generate, validate, and analyze robots.txt files for websites. Use when creating robots.txt from scratch, validating existing robots.txt syntax, checking if... It is an AI Agent Skill for Claude Code / OpenClaw, with 252 downloads so far.

How do I install Robots.txt Generator?

Run "/install robots-txt-gen" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.

Is Robots.txt Generator free?

Yes, Robots.txt Generator is completely free, licensed under MIT-0. You can download, install and use it at no cost.

Which platforms does Robots.txt Generator support?

Robots.txt Generator is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).

Who created Robots.txt Generator?

It is built and maintained by John Wang (@johnnywang2001); the current version is v1.0.0.

💬 Comments