← Back to Skills Marketplace
Python Crawler Architect
by
strong-Cyber
· GitHub ↗
· v1.0.0
· MIT-0
65
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install python-crawler-architect
Description
资深Python爬虫与数据工程专家。当用户需要设计网络爬虫系统、构建数据采集管道、设计数据库模型(SQLAlchemy ORM)、实现反爬虫策略（代理池、断点续传、重试机制）、异步并发编程(asyncio/aiohttp)、或进行数据清洗时，使用此技能。关键词：爬虫、crawler、scraper、数据采集、代理...
Usage Guidance
This skill is coherent with its stated purpose and doesn't request credentials or install code, but it includes guidance on proxy pools and anti‑scraping evasion. Before using or executing any generated code: 1) review the code yourself (or have a developer review it) and do not run untrusted code in production environments; 2) ensure your scraping activities comply with target site terms of service, local laws, and privacy rules (avoid collecting personal data or bypassing access controls); 3) if deploying, isolate runtime environments, rotate and protect any credentials you supply, and follow responsible rate-limiting and polite crawling practices.
Capability Analysis
Type: OpenClaw Skill
Name: python-crawler-architect
Version: 1.0.0
The skill bundle provides a comprehensive and professional framework for an AI agent to act as a 'Python Crawler Architect'. It contains well-structured SQLAlchemy ORM templates, proxy management logic, rate limiting, and state persistence mechanisms (SKILL.md). The code follows industry best practices (asyncio, type hinting, environment variables) and includes explicit reminders regarding legal compliance and ethical scraping. No indicators of data exfiltration, malicious execution, or prompt injection were found.
Capability Tags
Capability Assessment
Purpose & Capability
The name and description (Python crawler architect) match the SKILL.md contents: database modeling, asyncio/aiohttp patterns, proxy pools, retry/continuation strategies, SQLAlchemy templates, and project structure. No unrelated credentials, binaries, or install steps are requested.
Instruction Scope
SKILL.md is extensive and stays on-topic (architecture, ORM models, proxy pools, state management, retries, data cleaning, code templates). It also explicitly discusses anti‑scraping countermeasures (proxy pools, evasion strategies). That is coherent with a crawler-architect role but is ethically sensitive — it could be used to help evade site protections. The instructions do not direct reading arbitrary local files, exfiltrating secrets, or contacting unexpected remote endpoints.
Install Mechanism
No install specification or code files are included (instruction-only), so nothing is written to disk or fetched at install time. This is the lowest-risk class of skill in terms of install mechanism.
Credentials
The skill declares no required environment variables, credentials, or config paths. The SKILL.md references using environment variables for sensitive config in general (a best practice) but does not require any specific secrets. No disproportionate credential access is requested.
Persistence & Privilege
Flags show default behavior (always: false, agent-invocable allowed). The skill does not request persistent installation, system-wide changes, or modification of other skill settings.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install python-crawler-architect - After installation, invoke the skill by name or use
/python-crawler-architect - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.0
python-crawler-architect 1.0.0
- Initial release of the Python 爬虫架构师 skill.
- Defines workflow and best practices for designing large-scale Python web crawlers.
- Includes detailed guidelines for database modeling with SQLAlchemy ORM.
- Provides templates for anti-blocking strategies, proxy pool management, fault tolerance, and data cleaning.
- Specifies technical stack, project structure, and code style norms for robust, production-grade crawler systems.
Metadata
Frequently Asked Questions
What is Python Crawler Architect?
资深Python爬虫与数据工程专家。当用户需要设计网络爬虫系统、构建数据采集管道、设计数据库模型(SQLAlchemy ORM)、实现反爬虫策略（代理池、断点续传、重试机制）、异步并发编程(asyncio/aiohttp)、或进行数据清洗时，使用此技能。关键词：爬虫、crawler、scraper、数据采集、代理... It is an AI Agent Skill for Claude Code / OpenClaw, with 65 downloads so far.
How do I install Python Crawler Architect?
Run "/install python-crawler-architect" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Python Crawler Architect free?
Yes, Python Crawler Architect is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Python Crawler Architect support?
Python Crawler Architect is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Python Crawler Architect?
It is built and maintained by strong-Cyber (@strong-cyber); the current version is v1.0.0.
More Skills