← Back to Skills Marketplace
Novel Scraper Pro
by
yuzhihui886
· GitHub ↗
· v2.0.3
· MIT-0
115
Downloads
0
Stars
0
Active Installs
4
Versions
Install in OpenClaw
/install novel-scraper-pro
Description
智能小说抓取工具 V6,支持自动翻页、分页补全、章节号自动解析、内存监控、中断续抓。 使用 curl+BeautifulSoup 抓取笔趣阁等小说网站,输出格式化 TXT 文件。 默认每 10 章合并为一个文档,避免文件零散分布。 自动检测分页并补全,智能跳过非小说内容(作者感言、抽奖预告等)。 内存监控和中断续...
Usage Guidance
This skill appears to be what it says: a local novel scraper that uses curl/BeautifulSoup and optionally an 'openclaw browser' CLI for SPA sites. Before installing, consider: 1) it will perform network requests to arbitrary URLs you pass (respect site terms of service); 2) it invokes subprocesses (curl and openclaw browser) — ensure you trust those CLIs and the environment where it will run; 3) it reads /proc/meminfo and writes files under ~/.openclaw/workspace and /tmp, including a progress.json you may delete to reset state; 4) there are small inconsistencies (some filenames/paths and version strings differ across files, e.g., fetch_catalog writes to a 'novel-scraper' path while other scripts use 'novel-scraper-pro') — these are not security-critical but may cause confusion or require you to adjust paths; 5) run the skill in an isolated environment or sandbox if you plan to scrape untrusted websites. If you need higher assurance, inspect or run the included scripts locally in a safe environment and verify the behavior of the 'openclaw browser' CLI before enabling SPA mode.
Capability Analysis
Type: OpenClaw Skill
Name: novel-scraper-pro
Version: 2.0.3
The novel-scraper-pro skill bundle is a functional utility designed to scrape web novels from specific sites like Biquge using curl and BeautifulSoup. The code in scraper.py, fetch_catalog.py, and other scripts is well-structured and includes security measures such as URL scheme validation and a blacklist of shell-injection characters (e.g., ';', '|', '&') before executing subprocess commands. The tool's behavior, including file I/O in the workspace directory and memory monitoring via /proc/meminfo, is entirely consistent with its stated purpose, and there is no evidence of data exfiltration, prompt injection, or malicious intent.
Capability Assessment
Purpose & Capability
Name/description match the code and SKILL.md: all scripts implement scraping, catalog fetching, URL extraction and merging. Required capabilities (curl, BeautifulSoup or bs4) are consistent with the stated purpose. No unrelated credentials or tools are requested.
Instruction Scope
Runtime instructions tell the agent to run the included Python scripts which read/write files under ~/.openclaw/workspace and /tmp, call curl and (optionally) an 'openclaw browser' CLI, and read /proc/meminfo for memory checks. These actions are within scope for a scraper, but the skill will perform network requests and create/modify local files (including progress/state).
Install Mechanism
No install spec that downloads code at install time; dependencies are standard Python packages (beautifulsoup4/bs4) listed in requirements.txt. The project runs subprocess curl/CLI commands rather than installing arbitrary binaries from remote URLs.
Credentials
The skill requests no environment variables or external credentials. It reads local system info (/proc/meminfo) and writes state/cache under the user's home (~/.openclaw and /tmp) which is expected for progress caching and output files.
Persistence & Privilege
always is false and the skill does not request platform-level persistence. It writes its own state/progress files in its workspace but does not modify other skills or global agent settings.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install novel-scraper-pro - After installation, invoke the skill by name or use
/novel-scraper-pro - Provide required inputs per the skill's parameter spec and get structured output
Version History
v2.0.3
修复缓存命中导致分页不完整:缓存<50 段时重新抓取
v2.0.2
修复分页补全超时问题
v2.0.1
修复:进度保存逻辑、check_memory 返回值
v2.0.0
V6 增强版:内存监控 + 中断续抓(默认开启)+ SPA 支持
Metadata
Frequently Asked Questions
What is Novel Scraper Pro?
智能小说抓取工具 V6,支持自动翻页、分页补全、章节号自动解析、内存监控、中断续抓。 使用 curl+BeautifulSoup 抓取笔趣阁等小说网站,输出格式化 TXT 文件。 默认每 10 章合并为一个文档,避免文件零散分布。 自动检测分页并补全,智能跳过非小说内容(作者感言、抽奖预告等)。 内存监控和中断续... It is an AI Agent Skill for Claude Code / OpenClaw, with 115 downloads so far.
How do I install Novel Scraper Pro?
Run "/install novel-scraper-pro" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is Novel Scraper Pro free?
Yes, Novel Scraper Pro is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does Novel Scraper Pro support?
Novel Scraper Pro is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created Novel Scraper Pro?
It is built and maintained by yuzhihui886 (@yuzhihui886); the current version is v2.0.3.
More Skills