功能描述

Builds a fully runnable MCP (Model Context Protocol) knowledge server from any website or documentation URL. Crawls the site, extracts concepts using Claude,...

使用说明 (SKILL.md)

GraphRAG Builder Skill

Name: GraphRAGBuilder
Author: nacmonad

Turns any documentation website into a runnable MCP knowledge server in 5 pipeline steps, each run on the user's local machine using scripts in the scripts/ folder.

Quick Reference

Step	Script	What it does
M1	`crawl.py`	BFS crawl → raw HTML + metadata per page
M2	`extract_concepts.py`	HTML → chunks → LLM concept extraction
M3	`build_graph.py`	Concepts + links → networkx knowledge graph
M4	`build_embeddings.py`	Chunks + concepts → numpy vector index
M5	`generate_mcp_server.py`	Graph + embeddings → standalone `server.py`

All scripts require Python 3.10+ and auto-install their own dependencies on first run.

Step 0: Clarify Requirements

Before running anything, ask the user:

URL: Which site to crawl (required — starting page)
Depth: How many link-hops to follow (default 3; suggest 2 for large sites)
Model: Which Claude model for concept extraction — haiku (fast/cheap) or sonnet (higher quality). Default: haiku

Set the output slug from the URL: https://strudel.cc → strudel-cc-mcp.

Step 1: Crawl (M1)

Provide this command for the user to run locally:

python scripts/crawl.py \
  --url \x3CURL> \
  --max-depth \x3CDEPTH> \
  --output ./output

What to expect:

Creates output/\x3Cslug>/raw_content/*.json (one per page)
Creates output/\x3Cslug>/crawl.json (state tracking)
Prints a summary: pages crawled, JS fallbacks used, failures

Common issues:

JS-heavy single-page apps → many Playwright fallbacks (normal, just slower)
Rate limiting → add --rate-limit 1.5 to slow down
First run needs: pip install playwright && playwright install chromium

Step 2: Extract Concepts (M2)

The user must set ANTHROPIC_API_KEY first. Provide this command:

ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \
  --input ./output/\x3Cslug>-mcp \
  --model haiku

Dry-run first (no API cost):

python scripts/extract_concepts.py --input ./output/\x3Cslug>-mcp --dry-run

This validates chunking quality before spending API budget. Show them chunk counts and section names from dry-run output.

What to expect:

Processes ~2–5 pages/minute on haiku
Creates output/\x3Cslug>-mcp/extracted/*.json (one per page)
Each file contains chunks with: concepts, tags, code examples, prerequisites, relationships
Skips already-extracted pages (safe to re-run after interruption)

Common issues:

Pages showing no_chunks → likely JS-rendered content not captured; acceptable for a minority of pages
API rate limiting → script retries automatically with exponential backoff
--max-pages 10 flag to test on a small sample first

Re-running after a partial run:

python scripts/extract_concepts.py --input ./output/\x3Cslug>-mcp --model haiku
# (automatically skips already-extracted pages)

Force re-extraction of everything:

python scripts/extract_concepts.py --input ./output/\x3Cslug>-mcp --force

Step 3: Build Graph (M3)

python scripts/build_graph.py --input ./output/\x3Cslug>-mcp

What to expect:

Reads all non-dry-run extracted/*.json files
Deduplicates concept names (case-insensitive, strips trailing ())
Creates output/\x3Cslug>-mcp/graph.json
Prints node/edge counts by type

Healthy output looks like:

Pages:     46
Chunks:    357
Concepts:  200+
Total edges: 1000+
  MENTIONS       600+
  REQUIRES       100+
  HAS_CHUNK      357
  LINKS_TO       80+
  RELATED        40+

If concepts = 0 and "Skipped N dry-run files" appears, M2 hasn't been run with a real API key yet.

Step 4: Build Embeddings (M4)

python scripts/build_embeddings.py --input ./output/\x3Cslug>-mcp

First run downloads all-MiniLM-L6-v2 (~80MB, cached after that).

Add --smoke-test to query both collections immediately after building:

python scripts/build_embeddings.py --input ./output/\x3Cslug>-mcp --smoke-test

What to expect:

Creates output/\x3Cslug>-mcp/embeddings/ with 5 numpy files (no database needed)
Two indexes: chunks (semantic search) and concepts (concept lookup)

Step 5: Generate MCP Server (M5)

python scripts/generate_mcp_server.py --input ./output/\x3Cslug>-mcp

Outputs:

output/\x3Cslug>-mcp/server.py — the runnable MCP server
output/\x3Cslug>-mcp/mcp_config.json — Claude Desktop config snippet

Install into Claude Desktop:

Open ~/Library/Application Support/Claude/claude_desktop_config.json
Merge the contents of mcp_config.json into the "mcpServers" key
Restart Claude Desktop
The server name (e.g., strudel-cc) appears in Claude's available tools

Test the server standalone:

python output/\x3Cslug>-mcp/server.py
# Should print "Loading ... knowledge graph... Ready: N nodes, M edges"

The 8 MCP Tools

Once installed, Claude can use these tools against the knowledge base:

Tool	Description
`search(query, n=5)`	Semantic search over all content chunks
`get_concept(name)`	Concept details + chunks where it appears
`get_related(concept, n=5)`	Related concepts via graph edges
`get_learning_path(start, goal)`	Shortest concept path between topics
`get_prerequisites(concept)`	What must be understood first
`get_examples(concept)`	Code examples for a concept
`list_concepts(tag?, limit=20)`	Browse all indexed concepts
`get_page(url)`	All chunks for a specific doc page

Complete Pipeline Command Sequence

For a fresh install, provide the user with all commands in order:

# 0. Install system deps (once)
pip install requests beautifulsoup4 lxml playwright anthropic \
    networkx numpy sentence-transformers mcp
playwright install chromium

# 1. Crawl
python scripts/crawl.py --url \x3CURL> --max-depth 3 --output ./output

# 2. Extract concepts (dry-run first)
python scripts/extract_concepts.py --input ./output/\x3Cslug>-mcp --dry-run
# Then real run:
ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \
  --input ./output/\x3Cslug>-mcp --model haiku

# 3. Build graph
python scripts/build_graph.py --input ./output/\x3Cslug>-mcp

# 4. Build embeddings
python scripts/build_embeddings.py --input ./output/\x3Cslug>-mcp --smoke-test

# 5. Generate server
python scripts/generate_mcp_server.py --input ./output/\x3Cslug>-mcp

# 6. Test server
python output/\x3Cslug>-mcp/server.py

Output Directory Layout

output/\x3Cslug>-mcp/
├── crawl.json              State tracking (incremental re-runs)
├── raw_content/            One JSON per crawled page (HTML + links)
├── extracted/              One JSON per page (chunks + LLM concepts)
├── graph.json              networkx knowledge graph
├── embeddings/             numpy indexes (chunks.npy, concepts.npy + JSON)
├── server.py               The runnable MCP server ← share this
└── mcp_config.json         Claude Desktop config snippet ← install this

The entire output/\x3Cslug>-mcp/ folder is the deliverable. The user can move it anywhere as long as server.py, graph.json, and embeddings/ stay together.

Troubleshooting

"No module named X" → The script auto-installs deps, but if it fails:

pip install \x3Cpackage> --break-system-packages

Crawl gets 0 pages → Check robots.txt and try --force to bypass the crawl cache.

extract_concepts produces tiny concepts count → The page content may be JS-only. Check fetched_with field in raw_content/*.json — pages fetched via requests with very little text should have been picked up by Playwright. Re-crawl with --force.

Server fails to start → Run python output/\x3Cslug>-mcp/server.py directly and check stderr for import errors. Most common cause: mcp package not installed.

Claude Desktop doesn't show the server → Verify the path in mcp_config.json is absolute and the file exists. Restart Claude Desktop after any config change.

Deferred Features

See TODO.md for planned improvements including:

YouTube transcript fetching
Neo4j export for large graphs
OpenAI/Voyage embedding API support
Scheduled re-crawls
Graph visualization

安全使用建议

What to consider before installing/running this skill: - The skill appears to do what it says: crawling, LLM-based concept extraction (Anthropic), graph construction, embeddings, and a generated MCP server. That part is coherent. - However: SKILL.md and the scripts require an ANTHROPIC_API_KEY (for concept extraction) but the skill metadata did not declare any required environment variables. Treat that as a metadata omission; you must supply an Anthropic key to run the core step. - The code auto-installs Python packages at runtime (pip install ...) and will install Playwright and Chromium and download the sentence-transformers model (~80MB) when run. Those actions fetch code/binaries from the internet and can modify your Python environment; run in an isolated environment (virtualenv, venv, or VM/container) and do not run as root. - Review the scripts before running. The auto-install and download behavior is explicit in the code (os.system calls). If you are uncomfortable with automatic installs, manually create a clean virtualenv, install the pinned requirements.txt, and run the scripts yourself. - Network and cost implications: the extraction step makes many LLM API calls (haiku/sonnet). Dry-run mode is available to validate chunking without using API credits — use it first. - Generated server.py runs locally and currently has no built-in authentication (TODO mentions adding optional API key). If you plan to expose the server beyond localhost, add authentication or run behind a firewall. - If you intend to crawl private or internal documentation, consider privacy and legal implications (and whether you should store that data). The crawler respects robots.txt per the plan, but you are still executing broad scraping on a target origin. Recommendations: run only in an isolated environment, inspect/modify the scripts to remove automatic pip installs if you prefer manual dependency control, ensure you provide and secure your ANTHROPIC_API_KEY, and verify generated server code before making it accessible beyond your machine.

功能分析

Type: OpenClaw Skill Name: graph-rag-builder Version: 1.0.0 The skill bundle implements a functional GraphRAG pipeline but contains high-risk patterns, specifically the use of 'os.system' to execute shell commands for automated dependency installation across almost all scripts (crawl.py, build_graph.py, build_embeddings.py, extract_concepts.py, and the generated server.py). While these appear to be intended for user convenience to 'auto-bootstrap' the environment, they represent a significant security vulnerability (potential RCE) and bypass standard package management safety. There is no clear evidence of malicious intent or data exfiltration, but the practice of scripts self-installing packages with '--break-system-packages' via shell execution warrants a suspicious classification.

能力标签

cryptocan-make-purchasesrequires-sensitive-credentials

能力评估

✓ Purpose & Capability

The name/description (build an MCP knowledge server from a URL) align with the included scripts: crawler, concept extractor (LLM calls), graph builder, embeddings builder, and MCP server generator. The pipeline and outputs described in SKILL.md match the code files present.

⚠ Instruction Scope

SKILL.md and README instruct the user to run the provided scripts locally and to set ANTHROPIC_API_KEY for concept extraction. However the skill metadata declared no required environment variables. The instructions also tell users to edit a specific local Claude Desktop config path (~/Library/Application Support/Claude/claude_desktop_config.json). The agent instructions are broad (crawl arbitrary sites) and will direct the user to run network- and I/O-heavy operations; that scope is expected for this task but the missing env-var declaration and the fact the code bootstraps installs at runtime are concerning.

⚠ Install Mechanism

There is no formal install spec, but many scripts perform runtime dependency bootstrapping via os.system('pip install ...'). generate_mcp_server's emitted server also attempts to auto-install missing packages. The crawler will call 'playwright install chromium' when needed. These runtime installs will fetch and execute code from PyPI and download large browser/model artifacts — this increases attack surface and may modify the system environment. The use of pip with --break-system-packages was observed in the code, which can affect system-managed Python packages; that's higher-risk than assuming a controlled virtualenv.

⚠ Credentials

Functionally the only actual secret required for normal operation is ANTHROPIC_API_KEY (used by extract_concepts.py and documented in .env.example and SKILL.md). However the skill's registry metadata lists no required env vars or primary credential — a mismatch. Aside from that single API key, the pipeline is local (uses local sentence-transformers by default) and does not require unrelated credentials. The omission of ANTHROPIC_API_KEY from the declared requirements is an inconsistency worth noting.

ℹ Persistence & Privilege

The skill is not marked 'always:true' and autonomous invocation is allowed (the platform default). The skill's scripts create files in an output directory, download models and Chromium, and write a generated server.py that a user can run. The skill itself does not request system-wide persistent privileges or attempt to modify other skills' configs, but it does instruct users to merge generated JSON into a local Claude Desktop config file (a manual step). Consider the broader risk because the skill will cause persistent artifacts on disk and perform network downloads if executed.

版本历史

v1.0.0

GraphRAG Builder Skill 1.0.0 - Initial release: build a full MCP knowledge server from any website or documentation URL in five local pipeline steps. - Includes scripts for crawling sites, extracting concepts with LLM (Opus, Sonnet, etc), constructing a knowledge graph, generating vector embeddings, and outputting a ready-to-run MCP server. - Supports semantic search and eight graph-based retrieval tools out of the box. - Guides users step-by-step, including setup, requirements, error handling, and sample command sequences. - Designed for easy integration with agent harnesses via generated config snippets.

元数据

Slug graph-rag-builder

版本 1.0.0

许可证 MIT-0

累计安装 0

当前安装数 0

历史版本数 1

常见问题

GraphRAGBuilder 是什么？

Builds a fully runnable MCP (Model Context Protocol) knowledge server from any website or documentation URL. Crawls the site, extracts concepts using Claude,... 它是一个面向 Claude Code / OpenClaw 的 AI Agent Skill 插件，目前累计下载 80 次。

如何安装 GraphRAGBuilder？

在 OpenClaw 或 Claude Code 对话框中运行命令「/install graph-rag-builder」即可一键安装，无需额外配置。

GraphRAGBuilder 是免费的吗？

是的，GraphRAGBuilder 完全免费，采用 MIT-0 许可证，可自由下载、安装和使用。

GraphRAGBuilder 支持哪些平台？

GraphRAGBuilder 跨平台运行，可在任意部署了 OpenClaw / Claude Code 的环境中使用（cross-platform）。

谁开发了 GraphRAGBuilder？

由 nacmonad（@nacmonad）开发并维护，当前版本 v1.0.0。

GraphRAGBuilder

GraphRAG Builder Skill

Quick Reference

Step 0: Clarify Requirements

Step 1: Crawl (M1)

Step 2: Extract Concepts (M2)

Step 3: Build Graph (M3)

Step 4: Build Embeddings (M4)

Step 5: Generate MCP Server (M5)

The 8 MCP Tools

Complete Pipeline Command Sequence

Output Directory Layout

Troubleshooting

Deferred Features

GraphRAGBuilder 是什么？

如何安装 GraphRAGBuilder？

GraphRAGBuilder 是免费的吗？

GraphRAGBuilder 支持哪些平台？

谁开发了 GraphRAGBuilder？

💬 留言讨论