← Back to Skills Marketplace
数据管道工具箱
by
yesong-Hue
· GitHub ↗
· v1.0.0
· MIT-0
47
Downloads
0
Stars
0
Active Installs
1
Versions
Install in OpenClaw
/install data-pipeline-toolkit-v2
Description
快速构建ETL数据管道 — 提取(APIs/数据库/文件)、转换(清洗/过滤/聚合)、加载(数据仓库)，支持定时调度和监控告警。
README (SKILL.md)
数据管道工具箱
快速构建ETL数据管道：提取 → 转换 → 加载 → 调度
核心功能
- 多源提取 — REST APIs、GraphQL、SQL数据库、CSV/JSON/Parquet文件、S3/云存储、Kafka/SQS
- 数据转换 — 清洗、过滤、聚合、关联、跨表Join
- 多目标加载 — PostgreSQL/MySQL、Snowflake/BigQuery、S3、数据仓库
- 定时调度 — Cron任务或事件触发
- 监控告警 — 失败自动通知，可视化运行状态
快速开始
# 创建数据管道
./pipeline.sh create my-pipeline
# 添加数据源
./pipeline.sh extract my-pipeline api --url https://api.example.com/data
# 添加转换规则
./pipeline.sh transform my-pipeline filter "status == 'active'"
./pipeline.sh transform my-pipeline aggregate "group by category, sum(amount)"
# 添加目标存储
./pipeline.sh load my-pipeline postgres --connection $DATABASE_URL
# 运行管道
./pipeline.sh run my-pipeline
支持的数据源
| 类型 | 具体来源 |
|---|---|
| APIs | REST API, GraphQL, 内部服务 |
| 数据库 | PostgreSQL, MySQL, MongoDB, SQL Server |
| 文件 | CSV, JSON, Parquet, Excel |
| 云存储 | AWS S3, Google Cloud Storage |
| 消息队列 | Kafka, AWS SQS |
支持的目标存储
| 类型 | 具体目标 |
|---|---|
| 数据库 | PostgreSQL, MySQL, BigQuery, Snowflake |
| 数据仓库 | ClickHouse, DuckDB, TimescaleDB |
| 文件存储 | S3, GCS, 本地文件 |
| API | 第三方API回传 |
典型使用场景
场景1：每日销售数据汇总
# 从CRM API提取昨日销售数据
./pipeline.sh extract daily-sales api \
--url "https://crm.example.com/api/orders?date=yesterday"
# 转换：按产品分类汇总
./pipeline.sh transform daily-sales aggregate \
--group-by "product_category" \
--sum "quantity,amount"
# 加载到数据仓库
./pipeline.sh load daily-sales bigquery \
--project "my-project" --dataset "sales" --table "daily_summary"
# 设置每日定时任务
./pipeline.sh schedule daily-sales "0 6 * * *"
场景2：用户行为数据同步
# 从日志文件提取
./pipeline.sh extract user-logs file --path "/var/logs/app/*.json"
# 清洗和转换
./pipeline.sh transform user-logs filter "event_type != 'heartbeat'"
./pipeline.sh transform user-logs add-column "timestamp:parse_timestamp(time)"
# 加载到ClickHouse
./pipeline.sh load user-logs clickhouse --connection $CH_URL
监控与告警
查看运行状态
./pipeline.sh status my-pipeline
# 输出：
# Status: ✅ Running
# Last Run: 2026-05-05 06:00:00
# Duration: 45s
# Records Processed: 12,847
# Errors: 0
配置告警
# 失败时发送邮件
./pipeline.sh alert my-pipeline email --to [email protected]
# 失败时发送飞书消息
./pipeline.sh alert my-pipeline webhook --url "https://open.feishu.cn/..."
推荐资源
- ShadowAI API（数据管道配套）: https://referer.shadowai.xyz/r/1056448
由 AI智造工坊 (http://ai.qnitgroup.com) 整理发布 | 安装源: ClawHub
Usage Guidance
Before installing or using this skill, verify what ./pipeline.sh is, confirm the correct package slug/source, use least-privilege credentials, approve each data source and destination, and make sure any schedules or webhooks can be disabled.
Capability Analysis
Type: OpenClaw Skill
Name: data-pipeline-toolkit-v2
Version: 1.0.0
The bundle consists only of metadata and documentation (SKILL.md) describing an ETL data pipeline tool. No executable code or scripts (such as the referenced 'pipeline.sh') are included in the provided files. While the documentation contains an affiliate referral link (referer.shadowai.xyz), there is no evidence of malicious intent, prompt injection, or unauthorized data access.
Capability Assessment
Purpose & Capability
The described API/database/file extraction, transformation, loading, scheduling, and monitoring are coherent with an ETL toolkit, but they can affect real business data and external systems.
Instruction Scope
The instructions are mainly user-facing CLI examples and do not contain hidden goal overrides, but the run/load/schedule examples are high-impact actions that should remain user-approved.
Install Mechanism
There is no install spec or bundled code, yet SKILL.md references a local ./pipeline.sh helper and an install slug that differs from the evaluated registry slug, so users should verify the actual executable/source before running commands.
Credentials
Use of API, database, file, cloud storage, connection-string, and webhook inputs is expected for ETL, but required credentials/env vars are not declared in metadata and should be least-privilege.
Persistence & Privilege
Cron/event scheduling and failure alerts are explicit and purpose-aligned; any created jobs or webhooks should be reversible and monitored.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install data-pipeline-toolkit-v2 - After installation, invoke the skill by name or use
/data-pipeline-toolkit-v2 - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.0
首发版，ETL数据管道：提取-转换-加载-调度
Metadata
Frequently Asked Questions
What is 数据管道工具箱?
快速构建ETL数据管道 — 提取(APIs/数据库/文件)、转换(清洗/过滤/聚合)、加载(数据仓库)，支持定时调度和监控告警。 It is an AI Agent Skill for Claude Code / OpenClaw, with 47 downloads so far.
How do I install 数据管道工具箱?
Run "/install data-pipeline-toolkit-v2" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is 数据管道工具箱 free?
Yes, 数据管道工具箱 is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does 数据管道工具箱 support?
数据管道工具箱 is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created 数据管道工具箱?
It is built and maintained by yesong-Hue (@yesong-hue); the current version is v1.0.0.
More Skills