← 返回 Skills 市场

数据管道工具箱

Name: 数据管道工具箱
Author: yesong-hue

作者 yesong-Hue · GitHub ↗ · v1.0.0 · MIT-0

cross-platform ✓ 安全检测通过

总下载

当前安装

版本数

在 OpenClaw 中安装

/install data-pipeline-toolkit-v2

功能描述

快速构建ETL数据管道 — 提取(APIs/数据库/文件)、转换(清洗/过滤/聚合)、加载(数据仓库)，支持定时调度和监控告警。

使用说明 (SKILL.md)

数据管道工具箱

快速构建ETL数据管道：提取 → 转换 → 加载 → 调度

核心功能

多源提取 — REST APIs、GraphQL、SQL数据库、CSV/JSON/Parquet文件、S3/云存储、Kafka/SQS
数据转换 — 清洗、过滤、聚合、关联、跨表Join
多目标加载 — PostgreSQL/MySQL、Snowflake/BigQuery、S3、数据仓库
定时调度 — Cron任务或事件触发
监控告警 — 失败自动通知，可视化运行状态

快速开始

# 创建数据管道
./pipeline.sh create my-pipeline

# 添加数据源
./pipeline.sh extract my-pipeline api --url https://api.example.com/data

# 添加转换规则
./pipeline.sh transform my-pipeline filter "status == 'active'"
./pipeline.sh transform my-pipeline aggregate "group by category, sum(amount)"

# 添加目标存储
./pipeline.sh load my-pipeline postgres --connection $DATABASE_URL

# 运行管道
./pipeline.sh run my-pipeline

支持的数据源

类型	具体来源
APIs	REST API, GraphQL, 内部服务
数据库	PostgreSQL, MySQL, MongoDB, SQL Server
文件	CSV, JSON, Parquet, Excel
云存储	AWS S3, Google Cloud Storage
消息队列	Kafka, AWS SQS

支持的目标存储

类型	具体目标
数据库	PostgreSQL, MySQL, BigQuery, Snowflake
数据仓库	ClickHouse, DuckDB, TimescaleDB
文件存储	S3, GCS, 本地文件
API	第三方API回传

典型使用场景

场景1：每日销售数据汇总

# 从CRM API提取昨日销售数据
./pipeline.sh extract daily-sales api \
  --url "https://crm.example.com/api/orders?date=yesterday"

# 转换：按产品分类汇总
./pipeline.sh transform daily-sales aggregate \
  --group-by "product_category" \
  --sum "quantity,amount"

# 加载到数据仓库
./pipeline.sh load daily-sales bigquery \
  --project "my-project" --dataset "sales" --table "daily_summary"

# 设置每日定时任务
./pipeline.sh schedule daily-sales "0 6 * * *"

场景2：用户行为数据同步

# 从日志文件提取
./pipeline.sh extract user-logs file --path "/var/logs/app/*.json"

# 清洗和转换
./pipeline.sh transform user-logs filter "event_type != 'heartbeat'"
./pipeline.sh transform user-logs add-column "timestamp:parse_timestamp(time)"

# 加载到ClickHouse
./pipeline.sh load user-logs clickhouse --connection $CH_URL

监控与告警

查看运行状态

./pipeline.sh status my-pipeline
# 输出：
# Status: ✅ Running
# Last Run: 2026-05-05 06:00:00
# Duration: 45s
# Records Processed: 12,847
# Errors: 0

配置告警

# 失败时发送邮件
./pipeline.sh alert my-pipeline email --to [email protected]

# 失败时发送飞书消息
./pipeline.sh alert my-pipeline webhook --url "https://open.feishu.cn/..."

推荐资源

ShadowAI API（数据管道配套）: https://referer.shadowai.xyz/r/1056448

由 AI智造工坊 (http://ai.qnitgroup.com) 整理发布 | 安装源: ClawHub

安全使用建议

Before installing or using this skill, verify what ./pipeline.sh is, confirm the correct package slug/source, use least-privilege credentials, approve each data source and destination, and make sure any schedules or webhooks can be disabled.

功能分析

Type: OpenClaw Skill Name: data-pipeline-toolkit-v2 Version: 1.0.0 The bundle consists only of metadata and documentation (SKILL.md) describing an ETL data pipeline tool. No executable code or scripts (such as the referenced 'pipeline.sh') are included in the provided files. While the documentation contains an affiliate referral link (referer.shadowai.xyz), there is no evidence of malicious intent, prompt injection, or unauthorized data access.

能力评估

ℹ Purpose & Capability

The described API/database/file extraction, transformation, loading, scheduling, and monitoring are coherent with an ETL toolkit, but they can affect real business data and external systems.

ℹ Instruction Scope

The instructions are mainly user-facing CLI examples and do not contain hidden goal overrides, but the run/load/schedule examples are high-impact actions that should remain user-approved.

ℹ Install Mechanism

There is no install spec or bundled code, yet SKILL.md references a local ./pipeline.sh helper and an install slug that differs from the evaluated registry slug, so users should verify the actual executable/source before running commands.

ℹ Credentials

Use of API, database, file, cloud storage, connection-string, and webhook inputs is expected for ETL, but required credentials/env vars are not declared in metadata and should be least-privilege.

ℹ Persistence & Privilege

Cron/event scheduling and failure alerts are explicit and purpose-aligned; any created jobs or webhooks should be reversible and monitored.

如何使用

确保已安装 OpenClaw（本地或 Docker 部署）
在对话框中输入安装命令：/install data-pipeline-toolkit-v2
安装完成后，直接呼叫该 Skill 的名称或使用 /data-pipeline-toolkit-v2 触发
根据 Skill 的参数说明提供必要输入，即可获得结构化输出

版本历史

v1.0.0

首发版，ETL数据管道：提取-转换-加载-调度

元数据

Slug data-pipeline-toolkit-v2

版本 1.0.0

许可证 MIT-0

累计安装 0

当前安装数 0

历史版本数 1

常见问题