SkillProbe

A/B evaluates any AI agent skill's real impact through three-role isolation (orchestrator + two sub-agents). Generates skill profiles, synthetic test tasks,...

This skill is internally coherent and does what it claims: it A/B evaluates other skills by sending tasks and the target skill's content to your configured LLM provider and scoring the outputs locally. Before running: (1) do not include secrets, API keys, or sensitive credentials in the SKILL.md or the skill bundle you evaluate — those will be sent to the LLM provider; (2) confirm which LLM provider/runtime is configured and whether you trust its data handling; (3) if using the local CLI helper, be aware it may read runtime/provider config or env vars needed by your LLM client; (4) run evaluations in a sandbox or with redacted skill content if you need to protect sensitive artifacts. If you want an extra safety check, inspect the specific skill bundle being evaluated and remove any embedded secrets before using SkillProbe.

Type: OpenClaw Skill Name: skillprobe Version: 1.0.0 The skillprobe bundle provides a structured framework for A/B testing and evaluating other AI agent skills using a multi-agent orchestration approach. The logic involves an orchestrator agent generating tasks and dispatching them to isolated sub-agent sessions to compare performance with and without a specific skill, as detailed in SKILL.md and DISPATCH_PROTOCOL.md. While the evaluate.sh script performs basic input sanitization for its CLI arguments, no evidence of malicious intent, data exfiltration, or unauthorized remote execution was found; the behavior is consistent with its stated purpose of skill benchmarking.