rag-eval

Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).

This skill appears to do what it claims, but take these precautions before installing or running it: 1) Inspect the included scripts locally (scripts/run_eval.py, scripts/batch_eval.py, scripts/setup.sh) — don't run arbitrary shell scripts without review. 2) Use a Python virtual environment (python -m venv .venv; source .venv/bin/activate) before running setup.sh to avoid global pip installs. 3) Protect your LLM keys — the skill uses OPENAI_API_KEY/ANTHROPIC_API_KEY to call remote LLMs; grant least-privilege keys where possible and monitor usage. 4) The tool writes evaluation files to memory/eval-results in the working directory; verify this location suits your data-retention policies. 5) There is a truncated/possibly buggy section in the provided run_eval.py excerpt (the sample here was truncated) — ensure you have the complete, reviewed script before running explain/advanced features. 6) Expect runtime costs for LLM judge calls. If you need higher assurance, ask for a line-by-line code review or a reproducible test run in an isolated environment.

Type: OpenClaw Skill Name: rag-eval Version: 1.2.1 The skill is classified as suspicious due to a potential prompt injection vulnerability against the LLM judge in `scripts/run_eval.py`. The `explain_faithfulness` function constructs a prompt using user-controlled `answer` and `contexts`, which could allow an attacker to manipulate the LLM judge's behavior or extract information from it. While the skill developers have demonstrated good security practices by explicitly fixing a previous shell injection vulnerability (noted in `CHANGELOG.md`) and implementing secure input handling (JSON parsing instead of shell interpolation, as instructed in `SKILL.md`), the LLM interaction pattern remains a vulnerability inherent to its design.