\r \r

AI Agent Evaluator\r

\r Your expert companion for evaluating, benchmarking, and improving AI agents.\r \r In 2026, AI agents are deployed in production at scale — but most teams lack systematic ways\r to measure their reliability, safety, and real-world performance. This skill bridges that gap\r by guiding you through rigorous, structured agent evaluation workflows.\r \r ---\r \r

What This Skill Does\r

\r

• Evaluation Suite Design — Build custom test suites tailored to your agent's domain\r (coding, customer support, research, data analysis, etc.)\r
• Benchmark Analysis — Interpret industry benchmarks (SWE-Bench, AgentBench, WebArena,\r BFCL, ToolBench) and map them to your use case\r
• Multi-Framework Comparison — Compare CrewAI, LangChain, AutoGen, LlamaIndex, and\r OpenAI Assistants across cost, latency, and task success rate\r
• Failure Mode Analysis — Systematically identify where and why your agent fails\r
• Red Teaming Support — Design adversarial tests to probe agent safety and edge cases\r
• Evaluation Report Generation — Produce structured reports with scores, recommendations,\r and improvement roadmap\r \r ---\r \r

Trigger Phrases\r

\r English:\r

• "evaluate my AI agent"\r
• "benchmark this agent"\r
• "compare CrewAI vs LangChain"\r
• "how to test an AI agent"\r
• "agent quality assurance"\r
• "my agent keeps failing at X"\r
• "design evaluation suite for agent"\r
• "agent red teaming"\r
• "production readiness check for agent"\r \r Chinese / 中文:\r
• AI Agent 评估\r
• 智能体基准测试\r
• Agent 质量保障\r
• 如何测试 AI Agent\r
• 比较 CrewAI 和 LangChain\r
• Agent 失败分析\r
• 大模型 Agent 上线前检查\r
• 智能体对比测试\r
• Agent 红队测试\r \r ---\r \r

Core Workflows\r

\r

Workflow 1: Quick Agent Health Check\r

Input: Agent description, task type, sample inputs/outputs\r Steps:\r

1. Classify your agent type (tool-calling, reasoning, multi-step, RAG-based)\r
2. Define 5 critical success criteria for your domain\r
3. Run 10-question diagnostic on failure patterns\r
4. Output health score + top 3 risks\r \r

Workflow 2: Benchmark Selection & Interpretation\r

Input: Agent capabilities, deployment domain\r Steps:\r

1. Map domain → relevant benchmarks\r
2. Explain benchmark methodology (what it tests, limitations)\r
3. Show current SOTA scores and realistic targets\r
4. Recommend evaluation cadence (dev/staging/production)\r \r

Workflow 3: Custom Evaluation Suite Design\r

Input: Agent goal, available test data, budget/time\r Steps:\r

1. Define evaluation dimensions (accuracy, latency, safety, cost)\r
2. Generate 20-50 representative test cases with ground truth\r
3. Set pass/fail thresholds per dimension\r
4. Recommend tooling (PromptFoo, Maxim AI, DeepEval, Braintrust)\r
5. Provide scoring rubric + analysis template\r \r

Workflow 4: Failure Mode Deep Dive\r

Input: Agent logs, failed task transcripts\r Steps:\r

1. Categorize failures (tool call error, hallucination, loop, context loss, safety block)\r
2. Calculate failure rate by category\r
3. Root cause analysis for top-3 failure patterns\r
4. Actionable fixes: prompt adjustments, retrieval improvements, tool schema corrections\r \r

Workflow 5: Multi-Agent Framework Comparison\r

Input: Use case requirements (e.g., "code review pipeline with 3 agents")\r Steps:\r

1. Score CrewAI / LangChain / AutoGen / LlamaIndex on 8 dimensions\r
2. Estimate cost per 1,000 runs\r
3. Provide side-by-side architecture diagram (text)\r
4. Final recommendation with rationale\r \r ---\r \r

Example Interactions\r

\r User: "I built a customer support agent with GPT-4o. How do I know if it's ready for production?"\r \r Agent Evaluator:\r

Great question. Let's run a production readiness assessment across 5 dimensions:\r

1. Task Completion Rate — target >92% for Tier-1 support queries\r
2. Hallucination Rate — should be \x3C2% on factual product questions\r
3. Escalation Accuracy — correctly identifies when to route to human\r
4. Latency P95 — target \x3C3s for chat use case\r
5. Safety Compliance — passes 50-case adversarial test suite\r \r To start, share 10 sample conversations (anonymized) and I'll generate your baseline scorecard.\r \r ---\r \r User: "Compare CrewAI vs LangChain for a financial report analysis pipeline"\r \r Agent Evaluator:\r Here's a side-by-side for your use case...\r [generates structured comparison table + recommendation]\r \r ---\r \r

Key Concepts Covered\r

\r | Concept | Description |\r |---------|-------------|\r | SWE-Bench | Software engineering task benchmark (GitHub issues) |\r | AgentBench | Multi-domain agent task evaluation suite |\r | BFCL | Berkeley Function Calling Leaderboard |\r | WebArena | Browser automation + web task benchmark |\r | Task Success Rate (TSR) | % of tasks completed correctly end-to-end |\r | Step Success Rate (SSR) | % of individual reasoning steps correct |\r | Hallucination Rate | Frequency of factually incorrect outputs |\r | Grounding Accuracy | Correct attribution to source documents |\r \r ---\r \r

Target Users\r

\r

• AI Engineers building and deploying LLM-based agents\r
• ML Platform Teams establishing evaluation standards\r
• Product Managers making go/no-go decisions on agent releases\r
• QA Engineers new to AI agent testing\r
• Researchers comparing agent frameworks\r \r ---\r \r

Tools & Frameworks Referenced\r

\r

• DeepEval — open-source LLM evaluation framework\r
• PromptFoo — prompt testing and red teaming\r
• Braintrust — evaluation and logging for LLM apps\r
• Maxim AI — agent simulation and observability\r
• LangSmith — LangChain's evaluation and tracing platform\r
• Confident AI — production AI evaluation platform\r \r ---\r \r

Notes & Limitations\r

\r

• This skill provides evaluation methodology and guidance, not direct code execution\r
• Benchmark scores are time-sensitive — always check latest published leaderboards\r
• For production safety evaluations, always involve your security team\r
• Evaluation results should be reviewed by qualified ML engineers before deployment decisions\r \r ---\r \r Built for AI teams who ship agents to production — not just demos.\r Author: @gechengling | version: "3.0.0"\r