Agent Learner

An AI toolkit for configuring, benchmarking, comparing, and optimizing agent prompts and evaluation results. Agent Learner provides persistent, file-based logging for each command category with timestamped entries, summary statistics, multi-format export, and full-text search across all records.

Commands

Each data command (configure, benchmark, compare, etc.) works in two modes:

• Without arguments: displays the 20 most recent entries from that category
• With arguments: saves the input as a new timestamped entry and reports the total count

Data Storage

All data is stored in plain text files under the data directory:

• Category logs: $DATA_DIR/\x3Ccommand>.log — one file per command (e.g., configure.log, benchmark.log, prompt.log), each entry is timestamp|value
• History log: $DATA_DIR/history.log — audit trail of every command executed with timestamps
• Export files: $DATA_DIR/export.\x3Cfmt> — generated by the export command in json, csv, or txt format

Default data directory: ~/.local/share/agent-learner/

Requirements

• Bash (with set -euo pipefail support)
• Standard Unix utilities: grep, cat, date, echo, wc, du, head, tail, basename
• No external dependencies or API keys required

When to Use

1. Benchmarking agent performance — When you need to track and compare benchmark results across different agent configurations, models, or prompt strategies
2. Prompt engineering iteration — When you're testing multiple prompt variations and want to log each version with results for later comparison
3. Cost and usage tracking — When you need to monitor API costs and usage metrics over time to optimize spending
4. Fine-tuning experiments — When running fine-tuning sessions and you want to log parameters, results, and observations for reproducibility
5. Cross-category analysis — When you need to search across all logged data (benchmarks, prompts, evaluations, costs) to find patterns or specific entries

Examples

# Initialize and check status
agent-learner status

# Log a benchmark result
agent-learner benchmark "GPT-4o on MMLU: 88.7% accuracy, 1.2s avg latency"

# Log a prompt variation
agent-learner prompt "System: You are a helpful coding assistant. Always explain your reasoning step by step."

# Compare two configurations
agent-learner compare "GPT-4o vs Claude-3.5: GPT-4o 12% faster, Claude 5% more accurate on code tasks"

# Track costs
agent-learner cost "March batch: 12,450 tokens input, 3,200 tokens output, $0.47 total"

# View all recent benchmarks
agent-learner benchmark

# Search across all logs for a specific term
agent-learner search "accuracy"

# Export all data as JSON
agent-learner export json

# View summary statistics
agent-learner stats

# Show recent activity
agent-learner recent

Output

All commands return output to stdout. Export files are written to the data directory:

agent-learner export json   # → ~/.local/share/agent-learner/export.json
agent-learner export csv    # → ~/.local/share/agent-learner/export.csv
agent-learner export txt    # → ~/.local/share/agent-learner/export.txt

Every command execution is logged to $DATA_DIR/history.log for auditing purposes.

Powered by BytesAgain | bytesagain.com | [email protected]