agent-autoresearch

Any agent can run this. The experiment is always: change something → measure it → keep what works.

The Core Idea

Karpathy's insight: give an agent a fixed time budget, let it modify one file, measure if things got better, keep or discard, repeat.

Applied to agents: your workspace is train.py. Your SOUL.md, scripts, and skills are the experiment substrate.

PROPOSE → IMPLEMENT → MEASURE → KEEP/KILL → INTEGRATE → REPEAT

You are not just optimizing content. You are optimizing the agent itself.

What Can Be Mutated

The agent can propose changes to any file it owns:

The agent cannot mutate: safety rules, constitution, security boundaries, or files it doesn't own.

Project Structure

agent-autoresearch/
├── SKILL.md                    ← you are here
├── program.md                  ← 🧠 the experiment agent's instructions
├── prepare.py                  ← establish baseline metrics
├── evolve.py                   ← integrate KEEP verdict into agent files
├── analyze.py                  ← compute verdict from measurements
├── baseline.json               ← current agent baseline (performance + strategy)
├── results.tsv                 ← all experiment results (append-only log)
└── experiments/
    ├── meta.json               ← experiment state (next_exp_id, kill_streak)
    ├── active.md               ← one active experiment at a time
    └── archive/                ← completed experiments

🚀 Quick Start

# 1. Establish baseline (measure current agent performance)
python3 prepare.py --metric task_completion_rate --baseline 0.75

# 2. Read the experiment brief
cat program.md

# 3. Start the experiment loop
#    Agent reads program.md, proposes a self-improvement, implements it,
#    measures results, and executes KEEP/KILL verdict.

# Check current state
python3 prepare.py --status

Baseline Metrics

Track what matters for the agent's mission. Examples:

Establish baseline with ≥ 10 measurements before running experiments.

Verdict Logic

improvement = (experiment_score - baseline_score) / baseline_score

≥ +10%  → KEEP  (integrate the change into the agent)
≤ -10%  → KILL  (discard, revert to previous state)
-10% to +10% → MODIFY (extend evaluation or treat as KILL)

For quality/rating metrics (higher is better): above thresholds apply. For cost/latency metrics (lower is better): flip the sign in calculation.

Key Rules

• ❌ One mutation at a time — test one change per experiment
• ❌ No baseline — need ≥10 measurements before experimenting
• ❌ Vibes verdicts — use actual measurements
• ❌ Mutate safety/constitution files — never
• ❌ Kill streak ≥ 3 → pause and wait for human review
• ❌ Infinite MODIFY — max one extension
• ❌ Revert a KEEP — only a newer KEEP overrides

Commands

Security

• Agents can only mutate files within their own workspace
• Safety rules and constitution are always excluded from mutation
• External API calls require human approval
• Destructive operations (rm, git reset --hard) require explicit confirmation