Skip to content

lesson-reuse-benchmark.md

Latest commit

 

History

History
133 lines (100 loc) · 4.06 KB

lesson-reuse-benchmark.md

File metadata and controls

133 lines (100 loc) · 4.06 KB
Raw

LessonReuseBench

Evaluate whether AI agents can learn from past failures and reuse lessons when encountering similar problems.

Why this benchmark exists

Traditional coding benchmarks test: Can the agent fix this bug?

LessonReuseBench tests something more important: Can the agent fix this bug using prior experience, or does it re-debug from scratch?

An agent that forgets past failures will:

  • Re-discover the same fix repeatedly
  • Waste tokens on known dead ends
  • Never accumulate organizational knowledge

An agent that reuses lessons will:

  • Retrieve the right fix faster
  • Avoid known-bad approaches
  • Build on prior debugging sessions

How it works

Each benchmark pair consists of two tasks:

  1. Task A — Agent encounters a failure, fixes it, and generates a lesson
  2. Task B — Agent encounters a similar (but different) failure in a new context

The agent must retrieve and apply the lesson from Task A to solve Task B efficiently.

With vs Without

Each pair is run twice:

  • With lesson pool: Agent has access to MisakaNet search
  • Without lesson pool: Agent must debug from scratch

The score delta measures the value of experience reuse.

Task Pairs

Pair Task A Task B Domain
DCO PR fails DCO check (missing sign-off) Different repo, wrong email in sign-off devops
Secret Scan Token leaked in commit, push blocked GitHub Actions secret exposure variant devops
DB Lock SQLite locked during agent operation Agent state DB locked with different trigger development

Scoring

total = 40% × task_b_pass
      + 20% × correct_lesson_retrieved
      + 15% × avoided_known_bad_path
      + 15% × generated_reusable_lesson
      + 10% × ci/pr_compliance

Score definitions

Dimension What it measures How to check
task_b_pass Did agent solve Task B? Exit code, test result
correct_lesson_retrieved Did agent find the right lesson? Search query logs
avoided_known_bad_path Did agent skip the failed approach? Command/trace analysis
generated_reusable_lesson Did agent write a lesson after Task A? Lesson file exists, passes schema
ci/pr_compliance DCO, format, frontmatter valid CI checks pass

Running

# Dry-run (no API keys needed)
python3 scripts/lesson_reuse_bench.py --dry-run

# Full run
python3 scripts/lesson_reuse_bench.py --agent openai --tasks tasks/reuse/

# Compare with/without lesson pool
python3 scripts/lesson_reuse_bench.py --agent claude --compare

Output

{
  "agent": "openai",
  "timestamp": "2026-07-14T10:00:00Z",
  "pairs": [
    {
      "name": "dco",
      "task_a_pass": true,
      "task_b_pass": true,
      "lesson_retrieved": true,
      "bad_path_avoided": true,
      "lesson_generated": true,
      "ci_compliant": true,
      "score": 0.95
    }
  ],
  "total_score": 0.92,
  "delta_vs_no_lesson": 0.35
}

How to participate

  1. Run the benchmark:

    git clone https://github.com/Ikalus1988/MisakaNet.git
cd MisakaNet
python3 scripts/lesson_reuse_bench.py --dry-run  # validate
python3 scripts/lesson_reuse_bench.py --agent your-agent --compare

  2. Share your results: Open an issue titled [Benchmark] <agent-name> LessonReuseBench results with:

    • Agent name and version
    • Score with lessons vs without lessons
    • Delta (difference)
    • Observations

  3. Submit new task pairs: PR with tasks/reuse/<name>-a.json and tasks/reuse/<name>-b.json

Adding new task pairs

  1. Create tasks/reuse/<name>-a.json (Task A)
  2. Create tasks/reuse/<name>-b.json (Task B)
  3. Each task file needs: name, description, setup, expected_outcome, validation

Related