|
| 1 | +# Crawler A/B Experiment — 2026-07-14 |
| 2 | + |
| 3 | +## Hypothesis |
| 4 | + |
| 5 | +v2.8.0 had crawler PRs because issues had `bounty` label and standard bounty formatting. v2.9.0+ lost crawlers when labels changed to `zero-bounty` / `status:competition`. |
| 6 | + |
| 7 | +## Experiment Design |
| 8 | + |
| 9 | +**Start:** 2026-07-14 |
| 10 | +**End:** 2026-07-21 (7 days) |
| 11 | +**Metric:** External PRs, issue comments, claims |
| 12 | + |
| 13 | +### Experimental Group (6 issues) |
| 14 | + |
| 15 | +Enhanced with: `bounty` label + `[Bounty][$0]` title prefix + reward text in body |
| 16 | + |
| 17 | +| Issue | Title | Type | |
| 18 | +|-------|-------|------| |
| 19 | +| #429 | [Bounty][$0][agent competition] Blind test homepage SAG-Lite search | Test | |
| 20 | +| #459 | [Bounty][$0][Benchmark] Run LessonReuseBench | Benchmark | |
| 21 | +| #290 | [Bounty][$0][Docs] 5-minute quickstart | Docs | |
| 22 | +| #302 | [Bounty][$0][Search] Show match reason | Search | |
| 23 | +| #304 | [Bounty][$0][Search] --json output | Search | |
| 24 | +| #257 | [Bounty][$0] Translate Chinese lessons | Content | |
| 25 | + |
| 26 | +### Control Group (remaining issues) |
| 27 | + |
| 28 | +Standard labels: `status:competition`, `agent-friendly`, no `bounty` label |
| 29 | + |
| 30 | +| Issue | Title | Type | |
| 31 | +|-------|-------|------| |
| 32 | +| #352 | Decide verified/ semantics | Protocol | |
| 33 | +| #354 | Add trust_tiers | Protocol | |
| 34 | +| #318 | Cursor integration | Ecosystem | |
| 35 | +| #317 | VS Code extension | Ecosystem | |
| 36 | +| #313 | Query expansion | Search | |
| 37 | +| #314 | Typo tolerance | Search | |
| 38 | +| ... | (others) | Various | |
| 39 | + |
| 40 | +## Baseline (pre-experiment) |
| 41 | + |
| 42 | +| Metric | Experimental | Control | |
| 43 | +|--------|-------------|---------| |
| 44 | +| External PRs | 0 | 0 | |
| 45 | +| Issue comments | 0 | 0 | |
| 46 | +| Claims | 0 | 0 | |
| 47 | + |
| 48 | +## Day 1 (2026-07-14) |
| 49 | + |
| 50 | +- Added `bounty` label to all 22 competition issues |
| 51 | +- Updated 6 experimental issues with `[Bounty][$0]` title + reward text |
| 52 | +- Waiting for crawler discovery |
| 53 | + |
| 54 | +## Day 7 (2026-07-21) |
| 55 | + |
| 56 | +(TODO: fill in results) |
| 57 | + |
| 58 | +| Metric | Experimental | Control | |
| 59 | +|--------|-------------|---------| |
| 60 | +| External PRs | | | |
| 61 | +| Issue comments | | | |
| 62 | +| Claims | | | |
| 63 | +| Unique contributors | | | |
| 64 | + |
| 65 | +## Conclusion |
| 66 | + |
| 67 | +(TODO: fill in after 7 days) |
0 commit comments