Skip to content

Commit 5ca6399

Browse files
docs: add crawler A/B experiment tracking (7 days)
Signed-off-by: Ikalus1988 <sheldonisspark@gmail.com>
1 parent 1267bc0 commit 5ca6399

1 file changed

Lines changed: 67 additions & 0 deletions

File tree

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Crawler A/B Experiment — 2026-07-14
2+
3+
## Hypothesis
4+
5+
v2.8.0 had crawler PRs because issues had `bounty` label and standard bounty formatting. v2.9.0+ lost crawlers when labels changed to `zero-bounty` / `status:competition`.
6+
7+
## Experiment Design
8+
9+
**Start:** 2026-07-14
10+
**End:** 2026-07-21 (7 days)
11+
**Metric:** External PRs, issue comments, claims
12+
13+
### Experimental Group (6 issues)
14+
15+
Enhanced with: `bounty` label + `[Bounty][$0]` title prefix + reward text in body
16+
17+
| Issue | Title | Type |
18+
|-------|-------|------|
19+
| #429 | [Bounty][$0][agent competition] Blind test homepage SAG-Lite search | Test |
20+
| #459 | [Bounty][$0][Benchmark] Run LessonReuseBench | Benchmark |
21+
| #290 | [Bounty][$0][Docs] 5-minute quickstart | Docs |
22+
| #302 | [Bounty][$0][Search] Show match reason | Search |
23+
| #304 | [Bounty][$0][Search] --json output | Search |
24+
| #257 | [Bounty][$0] Translate Chinese lessons | Content |
25+
26+
### Control Group (remaining issues)
27+
28+
Standard labels: `status:competition`, `agent-friendly`, no `bounty` label
29+
30+
| Issue | Title | Type |
31+
|-------|-------|------|
32+
| #352 | Decide verified/ semantics | Protocol |
33+
| #354 | Add trust_tiers | Protocol |
34+
| #318 | Cursor integration | Ecosystem |
35+
| #317 | VS Code extension | Ecosystem |
36+
| #313 | Query expansion | Search |
37+
| #314 | Typo tolerance | Search |
38+
| ... | (others) | Various |
39+
40+
## Baseline (pre-experiment)
41+
42+
| Metric | Experimental | Control |
43+
|--------|-------------|---------|
44+
| External PRs | 0 | 0 |
45+
| Issue comments | 0 | 0 |
46+
| Claims | 0 | 0 |
47+
48+
## Day 1 (2026-07-14)
49+
50+
- Added `bounty` label to all 22 competition issues
51+
- Updated 6 experimental issues with `[Bounty][$0]` title + reward text
52+
- Waiting for crawler discovery
53+
54+
## Day 7 (2026-07-21)
55+
56+
(TODO: fill in results)
57+
58+
| Metric | Experimental | Control |
59+
|--------|-------------|---------|
60+
| External PRs | | |
61+
| Issue comments | | |
62+
| Claims | | |
63+
| Unique contributors | | |
64+
65+
## Conclusion
66+
67+
(TODO: fill in after 7 days)

0 commit comments

Comments
 (0)