Skip to content

pr-genius-observation.md

Latest commit

 

History

History
123 lines (95 loc) · 5.22 KB

pr-genius-observation.md

File metadata and controls

123 lines (95 loc) · 5.22 KB
Raw
type Maintainer Document
title PR Genius Advisory Signal — 10 PR Observation Report
description Empirical evaluation of PR Genius v1.3.1 advisory signal after 10+ PRs
created 2026-08-03
author zsxh1990
related_issue Ikalus1988#756
related_pr Ikalus1988#773

PR Genius Advisory Signal — Observation Report

Context

This document records observations from 12 real PRs monitored by PR Genius v1.3.1 (Status Mode + GraphQL batch query + Action JSON contract).

Observation period: 2026-07-03 to 2026-08-03 (31 days).

PR Observations

# PR Repo PR Genius Risk Human Conclusion Useful? Type
1 #773 Ikalus1988/MisakaNet high_risk DCO + audit failure, needed signoff fix ✅ Yes TP
2 #2121 harbor-framework/harbor high_risk CI failure (uv.lock drift), needed lockfile update ✅ Yes TP
3 #2305 HolmesGPT/holmesgpt high_risk CLA + DCO + FOSSA compliance issues ✅ Yes TP
4 #2902 soxoj/maigret medium_risk Changes requested, fix pushed, waiting re-review ✅ Yes TP
5 #1309 mongodb-js/mongodb-mcp-server low_risk Stale review, approved but blocked by branch protection ✅ Yes TP
6 #2 odebo/mindbook low_risk 30d no review, maintainer unresponsive ✅ Yes TP
7 #38 Rose22/openlumara low_risk 20d no review, maintainer unresponsive ✅ Yes TP
8 #143 qdrant/mcp-server-qdrant low_risk 24d no review, stale ✅ Yes TP
9 #801 plastic-labs/honcho low_risk 17d waiting for review ✅ Yes TP
10 #1190 RailtownAI/railtracks low_risk In review, no issues ✅ Yes TN
11 #221 JasperHG90/memex low_risk In review, no issues ✅ Yes TN
12 #3473 microg/GmsCore high_risk 49d stale, NEEDS_REBASE, likely abandon candidate ✅ Yes TP

Legend

  • TP = True Positive (PR Genius correctly flagged a real issue)
  • TN = True Negative (PR Genius correctly reported no issue)
  • FP = False Positive (PR Genius flagged an issue that wasn't real)
  • FN = False Negative (PR Genius missed a real issue)

Results

Metric Value
Total PRs observed 12
True Positive (correctly flagged) 10
True Negative (correctly clear) 2
False Positive 0
False Negative 0
Accuracy 100% (12/12)
Actionable rate 83% (10/12 needed action or awareness)

Key Findings

1. CI/DCO/CLA detection is the strongest signal

PRs #773, #2121, #2305 all had compliance issues that PR Genius caught immediately. Without PR Genius, #773's DCO failure would have been discovered later (or not at all). The lockfile drift in #2121 would have required manual CI log inspection.

2. Stale detection prevents forgotten PRs

PRs #2, #38, #143, #3473 were correctly flagged as stale. Without monitoring, these could sit indefinitely. The STALE_NO_REVIEW vs STALE_REVIEW distinction is valuable: it tells you whether to ping the maintainer (stale review, they reviewed but you fixed and they didn't re-review) vs consider abandoning (no one ever looked).

3. No false positives observed

In 31 days of monitoring, PR Genius did not flag any false alarms. Every "actionable" item was genuinely actionable. This is important for trust: if the tool cried wolf, it would be ignored.

4. Transition tracking (new in v1.3.1) needs more data

The --save-snapshot + transition detection was added late in the observation period. After 2+ runs, it correctly identifies status changes (e.g., WAITING → NEEDS_REBASE). This feature is promising but needs more observation cycles.

5. v1.0.0 was not useful

The original v1.0.0 action mostly reported unknown tier because analyze --format json had a silent fallback bug. This is fixed in v1.3.1.

Recommendation

Keep PR Genius v1.3.1 as advisory-only in MisakaNet CI.

Reasons:

  • 100% accuracy over 12 PRs (no false positives/negatives)
  • Catches real compliance issues (DCO, CLA, lockfile drift)
  • Identifies stale PRs that would otherwise be forgotten
  • Low overhead (GraphQL batch query, ~5s per heartbeat)
  • Advisory-only means zero risk: it never blocks merge

Constraints to maintain:

  • Advisory only: never become a merge blocker
  • Low permissions: contents: read only
  • Skip draft/docs-only PRs
  • No auto-close or auto-request-changes
  • If the tool has an internal failure, it should not block the PR workflow

Suggested next step: pin action to commit SHA for reproducibility.

Appendix: Observation Timeline

Date Event
2026-07-03 PR Genius v1.0.0 deployed to MisakaNet CI
2026-07-14 First observation: PR #3 (FANUC lessons), tier=unknown ❌
2026-07-18 v1.2.0: Status Mode MVP released
2026-07-24 v1.3.0: Daily content expansion
2026-07-28 v1.3.1: Action JSON contract fix (tier no longer unknown) ✅
2026-08-02 MisakaNet #773: tier=high_risk, caught DCO/audit failure ✅
2026-08-02 harbor #2121: CI_FAILING, caught lockfile drift ✅
2026-08-02 holmesgpt #2305: CI_FAILING, caught CLA/DCO issues ✅
2026-08-03 This report: 12 PRs, 100% accuracy
2026-08-06 Batch 2 observation started (#781): 5 PRs queued (#852, #851, #850, #848, #837). See pr-genius-v1.3.1-observation-batch2.md