Skip to content

0027-scheduled-poll-bridge.md

Latest commit

 

History

History
51 lines (43 loc) · 3.49 KB

0027-scheduled-poll-bridge.md

File metadata and controls

51 lines (43 loc) · 3.49 KB
Raw

ADR 0027: Scheduled polling — the bridge between due-work scheduling and the pipeline

  • Status: Accepted (2026-08-23)
  • Implements: roadmap M4; closes the last gap ADR-0025/0026 both left open

Context

Three things existed independently by the end of ADR-0026: scheduling.DueWorkIndex/ Dispatcher (ADR-0018) — fully tested, zero dependency on anything downstream; pipeline. run_for_org (ADR-0025) — fully tested, zero dependency on cadence; and tenancy. WatchlistService (ADR-0026) — the source of the WatchlistEntry the pipeline needs. Nothing connected them. A poll only ever happened because a test (or a human) called run_for_org directly; there was no path from "an org is due" to "the pipeline actually ran for it."

Decision

scheduled_poll.py is two functions, not a new subsystem. Consistent with how pipeline.py itself is "wiring, not new judgment" (ADR-0025), this module adds no decisions of its own:

  1. sync_watchlist_due_work — ensures every watchlist entry has a due-work schedule. Idempotent: an already-scheduled org is left untouched, so calling this on every watchlist change never resets a source's cadence back to its base interval.
  2. run_due_item — what a dispatcher-triggered worker calls for one DispatchedItem: resolve the WatchlistEntry by the due-work entry's work_key (the org slug), run the pipeline, reschedule via scheduling.cadence.advance using the org's own tier. "Found something new" for cadence purposes is any(outcome.stage != "already_seen" for outcome in result.outcomes) — a posting reaching the pipeline at all (whether it's later filtered or scored) means the source yielded something the dedupe ledger hadn't seen, which is the signal cadence should speed up on; only a cycle where everything was already-seen counts as a miss.
  3. A removed org fails soft, not hard. If the watchlist entry no longer exists by the time its due-work entry gets dispatched (the user deleted it in between), run_due_item returns None and unschedules the due-work entry instead of raising — the dispatcher already committed to processing this item this cycle; there's nothing useful to retry.
  4. Per-user config resolution is still the caller's job. run_due_item takes **pipeline_kwargs and passes them straight through to run_for_org — which profile, rubric, scorer, and settings a real batch worker resolves for which user in a dispatch cycle is genuinely separate work (a SettingsService/EntitlementsService-driven lookup per DispatchedItem.entry.user_id), not something this bridge module should decide.

Consequences

  • The chain the roadmap's M4 exit criterion describes now runs for real, tested end to end: add a watchlist entry → it gets scheduled → a fair-share dispatch cycle picks it up → the pipeline runs → cadence adapts (resets on a hit, backs off on an all-already-seen miss) → the org is due again later, automatically.
  • What's still missing to make this run unattended is exactly one thing: a compute trigger (EventBridge on a fixed interval, invoking a Lambda that calls Dispatcher.dispatch then run_due_item per item) — pure infra wiring at this point, not a new design decision, the same relationship ApiStack/lambda_handlers/me.py (ADR-0024) already has to tenancy.
  • Nothing here renders or sends anything — PipelineResult.scored entries carry a NotifyDecision, but delivery (SES, a digest email) is still separate, not-yet-built work.