Normative reference (WS-A). This document is the single source of truth for the canonical memory vocabulary, Memories record schema, and legal state-combination matrix. It supersedes scattered legacy codename-era docs for domain terminology. Implementation types live in
backend/models/memory_domain.py.Runtime architecture (capture → consolidate → promote → read): docs/doc/developer/backend/canonical_memory_architecture.md (visual: HTML companion).
flowchart TD
subgraph upstream [Upstream - not memory]
A[audio / screen / files]
Conv[Conversation session record]
Transcript[processed transcript_segments]
Derived[structured overview / action_items on doc]
end
subgraph memories [Memories - one store, layer-tagged]
ST[Short-term layer: extractions + TTL/decay]
LT[Long-term layer: durable facts]
AR[archive state on long-term]
end
subgraph workflow [Workflow - not memory]
Tasks[action_items]
Goals[goals]
end
A --> Transcript
Transcript --> Conv
Conv --> ST
Conv --> Tasks
Conv --> Goals
ST -- promote --> LT
LT -- age-out --> AR
| Term | Means | Lifecycle | Default-visible? |
|---|---|---|---|
| Conversation | Persisted session record at users/{uid}/conversations: processed transcript_segments, session metadata (structured, apps_results), audio/photo linkage. Upstream of memory. |
in_progress → processing → completed; user can delete whole session |
N/A — Conversations tab, not Memories |
| Capture session | Ephemeral listen/recording window (WebSocket lifetime). For voice paths, 1:1 with a Conversation stub created at listen start. Use this term when distinguishing runtime capture from the persisted record. | Ends when recording stops | N/A |
| Raw input | True source capture: audio in GCS, screenshots/files. Conversation docs hold processed transcripts (STT, diarization, speaker attribution) — not pristine raw audio/text. | Retained per recording/privacy policy | N/A — never surfaced as "memory" |
| Short-term memory (Layer 1) | Structured extractions in Memories, tagged layer=short_term. Observations tied to a source (usually a Conversation via evidence[].source_id). |
Born on extraction; TTL/decay; promoted to Long-term or expires | Yes |
| Long-term memory (Layer 2) | Durable facts in Memories, tagged layer=long_term (e.g. "Name is David Zhang", "Based in Seattle"). |
Promotion/consolidation or direct user assertion; may age to Archive | Yes |
| Archive | Aged-out long-term (layer=archive or terminal state); kept for recall but not shown by default. |
Terminal unless explicitly resurfaced | No (explicit opt-in only) |
| Workflow | Action items and goals — task state, due dates, integrations, progress. Not memory layers. | Task: pending → done; Goal: active → ended | Yes (dedicated UX) |
| Conversation | Session (informal) | |
|---|---|---|
| Exists in code? | Yes — Conversation model, Firestore collection, API, UI |
No persisted memory-domain type; overloaded elsewhere (ChatSession, auth session, focus session) |
| Role | Concrete session record for transcript/audio capture | Abstract provenance boundary or ephemeral capture window |
| Relationship | For voice/listen: capture session creates → Conversation doc | Memory extractions cite Conversation as source_id |
| Merge with Memories? | No — stays upstream | N/A |
Unrelated "session" domains (do not conflate with Conversation): ChatSession (AI chat),
StoredFocusSession (desktop focus/screen), auth/checkout/MCP protocol sessions.
| Use of "archive" | Means | Canonical handling |
|---|---|---|
layer=archive |
Aged-out long-term memory, kept for recall, hidden by default | The only product meaning of "archive" |
L1MemoryArchiveItem / working-memory "archive" |
A processing-pipeline extraction artifact (working_memory.py) |
Internal only; rename per terminology retirement; not the product Archive layer |
| Audio / conversation retention "archive" | Raw-input storage/retention policy | Upstream (not memory); never layer=archive |
- Memories is one store; layer (
short_term/long_term/archive) is a field on each record. Layer drives lifecycle, TTL, promotion, and UI badges — not which collection you query. - A client cache record without an authoritative canonical lifecycle is not an implicit Long-term memory. Canonical product surfaces may read only explicitly layered records; untiered legacy and local-pending records remain a compatibility/provenance concern until an authoritative read or create receipt supplies their lifecycle.
- Conversations are never Memories. No merge of Conversations tab into Memories.
- Promotion is an explicit Short-term → Long-term transition (corroboration, consolidation, or user assertion) within Memories — audited, not a silent flag flip.
- Non-durable / rejected extractions stay Short-term or are pruned; they never reach Long-term.
- Workflow (
action_items,goals) is extracted from the same seam as Memories but stored separately. Long-term may absorb a fact about a commitment; the task/goal row stays in workflow. - Conversation delete cascades to evidence tombstoning on linked Short-term items (
tombstone_source).
| Old / internal | Canonical |
|---|---|
layer 1, L1, "extracted conversation" |
Short-term memory (layer=short_term) |
layer 2, L2, durable memories rows |
Long-term memory (layer=long_term) |
memory, "new memory system" |
(drop the codename) the canonical system |
memory_items + short_term + legacy memories |
One Memories store with layer field (canonical cohort) |
| bare "session" in memory docs | Conversation (persisted) or capture session (ephemeral) |
action_items, goals |
Workflow — unchanged collections |
| Era | What it is | Key identifiers today | Canonical mapping | Disposition |
|---|---|---|---|---|
| Legacy flat memories | Original production store + extractor | users/{uid}/memories, MemoryDB, new_memories_extractor, /v3/memories |
Long-term in unified Memories (layer=long_term) |
Migrate → Retire store |
| Legacy categories | Old taxonomy on legacy rows | core, hobbies, lifestyle, work, skills, learnings, … |
Keep as category metadata; UI filters use primary four (interesting, system, manual, workflow) |
Keep (not layers) |
| Shadow short_term (retired) | Was interim shadow write path (OMI_MEMORY_SHORT_TERM_SHADOW_ENABLED) |
users/{uid}/short_term collection may still hold historical rows |
Short-term in unified Memories (layer=short_term) |
Retired write path; collection cleanup is separate |
| Canonical product memory | Tiered store + ledger | memory_items, MemoryTier, memory_commits, neutral memory_* modules |
Canonical Memories store | Rename complete; store is canonical |
| Rollout modes | Gradual rollout control | off / shadow / write / read, MEMORY_MODE (compat), MEMORY_ENABLED_USERS, memory_control/state |
MemorySystem + resolve_memory_system(uid) |
Collapse |
tier product field |
Persisted item field | short_term / long_term / archive on memory_items |
layer (same semantics) |
Rename API + clients |
memory_reads.py |
Merges legacy + shadow for reads | split-brain reader shim | Single Memories query by layer |
Retire |
Normative reference (locked 2026-06-18): docs/epics/memory_normative_architecture.md
— product tiers are exactly short_term, long_term, archive; context_only is not a tier.
The legacy pipeline introduced L1/L2 as processing stages — not the same as product Short-term/Long-term.
| Internal term (retire in product/docs) | Code locations | Means | Canonical term |
|---|---|---|---|
L1, L1MemoryArchiveItem, WorkingMemoryObservation |
working_memory.py, memory_contracts.py |
Working-memory / archive extraction candidates | Working observation or short-term candidate |
L2, L2MemoryRoute, durable_memory_patch* |
l2_memory_routes.py, durable_memory_patches.py |
Durable synthesis / promotion routing | Promotion proposal / consolidation route |
LifecycleState.working |
memory_contracts.py |
In-flight extraction state | Internal only; not a product layer |
context_only |
projections, route hints | Processing outcome | Not a tier — normalize to Archive or non-default outcome |
processing_state |
pending / processed / blocked |
Item processing pipeline | Keep internal; separate from layer |
status |
active / superseded / tombstoned |
Record lifecycle | Keep; distinct from layer |
| System | Location | Relationship |
|---|---|---|
memory_ingestion pipeline |
backend/utils/memory_ingestion/ |
Benchmark-oriented extraction (WorkingMemoryCandidate, working_memory_candidate.v1). Align source_type; not a separate product store |
| Benchmark v10–v15 | omi-ingestion-benchmark repo |
Memory cards, L1 spike, L2 evidence packaging. Feeds durable_memory_patches via drift guard. Benchmark-only — never leak v13/v14 into production domain |
| System | Store / module | Disposition |
|---|---|---|
| Conversations | users/{uid}/conversations |
Upstream session records |
| Action items / goals | action_items, goals |
Workflow — unchanged |
| Knowledge graph | Neo4j / knowledge_graph.py |
Derived from long-term facts; invalidation on delete/reprocess (WS-J) |
| Trends | trends_db |
Separate derived index from conversations |
| Legacy conversation shims | plugins_results, processing_memory_id |
Mirrored from apps_results / processing_conversation_id; retire when old clients age out |
| API today | Role | After migration |
|---|---|---|
/v3/memories |
Primary legacy REST | Keep route shape for parity; dispatch via MemoryService |
/memory/memory/search, /vector/search, /archive/search |
Canonical-memory reads (legacy /memory/ prefix retained pending sign-off) |
Fold into neutral memory API; drop memory path prefix when clients migrate |
/v1/mcp/memories, /v1/tools/memories |
Surface adapters | Route through seam (WS-L) |
No active /v1 or /v2 memories REST API — /v3 is the legacy product surface.
| Retire | Replace with | WS |
|---|---|---|
memvec: vector prefix (stored IDs — migrate per rollout Q5) |
memory / canonical_memory / neutral vector IDs |
WS-G, WS-J |
tier (product field on items) |
layer |
WS-G, WS-F |
L1, L2, layer1, layer2 in product/UI context |
Short-term / Long-term / promotion | WS-G, WS-F |
L1MemoryArchiveItem, WorkingMemoryObservation in docs/comments |
working observation / short-term candidate | WS-G |
durable_memory_patch, L2MemoryRoute in docs/comments |
promotion proposal / consolidation route | WS-G |
context_only as a user-visible tier |
Archive or internal processing outcome only | WS-B, WS-G |
Rollout off / shadow / write / read |
MemorySystem = { legacy, canonical } + cohort record |
WS-E |
memory_items collection name (optional) |
memories or neutral canonical name (decide in WS-G) |
WS-G |
plugins_results, processing_memory_id |
Already mirrored — document sunset timeline | WS-D |
Legacy category values (core, hobbies, …) |
Keep in DB; map to primary four in UI filters | WS-F |
These read like memory-domain terms but are fossils from when "memory" meant "conversation" or are externally-observable API strings. WS-G must not "correct" them toward the canonical vocabulary.
| Frozen name | Where | What it actually is | Action |
|---|---|---|---|
WebhookType.memory_created (+ payload conversation_to_dict) |
utils/webhooks.py, developer webhook config |
Developer-facing webhook that fires on Conversation creation, ships a Conversation payload | Keep string; document as legacy alias of "conversation created"; deprecation path only via versioned webhook, never an in-place rename |
UsageHistoryType.memory_created_external_integration |
utils/app_integrations.py |
Usage/billing event keyed off Conversation creation | Keep string; freeze for analytics/billing continuity |
plugins_results, processing_memory_id |
conversation docs | Mirror of apps_results / processing_conversation_id |
Keep; sunset only when old clients age out |
The single record shape every canonical-cohort store/client converges on.
| Field | Type | Meaning | Notes |
|---|---|---|---|
id |
string | Stable record id | Neutral scheme (no memvec:); see rollout §10 Q5 |
content |
string | The fact/observation text | — |
layer |
short_term | long_term | archive |
Product lifecycle layer; drives UI badge, default visibility, TTL | The only axis users/clients see |
status |
active | superseded | tombstoned |
Record lifecycle; non-active excluded from normal reads |
Distinct from layer |
processing_state |
pending | processed | blocked |
Internal pipeline state | Never surfaced to clients |
category |
legacy taxonomy value | Metadata only (core/hobbies/… → primary four in UI) |
Not a layer |
evidence[] |
array of { source_type, source_id, … } |
Provenance; for voice paths source_id = Conversation id |
Drives cascade/tombstone on Conversation delete |
source_id |
string | Primary upstream source (usually a Conversation) | Indexed for cascade |
promotion |
{ from_layer, to_layer, reason, at, by } | null |
Audit record of Short→Long transitions | Promotion is never a silent flag flip |
ttl / expires_at |
timestamp | null | Short-term decay deadline | Null for long-term/archive |
created_at / updated_at |
timestamp | — | — |
LifecycleState.working is an in-flight extraction state, not a stored field on this record — it
exists only inside the extraction pipeline and resolves to a layer before the record is durable.
The state axes are orthogonal but not freely combinable. Only these combinations are legal; anything else is a bug a validator should reject.
layer |
legal status |
legal processing_state |
Default-visible read? |
|---|---|---|---|
short_term |
active, superseded, tombstoned |
pending, processed, blocked |
active + processed only |
long_term |
active, superseded, tombstoned |
processed (must be settled before promotion) |
active + processed only |
archive |
active, tombstoned |
processed |
No — explicit opt-in only |
- Promotion requires
processing_state=processed— apending/blockeditem never reacheslong_term. archiveitems are neversuperseded(terminal) — they tombstone or are resurfaced.status=tombstonedoverrides visibility at every layer (hard-excluded from default reads).- Physical storage may carry
status=hidden(canonical pipeline outcome for secret/rejected items). It has no §1.3 axis value — boundary-map totombstonedat validation/materialization (physical_status_to_record_status()inmemory_domain.py). Persisted rows keephidden; canonical validation treats them as tombstoned (same default-read exclusion). context_onlyis not a value on any axis — normalize tolayer=archiveor a non-default outcome (§1.1).- A read surface requesting
layer=archivestill honorsstatusfiltering.
Implementation: is_legal_state_combination() and assert_legal_state() in backend/models/memory_domain.py.
Normative lock (WS-D). Conversations are upstream session records — never Memories. This section documents the current extraction seam as implemented today. WS-I will route writes through
MemoryServicebut must preserve these boundaries.
Conversations (users/{uid}/conversations, database/conversations.py) are persisted
session records: processed transcript_segments, session metadata (structured,
apps_results), audio/photo linkage, and status. They are never stored as, surfaced as,
or merged into Memories.
On a Conversation document:
| Field | Role | Memory? |
|---|---|---|
transcript_segments |
Processed STT input to extraction | No — upstream processed input |
structured (title, overview, action_items, events, …) |
Derived session artifacts from post-processing | No — session summary, not extracted facts |
apps_results |
Per-app plugin output on the session | No — derived session artifacts |
plugins_results, processing_memory_id |
Legacy mirrors of apps_results / processing_conversation_id |
No — frozen legacy names (§1.1) |
Memories are extracted facts written to separate stores (users/{uid}/memories, and
interim shadow users/{uid}/short_term) with provenance pointing back at the Conversation.
| Domain | Collection constant | Module |
|---|---|---|
| Conversation (upstream) | conversations |
database/conversations.py |
| Long-term memories (interim) | memories |
database/memories.py |
| Short-term shadow (interim) | short_term |
database/short_term_memories.py |
| Workflow — action items | action_items |
database/action_items.py |
| Workflow — goals | goals |
database/goals.py |
No code path may point the Conversation store and a Memory store at the same collection name.
Enforced by backend/tests/unit/test_upstream_boundary.py.
When a Conversation completes post-processing, process_conversation() in
backend/utils/conversations/process_conversation.py fans out one upstream record into
separate downstream destinations:
Conversation (completed)
├─► Memories _extract_memories → _extract_memories_inner
│ → new_memories_extractor / extract_memories_from_text
│ → memories_db.save_memories (+ optional short_term shadow)
├─► action_items _save_action_items
│ → action_items_db.create_action_items_batch
└─► goals _update_goal_progress
→ extract_and_update_goal_progress (utils/llm/goals.py)
The three WS-D fan-out calls (_extract_memories, _save_action_items, _update_goal_progress)
are submitted separately from process_conversation() at lines ~946 and ~948–949 via
submit_with_context(postprocess_executor, …) — the Conversation is decomposed, not
persisted verbatim as a memory row.
action_items and goals are Workflow domains: downstream of the same seam, stored in
their own collections, not inside Memories. Long-term memory may later absorb a fact about a
commitment; the task/goal row stays in workflow.
Extraction builds MemoryDB rows via MemoryDB.from_memory() (models/memories.py):
conversation_id— primary upstream link (set toconversation.id).evidence[].source_id— provenance id (alsoconversation.idfor voice paths;source_type="conversation",source_signal="transcription").- Legacy mirror
memory_id— set equal toconversation_idinMemoryDB.__init__for older query paths (e.g.get_memory_ids_for_conversationfilters onmemory_id).
Conversation delete cascade keys on this provenance: memories_db.delete_memories_for_conversation
→ ripple_source_deletion(uid, conversation_id) tombstones evidence where
evidence[].source_id == conversation_id and retracts facts with no surviving evidence; shadow
short-term rows are tombstoned via short_term_db.tombstone_source(uid, source_id).
Action items link via conversation_id on each action_items row (_save_action_items).
- No Conversation-as-Memory persistence — no code path may write a Conversation document
(or its full
dict()) intomemories/short_term/ canonical Memories as a memory row. - No Conversation-as-Memory reads — memory read surfaces must not return a raw Conversation doc shaped as a memory item.
- No Conversations tab merge — merging the Conversations timeline into the Memories tab is permanently out of scope; Conversations stay upstream.
- No workflow-in-memory —
action_itemsandgoalsremain separate workflow collections; they are not memory layers.
WS-I (write convergence) may relocate where memory rows are written (MemoryService), but must
preserve: separate stores, separate fan-out, extracted-fact payloads (not session records), and
provenance via conversation_id / evidence[].source_id.
Authoritative record of the §10 blocking decisions. Ratified by product owner before WS-I start.
| # | Decision | Ratified choice | Notes / implications |
|---|---|---|---|
| Q1 | Write convergence for process_conversation |
Hard cutover | Canonical-cohort extraction writes go to the canonical store only — no legacy write/fallback for canonical users. (Override of the dual-write default.) |
| Q2 | Interim split-brain tolerance | None — disallowed | A canonical-cohort user must read AND write the canonical store. Consequence: WS-I must also route that cohort's reads to canonical (cannot leave reads on legacy), else the user loses freshly written memories. |
| Q3 | Promotion trigger Short→Long | Batch-or-daily | Short-term promotes to Long-term in batches once enough accumulate to process a full batch, OR once per day, whichever comes first. (Implemented in WS-B, not WS-I.) |
| Q4 | Backfill dedup (both stores) | Hash-based idempotency key | Deterministic id derived from source so re-runs never duplicate. (WS-C.) |
| Q5 | Vector ID strategy | Neutral prefix | Drop memvec:; canonical items use neutral vector IDs. (WS-J/WS-G.) |
| Q6 | API field name for layer axis | layer |
Desktop aliases tier during WS-G. |
| Q7 | Reprocess semantics | Full retract | Reprocess/sync-merge retracts conversation-sourced items across ALL stores + vectors before re-extract. |
| Q8 | Conversation-delete cascade | Server-default cascade=true + fix clients |
Belt-and-suspenders; desktop currently omits the flag. (WS-J/WS-K.) |
WS-I scope consequence of Q1+Q2: WS-I is NOT a dual-write add-on. For the canonical cohort it must
make the CanonicalMemoryBackend real for both write and read, route process_conversation
extraction to canonical-only, and route the cohort's read path (at least /v3 GET) to canonical — so a
canonical user is fully self-consistent. The canonical cohort stays empty in production
(CANONICAL_MEMORY_USERS empty in memory_system.py); only explicitly-added test users are affected. Legacy-cohort behavior
must remain byte-unchanged.
- Backfill is NON-DESTRUCTIVE. Legacy
memories/short_termrows are retained for a migrated user, not moved or deleted. WS-C copies legacy → canonical (layer=long_term) idempotently (hash key, Q4); the legacy data stays intact as a fallback. - Reversibility: because legacy data is preserved, flipping a user
canonical → legacyis a pure routing change with the original data fully available — no restore step needed. - Clean cut-over / decommission (WS-H) is GATED: legacy stores are deleted ONLY once all users are fully migrated and verified. Until then, legacy is the durable fallback. No agent may run WS-H (legacy store deletion) without explicit owner sign-off at full-migration time.
- Consequence for Q1 cutover: "cutover" governs the write/read routing for a canonical user (their NEW writes go canonical-only), not destruction of their pre-existing legacy data.
Status legend: ✅ handled in code today ·
| Trigger | legacy memories |
canonical memory_items |
memory_evidence |
memory_operations |
Pinecone ns2 (legacy {uid}-{id}) |
Pinecone ns2 (canonical neutral mem_…) |
Pinecone ns2 (legacy memvec:…) |
review_queue |
Neo4j KG | WS-M keyword index |
|---|---|---|---|---|---|---|---|---|---|---|
Conversation delete, cascade=false (current server default) |
— (no cascade) | — | — | — | — | — | — | — | — | — |
Conversation delete, cascade=true |
✅ ripple_source_deletion / tombstone evidence |
✅ MemoryService.retract_conversation_memories (canonical cohort only) |
✅ tombstone in retract | — | ✅ delete_memory_vector for retracted legacy ids |
✅ purge outbox via retract (neutral_vector_id_for_memory) |
🔜 legacy memvec: vectors not purged on cascade yet |
🔜 no cascade purge wired | 🔜 invalidate_kg_for_memory_retraction logs deferral |
🔜 TBD (WS-M) |
| Account delete | ✅ Firestore recursive wipe + delete_memory_vectors_batch |
✅ Firestore recursive wipe | ✅ Firestore recursive wipe | ✅ Firestore recursive wipe | ✅ _purge_derived_user_data |
✅ purge_canonical_derived_user_data (canonical cohort only) |
🔜 not enumerated on account delete | ✅ subcollection wipe | ✅ knowledge_nodes / knowledge_edges wiped |
🔜 TBD (WS-M) |
| Reprocess / sync-merge (Q7 full retract) | ✅ legacy path in _extract_memories_inner |
✅ retract_conversation_sourced_memories |
✅ evidence tombstone in retract | — | ✅ legacy delete in reprocess inner | ✅ purge outbox in retract | 🔜 apply vector_sync still writes memvec: (carry-forward WS-G) |
🔜 | 🔜 KG hook defers to rebuild | 🔜 TBD (WS-M) |
| Supersede / tombstone (single memory) | ✅ invalidate_memory / ripple |
✅ delete_canonical_memory |
✅ per-item tombstone | ✅ via apply path | ✅ router delete | ✅ purge outbox | 🔜 | 🔜 | 🔜 | 🔜 TBD (WS-M) |
| Archive transition | 🔜 legacy archive path | 🔜 canonical archive workers | — | — | 🔜 archive vector filter exists; purge on transition not wired | 🔜 | 🔜 | — | — | 🔜 TBD (WS-M) |
Ratified (Q8): server-default cascade=true + client parity (desktop omits flag today).
Shipped (WS-J): default remains cascade=false — intentional; flipping the default is a
production behavior change for every user and requires explicit owner sign-off while asleep.
Characterization test: test_conversation_delete_cascade_default_is_false in
backend/tests/unit/test_ws_j_delete_privacy.py.
When approved: change Query(False) → Query(True) in routers/conversations.py and land
desktop client fix (WS-K).
Canonical purge paths use neutral_vector_id_for_memory (mem_… = memory_id). Legacy apply
vector_sync / repair worker still upserts memvec:… via deterministic_memory_vector_id —
carry-forward to WS-G; do not break existing memvec: vectors in shared Pinecone ns2.
Devices are capture surfaces only — provenance metadata, not memory authority or dedup keys.
| Field / header | Shape | Notes |
|---|---|---|
client_device_id |
{platform}_{hash} |
Same shape as FCM device_key in notifications.py |
hash |
sha256 → first 8 hex chars | From a stable per-install id (Keychain UUID on macOS; persisted local install id on Windows; IDFV/Android id on mobile) |
X-Device-Id-Hash |
HTTP / WS upgrade header (first auth message for browser WS) | Raw hash component only |
X-App-Platform |
macos / windows / ios / android / web |
Platform component |
X-App-Version |
optional | Stored on registry client_devices doc |
Nullability: all device fields optional. Absent headers or legacy data ⇒ client_device_id=null ⇒ UI shows unknown device. Device id is never folded into evidence_id hash inputs (legacy dedup must stay byte-identical when device is absent).
Registry: users/{uid}/client_devices/{client_device_id} — platform, device_class, label, first_seen_at, last_seen_at, app_version. Upsert throttled like record_user_platform().
Provenance path: Conversation (client_device_id, client_platform) → Evidence.client_device_id / MemoryEvidence.client_device_id (not in artifact_ref or evidence_id hash inputs) → optional denormalized capture_device_ids / primary_capture_device on MemoryItem → retrieval filter device_scope=current|all|explicit (canonical memory users only; see X-Omi-Memory-Device-Scope-Supported response header).