Hermetic Backend E2E Harness

A manually runnable integration test suite that imports the real omi FastAPI backend and exercises selected routes against faked or disabled external dependencies. It is intended as a local dogfood harness and a required GitHub Actions check for backend PRs that touch backend/**.

Current dogfood status:

113 passed, 3 skipped, 24 warnings

The run installs a local-only socket guard before importing backend code. Any non-local DNS/socket attempt raises an assertion, so real API calls fail the harness instead of silently leaking. The runner also wraps pytest in a process-level timeout (E2E_PYTEST_TIMEOUT, default 120s) so websocket/provider-seam regressions fail instead of hanging indefinitely.

Run it with:

bash backend/testing/e2e/run.sh

Install e2e-only dependencies once with:

cd backend
python -m pip install -r testing/e2e/requirements.txt

run.sh verifies these dependencies are present but does not install them dynamically, so the test entrypoint itself does not reach PyPI before pytest imports the socket guard.

Scope of v1

This version proves the backend can boot hermetically and that selected core CRUD, mobile-facing lifecycle, user/account, storage, webhook, task-integration, listen-routing, sync-job, conversation lifecycle, retrieval/search, deterministic processing-seam, and legacy-shape paths can execute without real Firestore, Redis, GCS, Pinecone, Typesense, Google ADC, or production API keys.

Scenario	Status	Notes
CRUD golden path	✅ Green	Conversations are seeded directly because `POST /v1/conversations` processes an existing in-progress conversation; action items and memories use real create/update/delete routes.
Mobile lifecycle / client compatibility	✅ Green	Canonical Flutter/desktop-facing flows assert response shapes for conversation lists/details, memory create/list/visibility/delete, action-item list/update/batch/pending-sync/delete, and language/transcription prefs. Conversations are still seeded directly where no generic creation route exists.
Deterministic conversation-processing seam	✅ Partial	Reprocess and finalize routes, auth, model serialization, Firestore update, memory readback, and action-item queryability run with the provider-heavy processing function replaced by deterministic output. Full LLM-client wiring remains v2.
Listen/STT route seam	✅ Partial	`/v4/web/listen` websocket auth/query parsing/custom-STT dispatch is covered with a fake stream handler; custom-STT suggested transcript events also run through the real listen websocket loop into client emission, reconnect behavior, decrypted conversation readback, and finalize lifecycle. Full Deepgram-compatible streaming fake remains v2.
Sync v2 job lifecycle	✅ Partial	`/v2/sync-local-files` fast path, Redis job creation, deterministic background pipeline completion, job polling, and conversation persistence run with decode/VAD/STT/provider-heavy segment work replaced by deterministic seams. Full audio decoding and provider transcription remain lower-level/unit or v2 fake work.
Storage / speech profile	✅ Green	`google.cloud.storage.Client` is patched to a temp-dir fake; speech-profile presence, signed URL, sample list, and delete paths run through real routes/helpers.
Webhooks	✅ Partial	Developer webhook config/status routes, disabled no-op behavior, realtime delivery payload, non-2xx failure health recording, timeout/exception health recording, and threshold auto-disable are covered with `httpx.MockTransport`. Marketplace app webhook retry/circuit-breaker behavior remains v2.
Task integrations	✅ Green	CRUD/default/delete paths plus connected/disconnected/no-token and Todoist success, provider 500, 401 disconnect, and timeout failure paths exercise real task-integration database helpers against fake Firestore.
User/auth/profile/account	✅ Green	Auth guard, profile, onboarding, language/transcription prefs, people CRUD, notification/assistant settings, AI profile, and BYOK activation/deactivation routes are covered. Account deletion additionally exercises its real admission route, durable marker, opaque Cloud Tasks payload, worker claim, required-purge retry, and idempotent redelivery against local fakes. Firebase deletion, billing lookup, Twilio, and derived-data purge stay controlled test seams.
Retrieval/search	✅ Partial	Memory, action-item, conversation summary, and transcript-chunk retrieval routes run through real public APIs with Firestore-backed records and a deterministic in-memory replacement for Pinecone/OpenAI embeddings at the `database.vector_db` client seam. Full Pinecone/Typesense service compatibility remains out of scope.
Failure / edge modes	✅ Partial	Invalid input and edge-case coverage runs. Redis-unavailable, LLM 500, and STT timeout cases are explicitly skipped or deferred until per-test failure fakes are wired.
Legacy shape compatibility	✅ Green	Exercises legacy conversation/memory shapes and deterministic fake-store repeated writes. It does not execute production migration scripts.

What is faked or disabled

Dependency	v1 behavior	Why
Firestore	`fake-firestore` `MockFirestore`	In-memory datastore backing the real database modules.
Redis	`fakeredis`	In-memory Redis replacement.
Google Cloud Storage	`google.cloud.storage.Client` patched to a filesystem-backed fake	Enables storage-backed routes without GCS credentials/network.
Cloud Tasks / OIDC	Strict in-memory `tasks_v2.CloudTasksClient` plus a local token-verification seam in the account-deletion lifecycle test	Exercises the production task protobuf, queue payload, OIDC identity/audience, and retry headers without a Cloud Tasks control plane or Google token verification.
Google ADC	`google.auth.default` returns anonymous credentials	Prevents real credential lookup at import time.
Pinecone	`PINECONE_API_KEY` removed globally; targeted retrieval/search tests monkeypatch `database.vector_db.index` to a deterministic in-memory fake	Keeps app import hermetic while allowing route-level vector upsert/query/delete assertions without real Pinecone.
Typesense	Dummy host/port/API key	Lets import-time Typesense client construction succeed; retrieval/search tests rely on vector results and fail-open keyword search rather than real Typesense compatibility.
Google Translate	Anonymous Google credentials	Allows import-time client construction; v1 tests do not call live translation.
LLM/STT/VAD/embeddings	Fake modules scaffolded; route and custom-STT suggested-transcript seams covered where deterministic patching is practical	Kept as v2 work where scenarios need real outbound HTTP/WS/provider assertions.
Webhook/task external HTTP	`httpx.MockTransport` in targeted tests	Captures outbound payloads without network.

What's real

FastAPI app import via main.app
Routers, middleware, auth dependency, websocket route entrypoints, Pydantic request/response validation
Database modules and model serialization/deserialization
Firestore query/update/delete code paths, backed by MockFirestore
Redis client construction and delegated fakeredis operations
Storage helper code paths, backed by temp-dir fake GCS

Running individual scenarios

# CRUD / data shape
bash backend/testing/e2e/run.sh -k "test_crud"

# Conversation processing and state seams
bash backend/testing/e2e/run.sh -k "conversation_processing"

# Core listen/sync/conversation lifecycle seams
bash backend/testing/e2e/run.sh -k "core_flow_expansion"

# Listen/STT websocket route seam
bash backend/testing/e2e/run.sh -k "listen_stt"

# Storage-backed speech profile routes
bash backend/testing/e2e/run.sh -k "storage_speech_profile"

# Webhook and task integration seams
bash backend/testing/e2e/run.sh -k "webhooks or task_integrations"

# Retrieval/search seams
bash backend/testing/e2e/run.sh -k "search or retrieval or embedding or vector"

# User/auth/profile/account routes
bash backend/testing/e2e/run.sh -k "user_auth_profile"

# Durable account-deletion Cloud Tasks lifecycle
bash backend/testing/e2e/run.sh -k "account_deletion_cloud_tasks"

# Mobile-facing lifecycle / client compatibility
bash backend/testing/e2e/run.sh -k "mobile_lifecycle"

# Failure / edge modes
bash backend/testing/e2e/run.sh -k "test_failure_modes"

# Legacy shape compatibility
bash backend/testing/e2e/run.sh -k "test_migration_safety"

Architecture

run.sh
  └── pytest testing/e2e/
        ├── conftest.py                         # env, auth, fake setup, TestClient
        ├── fakes/
        │   ├── firestore.py                    # MockFirestore + seed/read helpers
        │   ├── redis.py                        # FakeRedis + redis.Redis patch
        │   ├── storage.py                      # filesystem-backed fake GCS client
        │   ├── vector_search.py                # deterministic embeddings + in-memory Pinecone-like index
        │   ├── llm.py                          # deterministic LLM fake scaffold
        │   ├── stt.py                          # deterministic custom-STT event helper; Deepgram WS TODO
        │   └── embeddings.py                   # VAD/diarization/embedding fake scaffold
        ├── fixtures/
        │   ├── conversations.json
        │   ├── memories.json
        │   └── action_items.json
        ├── test_core_flow_expansion.py
        ├── test_crud.py
        ├── test_conversation_processing.py
        ├── test_conversation_processing_deterministic.py
        ├── test_account_deletion_cloud_tasks.py
        ├── test_failure_modes.py
        ├── test_harness_guards.py
        ├── test_listen_stt.py
        ├── test_migration_safety.py
        ├── test_mobile_lifecycle_compatibility.py
        ├── test_retrieval_search.py
        ├── test_storage_speech_profile.py
        ├── test_task_integrations.py
        ├── test_user_auth_profile.py
        └── test_webhooks.py

Test lifecycle

Set hermetic env vars before importing backend modules.
Patch Google auth before any Firestore/Translate client construction.
Build in-memory Firestore/Redis fakes and temp-dir fake GCS.
Disable dotenv loading so local .env files cannot rehydrate real credentials.
Patch Firestore/Redis/Storage client constructors before import main.
Import the real FastAPI app and wrap it with TestClient.
Clear fake Firestore/Redis/Storage state around each test.
Seed data where the backend has no generic create endpoint.
Run route-level assertions through the real app.

Adding tests

Prefer real public routes. If no route exists for setup (for example, arbitrary conversation creation), seed via fakes.firestore.seed_* and then exercise the route under test.

from fakes.firestore import seed_conversation


def test_read_seeded_conversation(client, auth_headers, sample_conversation_data):
    seed_conversation("123", sample_conversation_data)

    resp = client.get(
        f"/v1/conversations/{sample_conversation_data['id']}",
        headers=auth_headers,
    )

    assert resp.status_code == 200

Current limitations / v2 work

Dependencies

omi backend dependencies from requirements.txt
e2e-only dependencies from testing/e2e/requirements.txt

FilesExpand file tree

README.md

Latest commit

History