Skip to content

AGENTS.md

Latest commit

 

History

History
170 lines (133 loc) · 11.2 KB

AGENTS.md

File metadata and controls

170 lines (133 loc) · 11.2 KB
Raw

App (Flutter) — Operational Playbook

Inherits ../AGENTS.md; adds app-specific operational guidance.

Build Bootstrap

Flavors

  • dev: Android com.friend.ios.dev, iOS com.friend-app-with-wearable.ios12.development — uses .dev.env, Firebase project based-hardware-dev
  • prod: Android com.friend.ios, iOS com.friend-app-with-wearable.ios12 — uses .env, Firebase project based-hardware-prod
  • raybanDat: camera-capable iOS target with the same iOS development identity; scripts/rayban_dat.sh excludes mcumgr only for that transaction, then restores the default graph.

Generated Files (never edit manually)

Generator Source Output Command
envied lib/env/dev_env.dart, lib/env/prod_env.dart *.g.dart (obfuscated secrets) flutter pub run build_runner build
json_serializable @JsonSerializable models *.g.dart (fromJson/toJson) flutter pub run build_runner build
pigeon lib/pigeon_interfaces.dart lib/gen/pigeon_communicator.g.dart + iOS/Android stubs flutter pub run build_runner build
flutter_gen pubspec.yaml assets/fonts lib/gen/assets.gen.dart, lib/gen/fonts.gen.dart flutter pub run build_runner build
flutter_localizations lib/l10n/*.arb lib/gen_l10n/app_localizations*.dart flutter gen-l10n

Setup Sequence

bash setup.sh ios    # or: bash setup.sh android

This handles: pub get, build_runner, gen-l10n, and flavor configuration.

For physical-device builds, use the wrapper: it owns dev + local_dev and prod + mobile_beta pairing plus auth env setup. Direct builds must first run scripts/validate_mobile_build_config.sh --flavor <dev|prod> --profile <profile> with the matching OMI_APP_PROFILE; release/profile helpers do this too. OMI_MOBILE_BUILD_MODE=profile installs an AOT build that opens untethered (debug builds need flutter run attached on a physical iPhone; see README).

Firebase Config

Never run flutterfire configure — it overwrites prod credentials. Config files:

  • Dev: android/app/src/dev/
  • Prod: android/app/src/prod/
  • Local emulator: lib/firebase_options_local.dart

Native Bridge

Pigeon Interface (bidirectional, iOS ↔ Dart)

  • Contract: lib/pigeon_interfaces.dart — paired host/Flutter APIs for the watch recorder, BLE, and Ray-Ban Meta
  • Dart side: lib/gen/pigeon_communicator.g.dart
  • iOS side: ios/Runner/PigeonCommunicator.g.swift
  • Android side: android/app/src/main/kotlin/com/friend/ios/PigeonCommunicator.g.kt
  • Implementation: ios/Runner/RecorderHostApiImpl.swift
  • After editing the contract, regenerate: flutter pub run build_runner build

MethodChannel (Phone Calls)

  • Channel: com.omi/phone_calls + EventChannel com.omi/phone_calls/events
  • Dart: lib/services/phone_call_service.dart
  • iOS: ios/Runner/PhoneCalls/OmiPhoneCallsPlugin.swift
  • Android: android/app/src/main/kotlin/com/friend/ios/phonecalls/PhoneCallsPlugin.kt
  • Methods: initialize, makeCall, endCall, toggleMute, toggleSpeaker

Pigeon (Phone Mic — conversation capture)

  • Contract: lib/phone_mic_interface.dart → lib/gen/phone_mic_pigeon.g.dart + ios/Runner/PhoneMic/PhoneMicPigeon.g.swift + android/app/src/main/kotlin/com/friend/ios/phonemic/PhoneMicPigeon.g.kt
  • Regenerate: dart run pigeon --input lib/phone_mic_interface.dart
  • iOS module: ios/Runner/PhoneMic/ — self-healing AVAudioEngine capture (interruptions/route changes recover natively; Dart only mirrors state)
  • Android module: android/app/src/main/kotlin/com/friend/ios/phonemic/ — AudioRecord capture with a self-healing rebuild loop + silencing detection (calls/assistant recover natively; Dart only mirrors state); PhoneMicForegroundService (microphone FGS) keeps background capture alive; batch opus encode via a JNI shim over the plugin-shipped libopus
  • Dart service: lib/services/mic/native_mic_recorder_service.dart behind ServiceManager.phoneMic; chat memos/speech profile stay on flutter_sound via ServiceManager.mic; MicArbiter prevents the two stacks contending
  • Events carry a Dart-minted session id (start(mode, sessionId)); Dart drops events with a foreign id so a stale native event can't clobber a fresh session; start() onto a live native session adopts the new id and re-emits state so the caller converges; stop() always forwards to native (kills an orphaned session) and runs local teardown once
  • Two capture modes, fixed per session at start(mode): stream (realtime frames → Dart → socket/WAL) and batch (Transcribe Later — native opus encode (OpusKit iOS, libopus JNI shim Android) → WAL-compatible audio_omibatchphone[auto]_…bin; no frames cross to Dart; liveness = 1Hz onBatchProgress). Mode selection lives in CaptureController.streamRecording (explicit batchModeEnabled or auto offline fallback; iOS + Android); omibatchphoneauto recordings auto-upload on reconnect

On-device speech deadlines and cleanup: contract.

Permission Matrix

Permission Android iOS Feature
Microphone RECORD_AUDIO NSMicrophoneUsageDescription Recording, speech profile
Bluetooth BLUETOOTH_SCAN, BLUETOOTH_CONNECT NSBluetoothAlwaysUsageDescription Omi device connection
Location ACCESS_FINE_LOCATION NSLocationUsageDescription Background features
Contacts READ_CONTACTS NSContactsUsageDescription People recognition
Calendar READ/WRITE_CALENDAR NSCalendarsUsageDescription Calendar integration
Camera — NSCameraUsageDescription QR/photo features
Notifications POST_NOTIFICATIONS (automatic) Push notifications
Background FOREGROUND_SERVICE_* (4 types) UIBackgroundModes (7 modes) Continuous capture

Android: 26 permissions in AndroidManifest.xml; iOS: 11 background modes + 10 consent strings.

Test Strategy

Test Structure

  • test/unit/ — Auth, tokens, preferences, audio utils
  • test/widgets/ — UI components (shimmer, waveform, transcript)
  • test/providers/ — State management (capture_provider, device_provider)
  • test/utils/ — Utility functions (localization helpers)

Running Tests

bash test.sh           # runs all tests
flutter test           # same thing
flutter test test/unit/  # specific directory
# Android native tests: JDK 21, SDK 36, Flutter bootstrap
(cd android && ./gradlew :app:testDevDebugUnitTest)

bash test.sh bootstraps missing local generated files with an empty API_BASE_URL so test/ stays hermetic.

Native batch contracts: ruby ios/test/batch_audio_energy_test.rb runs the production Swift writers (macOS manifest, local + CI).

CI runs flutter test and app/scripts/analyze_ratchet.sh: errors fail; new info/warning occurrences above app/analysis_baseline.json fail. Run the ratchet before committing Dart changes. Update intentional baselines with --update-baseline in that PR.

Test Patterns

  • Mock singletons (SharedPreferencesUtil, AuthService, FirebaseAuth) since they aren't injectable
  • Test state machine logic via minimal abstractions mirroring production flow
  • Everything under test/ must be hermetic — no network, live backends, or real devices — because bash test.sh (the CI suite) runs all of it.
  • Chat transcript layout: pumping only AIMessage in a SingleChildScrollView misses scroll-extent bugs; chat list changes must keep test/widgets/chat_scroll_layout_test.dart green (ListView drag + citation/markdown sizes) — it is the Mobile App Checks contract for this class.
  • A test that needs a live service, device, or real API goes under integration_test/, which test.sh/CI never runs. For integration tests against a local backend, set OMI_APP_TEST_API_BASE_URL=http://127.0.0.1:<port>/; use OMI_APP_TEST_USE_PROD_API_DEFAULT=1 only when a test needs the prod API default. State in the PR how you ran it; it can't be the only evidence the change works.
  • Coverage rules (bug fix → regression test; feature → core + main error path): see root AGENTS.md → Testing.

Localization (l10n)

  • All user-facing strings must use context.l10n.keyName
  • 49 locales: English (template) + 48 translations in lib/l10n/ — never trust a remembered count; enumerate with ls lib/l10n/app_*.arb.
  • Template: lib/l10n/app_en.arb
  • Add keys via jq (never read full ARB — they're large). Use skill add-a-new-localization-key-l10n-arb
  • Translate all locales — use skill omi-add-missing-language-keys-l10n for real translations
  • Regenerate after changes: flutter gen-l10n; done only when it emits zero "untranslated message(s)" warnings. To get the exact missing-key list, temporarily add untranslated-messages-file: /tmp/untranslated.json to l10n.yaml and re-run.

Auth & Security

Token Lifecycle

  1. getAuthHeader() in lib/backend/http/shared.dart checks token expiry (5-minute buffer)
  2. If expired, calls AuthService.instance.getIdToken() for Firebase refresh
  3. Token stored via SharedPreferencesUtil in flutter_secure_storage (Keychain / EncryptedSharedPreferences); expiration timestamp stays in SharedPreferences. One-time migrateAuthTokenFromPrefs() runs at SharedPreferencesUtil.init() so existing sessions keep their token.
  4. 401 responses trigger automatic refresh + retry

Auth Methods

  • Google Sign In (google_sign_in package)
  • Apple Sign In (sign_in_with_apple package, includes PKCE via nonce+sha256)
  • Firebase Auth as the identity layer

Request Headers

All API requests include: X-Request-Start-Time, X-App-Platform, X-Device-Id-Hash, X-App-Version, plus Bearer token.

API Base URLs

  • Dev: configured in .dev.env → Env.apiBaseUrl
  • Prod: configured in .prod.env → Env.apiBaseUrl

Codegen Rules

  • Run flutter pub run build_runner build after changing: env files, model annotations, pigeon contracts, or pubspec assets
  • Run flutter gen-l10n after changing ARB files
  • Never edit files ending in .g.dart or .gen.dart
  • If build_runner fails with conflicts: flutter pub run build_runner build --delete-conflicting-outputs

App Flows & E2E

  • See e2e/SKILL.md for navigation architecture, screen map, widget patterns, and 34 reference flows
  • See e2e/flows/*.yaml for individual flow definitions

Verifying UI Changes (agent-flutter)

After any Flutter UI edit, verify with agent-flutter (Marionette is integrated in debug builds). Install once: npm install -g agent-flutter-cli.

Edit → Verify → Evidence loop:

  1. Edit code, hot restart: kill -SIGUSR2 $(pgrep -f "flutter run" | head -1)
  2. Connect: AGENT_FLUTTER_LOG=/tmp/flutter-run.log agent-flutter connect
  3. Verify: agent-flutter snapshot -i
  4. Interact: agent-flutter press @e3 / press 540 1200 / find type button press / fill @e5 "text" / dismiss
  5. Evidence: agent-flutter screenshot /tmp/evidence.png

Key rules:

  • Must reconnect after every hot restart (kills VM Service session).
  • Refs go stale frequently — always re-snapshot before every interaction. Use press x y as fallback.
  • AGENT_FLUTTER_LOG must point to flutter run stdout (not logcat).
  • Prefer find type X / find key "name" over hardcoded @ref. Add Key('descriptive_name') to new interactive widgets.
  • Full command reference: agent-flutter schema.