Skip to main content

Broker Caveats

This page highlights broker-specific constraints that affect routing, priorities, and control-plane behavior. These caveats are based on the adapter implementations.

In-memory broker​

  • No priority buckets: supportsPriority is false, so priorities are not enforced.
  • Single-queue consumption: only one queue can be consumed per subscription.
  • Not durable: data is lost when the process exits.

SQLite broker​

  • Broadcast scope is in-process: fan-out works for subscribers running in the same process, but cross-process worker control broadcasts are not supported.
  • Single-queue consumption: only one queue can be consumed per subscription.
  • Polling-based delivery: tasks are polled on pollInterval and claimed via row locks; latency depends on the poll interval.
  • Single-writer constraint: SQLite allows one writer at a time. Use separate broker/backend files and avoid producer writes to the backend.
  • Native assets: build CLI bundles (dart build cli) when using sqlite3 to ensure the native library is packaged reliably.
  • Local disk only: avoid network filesystems for WAL-backed SQLite files.

Redis Streams broker​

  • Single-queue consumption: only one queue can be consumed per subscription.
  • Priority uses per-queue streams: each priority bucket maps to a dedicated stream key.
  • Delayed delivery: delayed tasks are stored in a sorted set and re-enqueued when due.
  • Broadcast channels: broadcasts are stored in per-channel streams and consumed via dedicated consumer groups.
  • Visibility timeouts: the broker reclaims idle deliveries via XAUTOCLAIM. Extending a lease requeues the task into the delayed set (it does not update the original stream entry).
  • Key eviction risk: Redis eviction policies can drop stream, delayed, or dead-letter keys. Use a maxmemory policy that avoids evicting Stem keys, or isolate Stem data in a dedicated Redis instance.

Postgres broker​

  • Single-queue consumption: only one queue can be consumed per subscription.
  • Polling-based delivery: workers poll for due jobs on an interval.
  • Visibility timeouts: tasks are locked with a locked_until lease; if a worker dies or stops heartbeating, jobs become visible again after the lease expires.
  • Dead letter retention: dead letters are retained for a default window (7 days) unless configured otherwise.
  • Broadcast channels: broadcasts are stored in a separate table and read alongside queue deliveries.

Result backend caveat (ordering)​

  • Group result ordering: group/chord results are stored as maps (Redis hashes / Postgres tables) and returned without ordering guarantees. If you need stable ordering, sort results by task id or track ordering in group metadata.

Shutdown semantics (broker impact)​

  • Soft shutdowns are cooperative: brokers only see acknowledgements (or requeues). If a worker stops without acking a delivery, the task becomes visible again after the visibility lease expires (Redis reclaim interval / Postgres locked_until).
  • Long-running tasks should emit heartbeats or extend leases so the broker does not re-deliver them mid-execution.

Tips​

  • Use routing subscriptions to pin workers to a single queue when using Redis or Postgres.
  • Prefer Redis when you need low-latency delivery and high throughput.
  • Prefer Postgres when you need SQL visibility and a single durable store.

Example entrypoints​

brokers.dart
InMemoryBroker createInMemoryBroker() {
  return InMemoryBroker();
}
brokers.dart
Future<RedisStreamsBroker> connectRedisBroker() {
  return RedisStreamsBroker.connect('redis://127.0.0.1:6379/0');
}
brokers.dart
Future<PostgresBroker> connectPostgresBroker() {
  return PostgresBroker.connect('postgres://localhost:5432/stem');
}
brokers.dart
Future<SqliteBroker> connectSqliteBroker() {
  final file = File('stem_broker.sqlite');
  return SqliteBroker.open(file);
}