Cross-process reactions
How Act keeps reaction latency low when more than one process shares the backing event store. The short version: the configured store may implement an optional notify(handler) hook; the orchestrator wires it in at build time and wakes settle() immediately on commits from other processes โ no polling lag, no extra code.
The problemโ
In a single-process app, do() arms the drain locally and settle() runs reactions on the same Node event loop. Latency is bounded by reaction work, not by the framework.
Two or more processes against the same store don't have that luxury. A commit on Worker A is invisible to Worker B until Worker B asks the store. The default mechanism is polling โ start_correlations runs on a timer, or the user calls settle() periodically. The polling interval becomes a floor on reaction latency:
start_correlationsdefault: 10 s.- Common explicit poll loops: 50โ500 ms.
- Local single-process: 0 ms (event loop turn).
For event-driven workloads where reactions matter, that 10 s floor is a deal-breaker. Even 50 ms is loose for things like SSE fan-out or near-real-time projections.
The hookโ
The Store interface has an optional method:
interface Store extends Disposable {
// ...existing...
notify?(
handler: (notification: StoreNotification) => void
): NotifyDisposer | Promise<NotifyDisposer>;
}
type StoreNotification = {
readonly stream: string;
readonly events: ReadonlyArray<{
readonly id: number;
readonly name: string;
}>;
};
When present, the orchestrator subscribes once at build() time and routes notifications to wake settle() automatically. The hook is opt-in at the adapter level โ PostgresStore defaults notify: false so single-instance deployments pay zero overhead. Multi-process apps enable it explicitly:
store(new PostgresStore({ /* ... */, notify: true })); // โ opt in
const app = act()
.withState(Order)
.on("OrderPlaced").do(reduceInventory).to("inventory")
.build();
// Cross-process commits wake reactions on this process.
Optionally, the user can also subscribe to the notified lifecycle event for SSE fan-out, dashboards, or audit:
app.on("notified", (n) => sse.broadcast(n));
Self-filter โ a clean cross-process semanticโ
Every store instance carries a per-instance UUID (_by) embedded in the NOTIFY payload. The LISTEN handler skips payloads where by === this._by. Result:
- Local commits never echo back through
notified(the local fast path insidedo()already arms drain โ no double signal). - The
notifiedlifecycle event surfaces only another process writing to the same store. That gives consumers a clean signal for cross-process visibility.
The alternative (broadcast everything, let the consumer filter) was rejected as messier โ it pollutes the local fast path with self-echoes and forces every listener to know about the filter.
Self-filtering is a portable Store.notify contract, not a Postgres detail: the TCK's notify capability suite enforces it (plus one-notification-per-commit batch delivery) against every adapter that declares the capability (#1184).
Payload cap โ oversize commits degrade to pollโ
PostgreSQL rejects NOTIFY payloads at or above 8000 bytes (payload string too long), and pg_notify runs inside the commit transaction โ an
unguarded oversize payload would abort the whole INSERT batch. commit()
therefore measures the serialized payload first and skips the NOTIFY when
it would not fit (very large batches, long stream/event names). The commit
succeeds, no notification goes out, and listeners pick the events up on
their next poll cycle โ delivery degrades in latency, never in guarantees.
At-least-once is preserved because NOTIFY is only ever a latency
optimization over the poll path, not the delivery mechanism itself.
Connection loss โ the LISTEN client self-healsโ
The dedicated LISTEN client is a long-lived checkout, and node-postgres
drops its idle-error guard the moment a client leaves the pool. A backend
restart, a failover, or a dropped network connection surfaces as an
error event on that client. Left unhandled it is an uncaught exception โ
a process crash; and even a process that survived would keep a client
that looks healthy but never receives another notification, so
cross-process wakeups silently stop.
PostgresStore handles that error: it logs, tears the dead client down,
and re-establishes the LISTEN on a fresh client with capped
exponential backoff (250 ms base, doubling to a 30 s ceiling). Between
attempts the store degrades to the poll path โ the same graceful fallback
as an oversize payload โ so no events are lost, only wakeup latency
climbs until the subscription is back. A healthy re-LISTEN resets the
backoff, so an unrelated blip later starts fresh. Disposal (dispose() or
a re-subscribe) cancels any pending reconnect, so a reconnect never fires
after teardown.
A subtlety the teardown has to respect: a node-postgres socket routinely
emits error more than once on the way down โ the in-flight LISTEN
rejection first, then the ECONNRESET/end that follows. If the dead
client were ever left without an error listener during the backoff
window, that second emission would re-raise as an uncaught exception โ the
exact crash the self-heal set out to prevent. So the reconnect never
strips the dead client bare: it swaps the live handler for a swallow
listener that stays attached through release(true). The re-entrant path
is guarded the same way โ a second trigger while a reconnect timer is
already pending cancels the pending timer before scheduling, so two timers
never race to re-LISTEN on two fresh clients.
This makes notify durable across the connection blips that are routine
in managed Postgres (patching windows, failovers) without operator
intervention โ the subscription that was healthy before the blip is
healthy after it, no restart required.
Adapter statusโ
| Adapter | notify | Why |
|---|---|---|
PostgresStore | implemented (opt-in via notify: true) | LISTEN/NOTIFY on a per-(schema, table) channel (act_commit_<schema>_<table>). One NOTIFY per commit transaction with the full event batch as JSON payload. Default off โ single-instance deployments pay zero overhead, existing callers keep their current behavior on upgrade. |
InMemoryStore | not implemented | Single-process by definition โ there is no remote writer. |
SqliteStore | not implemented | Single-node by design. Use @rotorsoft/act-pg for multi-process. |
Build-time contractโ
Inject the store via store(adapter) before calling act()...build(). The orchestrator wires the notify subscription against whichever store is current at construction; late injection won't take effect.
// โ
Correct
store(new PostgresStore({...}));
const app = act().withState(Order).build();
// โ Wrong โ orchestrator binds before injection
const app = act().withState(Order).build();
store(new PostgresStore({...})); // too late; notify wasn't wired
Tests that build the app at module-load time should refactor to a buildApp() factory called inside beforeAll after store injection.
Hint, not a contractโ
notify is a performance hint. The orchestrator never depends on it for correctness:
- If the store doesn't implement
notify, the existing debounce/poll path still drains correctly. - If a notification is dropped (network hiccup, pool exhaustion, misconfigured channel), the existing debounce/poll path still drains correctly.
This means you can run notify as the happy-path optimization and keep start_correlations (or a periodic settle() timer) as the safety net. Lost wakeups cost latency, never correctness.
Topology and connection budgetโ
LISTEN checks out a dedicated client from the pool. Each subscribed process holds one extra connection for the lifetime of the subscription. Three common topologies:
- Fat single process: simplest, no notify needed. Easily handles thousands of events/sec.
- Symmetric workers: N identical processes, all running the same reactions, sharing a DB. Notify wakes them all; competing consumers via
claim()(FOR UPDATE SKIP LOCKED) ensures exactly-once-per-event per logical reaction. Scales linearly until the connection budget bites. - Specialized sidecars: each process subscribes to a different reaction subset. Notify wakes everyone but only the relevant subscriber does work.
For the symmetric-workers topology, watch for the thundering herd: every process wakes on every cross-process commit and races for the same lease. Only one wins per stream โ the rest see no work and go back to sleep. That's correct but introduces some redundant claim attempts. A small debounce on notified helps under bursty load.
Performanceโ
Benchmark in @rotorsoft/act-pg's PERFORMANCE.md. Single run, docker PG on localhost, 30 single-event commits:
| Mode | p50 | p95 | p99 |
|---|---|---|---|
| notify | 11 ms | 15 ms | 25 ms |
| polling (50 ms) | 27 ms | 54 ms | 77 ms |
At 50 ms polling, notify is ~3ร faster across percentiles. At the default start_correlations 10 s interval, the gap blows out to ~1000ร.
See alsoโ
- Correlation and drain โ how
settle()actually runs. - Extension points โ
Storecontract reference. @rotorsoft/act-pg/PERFORMANCE.mdโ benchmark methodology and numbers.