+ Last updated {lastUpdatedDate} +
+ ) + } ++ Error 404 +
++ The page you're looking for doesn't exist or has been moved. Try one of + the popular destinations below, or search the docs with + ⌘K. +
+
+ Two Connect services live in
+ proto/notify/v1.
+ All RPCs accept JSON over HTTP/1 + HTTP/2 (Connect) and
+ binary gRPC — the wire format is negotiated by the
+ Content-Type header. Field numbers are frozen forever;
+ additive fields land at the end with a new number, never reuse,
+ never renumber.
+
+ The string form used in NotifyRequest.addresses.byChannel
+ keys is the lowercase suffix: in_app,
+ email, web_push, mobile_push,
+ sms, whatsapp. (In-app needs no entry —
+ destination is the user id itself.)
+
+ The platform supports at most one registered token per
+ (user, device_type). Re-registering rotates the token
+ in place; it does not create a new device row.
+
+ Backend-to-backend surface. Speaks gRPC and Connect.
+ Auth is X-Notify-Internal-Token at the transport layer.
+
addresses = 9; // userID → {byChannel: {channel: address}}
+}
+
+message ChannelAddresses {
+ map by_channel = 1;
+}
+
+message NotifyResponse {
+ int32 delivered = 1; // provider accepted at least one (user, channel) row
+ int32 pending = 2; // stored but no active provider or no destination
+ int32 failed = 3; // provider returned an error
+}`} lang="proto" />
+
+ Wire path: POST /elloloop.notify.v1.NotificationInternalService/Notify
+
+ Errors:
+
+ InvalidArgument — tenant_id, notification_id, or user_ids empty.Unauthenticated — missing or mismatched X-Notify-Internal-Token.Internal — store error (DB unreachable, schema migration not applied, etc.).
+
+ NotificationClientService
+
+ Client-facing surface for browsers and mobile apps. Speaks Connect
+ HTTP/2. Auth is Authorization: Bearer <JWT>.
+ Tenant is derived from the JWT — request bodies never carry a tenant
+ field.
+
+
+ StreamEvents
+
+ Long-lived server-stream. The first event is always a
+ HeartbeatEvent carrying the assigned
+ session_id so the client can target
+ AckDataChange back at this exact connection.
+
+
+ Errors:
+
+ Unauthenticated — missing / invalid / expired JWT.Unimplemented — NOTIFY_LIVE_CONNECTIONS_ENABLED=false.Canceled — client disconnect, server shutdown.
+
+ GetNotifications
+
+ Newest-first. Cursor is strict < on
+ created_at_ms. unread_count is the global
+ not-yet-read total, not the count on this page.
+
+
+ AckNotification
+
+ Marks one row as Read and stamps
+ read_at_ms. Idempotent — repeated calls are no-ops.
+ Cross-user / cross-tenant attempts return NotFound
+ (never PermissionDenied, which would leak the existence
+ of someone else's row).
+
+
+ AckDataChange
+
+ Cancels any pending retries for the
+ (idempotency_key, session_id) pair on the in-memory
+ RetryTracker. Sessions older than the process lifetime are silently
+ ignored (the tracker treats unknown entries as no-ops).
+
+
+ RegisterPushToken
+
+ Upserts the (user, device_type) → token
+ row. Re-calling rotates the token in place. For Web Push the
+ token is the JSON-stringified
+ PushSubscription; for FCM it is the FCM registration
+ token.
+
+
+ Error codes
+
+ Code When InvalidArgumentMissing required field, malformed cursor, empty user_ids on Notify, empty id on AckNotification. UnauthenticatedMissing or invalid Authorization / X-Notify-Internal-Token header. Includes expired JWT. NotFoundAcking a row that doesn't belong to the calling user (or doesn't exist). UnimplementedStreamEvents when NOTIFY_LIVE_CONNECTIONS_ENABLED=false.CanceledStream interrupted (client close, server shutdown). InternalStore error, provider error not absorbed by the orchestrator, panic recovery.
+
+ Generating clients
+
+ Use buf against the published proto bundle on a tagged
+ release:
+
+
+ Or import the generated stubs from the Go module:
+
+ Related
+
+
+ The email channel delivers a notification as an outbound email. v0.1
+ ships one provider — emailservice, a thin translator over
+ a caller-supplied Sender interface — with slots for
+ ses, acs, smtp, and
+ sendgrid in later waves.
+
+ Any time a user needs an out-of-band message they will see whether or + not your app is open. Password resets, comment notifications, billing + receipts, weekly summaries — anything where the email is the + delivery rail itself, not a fallback hint. +
+ +
+ emailservice is intentionally small: it maps a
+ notify.Message onto a SendEmailInput and
+ hands it to a Sender the caller supplies. The
+ Sender contract is narrow on purpose so the provider
+ package never takes a dependency on any specific email proto.
+
Wire it like this in library mode:
+
+ The Name argument is the label that surfaces in
+ observability (notify_send_total{`{provider="emailservice"}`}).
+ Pass "" to default to "emailservice". For
+ an SES-backed Sender, pass "ses"; the channel kind
+ disambiguates.
+
Set the provider block in env vars:
+
+ With NOTIFY_EMAIL_PROVIDER=none (the default) the channel
+ is disabled. Email rows in a Notify call are stored but
+ counted as Pending rather than dispatched.
+
+ Producer-side, every email send rides the standard
+ Notify RPC. The only additional thing the producer
+ supplies is the address.
+
+ Response on a successful send: +
++ The provider maps the notify fields straight through: +
+From = the provider's configured default From address.To = Message.To.Subject = Title.BodyText = Body.BodyHTML = "". (HTML rendering is the Sender's concern; wrap the Sender with a template step if you need it.)
+ On success, Provider.Send returns a
+ Receipt{`{ProviderMessageID, StatusDelivered}`} populated
+ with the Sender's MessageID. The orchestrator persists
+ StatusDelivered on the row and stamps
+ DeliveredAtMS.
+
+ On error, the receipt is StatusFailed and the error is
+ wrapped with %w so the caller can
+ errors.Is / errors.As against the
+ underlying transport error.
+
+ Empty Message.To short-circuits with
+ StatusFailed — every email transport rejects an empty
+ recipient and the platform avoids a wasted RPC.
+
+ All four slot into the same ChannelEmail registry entry
+ via different Sender implementations. None of them
+ require changes to notify.Notifier; the contract
+ stops at the provider boundary.
+
+ In-app delivery is a real-time push to any of the user's currently
+ open client connections via the
+ NotificationClientService.StreamEvents server-stream.
+ No provider configuration is needed — when the live-connections
+ subsystem is on, the in-app provider is auto-registered.
+
+ Anything you want to surface immediately in the active session: a new
+ chat message, an updated task list, a "your colleague joined the
+ document" presence event. The same rows are stored for catch-up via
+ GetNotifications, so a user who comes back online sees
+ the missed traffic.
+
+ The orchestrator's in-app provider holds a reference to the
+ realtime Registry:
+
Notify with DELIVERY_CHANNEL_IN_APP.Send.StreamEvent to every live connection for the recipient via Registry.Push.StatusDelivered. If 0 connections (user offline) → StatusPending.+ Why this design. + An in-app row to a user with no live connections gets ++ +StatusPending— not a lying +StatusDelivered. The status field is used downstream + (unread filter, read-receipt analytics, dashboards) and it would + silently mislead consumers if "pushed to nobody" looked the same as + "successfully delivered". The row remains available via +GetNotificationsfor catch-up; when the user reconnects + they see it via the regular inbox flow. +
+ StreamEvents is a server-streaming RPC that multiplexes
+ three event kinds plus heartbeats over one connection:
+
notification — the platform delivered a new notification row to this user.data_change — an opaque "something upstream that you care about may have changed" hint. The client re-fetches via its own API. Carries an idempotency_key for at-least-once retry.heartbeat — keepalive carrying the server clock. First heartbeat after handshake carries the assigned session_id so the client can target AckDataChange at this exact stream.Authorization: Bearer <JWT> and a DeviceType.Conn, registers it under the user.HeartbeatEvent carrying the assigned session_id.NOTIFY_LIVE_HEARTBEAT_INTERVAL (default 30s); the in-app provider pushes notifications + data-change events as they arrive.
+ Setting NOTIFY_LIVE_CONNECTIONS_ENABLED=false disables
+ the subsystem entirely. StreamEvents returns
+ CodeUnimplemented, the in-app provider is not
+ registered, and in-app rows stay StatusPending for
+ inbox-only consumption.
+
async (req) => {
+ req.header.set("Authorization", \`Bearer \${getAccessToken()}\`);
+ return next(req);
+ },
+ ],
+});
+const client = createClient(NotificationClientService, transport);
+
+let sessionId = "";
+
+for await (const ev of client.streamEvents({ deviceType: "DEVICE_TYPE_BROWSER" })) {
+ sessionId = ev.sessionId || sessionId;
+
+ switch (ev.event.case) {
+ case "notification": {
+ const n = ev.event.value.notification!;
+ console.log("notification:", n.title, "—", n.body);
+ showInUI(n);
+ break;
+ }
+ case "dataChange": {
+ const dc = ev.event.value;
+ console.log("data change:", dc.subjectRef);
+ await refreshUpstream(dc.subjectRef);
+ // ACK so the server stops retrying.
+ await client.ackDataChange({
+ idempotencyKey: dc.idempotencyKey,
+ sessionId,
+ });
+ break;
+ }
+ case "heartbeat":
+ console.log("heartbeat @", ev.event.value.timestampMs);
+ break;
+ }
+}`} lang="ts" title="stream.ts" />
+
+ Reconnect strategy
+
+ Streams die for legitimate reasons: laptop sleep, server rolling
+ deploy, mobile carrier handover. The client should reconnect with
+ exponential backoff and a cap:
+
+ setTimeout(r, backoffMs + Math.random() * 250));
+ backoffMs = Math.min(backoffMs * 2, maxMs);
+ }
+ }
+}`} lang="ts" />
+
+
+ Reconnecting is safe: any in-flight events the client missed are
+ still in the store and available via
+ GetNotifications. Data-change events the client did not
+ ack will be retried by the server's RetryTracker until
+ either the ack arrives or
+ NOTIFY_LIVE_RETRY_MAX_ATTEMPTS is exhausted.
+
+
+ Sending an in-app notification
+
+ No address required — the in-app channel's destination is the user
+ id itself.
+
+ Backpressure
+
+ Each Conn has a buffered EventCh (default
+ 64 deep). Registry.Push does a non-blocking send; a
+ connection whose buffer is full has the event dropped and logged
+ (event_queue_full). It stays registered for future
+ events — one slow client never blocks the producer or the rest of
+ the recipients.
+
+
+ The buffer size is currently fixed at
+ DefaultEventBuffer = 64. If a real client genuinely
+ needs deeper queueing, it's a follow-up wave; for now the right move
+ is to keep the client UI thread non-blocking on event arrival.
+
+
+ Related
+
+ - Realtime Engine — Registry / RetryTracker internals
+ - Subscribe over SSE — full client example with reconnect
+ - Ack a notification — marking the row read
+
+
+ Mobile push delivers notifications to a native Android or iOS app via
+ Firebase Cloud Messaging (HTTP v1). v0.1 ships the FCM provider;
+ APNs / Azure / AWS slot into the same ChannelMobilePush
+ registry entry in later waves.
+
+ Anything you'd want to push to a native app whose process isn't + currently running. Chat messages, calendar reminders, system + alerts, anything that needs the OS notification surface. +
+ +
+ The provider is a ~150-line hand-rolled HTTP v1 client — we
+ deliberately do not use the Firebase Admin SDK because v1 send is
+ one POST and JWT auth is a single call into
+ golang.org/x/oauth2/google; the Admin SDK's transitive
+ closure isn't worth it for the surface notify exercises.
+
+ The provider parses the service-account JSON eagerly during
+ fcm.New, so a malformed credential blob fails at
+ construction — not on first send. The OAuth2 token source is built
+ once and reused across sends.
+
+ if (!task.isSuccessful) {
+ Log.w("notify", "Fetching FCM registration token failed", task.exception)
+ return@OnCompleteListener
+ }
+ val token = task.result
+ // Send token to notify.
+ notifyApi.registerPushToken(
+ RegisterPushTokenRequest(
+ deviceType = DeviceType.DEVICE_TYPE_ANDROID,
+ token = token,
+ )
+ )
+ })`} lang="kotlin" />
+
+ iOS — registering an FCM token
+
+ Apple Push Notification service (APNs) sits under FCM here — your
+ Xcode project registers for remote notifications and Firebase
+ surfaces an FCM token. notify treats it as one address, regardless
+ of upstream pipeline.
+
+
+ Sending a mobile push
+ Wire payload
+ The provider posts this body to /v1/projects/$PROJECT/messages:send:
+
+ The data map comes from Message.Data; it is
+ delivered to your client as a flat string-to-string map alongside
+ the notification.
+
+
+ Handling ErrUnregisteredToken
+
+ FCM signals "this device token is no longer valid" two ways:
+ HTTP 404 + status: NOT_FOUND, or any
+ response with details[].errorCode == "UNREGISTERED".
+ The provider folds both into ErrUnregisteredToken:
+
+
+ The provider itself never auto-purges. See
+ Channels & Providers
+ for the rationale.
+
+
+ Testing without hitting FCM
+
+ Both the HTTP client and the OAuth2 token source are injectable via
+ constructor options. Tests use an httptest.Server and a
+ static-token oauth2.TokenSource so they never reach
+ google.com:
+
+ Receipts
+
+ On success the receipt carries the FCM message name (e.g.
+ projects/my-project/messages/100) as
+ ProviderMessageID and StatusDelivered.
+
+
+ Future providers
+
+ - APNs direct — bypass FCM for iOS, P8 JWT auth.
+ - Azure Notification Hubs.
+ - AWS SNS Mobile Push.
+
+
+ Related
+
+
+ The SMS channel delivers a text message via Twilio's
+ Messages API. The same Twilio Client backs
+ both this provider and the WhatsApp
+ provider — the channel-specific rules live on the provider, not
+ the client.
+
+ OTP codes, urgent alerts, anything that needs to reach a user when + they don't have your app installed and aren't reading email. +
+ ++ Alternatively configure a Messaging Service for sender pools / pool-of-numbers: +
+
+ When both From and MessagingServiceSID are
+ set, the provider prefers the Messaging Service.
+
Response:
+
+ The provider does a cheap structural E.164 check before calling
+ Twilio — empty, missing +, non-digit body, or length
+ outside 6..15 short-circuits to
+ StatusFailed:
+
15 {
+ return fmt.Errorf("To: %q has invalid length", s)
+ }
+ for _, r := range digits {
+ if r < '0' || r > '9' {
+ return fmt.Errorf("To: %q contains non-digit characters", s)
+ }
+ }
+ return nil
+}`} lang="go" />
+
+
+ This is intentionally structural, not a phone-number validity check
+ — Twilio remains the authority on whether a number can actually
+ receive SMS. We just avoid burning a request on obvious garbage.
+
+
+ Body composition
+
+ SMS does not carry a separate subject line. The provider collapses
+ Title and Body with Body winning when non-empty:
+
+
+ - Both supplied → only
Body is sent.
+ - Only
Title → Title is sent.
+ - Neither → empty body; Twilio rejects with code 21602.
+
+
+ Testing with httptest
+
+ Because the provider uses Go's net/http via an
+ injectable http.Client, you can swap the Twilio endpoint
+ for an httptest.Server and drive any response shape from
+ a unit test. The full test lives in
+ channels/twilio/sms_test.go;
+ abbreviated shape:
+
+ Receipts and error handling
+
+ On success the receipt carries Twilio's SID as
+ ProviderMessageID and StatusDelivered. On
+ failure the wrapped error is propagated and the row is recorded as
+ StatusFailed.
+
+
+ Related
+
+ - WhatsApp channel — same client, different prefix
+ - Channels & Providers
+ - Send a notification
+
+
+ Web Push delivers notifications to a browser even when your tab is
+ closed, via the W3C Push API (RFC 8030) with VAPID (RFC 8292) for
+ application-server identification and the aes128gcm
+ content encoding (RFC 8291) for payload encryption.
+
+ Anything you'd push to a desktop or mobile-web user who isn't + currently looking at your app. Common cases: chat messages while the + tab is in the background, new email/comment alerts, urgent system + notifications. +
+ +
+ The provider implementation lives in
+ channels/webpush and wraps
+ github.com/SherClockHolmes/webpush-go for the on-wire
+ encryption work. notify owns:
+
endpoint / keys.auth / keys.p256dh early).mailto:, https://, or bare user@host).TTL (60s) and Urgency (normal), overridable via options.200/201/202 → StatusDelivered; 410 Gone → ErrSubscriptionGone; everything else → StatusFailed with the body text.
+ The contact email is required by RFC 8292; push services may reject
+ anonymous senders. mailto: and https://
+ URLs are both accepted; a bare user@host string is
+ auto-prefixed with mailto:.
+
{
+ const data = event.data?.json() ?? {};
+ event.waitUntil(
+ self.registration.showNotification(data.title || "Notification", {
+ body: data.body,
+ data: data.data, // your deep-link key/values
+ icon: "/icon-192.png",
+ })
+ );
+});
+
+self.addEventListener("notificationclick", (event) => {
+ event.notification.close();
+ event.waitUntil(
+ clients.openWindow(event.notification.data?.url || "/")
+ );
+});`} lang="js" title="public/sw.js" />
+
+ Browser side — subscribe + register
+ c.charCodeAt(0));
+}
+
+async function subscribePush(vapidPublicKey) {
+ const reg = await navigator.serviceWorker.register("/sw.js");
+ await navigator.serviceWorker.ready;
+
+ let sub = await reg.pushManager.getSubscription();
+ if (!sub) {
+ sub = await reg.pushManager.subscribe({
+ userVisibleOnly: true,
+ applicationServerKey: urlBase64ToUint8Array(vapidPublicKey),
+ });
+ }
+
+ // Register with notify.
+ await client.registerPushToken({
+ deviceType: "DEVICE_TYPE_BROWSER",
+ token: JSON.stringify(sub), // the whole PushSubscription JSON
+ });
+}`} lang="ts" title="subscribe.ts" />
+
+ Sending a web push
+
+ The provider expects the JSON-stringified PushSubscription
+ as the address. In production this comes from the device store
+ (RegisterPushToken writes it); in a one-off call you can
+ inline it in the Addresses map.
+
+ Handling ErrSubscriptionGone
+
+ HTTP 410 from the push service means the user has unsubscribed or
+ uninstalled the browser. The provider surfaces this as a sentinel
+ you can match with errors.Is:
+
+ Why the provider does not auto-purge
+
+ Why this design.
+ Both web push (ErrSubscriptionGone) and FCM
+ (ErrUnregisteredToken) expose token-is-dead as a typed
+ error. The provider never deletes the corresponding device row
+ itself because (a) the device store is owned by the orchestrator,
+ not the provider, and (b) a future caller may want to keep the row
+ for audit or to attempt re-registration. Providers stay stateless;
+ purge logic lives at the call site.
+
+
+ Receipts
+
+ Push services don't return a message id, so successful receipts
+ have an empty ProviderMessageID. The status is
+ StatusDelivered on 200/201/202.
+
+
+ Related
+
+ - Mobile push channel — sibling channel for native apps
+ - Register a push token — full round-trip example
+ - Channels & Providers
+
+
+ The WhatsApp channel delivers messages via Twilio's WhatsApp Business
+ pipeline. It reuses the same Twilio Client as
+ SMS — only the wire-level
+ formatting differs: both addresses are prefixed with
+ whatsapp: so Twilio routes through the WhatsApp pipeline
+ instead of SMS.
+
+ Where WhatsApp is the dominant messaging app (LatAm, India, + Indonesia, much of EMEA) and your users have opted in via a + pre-approved template flow on the Twilio side. +
+ +
+ Pass the From number without the whatsapp:
+ prefix — the provider adds it for you. Same Twilio credentials can
+ drive both the SMS and WhatsApp channels; the channel kind
+ disambiguates in the registry.
+
+ The producer supplies the recipient as a bare E.164 number — same as + SMS — and the channel decides the routing. +
+
+ The provider builds the Twilio request with the
+ whatsapp: prefix automatically:
+
+ ensureWhatsAppPrefix is idempotent — calling it on
+ "whatsapp:+15555550000" returns the same string.
+ Callers that have already prefixed their config (legacy code paths,
+ custom Sender wrappers) won't get a double prefix.
+
+ The provider strips the whatsapp: prefix for validation
+ and runs the same E.164 structural check used by SMS. Empty,
+ non-E.164, or length-out-of-range addresses short-circuit to
+ StatusFailed without burning a Twilio request.
+
+ Same as SMS: Body wins, falling back to Title.
+ Twilio's WhatsApp pipeline accepts plain text for in-session messages
+ and approved template references for out-of-session — at the proto
+ level notify does not model templates today; pass the rendered text
+ in Body.
+
+ Messaging Services do not take the whatsapp: prefix —
+ the prefix only applies to bare From numbers. The
+ provider handles this:
+
+ Same shape as SMS: success returns the Twilio SID as
+ ProviderMessageID, failure surfaces the error.
+
+ notify is one Go package (github.com/elloloop/notify) that
+ depends on nothing concrete, plus pluggable sub-packages for each
+ backend. The container (cmd/notifyd) is a thin wiring
+ layer; the library mode is the same logic with the wiring deferred to
+ the caller.
+
+ notify.Notifier is the orchestrator. It owns a
+ Store and a ProviderRegistry and exposes a
+ single method:
+
For each (user, channel) the orchestrator:
Notification row stamped with the request fields.Store.CreateNotification — idempotent on (TenantID, UserID, NotificationID). A repeat call returns the existing row.channel in the registry. If none, increments Pending and continues.req.Addresses (or, for in-app, the user id itself). Missing address ⇒ Pending.Provider.Send. On error, increments Failed and records StatusFailed via UpdateStatus. On success, records the receipt's status (default StatusDelivered).
+ Storage failures abort the whole call. Per-channel delivery failures
+ do not — the fan-out completes and the
+ (delivered, pending, failed) counters surface what
+ happened.
+
+ The root package has no concrete dependencies — it imports only
+ standard library and connect-go (for typed errors). That
+ is what makes it library-mode-friendly: a downstream service can
+ embed Notifier with its own Store and never
+ pull in Twilio or FCM or pgx.
+
+ cmd/notifyd is the production entry point. It does three
+ things and only three:
+
server.Config from environment variables.server.New and calls Run.
+ The internal/server package owns the Connect handlers,
+ auth middleware, lifecycle plumbing, and the in-app provider that
+ bridges the realtime engine. It depends on the root package
+ (interfaces), the generated proto stubs (wire types), and the
+ realtime + store / channel sub-packages — exactly the surface a
+ production deployment wires.
+
|
+| · StreamEvents (SSE / Connect server-stream) |
+| · GetNotifications, AckNotification, AckDataChange |
+| · RegisterPushToken |
+| │ |
+| ┌───────────┴────────────────────────────────┐ |
+| ▼ ▼ |
+| notify.Notifier ◄────── realtime.Registry / RetryTracker |
+| │ (only when LiveConnections.Enabled) |
+| ▼ |
+| notify.Store |
+| │ |
+| ▼ |
+| memory · postgres · entdb |
+| |
+| :9090 /healthz · /metrics (Prometheus exposition) |
++---------------------------------------------------------------------------+`} lang="text" />
+
+ Why two services, not one
+
+ The producer surface and the recipient surface have orthogonal auth
+ stories: producers are trusted backend code, recipients are
+ untrusted end users. Splitting them into two Connect services means:
+
+
+ - The internal token never touches a browser; the user JWT never reaches a producer.
+ - Operators can publish the recipient port behind a CDN/WAF and keep the producer port reachable only from the cluster network — at the listener level, not via middleware that could regress.
+ - The schemas evolve independently. Adding a producer-side field never expands the recipient API by accident.
+
+
+ Why notification IDs are caller-supplied
+
+ Why this design.
+ The platform is idempotent on
+ (tenant_id, user_id, notification_id) because the producer
+ knows when a logical event is "the same one" and the platform does not.
+ Examples: a webhook retry, a Kafka redelivery, an at-least-once event
+ fanout in the producer. If notify generated the id, the producer
+ would either have to send "same logical event" twice (duplicate
+ delivery) or maintain its own dedupe table that mirrors notify's. We
+ push the decision to the source of truth — the producer.
+
+
+ Related
+
+ - Channels & Providers — the channel ↔ provider taxonomy in detail
+ - Store & Conformance — the driver-agnostic spec
+ - Realtime Engine — Registry, RetryTracker, Conn
+ - Auth Model — JWT + internal token
+
++ notify validates two kinds of credentials, one per service: +
+NotificationClientService (browser / mobile) — Authorization: Bearer <JWT>, HS256 against NOTIFY_AUTH_JWT_SECRET.NotificationInternalService (backend producers) — X-Notify-Internal-Token, constant-time compare against NOTIFY_INTERNAL_TOKEN.
+ Anything calling Notify (sending a notification) is a
+ backend producer and uses the internal token. Anything streaming a
+ user's own notifications, reading their inbox, marking rows read or
+ registering a push token is the recipient SPA / mobile app and uses
+ a JWT issued for that user.
+
+ The validator is a thin wrapper around github.com/golang-jwt/jwt/v5:
+
Token requirements:
+jwt.WithValidMethods).sub claim must be a non-empty string. It becomes Claims.UserID.tenant (or tenant_id) claim must be a non-empty string. It becomes Claims.TenantID.NOTIFY_AUTH_JWT_ISSUER is set, the iss claim must match.NOTIFY_AUTH_JWT_AUDIENCE is set, the aud claim must match.exp and nbf are validated with NOTIFY_AUTH_JWT_LEEWAY (default 30s) slack.A valid token results in a typed Claims on the request context:
+ Handlers read it via ClaimsFromContext(ctx). The raw
+ jwt.MapClaims is intentionally not exposed — handlers
+ never reach for fields the platform did not promise.
+
+ Why this design. + Every client-side RPC scopes data by+ +(tenant, user). If a + client could pick the tenant via the request body, a logged-in user + could mint requests that read another tenant's data simply by typing + a different tenant_id. The platform takes +Claims.TenantIDfrom the JWT and ignores any + request-body tenant field on the client surface — the producer + surface is the only place tenant is supplied via the body, and that + surface is internal-token gated. +
+ subtle.ConstantTimeCompare avoids the timing side-channel
+ of ==. The container refuses to boot with
+ DevMode=false and no NOTIFY_INTERNAL_TOKEN
+ set — there is no "internal calls go unauthenticated" footgun.
+
+ NOTIFY_AUTH_DEV_MODE=true relaxes the boot-time validation
+ so local development doesn't need a JWT signer:
+
Bearer dev:<userid>:<tenant>[:<email>].NOTIFY_INTERNAL_TOKEN is also unset.Sample dev tokens:
++ Never enable dev mode in production. The container + rejects boots that try to disable both real auth modes + simultaneously. +
+ ++ Authentication answers "is this user who they say they are". notify + answers it via JWT verification. +
+
+ Authorization — "may this authenticated user read this row?" — is
+ enforced at the handler boundary by scoping every Store call to
+ (claims.TenantID, claims.UserID). The standout case is
+ AckNotification: Store.UpdateStatus takes
+ only tenantID + id, not userID,
+ so the handler does a GetNotification(claims.TenantID,
+ claims.UserID, id) first. A cross-user attempt surfaces as
+ ErrNotFound, which maps to CodeNotFound on
+ the wire — never CodePermissionDenied, which would leak
+ the existence of someone else's row.
+
+ A channel is the kind of delivery rail: + in-app, email, web push, mobile push, SMS, WhatsApp. A + provider is the concrete backend that handles that + rail — Twilio for SMS, VAPID for web push, the emailservice Sender for + email, etc. notify lets you mix and match. +
+ ++ Whenever you ask "how do I send X over Y": X is the channel, Y is + the provider. The channel decides which destination shape applies + (an email address, an E.164 phone number, a device token, the user id); + the provider decides which API does the talking and which credentials + sit on disk. +
+ +
+ The proto enum (DeliveryChannel) carries the same set,
+ with frozen field numbers. Adding a new channel is an additive proto
+ change plus a new ChannelKind constant in
+ notify/model.go — never reuse, never renumber.
+
| Channel | Shipped providers | Future providers |
|---|---|---|
email | emailservice (DI Sender) | ses, acs, smtp, sendgrid |
sms | twilio | sns, acs |
whatsapp | twilio | meta |
web_push | vapid | — |
mobile_push | fcm | apns, azure, aws |
in_app | built-in realtime engine | — |
+ Every provider satisfies one tiny contract: +
+
+ On error the orchestrator records the row as StatusFailed
+ and counts it in NotifyResult.Failed. The rest of the
+ fan-out continues.
+
+ A channel is active when a provider has been registered for it.
+ Unregistered channels are still legal in a NotifyRequest —
+ the row is stored and counted as Pending, but no provider
+ call happens.
+
In library mode you register explicitly:
+
+ In container mode cmd/notifyd registers providers based on
+ the configured NOTIFY_<CHANNEL>_PROVIDER blocks. An
+ empty block leaves the channel disabled.
+
+ The orchestrator resolves an address per (user, channel)
+ in this order:
+
NotifyRequest.Addresses[userID][channel] is set, use it.in_app, use the user id itself.Pending.+ The Addresses map is opt-in. For email/SMS/WhatsApp/web-push/mobile-push + you almost always supply it (or your producer reads it from your own + user store and assembles the map before calling Notify). For in-app + you almost never need it. +
+ ++ Why this design. + Both web push (+ +ErrSubscriptionGoneon HTTP 410) and FCM + (ErrUnregisteredTokenon UNREGISTERED / NOT_FOUND) expose + "the recipient endpoint is gone" as a sentinel error. The provider + never deletes the correspondingDevicerow itself — it + just surfaces the sentinel. Why? Because the device store is owned by + the orchestrator / caller, and a future caller may want to keep the + row for audit, retry from a fresher token, or analytics. Providers + are stateless from the platform's POV. +
+ notify carries a tenant_id on every persisted row and
+ every RPC. The model is deliberately lightweight: tenants are an
+ isolation key, not a billing identity or a feature gate.
+
tenant claim and want that to flow through to the inbox.+ Two paths, depending on the service: +
+NotificationInternalService) — the body's tenant_id field. The internal-token check confirms the caller is trusted; the caller is expected to know which tenant it is acting on behalf of.NotificationClientService) — the JWT's tenant (or tenant_id) claim. The request body has no tenant field at all on these RPCs.
+ Every Store method that reads or writes carries a tenantID
+ parameter, and every implementation honors it as a hard filter:
+
CreateNotification — the (TenantID, UserID, NotificationID) triple is the uniqueness key. Re-using a notification id under a different tenant produces a new row, not an idempotent hit.GetNotification — a row that exists under a different tenant returns ErrNotFound. The same goes for cross-user reads under the same tenant.QueryUserNotifications — scoped by (TenantID, UserID). The cursor and unread count are also scoped.UpsertDevice / ListDevices — same shape.
+ The conformance suite exercises this with a dedicated
+ UserIsolation subtest that creates three rows
+ ((acme, u1, x), (acme, u2, x),
+ (beta, u1, x)) and asserts that
+ QueryUserNotifications for (acme, u1)
+ returns exactly the first.
+
+ Notify is happy to serve many tenants from one container — the + tenant id is just data. There is no per-tenant configuration knob + today, no per-tenant rate limiting, no per-tenant provider override. + Provider configuration is global, and the same providers serve every + tenant. +
+
+ If you need per-tenant behaviour (e.g. different SMS sender numbers
+ per customer), the right shape today is library mode: wire one
+ Notifier per tenant with a tenant-specific
+ ProviderRegistry sharing a single underlying
+ Store. The container will grow this as a follow-up wave;
+ until then library mode owns the customization story.
+
+ EntDB itself is tenant-sharded. The notify EntDB driver maps onto a
+ single EntDB tenant per process via NOTIFY_ENTDB_TENANT_ID:
+
+ Inside notify's logical model, you can still serve many tenants over
+ that one EntDB shard — they all live under the same EntDB tenant,
+ differentiated by the tenant_id field on each row. The
+ EntDB tenant is the storage shard; the notify tenant is the
+ application-level isolation key.
+
+ Operationally large customers may want a dedicated EntDB shard per
+ customer. The shape is to run a notify container per shard,
+ pointing each at its dedicated NOTIFY_ENTDB_TENANT_ID.
+
+ The Postgres schema carries tenant_id as a column on
+ both tables. Every WHERE clause includes it; every UNIQUE constraint
+ starts with it. There is no Postgres-level row security or schema
+ fan-out — the application-level filter is the contract.
+
+ Why this design. + There is no "fanout to all tenants" RPC, no "list every tenant" + admin endpoint, no global query. The tenant boundary is a hard wall + because the auth boundary is — the JWT pins the recipient to one + tenant, and the producer surface is per-call tenant-scoped. + Cross-tenant operations would either need an admin role concept the + platform doesn't model (yet) or a producer with super-user reach + that defeats the isolation promise. Neither pays off enough at v0.1 + to be worth the footgun. ++ +
+ The realtime package is the in-memory engine that powers
+ in-app delivery: a per-user connection registry, a buffered per-conn
+ channel, and an at-least-once retry tracker. It is generic over the
+ event payload (Registry[T any]), so it has zero
+ dependency on the notify proto or any transport — the server
+ instantiates Registry[*pb.StreamEvent] at boot.
+
+ You only need to know this exists if you are: +
+NOTIFY_LIVE_CONNECTIONS_ENABLED).
+ The server constructs one Registry shared by every
+ connection, and one RetryTracker shared by every
+ in-flight retry. StreamEvents creates a fresh
+ Conn on each open, registers it, and tears it down on
+ exit. The in-app provider's Send looks up live
+ connections for the target user via Registry.Push and
+ returns StatusDelivered if any accepted the event,
+ StatusPending if none did.
+
+ Why this design. + A stalled client must never block the producer. Every+ +Push+ is a non-blocking{`select { case ch <- ev: ... default: ... }`}; + a connection whose buffer is full has the event dropped (logged via +event_queue_full) but stays registered for future + events. The alternative — backing up the orchestrator behind one slow + client — would let one misbehaving recipient stall delivery for + everyone else on the same notifier. +
+ This is a subtle invariant documented inline in
+ realtime/conn.go:
+
+ Data-change hints (DataChangeEvent) are best-effort hints
+ that something the client cares about may have changed upstream.
+ The client re-fetches its own state via its own API. For these
+ hints, the platform offers at-least-once via the
+ RetryTracker:
+
DataChangeEvent with an idempotency_key.RetryTracker.Track(ctx, key, []connID, sendCallback) spawns a goroutine per (key, connID) that re-sends on NOTIFY_LIVE_RETRY_INTERVAL ticks.{`AckDataChange{idempotency_key, session_id}`}.
+ Budget is bounded: NOTIFY_LIVE_RETRY_MAX_ATTEMPTS (default
+ 3) caps retries; a value of 0 disables retries entirely. On shutdown
+ CancelAll closes every in-flight retry so they don't
+ leak.
+
+ Set NOTIFY_LIVE_CONNECTIONS_ENABLED=false. The container
+ never constructs a Registry, the in-app provider is not registered,
+ and StreamEvents returns
+ CodeUnimplemented. In-app notifications remain
+ StatusPending; clients catch up via
+ GetNotifications.
+
+ This is a legitimate deployment shape when you only care about the + durable side of the platform — e.g. a backend running notify in + library mode purely for stored notification fanout, with the live + surface served by a separate gateway. +
+ +
+ notify treats persistence as a pluggable concern. There is one
+ Store interface, three shipped drivers
+ (memory, postgres, entdb), and a
+ shared conformance suite that every driver runs. Adding a new driver
+ is a matter of implementing the interface and passing the suite.
+
Two sentinel errors round out the contract:
++ Why this design. + The realtime registry, the retry tracker, and per-user channel + preferences (future) all live outside the Store. The Store + holds only what is durable and shared across replicas: notification + rows and device registrations. Keeping the interface tight makes new + drivers cheap, makes the conformance suite tractable, and prevents + "the store" from becoming a leaky god-object. ++ +
+ store/conformance.RunConformance exercises every driver
+ against the same 24 leaf subtests across six categories. A failure
+ path like TestConformance/postgres/Concurrency/ConcurrentCreate_SameKey_SingleWinner
+ points straight at the driver and the exact semantic that broke.
+
| Category | Subtests | What it catches |
|---|---|---|
| Core CRUD | 8 | create / get / idempotency / status transitions / paging basics / unread filter / device upsert / user isolation |
| Pagination | 2 | full-set traversal across many pages, strict-less-than cursor semantics |
| FreshTenant | 3 | queries against an unseen tenant return empty, not error |
| RoundTrip | 3 | adversarial string values (emoji, unicode, SQL-shaped, leading/trailing whitespace, 10k chars), int64 fidelity, large payload bodies |
| Concurrency | 5 | distinct-keys-no-lost-writes, same-key-single-winner, device-upsert-same-key-single-row, update-status concurrency, read-your-writes |
| KeyEdge | 3 | long notification ids (256 chars), separator-byte collisions in composite keys, case-sensitive device types |
+ All shipped drivers pass 24/24: +
+store/<driver>/ with the implementation.store/<driver>/<driver>_test.go that runs the suite:.github/workflows/conformance.yml's matrix.go test ./store/<driver>/... -race -count=1. Any failing subtest gets a one-line root-cause attribution in store/<driver>/CONFORMANCE.md — see the existing two as templates.
+ The CI check Conformance / <driver> is what branch
+ protection pins. A driver that doesn't pass cannot land.
+
+ Production-shape Docker deployment of notify. For the "what is the
+ image" overview see Installation
+ → Docker; this page covers networking, secrets, healthchecks,
+ multi-arch, and a worked docker-compose stack.
+
docker-compose.
+ The published image is built for linux/amd64 and
+ linux/arm64. Docker pulls the right variant
+ automatically. To verify:
+
+ Every published tag is signed via cosign keyless OIDC. Verify before + deploying: +
+
+ Compose's .env file is fine for dev. Production:
+
+ notify speaks plaintext HTTP/2 (h2c) inside the container — terminate + TLS upstream. Typical shapes: +
+app.example.com {`{`} reverse_proxy notify:8080 {`}`}.grpc_pass for the internal port; proxy_pass for the client port (Connect over HTTP/1 works fine). /tmp/h.json; then
+ cat /tmp/h.json
+ break
+ fi
+ sleep 1
+done
+
+docker rm -f notify-smoke`} lang="bash" />
+
+
+ The same smoke is what .github/workflows/release.yml
+ runs against every published image before tagging it for promotion.
+
+
+ Resource sizing
+
+ notify's hot path is small: one Notifier.Notify is one
+ store insert plus one provider call per channel. The realtime
+ engine is in-memory and a per-conn buffered channel. Reasonable
+ starting points:
+
+
+ - CPU: 0.5 vCPU is enough for moderate traffic; the upper bound is dominated by provider call latency.
+ - Memory: 64 MiB baseline; budget ~5 KiB per active stream connection.
+ - Network: long-lived streams open a TCP connection per recipient — size your ephemeral port range accordingly if you expect 10k+ concurrent clients.
+
+
+ Related
+
+
+ notify publishes container images via
+ .github/workflows/release.yml. A push to any
+ v* tag triggers test → docker-smoke → build-and-push →
+ cosign sign → Trivy scan → proto bundle release. The exact same
+ workflow is also workflow_dispatch-triggerable for
+ one-off rebuilds.
+
+ Re-runs every gate that PRs run. Plain
+ go build ./... + go vet ./... +
+ go test -race over everything except the
+ driver-specific tree. A broken build must not publish.
+
+ Builds the server target locally
+ (docker buildx build --load), runs the container with
+ NOTIFY_AUTH_DEV_MODE=true +
+ NOTIFY_STORE_DRIVER=memory, and curls
+ /healthz until it sees
+ {`{"status":"ok"}`}. Fails the run if the container
+ crashes during boot.
+
/tmp/notify-health.json; then
+ grep -q '"status":"ok"' /tmp/notify-health.json
+ exit 0
+ fi
+ sleep 1
+done
+docker logs "$cid"
+exit 1`} lang="bash" title="docker-smoke (abridged)" />
+
+ 3. build-and-push
+
+ Cross-builds for linux/amd64 + linux/arm64
+ via docker/setup-qemu-action +
+ docker/setup-buildx-action, then publishes to
+ ghcr.io/elloloop/notify tagged from
+ docker/metadata-action — semver components, branch
+ name, sha, and latest on a tag push.
+
+
+ Permissions are scoped: contents: read,
+ packages: write (to push to ghcr),
+ id-token: write (for cosign keyless OIDC).
+
+
+ 4. scan-image
+
+ Post-publish, Trivy is pinned by SHA (defending against the
+ March 2026 advisory) and scans the published image. HIGH and
+ CRITICAL CVEs cause the workflow to fail; results are uploaded as
+ SARIF to GitHub Code Scanning.
+
+
+ 5. release-protos
+
+ On a tag push only. Packs the proto/ tree plus
+ buf.yaml + buf.gen.yaml into a tarball +
+ zip + sha256 file and attaches them to the GitHub Release. Consumers
+ who don't want to vendor a git ref can pin against the proto
+ bundle for the exact version.
+
+
+ Cutting a release
+
+ - Update
VERSION (or rely on the workflow's "Inject version from git tag" step).
+ - Open and merge a PR with the changelog entry / version bump.
+ - From
main: git tag v0.1.0 && git push origin v0.1.0.
+ - Workflow runs end-to-end: test → smoke → push → cosign → Trivy → proto bundle.
+ - The published image is reachable at
ghcr.io/elloloop/notify:0.1.0.
+
+
+ Conformance workflow (PR gate)
+
+ .github/workflows/conformance.yml is the PR-side gate.
+ It runs:
+
+
+ - A Unit job —
go build / vet / test -race over everything except the driver-specific tree.
+ - A Conformance / <driver> matrix —
memory, postgres (via the postgres:16.13-alpine3.23 service container), entdb (via ghcr.io/elloloop/tenant-shard-db:2.0.5 service container with the realentdb build tag).
+
+
+
+ Branch protection pins the Conformance / memory,
+ Conformance / postgres, Conformance / entdb,
+ and Unit checks. Nothing lands without all four green.
+
+
+ Local equivalent
+ Related
+
++ Manifests for a production-shape Kubernetes deployment of notify: + one Deployment, two Services (one internal-only, one exposed via + Ingress), a Secret for the credentials, and an HPA tuned for the + realtime workload. +
+ ++ You already run Kubernetes and want notify to ride your existing + operational story — Helm-ish chart, GitOps via Argo / Flux, mesh-of- + your-choice, etc. +
+ +"
+ NOTIFY_INTERNAL_TOKEN: "<32-byte hex>"
+ NOTIFY_POSTGRES_DSN: "postgres://notify:password@db.notify:5432/notify?sslmode=require"
+ TWILIO_AUTH_TOKEN: "..."
+ FCM_SERVICE_ACCOUNT_JSON: |
+ { "type": "service_account", ... }`} lang="yaml" title="00-namespace-secret.yaml" />
+
+
+ Project these from your secret manager via External Secrets Operator,
+ Sealed Secrets, or whatever your shop uses. The point is that they
+ live outside the manifest tree.
+
+
+ Deployment
+ Services
+ Ingress (client surface)
+
+ The long proxy-{`{read,send}`}-timeout values matter for
+ StreamEvents — without them nginx will close the
+ long-lived SSE connection after 60s.
+
+
+ NetworkPolicy (internal port lockdown)
+ Horizontal Pod Autoscaler
+
+ The realtime workload is connection-count bound rather than CPU
+ bound. The HPA below uses CPU as a coarse signal; when the
+ Prometheus metric surface is wired (follow-up wave) you can switch
+ to a custom metric (e.g. notify_live_connections).
+
+ PodDisruptionBudget
+ Rolling deploy semantics
+
+ - New pod boots,
/healthz returns 503 until Server.Run binds listeners → marks ready.
+ - Service starts routing traffic to the new pod.
+ - Old pod receives SIGTERM,
/healthz immediately flips to 503 → Service stops sending new traffic.
+ Shutdown drains in-flight requests within NOTIFY_SHUTDOWN_TIMEOUT.- Open SSE streams: each handler exits on context cancel, client reconnects to whichever pod the LB picks next.
+
+
+ Related
+
+
+ Two distinct acknowledgements live on the client surface:
+ AckNotification marks a stored notification as
+ Read; AckDataChange cancels the
+ at-least-once retry of a data-change hint. Both are idempotent;
+ both surface as one Connect call.
+
AckNotification — user opened the notification, viewed the inbox item, or otherwise consumed it. Decrements unread count, stamps read_at_ms.AckDataChange — the client successfully refetched the upstream state the hint pointed at; tells the server to stop retrying.
+ The id field is the store id (the
+ Notification.id the server assigned), not the
+ producer's notification_id:
+
+ The store transitions the row to StatusRead and stamps
+ read_at_ms to the server clock. A follow-up
+ GetNotifications shows the new status and an
+ unread_count reduced by one.
+
+ Why this design. ++ +Store.UpdateStatustakestenantIDand +idonly — notuserID. Without an + ownership check, any user in tenant T could mark any other user's + notification as Read by guessing the id. The handler does an + explicitGetNotification(claims.TenantID, claims.UserID, + id)first and maps a miss toCodeNotFound— + neverCodePermissionDenied, which would leak the + existence of someone else's row. +
+ Calling AckNotification on an already-read row is a
+ no-op that returns success. The store update is a status-set, not a
+ counter increment; multiple calls do not multiply the effect.
+
+ The handler calls RetryTracker.Ack(key, sessionID) on
+ the server's in-memory tracker. The matching retry goroutine
+ receives a context cancel; the entry is removed from the map. If
+ the (key, sessionID) pair is not registered (already acked,
+ process restart, etc.), the call is a no-op and still returns
+ success.
+
+ When NOTIFY_LIVE_CONNECTIONS_ENABLED=false, there is no
+ retry tracker and no live stream. AckDataChange returns
+ success without doing anything — clients calling it speculatively
+ do not need to gate on the subsystem state.
+
void }) {
+ return (
+ {
+ await client.ackNotification({ id: n.id });
+ onAcked();
+ }}
+ className={n.status === "DELIVERY_STATUS_READ" ? "muted" : "unread"}
+ >
+ {n.title}
+ {n.body}
+
+ );
+}`} lang="tsx" />
+
+ Mark-all-read
+
+ v0.1 ships no bulk-ack RPC. The recommended client-side shape is to
+ page the inbox with unreadOnly: true and ack each row.
+ A native MarkAllRead RPC is a candidate follow-up; if
+ you need it before then, file an issue on the repo.
+
+
+ Related
+
+ - API Reference
+ - Subscribe over SSE
+ - Realtime engine — RetryTracker invariants
+
+
+ One Notify call, many recipients, many channels per
+ recipient. The orchestrator stores one row per
+ (user, channel), attempts immediate delivery per row,
+ and returns aggregate counts so the producer knows what landed.
+
+ Business events that deserve every available rail: incident pages, + onboarding milestones, multi-recipient activity events ("Bob + commented on Task 42 — page Alice and Carol via in-app + email + + SMS"). +
+ +| User | Channel | Address resolved? | Provider configured? | Result |
|---|---|---|---|---|
| alice | in_app | yes (user id) | yes (auto) | delivered if live, else pending |
| alice | yes (alice@example.com) | yes | delivered | |
| alice | sms | yes (+15555550199) | yes | delivered |
| bob | in_app | yes | yes | delivered if live, else pending |
| bob | yes (bob@example.com) | yes | delivered | |
| bob | sms | no | yes | pending (no address) |
| carol | in_app | yes | yes | delivered if live, else pending |
| carol | no | yes | pending (no address) | |
| carol | sms | yes (+15555550158) | yes | delivered |
+ The orchestrator stores all nine rows (3 users × 3 channels) and
+ reports a NotifyResponse like:
+
+ Two of those pending are "no address"; the other two
+ are "user offline for in-app". Both are recoverable: the missing
+ addresses can be supplied in a future retry; the in-app rows will
+ surface on the next StreamEvents open via
+ GetNotifications.
+
+ The single notification_id covers all nine
+ rows. A second call with the same
+ (tenant, notification_id) but a wider channel list
+ will not create new rows for the already-stored
+ (user, channel) pairs — the Store's idempotency key is
+ (tenant, user, notification_id), and that triple
+ matches per row. The second call's NotifyResponse will report
+ those rows as pending (skipped at the storage layer)
+ or re-deliver via the providers if a provider is now available
+ where one wasn't before.
+
+ The practical recommendation: choose the channel list once,
+ at the producer, when the event is generated. Retries should not
+ change the channel set; if you need to add a channel, mint a new
+ notification_id for the addition.
+
+ Omitting channels tells the orchestrator to use every
+ channel that has both an active provider AND a destination for the
+ user. For the example above with all three providers configured,
+ leaving channels off produces the same nine rows.
+
+ The difference matters when a provider is disabled: with
+ channels=[in_app, email, sms] on a deployment where
+ SMS is disabled, the SMS rows are stored as pending
+ (recovery is "wire a Twilio provider, re-run"). With empty
+ channels on the same deployment, SMS isn't part of the
+ effective set at all — no row stored, no recovery story needed.
+
+ A user who logs in after the fanout opens
+ GetNotifications and sees every row addressed to them,
+ regardless of delivery outcome. The in-app row with
+ StatusPending looks the same in the UI as a delivered
+ row — it just hadn't been pushed live yet.
+
+ For a real producer with a user-store integration, the addresses + map is usually built from a single query over your user table: +
+ 0 {
+ addrs[uid] = ¬ifyv1.ChannelAddresses{ByChannel: m}
+ }
+ }
+
+ _, err = c.Notify(ctx, ¬ifyv1.NotifyRequest{
+ TenantId: "acme",
+ NotificationId: "incident-" + incidentID,
+ UserIds: userIDs,
+ Channels: []notifyv1.DeliveryChannel{
+ notifyv1.DeliveryChannel_DELIVERY_CHANNEL_IN_APP,
+ notifyv1.DeliveryChannel_DELIVERY_CHANNEL_EMAIL,
+ notifyv1.DeliveryChannel_DELIVERY_CHANNEL_SMS,
+ },
+ SubjectRef: "incident:" + incidentID,
+ SubjectType: "incident",
+ Title: "P1: checkout service unavailable",
+ Body: "Pagerduty incident " + incidentID + " fired.",
+ Addresses: addrs,
+ })
+ return err
+}`} lang="go" />
+
+ Partial failure
+
+ A provider error on one (user, channel) row records that row as
+ StatusFailed and increments NotifyResponse.failed;
+ the rest of the fan-out continues. Storage errors abort the call
+ immediately (the platform cannot guarantee the remaining rows would
+ be stored either).
+
+
+ Producer-side strategy: log non-zero
+ failed for ops alerts; do not retry the whole call
+ automatically — the idempotency key ensures the stored rows aren't
+ duplicated, but it does not re-attempt the failed provider sends.
+ To re-send only the failed channel for the affected users, mint a
+ fresh notification_id with the narrowed audience.
+
+
+ Related
+
+
+ End-to-end flow for both Web Push (browser, VAPID) and mobile push
+ (FCM). The token is stored in notify's device table keyed by
+ (tenant, user, device_type); sending then just supplies
+ the user id and notify resolves the right token per channel.
+
+ First-launch of your client app, or whenever the browser /OS rolls
+ the underlying token (push services do this occasionally). Always
+ re-register on app start — UpsertDevice is idempotent;
+ a no-change call is a cheap no-op.
+
{
+ const data = event.data?.json() ?? {};
+ event.waitUntil(
+ self.registration.showNotification(data.title || "Notification", {
+ body: data.body,
+ data: data.data,
+ icon: "/icon-192.png",
+ })
+ );
+});
+
+self.addEventListener("notificationclick", (event) => {
+ event.notification.close();
+ event.waitUntil(
+ clients.openWindow(event.notification.data?.url || "/")
+ );
+});`} lang="js" title="public/sw.js" />
+
+ 3. Subscribe and register
+ c.charCodeAt(0));
+}
+
+export async function enableWebPush(): Promise {
+ if (!("serviceWorker" in navigator) || !("PushManager" in window)) {
+ throw new Error("Push API not available in this browser");
+ }
+
+ // 1. Service worker.
+ const reg = await navigator.serviceWorker.register("/sw.js");
+ await navigator.serviceWorker.ready;
+
+ // 2. Permission. Must be triggered by a user gesture.
+ const perm = await Notification.requestPermission();
+ if (perm !== "granted") {
+ throw new Error("Notification permission denied");
+ }
+
+ // 3. PushManager subscription.
+ let sub = await reg.pushManager.getSubscription();
+ if (!sub) {
+ sub = await reg.pushManager.subscribe({
+ userVisibleOnly: true,
+ applicationServerKey: urlBase64ToUint8Array(VAPID_PUBLIC_KEY),
+ });
+ }
+
+ // 4. Hand the subscription JSON to notify.
+ await client.registerPushToken({
+ deviceType: "DEVICE_TYPE_BROWSER",
+ token: JSON.stringify(sub),
+ });
+}`} lang="ts" title="features/push/web-push.ts" />
+
+ 4. Send via Web Push
+
+ From the producer, you don't need to thread the address — once
+ registered, notify will resolve the device token automatically when
+ a follow-up wave wires "store-side address lookup". Until then, in
+ v0.1 supply the subscription JSON in Addresses as
+ well:
+
+ Mobile push (FCM) — full round-trip
+
+ 1. Set up FCM on the server
+ 2. Android client — register the FCM token
+
+ if (!task.isSuccessful) {
+ Log.w("notify", "FCM token fetch failed", task.exception)
+ return@addOnCompleteListener
+ }
+ val token = task.result
+ notifyApi.registerPushToken(
+ RegisterPushTokenRequest(
+ deviceType = DeviceType.DEVICE_TYPE_ANDROID,
+ token = token,
+ )
+ )
+ }
+ }
+}
+
+// Token rotation — implement onNewToken so an OS-side token roll
+// re-registers automatically.
+class NotifyMessagingService : FirebaseMessagingService() {
+ override fun onNewToken(token: String) {
+ super.onNewToken(token)
+ notifyApi.registerPushToken(
+ RegisterPushTokenRequest(
+ deviceType = DeviceType.DEVICE_TYPE_ANDROID,
+ token = token,
+ )
+ )
+ }
+}`} lang="kotlin" />
+
+ 3. iOS client — register the FCM token
+ Bool {
+ FirebaseApp.configure()
+ Messaging.messaging().delegate = self
+ UNUserNotificationCenter.current().requestAuthorization(options: [.alert, .badge, .sound]) { _, _ in }
+ application.registerForRemoteNotifications()
+ return true
+ }
+
+ func messaging(_ messaging: Messaging, didReceiveRegistrationToken fcmToken: String?) {
+ guard let token = fcmToken else { return }
+ notifyApi.registerPushToken(
+ deviceType: .ios,
+ token: token,
+ )
+ }
+}`} lang="swift" />
+
+ 4. Producer-side send via FCM
+ Handling dead tokens
+
+ Push tokens go stale — users uninstall apps, browsers unsubscribe,
+ OSes roll tokens. Both providers surface "this token is gone" as a
+ typed error:
+
+
+ - Web Push:
webpush.ErrSubscriptionGone on HTTP 410.
+ - FCM:
fcm.ErrUnregisteredToken on UNREGISTERED / NOT_FOUND.
+
+
+
+ The provider does not purge the device row itself — that's
+ the orchestrator/caller's job. The recommended shape is to wire a
+ callback on NotifyResponse failures that calls
+ RegisterPushToken with an empty token to delete the
+ row (a follow-up wave will add an explicit
+ DeleteDevice RPC; for now the recommended path is for
+ the failing producer to remove the token from its own consumer
+ user store and stop sending to that channel).
+
+
+ Related
+
+
+ Producer-side recipes for the Notify RPC. Same wire
+ contract from every language; only the SDK and the auth-header
+ plumbing differ.
+
+ Any backend service that wants to push a notification to a user + via notify. Webhook receivers, business-logic services, scheduled + job runners — all of them call this RPC. +
+ ++ The simplest possible producer. Useful for one-off debugging, smoke + tests against a fresh deployment, or generating fixtures. +
+Response:
+
+ delivered=1 = the email send succeeded.
+ pending=1 = the in-app row was stored, but no live
+ connection is open for user-alice; it will surface on
+ their next StreamEvents open via GetNotifications.
+
+ Production-shape Go producer using the generated Connect client. This + is what an api-gateway-style consumer looks like. +
++ For services using Python. The same pattern works with the gRPC + stubs — Connect's JSON-over-HTTP is just easier to debug. +
+
+ Pick a notification_id that is deterministic from the
+ triggering event. Two examples:
+
"task-{`{taskID}`}-comment-{`{commentID}`}". Webhook retries of the same comment produce the same id; notify de-dupes."summary-{`{userID}`}-{`{YYYY-MM-DD}`}". A cron re-run within the same day produces the same id; notify de-dupes.+ Avoid wall-clock-derived ids for retryable events — a slow retry + crossing a millisecond boundary would create a duplicate. The + one place wall-clock is fine is event-source ids that include + their own monotonic offset (Kafka offsets, etc.). +
+ +
+ The Addresses map is per-user. Mix-and-match
+ destinations across recipients in one call:
+
+ Omit channels entirely (or pass an empty list) to fan
+ out to every channel that has both an active provider AND a
+ destination for the user. The orchestrator computes the effective
+ set at request time.
+
+ Open a long-lived stream of new notifications, data-change hints,
+ and heartbeats from a browser or mobile client. The platform speaks
+ Connect over HTTP/2; StreamEvents is a server-streaming
+ RPC, so it works equally over plain SSE-style framing.
+
+ Any client surface that wants real-time updates: a web SPA showing + chat messages without a refresh, a mobile app updating a task list + while the user has it open, a desktop tray app surfacing system + notifications. +
+ + Promise) {
+ const transport = createConnectTransport({
+ baseUrl,
+ interceptors: [
+ (next) => async (req) => {
+ const token = await getToken();
+ req.header.set("Authorization", \`Bearer \${token}\`);
+ return next(req);
+ },
+ ],
+ });
+ return createClient(NotificationClientService, transport);
+}`} lang="ts" title="shared/notify-client.ts" />
+
+ Promise;
+ onEvent: (ev: NotifyEvent) => void;
+ onError?: (err: unknown) => void;
+}): Stream {
+ const client = buildNotifyClient(opts.baseUrl, opts.getToken);
+ const abort = new AbortController();
+ let currentSession = "";
+
+ void (async () => {
+ let backoffMs = 500;
+ const maxMs = 30_000;
+
+ while (!abort.signal.aborted) {
+ try {
+ const it = client.streamEvents(
+ { deviceType: "DEVICE_TYPE_BROWSER" },
+ { signal: abort.signal },
+ );
+ for await (const ev of it) {
+ currentSession = ev.sessionId || currentSession;
+ backoffMs = 500; // reset on the first received event
+
+ switch (ev.event.case) {
+ case "notification": {
+ const n = ev.event.value.notification!;
+ opts.onEvent({
+ kind: "notification",
+ id: n.id,
+ title: n.title,
+ body: n.body,
+ subjectRef: n.subjectRef,
+ });
+ break;
+ }
+ case "dataChange": {
+ const dc = ev.event.value;
+ opts.onEvent({
+ kind: "dataChange",
+ idempotencyKey: dc.idempotencyKey,
+ subjectRef: dc.subjectRef,
+ subjectType: dc.subjectType,
+ });
+ // Ack so the server stops retrying.
+ await client.ackDataChange({
+ idempotencyKey: dc.idempotencyKey,
+ sessionId: currentSession,
+ });
+ break;
+ }
+ case "heartbeat":
+ opts.onEvent({
+ kind: "heartbeat",
+ timestampMs: ev.event.value.timestampMs,
+ });
+ break;
+ }
+ }
+ } catch (err) {
+ if (abort.signal.aborted) return;
+ opts.onError?.(err);
+ // Exponential backoff with jitter and a cap.
+ await new Promise((r) => setTimeout(r, backoffMs + Math.random() * 250));
+ backoffMs = Math.min(backoffMs * 2, maxMs);
+ }
+ }
+ })();
+
+ return {
+ cancel: () => abort.abort(),
+ sessionId: () => currentSession,
+ };
+}`} lang="ts" title="features/notifications/stream.ts" />
+
+ {
+ const stream = openStream({
+ baseUrl: "https://notify.example.com",
+ getToken: async () => localStorage.getItem("access_token") || "",
+ onEvent: (ev) => {
+ if (ev.kind === "notification") {
+ setCount((c) => c + 1);
+ }
+ },
+ onError: (err) => console.warn("stream error:", err),
+ });
+ return () => stream.cancel();
+ }, []);
+
+ return 🔔 {count};
+}`} lang="tsx" title="app/page.tsx" />
+
+ The handshake heartbeat
+
+ The server immediately sends one heartbeat after registering
+ your connection, before any traffic. That heartbeat carries the
+ server-assigned session_id, which is what you pass back
+ on AckDataChange to scope the cancellation to this
+ exact stream. The code above stores it in currentSession.
+
+
+ Reconnect strategy
+
+ Streams die for legitimate reasons: laptop sleep, server rolling
+ deploy, mobile carrier handover, network blip. The code above
+ reconnects with exponential backoff (500ms → 1s → 2s → ... → 30s)
+ with a 0–250ms jitter so a herd of clients does not synchronise
+ its reconnects.
+
+
+ Reconnecting is safe: any in-flight events the client missed are
+ still in the store and available via
+ GetNotifications. Data-change events the client did
+ not ack will be retried by the server's RetryTracker
+ until either the ack arrives or
+ NOTIFY_LIVE_RETRY_MAX_ATTEMPTS is exhausted.
+
+
+ Cold-start catch-up
+
+ A typical UI fetches the inbox once on app load to populate the
+ list, then opens the stream for live updates:
+
+ Recipe — raw cURL (for debugging only)
+
+ Useful for confirming that the stream works at all. Connect over
+ HTTP/2 uses length-prefixed JSON frames, not text/event-stream, so
+ cURL output is not human-friendly. Use --no-buffer and
+ expect binary framing.
+
+
+ For a readable debug stream, use the Connect-Go client in a small
+ Go test harness, or the connect-web client in
+ node --experimental-fetch.
+
+
+ Authorization on streams
+
+ The JWT is validated at handshake time and the resulting claims live
+ on the stream's context for its entire lifetime. The
+ platform does not re-validate mid-stream. If a token's
+ exp is reached during a long-lived stream, the existing
+ stream keeps flowing until disconnect; the client must mint a fresh
+ token before its next reconnect.
+
+
+ Related
+
+
+ Every container option is set via a NOTIFY_* environment
+ variable. cmd/notifyd calls
+ server.LoadConfigFromEnv() on boot; invalid values
+ (negative ports, missing required fields, unknown enums) fail fast
+ with a descriptive error so production never silently runs
+ misconfigured.
+
+ You're standing up a deployment, debugging a startup error, or moving + a provider configuration from one environment to another. The table + below is the full surface — there are no hidden flags. +
+ +| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_CLIENT_PORT | int | 8080 | Public Connect/HTTP/2 listener (NotificationClientService). |
NOTIFY_INTERNAL_PORT | int | 8081 | Private gRPC listener (NotificationInternalService). |
NOTIFY_METRICS_PORT | int | 9090 | /healthz + /metrics listener. |
NOTIFY_LOG_LEVEL | enum | info | debug · info · warn · error. |
NOTIFY_SHUTDOWN_TIMEOUT | duration | 30s | Graceful-shutdown deadline (Go duration syntax: 30s, 2m, etc.). |
NOTIFY_ALLOWED_ORIGINS | csv | — | Comma-separated CORS origins for the client listener. |
| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_STORE_DRIVER | enum | memory | memory · postgres · entdb. |
NOTIFY_POSTGRES_DSN | string | — | Required when driver=postgres. libpq-style URL. |
NOTIFY_POSTGRES_AUTOMIGRATE | bool | true | Apply pending migrations on connect. |
NOTIFY_ENTDB_ADDRESS | string | — | Required when driver=entdb. host:port. |
NOTIFY_ENTDB_TENANT_ID | string | — | Required when driver=entdb. EntDB tenant id (the storage shard). |
| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_AUTH_JWT_SECRET | string | — | HS256 verification key. Required unless dev mode. |
NOTIFY_AUTH_JWT_ISSUER | string | — | Pinned iss claim, if set. |
NOTIFY_AUTH_JWT_AUDIENCE | string | — | Pinned aud claim, if set. |
NOTIFY_AUTH_JWT_LEEWAY | duration | 30s | Allowed clock skew when validating exp / nbf. |
NOTIFY_INTERNAL_TOKEN | string | — | Shared secret for X-Notify-Internal-Token. Required unless dev mode. |
NOTIFY_AUTH_DEV_MODE | bool | false | Accepts Bearer dev:<uid>:<tenant>. Local-dev only. |
See Auth Model for the full validation flow.
+ +| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_LIVE_CONNECTIONS_ENABLED | bool | true | When false, StreamEvents returns Unimplemented. |
NOTIFY_LIVE_HEARTBEAT_INTERVAL | duration | 30s | Per-connection heartbeat cadence. |
NOTIFY_LIVE_RETRY_MAX_ATTEMPTS | int | 3 | At-least-once retry budget for data-change events. 0 disables. |
NOTIFY_LIVE_RETRY_INTERVAL | duration | 5s | Interval between retry attempts. |
| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_EMAIL_PROVIDER | enum | none | none · emailservice (others land in later waves). |
NOTIFY_EMAIL_FROM | string | — | Default From address. Required when a provider is set. |
NOTIFY_EMAIL_SERVICE_ADDRESS | string | — | host:port of the elloloop EmailService. Required for emailservice. |
NOTIFY_EMAIL_SMTP_HOST | string | — | SMTP relay host (future SMTP provider). |
NOTIFY_EMAIL_SMTP_PORT | int | — | SMTP port. |
NOTIFY_EMAIL_SMTP_USERNAME | string | — | SMTP username. |
NOTIFY_EMAIL_SMTP_PASSWORD | string | — | SMTP password. |
NOTIFY_EMAIL_API_KEY | string | — | API key for SES/SendGrid/ACS providers. |
NOTIFY_EMAIL_REGION | string | — | Provider region (SES). |
NOTIFY_EMAIL_ENDPOINT | string | — | Provider endpoint override. |
| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_SMS_PROVIDER | enum | — | twilio (others land later). |
NOTIFY_SMS_ACCOUNT_SID | string | — | Twilio Account SID. Required when provider=twilio. |
NOTIFY_SMS_AUTH_TOKEN | string | — | Twilio Auth Token. Required when provider=twilio. |
NOTIFY_SMS_MESSAGING_SERVICE_SID | string | — | Optional Twilio Messaging Service SID. |
NOTIFY_SMS_FROM | string | — | E.164 sender (e.g. +15555550000). |
NOTIFY_WHATSAPP_PROVIDER | enum | — | twilio. |
NOTIFY_WHATSAPP_ACCOUNT_SID | string | — | Required when provider=twilio. |
NOTIFY_WHATSAPP_AUTH_TOKEN | string | — | Required when provider=twilio. |
NOTIFY_WHATSAPP_FROM | string | — | E.164 sender; the whatsapp: prefix is added automatically. |
| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_WEBPUSH_PROVIDER | enum | — | vapid. |
NOTIFY_WEBPUSH_VAPID_PUBLIC | string | — | VAPID public key (base64url). |
NOTIFY_WEBPUSH_VAPID_PRIVATE | string | — | VAPID private key (base64url). |
NOTIFY_WEBPUSH_CONTACT_EMAIL | string | — | Contact email for the push service (RFC 8292). |
| Variable | Type | Default | Description |
|---|---|---|---|
NOTIFY_MOBILEPUSH_PROVIDER | enum | — | fcm (others land later). |
NOTIFY_FCM_CREDENTIALS_JSON | string | — | Service-account JSON. Required for FCM. |
NOTIFY_FCM_PROJECT_ID | string | — | Firebase project id. Required for FCM. |
NOTIFY_APNS_KEY_P8 | string | — | APNs auth key (P8 PEM). |
NOTIFY_APNS_KEY_ID | string | — | APNs key id. |
NOTIFY_APNS_TEAM_ID | string | — | Apple developer team id. |
NOTIFY_APNS_TOPIC | string | — | Bundle id / topic. |
NOTIFY_APNS_SANDBOX | bool | false | Use the APNs sandbox endpoint. |
+NOTIFY_INTERNAL_TOKEN=<32-byte hex>
+NOTIFY_LOG_LEVEL=info
+NOTIFY_ALLOWED_ORIGINS=https://app.example.com
+
+# Email via the elloloop EmailService container.
+NOTIFY_EMAIL_PROVIDER=emailservice
+NOTIFY_EMAIL_SERVICE_ADDRESS=emailservice.internal:50053
+NOTIFY_EMAIL_FROM=noreply@example.com
+
+# SMS via Twilio.
+NOTIFY_SMS_PROVIDER=twilio
+NOTIFY_SMS_ACCOUNT_SID=AC...
+NOTIFY_SMS_AUTH_TOKEN=...
+NOTIFY_SMS_FROM=+15555550000`} lang="ini" title="notify.env" />
+
+ Use it with:
+ Boot-time validation
+
+ On startup notify validates every value. Examples of what it catches:
+
+
+ NOTIFY_STORE_DRIVER=mongo → "unknown driver \"mongo\" (want memory|postgres|entdb)".NOTIFY_STORE_DRIVER=postgres without a DSN → "NOTIFY_POSTGRES_DSN is required when NOTIFY_STORE_DRIVER=postgres".NOTIFY_EMAIL_PROVIDER=emailservice without NOTIFY_EMAIL_SERVICE_ADDRESS → required-field error.NOTIFY_AUTH_DEV_MODE=false without NOTIFY_AUTH_JWT_SECRET → required-field error.NOTIFY_CLIENT_PORT=99999 → "NOTIFY_CLIENT_PORT: must be in 1..65535".
+
+ Related
+
+ - JWT Keys
+ - Store Setup
+ - Observability
+
+
+ The reference deployment shape is the published multi-arch container
+ image at ghcr.io/elloloop/notify. It is a
+ FROM scratch binary plus CA certificates — no shell, no
+ package manager, ~10 MB.
+
+ This is the recommended path for every environment except library mode. + Local dev, staging, production: same image, different env vars. If you + want to embed notify in another Go binary, see + library + usage in the README. +
+ +Pin to a specific tag in CI and production:
+| Port | Listener | Audience |
|---|---|---|
| 8080 | NotificationClientService | Browser / mobile recipients (Connect HTTP/2) |
| 8081 | NotificationInternalService | Backend producers (gRPC / Connect) |
| 9090 | /healthz and /metrics | Orchestrators / Prometheus |
+ The Dockerfile builds a FROM --platform=$BUILDPLATFORM
+ golang:1.26.3-alpine3.23 stage that produces a static
+ binary, then COPYs the binary and CA bundle into
+ FROM scratch. The final image runs as UID
+ 65532 (non-root):
+
+ The container has no shell — you can't HEALTHCHECK CMD curl …
+ against itself. Probe from the orchestrator side instead:
+
+ For Kubernetes use HTTP probes — see + Kubernetes. +
+ ++ Every published image is signed via cosign keyless OIDC and scanned by + Trivy with a HIGH/CRITICAL gate. Verify the signature on pull: +
+
+ notify validates client-facing requests with an HS256 JWT signature.
+ The verification key is a shared secret you generate, hand to the JWT
+ issuer (identity), and hand to notify via
+ NOTIFY_AUTH_JWT_SECRET.
+
+ You are setting up a production deployment, rotating a leaked secret, + or wiring an existing identity-issued JWT into notify. +
+ ++ Use a CSPRNG. Anything 32 bytes or larger is fine for HS256 (256 bits + of entropy matches the HMAC-SHA-256 output size). +
++ In production prefer a secrets manager (Kubernetes Secret, AWS + Secrets Manager, Azure Key Vault) and project the value into the + container as an env var. +
+ ++ When the JWT issuer is identity, set the matching expected claims + so notify rejects tokens minted by anyone else: +
+
+ If your identity tenant uses different values, copy them verbatim
+ from the issuer config. A missing or mismatched claim returns
+ CodeUnauthenticated.
+
+ notify expects the standard claims plus a tenant claim: +
+
+ sub populates Claims.UserID;
+ tenant (or the alternative tenant_id)
+ populates Claims.TenantID; email is
+ optional and surfaces in observability.
+
+ Tiny one-off generator in Python — useful for manual cURLs against a + locally-deployed notify with a real JWT secret: +
++ notify v0.1 validates against a single secret — there is no key ring + on the verifier. To rotate without a flap, the standard procedure is: +
+S2.S2, every legacy token still signed with S1 tracked by kid.NOTIFY_AUTH_JWT_SECRET=S1, one with NOTIFY_AUTH_JWT_SECRET=S2. Route traffic by inspecting the kid in the JWT header at the gateway.S1-signed tokens have expired (one access-token TTL window), retire the S1 replica and the issuer's S1 signing key.+ The simpler alternative — rolling everyone out, swapping the secret, + rolling back in — is acceptable for short outage windows but is what + rotation is supposed to avoid. A future wave will add a multi-key + verifier so notify can hold both keys natively. +
+ ++ v0.1 only validates HS256. If your identity provider signs RS256 + (e.g. you are using the + identity service's + JWKS surface), the recommended shape today is: +
+server.Dependencies.AuthValidator.+ Native RS256 + JWKS verification is on the roadmap; track + notify issues. +
+ +NOTIFY_AUTH_* var+ Pick the durable store driver that fits the deployment. Memory is the + test/dev reference; Postgres is the typical production shape; + EntDB is the right call if you already run tenant-shard-db. +
+ ++ You are sizing storage for a new environment, deciding which managed + Postgres SKU to provision, or wiring notify into an existing EntDB + cluster. +
+ +
+ No setup. NOTIFY_STORE_DRIVER=memory is the default;
+ everything lives in process memory and disappears on restart.
+
+ Use this for tests, demos, and CI fixtures. It is the conformance + reference — every other driver is verified against memory's + behaviour. +
+ +
+ Use a managed Postgres 14+ (we test against
+ postgres:16.13-alpine3.23). Auto-migration is on by
+ default and applies the schema under
+ pg_advisory_lock so concurrent boots serialise.
+
+ On first boot it creates notify_notifications,
+ notify_devices and the
+ notify_schema_migrations bookkeeping table. Subsequent
+ boots see the ledger and skip already-applied versions.
+
'read';
+
+CREATE TABLE notify_devices (
+ id uuid PRIMARY KEY,
+ tenant_id text NOT NULL,
+ user_id text NOT NULL,
+ device_type text NOT NULL,
+ token text NOT NULL DEFAULT '',
+ created_at_ms bigint NOT NULL DEFAULT 0,
+ last_active_ms bigint NOT NULL DEFAULT 0,
+ UNIQUE (tenant_id, user_id, device_type)
+);`} lang="sql" title="migration 1 — init" />
+
+
+ If you want to manage migrations out of band (a CI step, golang-migrate,
+ etc.) set NOTIFY_POSTGRES_AUTOMIGRATE=false. The schema
+ is two tables plus three indexes — easy to mirror verbatim.
+
+
+ Sizing
+
+ Reasonable defaults: MaxConns = 25,
+ MaxConnLifetime = 1h, MaxConnIdleTime = 30m —
+ cooperative with pgbouncer / Azure idle reapers. notify is read-heavy
+ on the inbox surface and append-heavy on Notify; the
+ notify_notifications_user_unread partial index keeps
+ unread counts fast even with millions of read rows.
+
+
+ Conformance
+
+ The Postgres driver is verified at 24/24 against the
+ shared suite. See Postgres driver
+ or the
+ CONFORMANCE.md
+ for the per-subtest table and the design rationale.
+
+
+ entdb (tenant-shard-db)
+
+ Use this when EntDB is already a primitive in your stack
+ (e.g. you run identity against it). The notify driver requires the
+ v2 schema-aware mode (ADR-031) — v1.32.1 cannot enforce the
+ composite-uniqueness constraint that idempotent
+ CreateNotification needs.
+
+
+ Boot a local EntDB
+
+ -data-dir is required on v2 (it was implicit in v1.32.1
+ and earlier).
+
+
+ Wiring it in
+ Schema-aware mode
+
+ notify's cmd/notifyd registers the schema on the SDK
+ client at boot:
+
+
+ entdb.SchemaMessages() returns the two proto descriptors
+ (UserNotification, DeviceRegistration) declared
+ in proto/entdb_notify/notify.proto. The v2 server
+ materializes the schema from those descriptors on the first
+ ExecuteAtomic and enforces unique: true on
+ the composite_key field — which closes the
+ concurrent-create race the conformance suite probes.
+
+
+ Sharding
+
+ One notify container talks to one EntDB tenant shard
+ (NOTIFY_ENTDB_TENANT_ID). All notify-application tenants
+ live under that one storage shard, differentiated by the
+ tenant_id row field. If a large customer needs its own
+ storage shard, run a second notify container pointed at a different
+ NOTIFY_ENTDB_TENANT_ID.
+
+
+ Conformance
+
+ EntDB driver passes 24/24 on v2.0.5; the two
+ formerly-red canaries (composite uniqueness under concurrent create
+ + device upsert) flipped to green when the schema-aware enforcement
+ landed in v2.0.1. See
+ EntDB driver or
+ store/entdb/CONFORMANCE.md.
+
+
+ Related
+
++ notify is a multi-channel notification platform packaged two ways: +
+import "github.com/elloloop/notify"
+ and embed the orchestrator in-process. Inject a Store and
+ one or more Providers; call Notifier.Notify with
+ a request.
+ ghcr.io/elloloop/notify:<version>,
+ a thin server wrapper that exposes the same orchestrator over gRPC
+ (for backend producers) and Connect/HTTP-2 (for browser + mobile
+ recipients).
+ + Both packaging modes share the same internals — the dual-mode pattern + is the same one elloloop/identity + and tenant-shard-db + use. +
+ +NOTIFY_* env vars is the
+ whole deployment surface. No bespoke deploy pipelines.
+ notify.Store interface and run the same conformance
+ suite.
+ Notification carries
+ an opaque SubjectRef + SubjectType. The
+ platform never interprets them — no todo_id,
+ message_id or any other consumer-specific fields leak
+ across the API.
+ Notify
+ call is keyed by (tenant_id, user_id, notification_id).
+ Retries are safe and never duplicate stored rows or provider
+ dispatches.
+ Title + Body + Data the caller
+ hands it. Localization, MJML rendering and HTML email composition are
+ the caller's concern (or a future wave).
+ "" to keep the row but suppress dispatch.
+ StatusPending rather than
+ a lying StatusDelivered.
+ + Two services live in the proto contract + (proto/notify/v1): +
+NotificationInternalService.Notify — the backend producer
+ RPC. Fan a payload out to one-or-many users across one-or-many
+ channels.
+ NotificationClientService — the client-facing surface
+ a browser SPA or mobile app calls. RPCs: StreamEvents,
+ GetNotifications, AckNotification,
+ AckDataChange, RegisterPushToken.
+ + Both are accessible via Connect-Go / Connect-Web / Connect-Python / + gRPC. There is no separate REST surface — Connect speaks JSON natively + over HTTP/1 and HTTP/2. +
+
+ notify exposes structured JSON logs to stderr, a /healthz
+ endpoint for liveness, and a /metrics endpoint for
+ Prometheus scrapers. The metrics endpoint always returns a parseable
+ body so a scraper hitting it never sees a 404.
+
+ Setting up dashboards, debugging a slow tenant, wiring an alerting + rule, or chasing an intermittent timeout in production. +
+ +
+ Logs are emitted via Go's log/slog with a JSON handler
+ writing to stderr at the level set by NOTIFY_LOG_LEVEL
+ (default info). One event per line, no header, ingest
+ directly into Loki / Datadog / Splunk / CloudWatch.
+
server_listen — one line per listener (client / internal / metrics).notifyd_starting — boot banner with version, commit, store driver, live-connections toggle, listener ports.stream_open / stream_close — every StreamEvents session, including connection_id, user_id, tenant_id, device_type.event_queue_full — one log per dropped event when a client buffer overflows.retry_failed — per-attempt error from the at-least-once retry tracker (key, connection_id, attempt, error).server_shutdown_signal / server_shutdown_error — graceful shutdown.{`{"status":"ok"}`} once Server.Run has bound all listeners.{`{"status":"not_ready"}`} after Shutdown begins and before the process exits.An alias /health is registered for tool-muscle-memory.
+ The readiness probe will start failing the instant
+ Shutdown begins, so an in-flight rolling update stops
+ sending new traffic to the pod immediately while
+ Shutdown drains in-flight requests.
+
+ /metrics on the metrics port returns
+ text/plain; version=0.0.4 — the standard Prometheus
+ exposition. v0.1 ships a placeholder body so scrapers don't 404;
+ real metric series land in a follow-up wave without changing the
+ route.
+
+ Once the registry is wired, expect these series. Labels are + intentionally cardinality-bounded — no per-tenant labels. +
+| Metric | Type | Labels |
|---|---|---|
notify_notify_requests_total | Counter | code |
notify_send_total | Counter | channel, provider, status |
notify_send_duration_seconds | Histogram | channel, provider |
notify_store_op_total | Counter | op, code |
notify_store_op_duration_seconds | Histogram | op |
notify_live_connections | Gauge | device_type |
notify_stream_events_total | Counter | kind |
+ notify does not emit OpenTelemetry spans today. The Connect
+ interceptors are the right place to add them; once the
+ follow-up lands, set OTEL_* env vars per the standard
+ Go OTel SDK auto-configuration and traces will surface in the
+ backend of your choice.
+
+ Loki + Grafana is the cheapest path for JSON logs. Drop this in your + promtail config: +
+
+ Server.Run traps SIGINT and
+ SIGTERM, marks the health endpoint not-ready, and runs
+ a single ordered Shutdown against the three HTTP
+ servers, the retry tracker, and the store. The total time is
+ capped by NOTIFY_SHUTDOWN_TIMEOUT (default 30s).
+
+ Tuning a rolling deploy, debugging requests cut off mid-flight, or
+ sizing a Kubernetes terminationGracePeriodSeconds.
+
Server.health.markNotReady() — /healthz immediately starts returning 503 with not_ready. Kubernetes readiness probes stop steering traffic to the pod within one probe interval (default ~5s).http.Server.Shutdown is called on the client server, the internal server, and the metrics server in turn. Each waits for in-flight handlers to return; new connections are rejected.RetryTracker.CancelAll cancels every in-flight at-least-once retry so no goroutine leaks across the process exit.pgx.Pool.Close for Postgres, sdk.Client.Close for EntDB, no-op for memory). This is the last step so any in-flight handler that's still touching the store has time to return first.+ The default 30s is a reasonable starting point. Two workloads should + bump it: +
+StreamEvents connection must drain. The handler exits cleanly on context cancel, but a TCP close handshake takes a few hundred ms times N connections. For 10k open streams plan ~60s.NotificationInternalService.Notify calls may need ~10s+ each. Set NOTIFY_SHUTDOWN_TIMEOUT a comfortable margin above the longest acceptable provider latency.
+ Set it slightly longer than NOTIFY_SHUTDOWN_TIMEOUT so
+ the kubelet doesn't SIGKILL before notify finishes draining:
+
+ Optional: a short preStop hook gives the load balancer / service + mesh time to stop sending new traffic before notify even sees SIGTERM: +
+
+ The container ships FROM scratch so there's no
+ /bin/sh; if you want a preStop sleep, project a
+ sidecar with a shell, or rely on the readiness-probe flip to drain
+ upstream proxies inside Server.Shutdown itself.
+
+ The at-least-once RetryTracker spawns one goroutine per
+ (key, connID) pair when a data-change event is
+ scheduled. CancelAll cancels every parent context and
+ drains the map. After shutdown:
+
<-ctx.Done().+ If you need at-least-once delivery across pod restarts for a + specific use case, push it through a durable channel (email, SMS, + web push, mobile push) — the in-app channel is real-time-or-nothing + by intent. +
+ +pgxpool.Pool.Close; waits for all checked-out connections to be returned before closing.sdk.Client.Close; closes the gRPC transport.+ The store close runs after the HTTP servers have + finished draining. This guarantees no in-flight Connect handler is + still touching the store when its connections get torn down — + avoiding spurious "connection closed" errors in the very last log + lines. +
+ +
+ If the shutdown completes within NOTIFY_SHUTDOWN_TIMEOUT,
+ the process exits with status 0. If the timeout fires first the
+ process logs server_shutdown_error with the underlying
+ cause and exits with the first non-nil error.
+
+ Five-minute "Hello, notify": run the container, send a notification + from a backend, receive it over SSE in a browser. +
+ ++ Use the memory driver and dev-mode auth — no Postgres, no EntDB, no + JWT secret. Never enable dev mode in production. +
+Confirm it's healthy:
+
+ Backend producers call the internal service on port
+ 8081. In dev mode the
+ X-Notify-Internal-Token check is skipped. The request is
+ plain Connect-JSON over HTTP, so cURL works.
+
Response:
+
+ pending=1 is correct: no client is connected yet, so the
+ in-app provider has nowhere to deliver. The row is stored and will
+ surface via GetNotifications or the live stream as soon
+ as a client subscribes.
+
+ Client connections go to the client service on port
+ 8080. The dev-mode token shape is
+ Bearer dev:<userid>:<tenant>.
+
async (req) => {
+ req.header.set("Authorization", "Bearer dev:user-alice:acme");
+ return next(req);
+ },
+ ],
+});
+const client = createClient(NotificationClientService, transport);
+
+for await (const ev of client.streamEvents({ deviceType: "DEVICE_TYPE_BROWSER" })) {
+ switch (ev.event.case) {
+ case "notification":
+ console.log("notification:", ev.event.value.notification?.title);
+ break;
+ case "heartbeat":
+ console.log("heartbeat @", ev.event.value.timestampMs);
+ break;
+ case "dataChange":
+ console.log("data change:", ev.event.value.subjectRef);
+ break;
+ }
+}`} lang="ts" title="browser.ts" />
+
+
+ Now repeat step 2 with a different notificationId: the
+ open stream prints the title within milliseconds.
+
+
+ 4. Page the inbox history
+
+ Anything sent while no client was connected is still available via
+ GetNotifications.
+
+ 5. Mark a notification as read
+ "}'`} lang="bash" />
+
+
+ A repeat GetNotifications shows
+ status: DELIVERY_STATUS_READ,
+ readAtMs stamped, and unreadCount: 0.
+
+
+ 6. Inspect /metrics
+
+ The endpoint exposes a parseable Prometheus exposition; v0.1 ships
+ a placeholder so scrapers don't 404. Real metric series land in a
+ follow-up wave without changing the route.
+
+
+ Next steps
+
+ - Configuration — every environment variable, defaults, validation rules
+ - JWT Keys — production-grade HS256 secret
+ - Store Setup — Postgres or EntDB driver
+ - Send a Notification — Go + Python producer code
+ - Subscribe over SSE — full browser/mobile flow
+
+
+ store/conformance.RunConformance is the driver-agnostic
+ spec for notify.Store. Memory is the differential
+ reference; every other driver passes the same 24 leaf subtests
+ across six categories or its CI check goes red and the merge is
+ blocked.
+
TestConformance/postgres/Concurrency/ConcurrentCreate_SameKey_SingleWinner tells you exactly which semantic broke.
+ RunConformance wraps everything in
+ t.Run(d.Name, …), so the test path on failure is
+ TestConformance/<driver>/<Category>/<Subtest>.
+
| Category | Subtest | What it catches |
|---|---|---|
| Core CRUD | CreateGet | basic round-trip; id is assigned by Store, not caller |
GetNotFound | unknown id returns ErrNotFound (not a typed driver error) | |
Idempotency | second create with same (tenant, user, notification_id) returns created=false, stamps existing id, leaves stored row unchanged | |
StatusTransitions | each status stamps its matching *_at_ms field; unknown id returns ErrNotFound | |
CursorWalk_ThreePages | multi-page traversal with cursors; last page returns empty cursor | |
UnreadFilterAndCount | unreadCount is independent of the page window and UnreadOnly filter | |
DeviceUpsertRotation | re-registering the same (tenant, user, device_type) rotates token in place; doesn't create a new row | |
UserIsolation | cross-user and cross-tenant reads surface as ErrNotFound | |
| Pagination | QueryUserNotifications_AllPagesReturnEveryRow | full-set traversal across many pages — no row dropped, no row duplicated |
StrictLessThanCutoff | cursor semantics are strict <, never <= (otherwise boundary rows repeat) | |
| FreshTenant | QueryUserNotifications_Empty | query against a never-seen tenant returns empty, not error |
GetNotification_NotFound | same for a single Get | |
ListDevices_Empty | same for device listing | |
| RoundTrip | StringFields_OnCreate | adversarial values — emoji, unicode, SQL-shaped, leading/trailing whitespace, 10k chars — survive byte-for-byte |
Int64_Fidelity_Timestamps | extreme int64 values (max_int64, 2^53+1) round-trip exactly — proves the driver does not silently coerce to float64 | |
LargePayload_Body | 64 KiB and 512 KiB bodies round-trip unmodified | |
| Concurrency | ConcurrentCreate_DistinctKeys_NoLostWrites | 16 concurrent creates with distinct keys land 16 rows — the WAL/applier serializes correctly |
ConcurrentCreate_SameKey_SingleWinner | The headline test. 16 concurrent creates with the same key land exactly one row; all callers see the same canonical id | |
ConcurrentUpsertDevice_SameKey_SingleRow | concurrent device upserts on the same (tenant, user, device_type) land exactly one row; token reflects one of the writers | |
ConcurrentUpdateStatus_NoError | concurrent status transitions don't error; the final state matches one of the racers | |
ConcurrentReadYourWrites_QueryAfterCreate | a writer's very next query sees its own row (no stale-read window) | |
| KeyEdge | NotificationID_LongValue | 256-char notification id round-trips; re-create is idempotent |
NotificationID_SeparatorBytesDoNotCollide | distinct triples with separator-byte payloads do not collide on the composite key (length-prefix encoding canary) | |
DeviceType_CaseSensitive_SeparateRows | "android" and "Android" are distinct composite keys (byte-exact equality) |
+ Concurrency/ConcurrentCreate_SameKey_SingleWinner is the
+ subtest that catches the broadest class of bugs. It launches 16
+ goroutines that all call CreateNotification with the
+ same (tenant, user, notification_id):
+
+ This catches: +
+
+ Each non-memory driver ships a CONFORMANCE.md with the
+ full per-subtest table, status, and a one-line root-cause note for
+ anything red. Templates to follow:
+
+ .github/workflows/conformance.yml declares one job per
+ driver in a matrix. The check name is
+ Conformance / <driver>, and that is what branch
+ protection pins. A new driver landing PR must add itself to the
+ matrix and turn the new check green.
+
+ The EntDB driver maps notify's Store contract onto
+ tenant-shard-db's
+ SDK. v0.1 targets v2 schema-aware mode (ADR-031) and
+ pins to v2.0.5 — the v2 server materializes the notify schema from
+ proto descriptors and enforces composite uniqueness, which is what
+ makes the concurrent-create conformance test pass without a
+ service-level mutex.
+
+ Pick EntDB if you already run tenant-shard-db for another service + (e.g. identity) and want notify to share the operational story — + same image management, same WAL pipeline, same scaling story. For + greenfield deployments, the Postgres driver is the more common + starting point. +
+ +
+ NOTIFY_ENTDB_TENANT_ID is the EntDB storage shard — one
+ notify container talks to one EntDB tenant. All notify-application
+ tenants live under that one EntDB tenant, differentiated by the
+ tenant_id field on each row.
+
+ Schema is declared as proto messages in
+ proto/entdb_notify/notify.proto. type_id
+ values are local to notify's tenant-shard-db instance (it runs its
+ own EntDB deployment), so the type ids do not need to coordinate
+ with identity's.
+
+ The composite_key field on each node is the unique
+ index the driver hits via sdk.GetByKey for the
+ idempotency guard. v2 server enforces the
+ unique: true annotation, closing the
+ composite-uniqueness race the v1.32.1 canary tests reproduced.
+
+ Why this design. + The composite key string is built by length-prefix encoding each + part rather than concatenating with a separator. A naive ++ +"u1|n1|n2"scheme collides for two distinct triples + that escape differently (e.g. one with a literal|in + the user id). The driver'slpEncodehelper writes +<len>:<part>for each piece so the + serialization is injective. The +KeyEdge/NotificationID_SeparatorBytesDoNotCollide+ conformance test pins this invariant — drivers that get it wrong + would silently treat two different rows as the same. +
+ The container wires the SDK with the registered schema at boot: +
+
+ entdb.SchemaMessages() returns the two proto descriptors
+ declared above. The v2 server materializes the schema from those
+ descriptors on the first ExecuteAtomic call and
+ enforces every annotation including unique: true.
+ Concurrent CreateNotification calls with the same
+ composite_key now produce exactly one row.
+
+ The Go SDK's Plan.Commit does not expose
+ wait_applied today (upstream item #U1, documented in
+ store/entdb/CONFORMANCE.md),
+ so the loser of a unique-constraint race receives a
+ success=true receipt with a pre-allocated node id that
+ never actually lands. The driver reconciles in code:
+
+ CreateNotification returns
+ (canonicalID, won) so the caller learns whether it was
+ the actual writer. UpsertDevice additionally runs a
+ no-wait UpdateFields against the canonical row in the
+ loser branch so each racer's token / last_active_ms gets a chance to
+ land (preserving memory's "last writer wins" semantics).
+
ghcr.io/elloloop/tenant-shard-db:2.0.5 (pin in docker-compose or your Helm chart).github.com/elloloop/tenant-shard-db/sdk/go/entdb/v2 v2.0.5.-addr=:50051 -data-dir=/tmp/entdb -wal-backend=memory (-data-dir is required on v2).sdk.WithSchema(entdb.SchemaMessages()...).+ 24/24 PASS on v2.0.5 — including the two + composite-uniqueness canaries that were intentionally red on v1.32.1 + (they flipped when the v2 schema-aware unique enforcement landed). + See store/entdb/CONFORMANCE.md + for the per-subtest table. +
+ +Intentional omissions documented for transparency:
+system:notify as the actor for every read and write; cross-tenant isolation is enforced by the tenantID argument the driver passes to the SDK transport.+ The memory driver is the in-process reference implementation. All + state lives in Go maps protected by a mutex; restart loses + everything. It is the differential reference for the conformance + suite — every other driver is verified against memory's exact + behaviour. +
+ +It is the default. No env vars required.
+Or explicitly:
+sync.RWMutex guards two maps.created=false.+ All 24 leaf subtests across the six categories pass. memory is the + fastest of the three (no testcontainer warm-up, no EntDB SDK + handshake) and is the gate that catches a botched conformance + refactor before the slower drivers run. +
+ ++ If you are writing tests that exercise notify's behaviour from your + consumer's POV — without spinning up Postgres or EntDB — wire + memory directly: +
+
+ The Postgres driver is the typical production shape: two tables, two
+ indexes per table, pgx v5 connection pool, embedded
+ migrations applied under pg_advisory_lock. Composite
+ uniqueness is enforced by Postgres, not the application — that is
+ what makes the conformance suite's same-key concurrency tests pass
+ cleanly.
+
+ Default choice for production unless you already run EntDB. Works + with any managed Postgres 14+ (RDS, Azure Database for PostgreSQL, + Cloud SQL). +
+ ++ See Store + Setup → Postgres for the DSN options and the provisioning SQL. +
+ + 'read';
+
+CREATE TABLE notify_devices (
+ id uuid PRIMARY KEY,
+ tenant_id text NOT NULL,
+ user_id text NOT NULL,
+ device_type text NOT NULL,
+ token text NOT NULL DEFAULT '',
+ created_at_ms bigint NOT NULL DEFAULT 0,
+ last_active_ms bigint NOT NULL DEFAULT 0,
+ UNIQUE (tenant_id, user_id, device_type)
+);
+
+CREATE INDEX notify_devices_user
+ ON notify_devices (tenant_id, user_id, device_type);`} lang="sql" />
+
+ Three things matter about this schema:
+
+ - Composite uniqueness is enforced by Postgres. The
UNIQUE(tenant_id, user_id, notification_id) constraint is what makes ON CONFLICT DO NOTHING safe for the idempotent CreateNotification and what makes the same-key concurrency test pass without an application-level mutex.
+ - The paging index is the tuple the query sorts by.
(tenant_id, user_id, created_at_ms DESC, id DESC) matches ORDER BY created_at_ms DESC, id DESC, so paging is an index scan, not a sort node.
+ - The unread partial index keeps unreadCount cheap. Even with millions of read rows, the unread count is a scan over the partial index.
+
+
+ Idempotency / upsert pattern
+
+ CreateNotification:
+
+
+ If RETURNING produces a row, the writer won the
+ insert and gets created=true. If pgx.ErrNoRows
+ comes back (ON CONFLICT absorbed the dup), a follow-up SELECT
+ recovers the canonical id and returns created=false.
+ This satisfies both the basic Idempotency test and the
+ same-key concurrency test: exactly one goroutine gets the
+ RETURNING row, the rest take the recovery path.
+
+
+
+ UpsertDevice:
+
+
+ created_at_ms is intentionally not touched on conflict
+ — token rotation preserves the original create time.
+
+
+
+ UpdateStatus stamps the matching *_at_ms in
+ one statement via a CASE:
+
+
+ A RowsAffected() == 0 result maps to
+ notify.ErrNotFound.
+
+
+ Pagination
+
+ - Order:
ORDER BY created_at_ms DESC, id DESC.
+ - Cursor:
strconv.FormatInt(lastRow.CreatedAtMS, 10) — same wire format as the memory driver, so cross-driver tests interchange.
+ - Next page:
WHERE created_at_ms < $cursor (strict <, per the contract).
+ - Detect next page:
LIMIT n+1. If you get back n+1 rows, slice to n and emit nextCursor = items[n-1].CreatedAtMS. Saves a second round-trip and a count(*).
+ unreadCount is a separate SELECT count(*) FROM ... WHERE status <> 'read' — independent of the page window and any UnreadOnly filter.
+
+ Migrations
+
+ Embedded as a Go slice in store/postgres/migrations.go;
+ applied under pg_advisory_lock(0x6E6F74696679_00) so
+ concurrent process boots serialise on the lock instead of racing
+ the DDL. The bookkeeping table notify_schema_migrations
+ is idempotent — boots after the first see it and skip applied
+ versions.
+
+
+
+ Migrations are append-only. A new schema change becomes a new
+ migration{`{}`} entry at the end; shipped versions never get
+ edited. The advisory lock key is unlikely to collide
+ ("notify\0" in hex), but if another driver in the same
+ database picks the same key its migrations would block on ours.
+
+
+ Connection pool
+
+ MaxConns = 25 by default — matches identity's DefaultMaxConns.MaxConnLifetime = 1h, MaxConnIdleTime = 30m — cooperate with pgbouncer / Azure idle reapers.
+
+ Conformance results
+
+ 24/24 pass under -race. Total wall time after the
+ testcontainers Postgres warm-up is ~3.3s on Apple Silicon. See
+ store/postgres/CONFORMANCE.md
+ for the per-subtest table and design notes.
+
+
+ Running the suite locally
+
+ The first run pulls postgres:16.13-alpine3.23; subsequent
+ runs reuse the layer cache.
+
+
+ Related
+
++ Multi-channel notification platform as a single, deployable container. + Pull a pinned image, point it at a durable store (memory · Postgres · + EntDB), + configure one or more channel providers, and you have in-app real-time + SSE, email, web push, mobile push, SMS and WhatsApp delivery — all over + Connect-RPC (HTTP/JSON) and gRPC. +
+ +Sender interface. Ships with the elloloop EmailService backend; slots for SES, ACS, SMTP, SendGrid.whatsapp: prefix.ErrSubscriptionGone sentinel on 410 responses for token-purge logic.ErrUnregisteredToken sentinel for the standard UNREGISTERED / NOT_FOUND cases.Store contract; all drivers run the same 24-subtest conformance suite.Notify call is keyed by (tenant, user, notification_id). Retries never duplicate deliveries.pending → delivered → acked → read (or failed) tracked per row./healthz, /metrics on a separate port.cmd/notifyd wires it all from environment variables; library mode is the same code, minus the wiring.+ One container, three listeners. Treat it like + tenant-shard-db + or identity: pin a + tag, deploy once per environment. Auth is JWT (HS256) against a shared + secret — no callback into any consumer. +
+ +The container exposes three ports:
+8080 — NotificationClientService (browser / mobile, Connect HTTP/2)8081 — NotificationInternalService (backend producers, gRPC)9090 — /healthz and /metrics (Prometheus exposition)+ The proto contract carries two services so the auth + transport story + cleanly splits the two callers: +
+| Service | Caller | Auth |
|---|---|---|
NotificationInternalService | Backend producers (your services that send notifications) | X-Notify-Internal-Token shared secret |
NotificationClientService | Browser + mobile clients (recipients reading + streaming) | Authorization: Bearer <JWT> |
+ All durable state — notification rows and device registrations —
+ lives in the configured store. Memory is the in-process reference
+ used for tests and dev; Postgres uses pgx; EntDB
+ targets the v2 schema-aware mode (ADR-031) and rides server-enforced
+ composite-key uniqueness.
+
| Type | Where (memory/postgres) | Where (entdb) |
|---|---|---|
| Notification | notify_notifications table | UserNotification node, type_id = 1 |
| Device | notify_devices table | DeviceRegistration node, type_id = 2 |
+ notify runs its own EntDB instance; type_id values are
+ scoped to that deployment and do not collide with the identity
+ namespace.
+
NOTIFY_* variable component.
+ Expressive-code now provides chrome (border/radius/copy buttons). We just
+ guarantee horizontal scroll inside the container so blocks never widen
+ the page on mobile. */
+ .prose pre.astro-code,
+ .prose pre {
+ max-width: 100%;
+ overflow-x: auto;
+ }
+ /* Fallback background for any non-Shiki (legacy raw blocks if any). */
+ .prose pre:not(.astro-code):not([class*="ec-"]):not(.expressive-code pre) {
+ @apply relative my-6 overflow-x-auto rounded-lg border border-zinc-800/80 px-5 py-4 text-[13px] leading-[1.65] shadow-md;
+ background-color: hsl(var(--code-bg));
+ color: hsl(var(--code-fg));
+ font-family: var(--font-mono);
+ }
+ .prose pre code {
+ @apply block border-0 bg-transparent p-0 font-normal;
+ font-family: var(--font-mono);
+ font-feature-settings: "ss02", "cv01", "cv02";
+ }
+ /* Custom thin scrollbars inside legacy code blocks */
+ .prose pre::-webkit-scrollbar {
+ height: 0.5rem;
+ }
+ .prose pre::-webkit-scrollbar-track {
+ background: transparent;
+ }
+ .prose pre::-webkit-scrollbar-thumb {
+ background: rgb(63 63 70);
+ border-radius: 0.25rem;
+ }
+ .prose pre::-webkit-scrollbar-thumb:hover {
+ background: rgb(82 82 91);
+ }
+
+ /* Expressive-code wrapper — never let the frame widen the page. */
+ .prose .expressive-code {
+ @apply my-6 max-w-full;
+ }
+ .prose .expressive-code .frame {
+ box-shadow: none;
+ }
+ .prose .expressive-code pre {
+ border-radius: 0.5rem;
+ }
+
+ /* Tables — wrapped in .table-scroll for horizontal overflow on mobile. */
+ .prose .table-scroll {
+ @apply -mx-2 my-6 overflow-x-auto rounded-lg border border-border;
+ position: relative;
+ }
+ /* Subtle gradient-fade on the right edge to hint scrollability. */
+ .prose .table-scroll::after {
+ content: "";
+ position: absolute;
+ top: 0;
+ right: 0;
+ bottom: 0;
+ width: 24px;
+ pointer-events: none;
+ background: linear-gradient(
+ to left,
+ hsl(var(--background)),
+ hsl(var(--background) / 0)
+ );
+ opacity: 0;
+ transition: opacity 200ms;
+ }
+ @media (max-width: 768px) {
+ .prose .table-scroll::after {
+ opacity: 0.85;
+ }
+ }
+ .prose table {
+ @apply w-full border-collapse text-[14px];
+ margin: 0;
+ border: 0;
+ }
+ .prose thead {
+ @apply bg-muted/60;
+ position: sticky;
+ top: 0;
+ z-index: 1;
+ }
+ .prose th {
+ @apply border-b border-border px-4 py-2.5 text-left text-xs font-semibold uppercase tracking-wider text-muted-foreground;
+ }
+ .prose td {
+ @apply border-b border-border/70 px-4 py-2.5 align-top text-foreground;
+ overflow-wrap: break-word;
+ }
+ .prose tbody tr {
+ transition: background-color 150ms;
+ }
+ .prose tbody tr:hover {
+ @apply bg-muted/30;
+ }
+ .prose tbody tr:last-child td {
+ @apply border-b-0;
+ }
+ .prose tbody tr:nth-child(even) {
+ @apply bg-muted/15;
+ }
+ .prose tbody tr:nth-child(even):hover {
+ @apply bg-muted/40;
+ }
+ .prose table code {
+ @apply whitespace-nowrap;
+ }
+
+ /* Bare table (one without a wrapper) — make it scroll on its own. */
+ .prose > table {
+ display: block;
+ overflow-x: auto;
+ white-space: nowrap;
+ }
+
+ /* Blockquotes / callouts fallback */
+ .prose blockquote {
+ @apply my-6 rounded-r-lg border-l-4 border-primary bg-primary/5 px-5 py-4 text-foreground;
+ }
+ .prose blockquote p {
+ @apply mb-0;
+ }
+}
+
+/* Dark-mode token overrides. Kept outside @layer base because Tailwind's
+ purge step can drop the `.dark` selector if it doesn't appear in a
+ content file — and the theme toggle adds it dynamically. */
+.dark {
+ --background: 222 47% 5%;
+ --foreground: 210 40% 98%;
+ --card: 222 47% 7%;
+ --card-foreground: 210 40% 98%;
+ --popover: 222 47% 7%;
+ --popover-foreground: 210 40% 98%;
+ --primary: 234 89% 74%;
+ --primary-foreground: 222 47% 11%;
+ --secondary: 217 33% 17%;
+ --secondary-foreground: 210 40% 98%;
+ --muted: 217 33% 17%;
+ --muted-foreground: 215 20% 65%;
+ --accent: 217 33% 17%;
+ --accent-foreground: 210 40% 98%;
+ --destructive: 0 63% 31%;
+ --destructive-foreground: 210 40% 98%;
+ --border: 217 33% 17%;
+ --input: 217 33% 17%;
+ --ring: 234 89% 74%;
+
+ --code-bg: 222 47% 3%;
+ --code-fg: 210 40% 98%;
+}
+
+/* Pagefind UI — recolor to match the theme. */
+:where(.pagefind-ui) {
+ --pagefind-ui-primary: hsl(var(--primary));
+ --pagefind-ui-text: hsl(var(--foreground));
+ --pagefind-ui-background: hsl(var(--background));
+ --pagefind-ui-border: hsl(var(--border));
+ --pagefind-ui-tag: hsl(var(--muted));
+ --pagefind-ui-border-width: 1px;
+ --pagefind-ui-border-radius: 0.5rem;
+ --pagefind-ui-image-border-radius: 0.5rem;
+ --pagefind-ui-image-box-ratio: 3 / 2;
+ --pagefind-ui-font: var(--font-sans);
+}
diff --git a/docs-site/tailwind.config.mjs b/docs-site/tailwind.config.mjs
new file mode 100644
index 0000000..ed1623b
--- /dev/null
+++ b/docs-site/tailwind.config.mjs
@@ -0,0 +1,28 @@
+import { refractionPreset } from "@refraction-ui/tailwind-config";
+
+/** @type {import('tailwindcss').Config} */
+export default {
+ content: [
+ "./src/**/*.{astro,html,js,jsx,md,mdx,svelte,ts,tsx,vue}",
+ // refraction-ui's components ship Tailwind utility classes inside
+ // node_modules. Without scanning them, classes like the AppShellOverlay's
+ // `bg-black/50` and `z-30` get purged and the mobile sidebar backdrop
+ // becomes invisible — making the hamburger appear broken.
+ "./node_modules/@refraction-ui/astro/dist/**/*.{astro,js,mjs,ts}",
+ "./node_modules/.pnpm/@refraction-ui+astro@*/node_modules/@refraction-ui/astro/dist/**/*.{astro,js,mjs,ts}",
+ ],
+ presets: [refractionPreset],
+ darkMode: "class",
+ // The .dark class is toggled by the theme script at runtime, and Tailwind's
+ // JIT only emits a base-layer rule if its selector appears in `content`.
+ // Safelist `dark` so the CSS-variable overrides survive the purge step.
+ safelist: ["dark"],
+ theme: {
+ extend: {
+ fontFamily: {
+ sans: ["var(--font-sans)"],
+ mono: ["var(--font-mono)"],
+ },
+ },
+ },
+};
diff --git a/docs-site/tsconfig.json b/docs-site/tsconfig.json
new file mode 100644
index 0000000..b7243b9
--- /dev/null
+++ b/docs-site/tsconfig.json
@@ -0,0 +1,7 @@
+{
+ "extends": "astro/tsconfigs/strict",
+ "compilerOptions": {
+ "jsx": "react-jsx",
+ "jsxImportSource": "react"
+ }
+}