Lumen

Lumen is a multi-tenant event ingestion and session analytics service built in Go. It ingests high-throughput telemetry streams over Connect RPC / HTTP/2, enriches events server-side, and stores them in ClickHouse for real-time analytical queries.

Postgres is used for transactional control plane metadata (tenant accounts and API keys), while ClickHouse stores append-only event logs, materialized sessions, and identity maps.

Open source, self-hostable, no limits. No event caps, no license key, no phone-home. The server is AGPL-3.0; the client SDKs are Apache-2.0 so you can embed them in closed-source apps. A small set of commercial-operations features (licensing, quotas, billing, SSO) live in a separate private module and are not required to run anything. See OPEN_CORE.md.

---

Architecture

                    ┌─────────────────────────┐
                    │ Client SDKs (Go/TS/Py)  │
                    └────────────┬────────────┘
                                 │ Connect RPC (HTTP/2)
                                 ▼
                    ┌─────────────────────────┐
                    │  Lumen Ingest Engine    │
                    │  - Auth Interceptor     │
                    │  - UA & UTM Enricher    │
                    └────┬───────────────┬────┘
                         │               │
  Async Batches          │               │ Key & Tenant Lookup
  (async_insert=1)       ▼               ▼
            ┌────────────────┐   ┌────────────────┐
            │   ClickHouse   │   │    Postgres    │
            │   (OLAP Logs)  │   │(Control Plane) │
            └────────────────┘   └────────────────┘

---

Quickstart

1. Run local dependencies

docker compose up -d clickhouse postgres

2. Set the admin token

ADMIN_TOKEN has no default — the admin port provisions tenants and mints API keys, so the server refuses to start without one.

export ADMIN_TOKEN="$(openssl rand -hex 32)"

For throwaway local work, LUMEN_DEV=1 generates an ephemeral token and prints it at startup instead.

3. Run the server

go run ./cmd/lumen

Or run via Docker:

docker run -p 50051:50051 -p 50052:50052 \
  -e ADMIN_TOKEN="$(openssl rand -hex 32)" \
  ghcr.io/synehq/lumen:latest

Never expose the admin port (50052) to the internet, and terminate TLS in front of the ingest port. See SECURITY.md for the full operator hardening checklist.

---

Client SDK Usage

Go

package main

import (
    "context"
    "time"
    "github.com/SyneHQ/lumen/sdk/go"
)

func main() {
    client, _ := lumen.New("lum_live_your_api_key",
        lumen.WithEndpoint("http://localhost:50051"),
        lumen.WithBatchSize(100),
    )
    defer client.Close(context.Background())

    // Track an event
    client.Track(context.Background(), "order_completed", lumen.P{
        "amount": 99.50,
        "currency": "USD",
    })

    // Identify user
    client.Identify(context.Background(), "usr_12345", lumen.P{
        "plan": "pro",
    })
}

TypeScript (@syne/lumen-js)

import { Lumen } from "@syne/lumen-js";

const lumen = new Lumen({
  apiKey: "lum_live_your_api_key",
  endpoint: "http://localhost:50051",
});

// Track pageview
lumen.track("page_view", { path: "/pricing" });

// Identify user
lumen.identify("usr_12345", { email: "alex@example.com" });

Web analytics script

A cookieless drop-in tag, served by the ingest server at /js/script.js:

<script defer data-key="YOUR_WEB_KEY" src="https://<lumen-host>/js/script.js"></script>

It sends a $pageview on load and on SPA navigation, and sends no visitor or session ids; it never uses cookies or browser storage. Optional attributes: data-api (endpoint, default is the script's origin), data-hash (count hash changes), data-allow-localhost (localhost and file: are skipped by default). Set window.lumenIgnore = true to opt a browser out.

Custom events: lumen("signup", { plan: "pro" }). To call it before the script loads, add this stub first:

<script>window.lumen = window.lumen || function () { (lumen.q = lumen.q || []).push(arguments) }</script>

Python

from lumen import Client

lumen = Client("lum_live_your_api_key", endpoint="http://localhost:50051")

lumen.track("signup_completed", {"plan": "growth"})
lumen.identify("usr_12345", {"company": "Acme Corp"})
lumen.close()

---

Configuration

Settings can be set via environment variables or loaded directly from Infisical at startup.

Environment Variable	Default	Description
INGEST_PORT	50051	Port for public Connect / gRPC telemetry ingestion
ADMIN_PORT	50052	Port for internal control plane RPCs
METRICS_PORT	9090	Prometheus metrics and health check port
ADMIN_TOKEN	required, no default	Token for admin provisioning RPCs. Min 32 chars. Server refuses to start without it.
LUMEN_DEV	false	Development only. Generates an ephemeral ADMIN_TOKEN instead of failing.
CLICKHOUSE_DSN	clickhouse://127.0.0.1:9000/lumen?dial_timeout=10s&compress=true	Native ClickHouse connection string
CLICKHOUSE_DSN_SOURCE	infisical	DSN selection: infisical or environment. See below.
CLICKHOUSE_COMPRESSION	unset	Set lz4 to override DSN compression. Unset preserves the DSN settings.
CH_HOST	unset	Optional host returned by tenant provisioning. Unset uses the first ClickHouse DSN host; the DSN still supplies the port.
POSTGRES_DSN	postgres://postgres:postgres@localhost:5433/lumen?sslmode=disable	Postgres connection string
GEOIP_DB_PATH	unset	Path to a MaxMind GeoLite2-City .mmdb. Wins over GEOIP_SOURCE.
GEOIP_SOURCE	unset	Set dbip-lite to download DB-IP City Lite in the background and refresh it monthly. Unset leaves geo fields blank.
TRUSTED_PROXY_HOPS	1	How many reverse proxies in front of ingest append to X-Forwarded-For. Set this to match your deployment (see below).
GEOIP_CACHE_DIR	$XDG_CACHE_HOME/lumen/geoip (~/Library/Caches/lumen/geoip on macOS)	Writable directory for the downloaded GeoIP database (~130 MB).

With GEOIP_SOURCE=dbip-lite, geo data comes from DB-IP under CC BY 4.0. Anywhere country, region or city is shown, display: IP Geolocation by DB-IP (link to https://db-ip.com).

Client IP and TRUSTED_PROXY_HOPS

Lumen takes the client IP from X-Forwarded-For, but only the part your own proxies wrote. Each trusted proxy appends one entry, so the client IP is the entry TRUSTED_PROXY_HOPS places from the right. Entries further left come from the client and are ignored, because anyone can send a fake one.

• 1 (default): a single proxy, such as the bundled Caddy, in front of ingest.
• 2 or more: add one per extra layer that appends to the header, for example a CDN or cloud load balancer in front of Caddy. Too low and every request looks like it comes from your own proxy; too high and clients can spoof their IP.
• 0: ingest is reached directly; the header is ignored and the TCP peer address is used.

If the header has fewer entries than the setting, the leftmost entry is used. This applies to all keys: it decides the IP used for GeoIP and the stored IP on server keys (when the tenant has store_ip), and for web keys the visitor hash and rate limit.

Managed connections and compression

By default, a nonempty Infisical CLICKHOUSE_DSN overrides the environment value. If Infisical does not supply one, Lumen uses the environment value, then the development default. Explicit infisical mode keeps this behavior.

For a platform-provided connection binding, set:

CLICKHOUSE_DSN_SOURCE=environment
CLICKHOUSE_COMPRESSION=lz4

The platform must supply a nonempty CLICKHOUSE_DSN. Environment mode ignores the Infisical DSN and refuses startup when the environment DSN is empty. It never substitutes a different credential source. Other Infisical settings continue to load normally. These two selector variables come from the environment, not Infisical.

lz4 replaces any DSN compression method and level with LZ4 at the driver default level. Leave CLICKHOUSE_COMPRESSION unset to preserve all DSN compression settings, including disabled compression or another codec. Unsupported values stop startup. The override preserves credentials, database selection, TLS verification, timeouts and pool settings. Continue to configure verified TLS and the managed database's public CA. Compression does not enable or disable TLS.

Infisical Secret Loading

If INFISICAL_CLIENT_ID and INFISICAL_CLIENT_SECRET (or INFISICAL_TOKEN) are set, Lumen automatically fetches configuration secrets directly into memory at startup without writing them to process environment variables.

export INFISICAL_CLIENT_ID="your-client-id"
export INFISICAL_CLIENT_SECRET="your-client-secret"
export INFISICAL_PROJECT_ID="your-project-id"
export INFISICAL_ENV="prod"

---

Testing

Run unit tests, E2E flow tests, and data race detection:

# Run all tests with Go data race detector
go test -race -v ./...

To run tests against live local ClickHouse and Postgres instances:

docker compose up -d clickhouse postgres
go test -v ./tests/live_db_test.go

---

Project layout

ee/            Extension interfaces + open no-op defaults (the commercial seam)
app/           Bootstrap: app.Run(ctx, hooks)
cmd/lumen/     Community binary
internal/      Engine: auth, enrich, ingest, ch, pg, provision, server
migrations/    ClickHouse + Postgres schema
proto/, gen/   Wire contract (Apache-2.0)
sdk/           Go, TypeScript, Python clients (Apache-2.0)
enterprise/    Private submodule, optional, not needed to build or run

---

Contributing

Bug reports and PRs welcome. See CONTRIBUTING.md for setup, DCO sign-off, and commit conventions. Security issues go to SECURITY.md — never a public issue.

By participating you agree to the Code of Conduct.

---

License

Path	License
Server (everything not listed below)	AGPL-3.0-or-later
sdk/, proto/, gen/	Apache-2.0
enterprise/ (private submodule)	Commercial, not part of this distribution

Using Lumen in your product does not put your code under AGPL — your app talks to the server over the network using Apache-2.0 SDKs. See OPEN_CORE.md for the full explanation, the open-core boundary, and the commitments we hold ourselves to.

Need a non-AGPL server license? sales@synehq.com. "Lumen" and "Syne" are trademarks — see TRADEMARK.md.