fix(synapse): keep credentials out of the config ConfigMap

[pigri]

Problem

platform.api_key, captcha.secret_key and captcha.jwt_secret are templated into synapse.config, which renders into a ConfigMap. That exposes them three ways:

• readable by anyone with get configmap in the namespace;
• copied verbatim into the kubectl.kubernetes.io/last-applied-configuration annotation;
• returned by helm get values.

The chart had no Secret template at all. Putting them in env: is no better — that block renders literal name/value pairs into the Deployment spec.

Fix

A secrets block, injected via envFrom:

secrets:
  existingSecret: synapse-credentials   # preferred
  # or
  create: true
  apiKey: "..."

• existingSecret — reference a Secret managed out of band (External Secrets, sealed-secrets, Vault). No credential ever enters values. Preferred.
• create — render a Secret from values. Better than a ConfigMap, but the credential then lives in your values file.

The Secret's keys are the env var names synapse reads: API_KEY, CAPTCHA_SECRET_KEY, CAPTCHA_JWT_SECRET.

Why no synapse change is needed

synapse parses config.yaml and then applies env overrides, assigning unconditionally:

cli.rs:2907   Self::load_from_file(config_path)?
cli.rs:2914   config.apply_env_overrides();
cli.rs:3228   if let Some(val) = get_env_arxignis("API_KEY") { self.platform.api_key = val; }

So env wins over the file, and the ConfigMap can simply stay empty.

Two documentation bugs, corrected

values.yaml claimed the exact opposite in two places:

# Note: YAML config has higher priority than env vars, so leave values empty
# Environment variable overrides (lowest priority; use only for settings not in config.yaml)

Both are backwards, and either would have talked someone out of this fix. Corrected, with the mechanism named.

Guard

Rendering now fails if a credential is still set inside synapse.config:

SECURITY: `api_key` is set inside synapse.config, which renders into a
ConfigMap (world-readable in the namespace, copied into
last-applied-configuration, and returned by `helm get values`). Blank it
there and inject it via the `secrets` values instead - env vars override
config.yaml, so the value still reaches synapse.

A hard fail rather than a warning is deliberate: the silent version of this is how a production api_key ended up in cleartext.

This is a breaking upgrade for anyone currently setting a credential in synapse.config — by design. Migration is: blank the field, move the value to secrets.

Verification

path	result
defaults (all blank)	renders clean
existingSecret=my-creds	envFrom: [{secretRef: {name: my-creds}}], no Secret rendered
create=true	Secret with keys API_KEY, CAPTCHA_JWT_SECRET; envFrom points at it
credential in ConfigMap	test value absent from rendered ConfigMap; api_key: ""
credential in synapse.config	render refused

All three charts lint; stack and both overlays render.

Known remaining exposure

Env vars are visible in /proc/<pid>/environ inside the container and in crash dumps. File-mounted secrets would be tighter, but that needs synapse to learn to read a credential from a file — worth a follow-up issue rather than blocking this.

Precedence was verified by reading the source, not at runtime.

[pigri]

Verified on a live k3s cluster, not just from source. The precedence claim was the load-bearing assumption in this PR, so it needed a runtime check.

Setup

Rendered this branch with the credential only in the Secret, and a deliberately different base_url in config.yaml so it's visible which one synapse actually uses:

ConfigMap: api_key: ""
           base_url: "https://config-loses.invalid"
Secret:    API_KEY   = dummy-key-from-secret
           BASE_URL  = https://env-wins.invalid
Deployment: envFrom: [{secretRef: {name: secrettest-credentials}}]

base_url is set by the identical mechanism on the line right after api_key (get_env_arxignis, unconditional assignment, cli.rs:3228 and :3231), and unlike api_key it's observable in logs — the agent_status worker targets it.

Results

1. The Secret reaches synapse. The worker logs "api_key or base_url empty — agent_status worker idle" when unset. Occurrences: 0. Since config.yaml had api_key: "", the only possible source was the Secret.

2. Env wins over config.yaml. References to env-wins.invalid: 6. To config-loses.invalid: 0.

INFO  synapse-telemetry: OTel metrics exporter → https://env-wins.invalid/v1/telemetry
INFO  synapse-telemetry: OTel logs exporter (SDK) → https://env-wins.invalid/v1/telemetry
WARN  Could not validate the API key (platform unreachable: request failed:
      error sending request for url (https://env-wins.invalid/authcheck))
INFO  [agent_status] capabilities (8) at boot: [firewall_rules, agent_status, ...]

The authcheck attempt confirms the api_key path is live too — that request only happens with a key set.

3. The ConfigMap is clean. Grepping the full in-cluster ConfigMap YAML (which includes the last-applied-configuration annotation) for the credential: 0 occurrences. It appears only in the Secret.

This also confirms the two values.yaml comments this PR corrects were backwards — config.yaml does not take priority over env.

Test resources removed afterwards.

[pigri]

Follow-up: per-endpoint credentials needed more than env injection

The env-var approach above only reaches fields synapse exposes an env override for. It does not cover per-endpoint keys.

There are three headers maps in the config — telemetry/OTel exporter, geoip sources (country/asn/city), and the threat/download source. They exist to carry per-endpoint auth; the doc comment shows Authorization: "Bearer xaat-...". And:

• no env override exists for any headers field (geoip has GEOIP_COUNTRY_URL/_PATH, but no _HEADERS)
• config.yaml has no ${VAR} interpolation

So a separate key for geoip or model/threat downloads could only be written into the file — straight back into a world-readable ConfigMap.

Fix

synapse.configSecret renders config.yaml into a Secret. synapse reads a path and doesn't care about the source kind, so no synapse change is needed. upstreams.yaml is not sensitive and stays a ConfigMap so the --ingress-mode operator can keep owning it.

Off by default.

One combination is refused outright

The operator's NetVarsResolver reads and rewrites config.yaml in a ConfigMap only (netvars_resolver.go: var cm corev1.ConfigMap … r.Update(ctx, &cm)). With config.yaml in a Secret it finds nothing to rewrite and silently stops filling ids.address_vars.HOME_NET/EXTERNAL_NET.

That is not cosmetic — an unfilled HOME_NET is what lets inline IDS blocking ban an internal or cluster source IP. So the chart fails the render rather than letting it degrade quietly. This matters for the agent alias in synapse-stack, which carries the resolve-netvars label.

Verified on a cluster

check	result
synapse parsed the Secret-sourced file	Using config file: /etc/synapse/config.yaml, and it names keys from that file as unused
header credential across all ConfigMaps in the namespace	0 occurrences
header credential in the Secret	1
ConfigMap contents	['upstreams.yaml'] only
guard fires on configSecret + resolve-netvars	yes; renders fine without the label

Boundary worth stating: the geoip fetch itself did not run during the test (0 hits on the bogus endpoint), so I proved the credential is stored and the file is loaded from a Secret, not that geoip transmits that header. That is synapse behaviour and unchanged by this PR.

Follow-up worth filing separately

Teaching synapse ${VAR} interpolation in config.yaml would let non-sensitive config move back to a ConfigMap while credentials stay env-injected, and would remove the NetVarsResolver conflict. Alternatively, teaching NetVarsResolver to handle a Secret-backed config would remove the restriction on its own.

[pigri]

Rebased onto main, which now carries #33 (Dragonfly auto-wire) and #34 (content_scanning keys + appVersion). Five conflicts, all version bumps or adjacent additions rather than disagreements:

• charts/synapse/Chart.yaml — version 0.8.2, appVersion stays 0.8.3 (main's, not this branch's stale 0.7.0)
• charts/synapse-stack/Chart.yaml — 0.9.2, dependency pin 0.8.2
• Chart.lock — regenerated with helm dependency update rather than hand-merged
• validate-values.yaml — both guards kept; this branch's credential check now sits alongside main's Dragonfly-auth check

One thing I changed rather than just merging. The credential guard's comment said "a warning, not a hard fail: existing installs must still be able to render while they migrate" — but the code called fail, which is a hard fail. Helm has no warning primitive, so the comment was describing something that cannot be built. Resolved in favour of the stated intent by adding an explicit escape hatch: allowConfigCredentials: true lets an unmigrated install keep rendering, which makes the exposure a recorded decision instead of an accident.

Verified after the rebase — all four guards fire on their own trigger and nothing else:

trigger	result
clean defaults	renders
stale valkey: block	fails
Dragonfly auth + auto-wire	fails
configSecret + resolve-netvars label	fails
api_key inside synapse.config	fails
…same, with allowConfigCredentials: true	renders

And the two features coexist: with secrets.create and dragonfly.enabled both on, the credential lands in Secret/rel-synapse-credentials (keys API_KEY, CAPTCHA_SECRET_KEY) consumed via envFrom, REDIS_URL is auto-wired to the Dragonfly Service, and the ConfigMap contains no credential.

helm lint clean on both charts.

[linear-code]

SYN-198