ingress-mode central layout: rendered upstreams take up to a kubelet sync period to reach the proxy

[pigri]

Summary

In the central layout (--upstreams-out-configmap), a changed render is written to the ConfigMap promptly, but the proxy does not see it for up to a full kubelet sync period — 60s with the default --sync-frequency. Every other hop in the chain is event-driven; this one is polled.

Why

Writing the ConfigMap does not put the bytes in the proxy's container. The kubelet re-projects a ConfigMap volume only during its periodic pod sync, and a ConfigMap change is not a pod event, so it never wakes that loop.

configMapAndSecretChangeDetectionStrategy: Watch (the default) does not help, and is an easy thing to misread: it keeps the kubelet's cache fresh via a watch, but that cache is only consulted when the volume is re-projected on the sync pass. Fresh cache, stale file.

The sidecar layout has no such problem — signalReload SIGHUPs the co-located process — and main.go correctly disables signalling in central mode (SignalReload: outCM.Name == ""), since there is no PID to signal across a pod boundary. So the central layout currently has no reload lever at all.

Measurements

Single-node k3s, one proxy replica. Timed from kubectl apply of an Ingress to the proxy actually serving the new route:

kubelet syncFrequency	propagation
1m0s (default)	59s
10s	7s, then 1s

The fact that changing only syncFrequency moves the number is itself the proof that the bottleneck is the polled sync pass rather than slow work anywhere in the operator.

Proposal

Nudge the consumers. Mutating the Pod object is an event the kubelet watches, and it wakes the same sync pass that re-projects volumes. After a changed render in central mode, annotate each Pod that projects the rendered ConfigMap.

Isolated verification that the mechanism does what it claims — same ConfigMap, same pod, with and without a no-op annotation patch:

trial	plain ConfigMap update	+ pod annotation patch
1	11.6s	0.6s
2	14.0s	0.6s
3	12.6s	0.6s

Critically, the container is not restarted. Across the run, restartCount stayed 0, the container's startedAt was unchanged, and PID 1's start time in /proc/1/stat was byte-identical — the same process observed the new file.

Notes:

• Annotating a Pod owned by a ReplicaSet is safe: the ReplicaSet controller reconciles ownership and labels, not annotations, so this neither fights it nor triggers a replacement.
• Consumer selection should match on spec.volumes rather than a label. The dataplane chart takes config.yaml from its own ConfigMap and upstreams.yaml from the operator-written one through a single projected volume, so matching only plain configMap volumes would silently nudge nothing.
• Needs pods: patch added to the ClusterRole. Consistent with the existing convention, this should sit behind an opt-in flag so the permission and the behaviour cannot drift.

Why not the alternatives

• Lower --sync-frequency — works (the table above), but it is a cluster-wide kubelet change that makes every ConfigMap and Secret projection re-check more often, to fix one controller's propagation.
• Switch to the sidecar layout — sub-second and already supported, but it costs shareProcessNamespace: true and the operator's RBAC on the pod that terminates TLS. That is real attack surface on an internet-facing workload to save a config-update delay.

The nudge needs neither, and works cross-pod and for many replicas.

Status

I have a working patch (flag + reconciler hook + gated RBAC + unit tests) validated end-to-end: with the flag on and the kubelet left at its default 1m0s, propagation went from 59s to 2-3s, with the operator log confirming it targeted only the one consuming pod. Happy to open it as a PR if the approach looks right.

A follow-up worth considering separately: --certs-out-secret projects TLS Secrets into the same kind of central mount, so certificate rotation has the identical delay.