A backend with weight 0 reaching KetamaHashing permanently kills the load balancer's background update task (or aborts the process under panic = "abort").
Where
pingora-load-balancing/src/selection/consistent.rs, KetamaHashing::build_with_config, passes b.weight as u32 straight into pingora_ketama::Bucket::new.pingora-ketama/src/lib.rs, Bucket::new: assert!(weight != 0, "weight must be at least one").
How it triggers
Backend::weight is a plain, unvalidated usize (pingora-load-balancing/src/lib.rs). Neither Backend::new_with_weight nor the ServiceDiscovery trait rejects 0. Some discovery backends use weight 0 as a draining/disabled marker, and a typo in a static config or DNS SRV record produces the same thing. Once one such backend is in the BTreeSet<Backend> passed to KetamaHashing::build_with_config, Bucket::new panics.
That call happens inside LoadBalancer::update() (pingora-load-balancing/src/lib.rs), driven by the background loop in pingora-load-balancing/src/background.rs (LoadBalancer::run). Nothing wraps it in catch_unwind or spawn_blocking. The newer LoadBalancerGroup rebuild path already runs build_selector inside spawn_blocking and checks JoinError::is_panic(), so this class of panic is handled there, just not on the plain LoadBalancer<S> path.
Impact
Once the panic fires, the spawned update task ends and is never restarted:
- Backend membership changes (new/removed hosts) stop being applied.
- Health check state freezes at whatever it was.
- Under
panic = "abort", the whole process goes down instead.
All from a single zero-weight entry in one discovery response, when the consumer uses consistent hashing (a common choice for cache or session-sticky routing).
Suggested fix
Clamp the weight to a minimum of 1 at the Backend to Bucket boundary in KetamaHashing::build_with_config, since Backend::weight is untrusted, unvalidated input and Bucket requires a positive weight by contract.
A backend with weight 0 reaching
KetamaHashingpermanently kills the load balancer's background update task (or aborts the process underpanic = "abort").Where
pingora-load-balancing/src/selection/consistent.rs,KetamaHashing::build_with_config, passesb.weight as u32straight intopingora_ketama::Bucket::new.pingora-ketama/src/lib.rs,Bucket::new:assert!(weight != 0, "weight must be at least one").How it triggers
Backend::weightis a plain, unvalidatedusize(pingora-load-balancing/src/lib.rs). NeitherBackend::new_with_weightnor theServiceDiscoverytrait rejects 0. Some discovery backends use weight 0 as a draining/disabled marker, and a typo in a static config or DNS SRV record produces the same thing. Once one such backend is in theBTreeSet<Backend>passed toKetamaHashing::build_with_config,Bucket::newpanics.That call happens inside
LoadBalancer::update()(pingora-load-balancing/src/lib.rs), driven by the background loop inpingora-load-balancing/src/background.rs(LoadBalancer::run). Nothing wraps it incatch_unwindorspawn_blocking. The newerLoadBalancerGrouprebuild path already runsbuild_selectorinsidespawn_blockingand checksJoinError::is_panic(), so this class of panic is handled there, just not on the plainLoadBalancer<S>path.Impact
Once the panic fires, the spawned update task ends and is never restarted:
panic = "abort", the whole process goes down instead.All from a single zero-weight entry in one discovery response, when the consumer uses consistent hashing (a common choice for cache or session-sticky routing).
Suggested fix
Clamp the weight to a minimum of 1 at the
BackendtoBucketboundary inKetamaHashing::build_with_config, sinceBackend::weightis untrusted, unvalidated input andBucketrequires a positive weight by contract.