What did you do?
Drain a TiCDC capture before restarting it during a rolling upgrade.
The drain sequence can reach the following state:
- Maintainers and dispatcher tasks have been moved away from the target capture.
- Maintainer-reported drain progress has converged to zero.
- The target capture has reported
STOPPING.
- Its local event brokers still have registered dispatchers.
Event broker registrations include both ordinary dispatchers and table trigger dispatchers. Their cleanup is not covered by maintainer-reported drain progress alone.
What did you expect to see?
Drain should remain incomplete until the target capture's event broker dispatcher count reaches zero, in addition to satisfying the existing drain completion conditions.
The coordinator should observe this count from a STOPPING heartbeat for the current capture epoch. A liveness response alone, or a delayed heartbeat from a previous process, should not authorize completion.
What did you see instead?
The drain completion check does not include event broker registrations. It can return zero remaining work after the existing liveness and maintainer progress conditions converge, even while event brokers still have registered dispatchers.
This allows a rolling upgrade controller to restart the capture prematurely, potentially causing avoidable replication interruptions and increased checkpoint lag.
This gap was identified from the drain logic. The proposed fix includes a unit test covering the state transition above; a cluster-level rolling upgrade reproduction has not been performed for this standalone change.
Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
(paste TiDB cluster version here)
Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)
TiCDC version (execute cdc version):
(paste TiCDC version here)
What did you do?
Drain a TiCDC capture before restarting it during a rolling upgrade.
The drain sequence can reach the following state:
STOPPING.Event broker registrations include both ordinary dispatchers and table trigger dispatchers. Their cleanup is not covered by maintainer-reported drain progress alone.
What did you expect to see?
Drain should remain incomplete until the target capture's event broker dispatcher count reaches zero, in addition to satisfying the existing drain completion conditions.
The coordinator should observe this count from a
STOPPINGheartbeat for the current capture epoch. A liveness response alone, or a delayed heartbeat from a previous process, should not authorize completion.What did you see instead?
The drain completion check does not include event broker registrations. It can return zero remaining work after the existing liveness and maintainer progress conditions converge, even while event brokers still have registered dispatchers.
This allows a rolling upgrade controller to restart the capture prematurely, potentially causing avoidable replication interruptions and increased checkpoint lag.
This gap was identified from the drain logic. The proposed fix includes a unit test covering the state transition above; a cluster-level rolling upgrade reproduction has not been performed for this standalone change.
Versions of the cluster
Upstream TiDB cluster version (execute
SELECT tidb_version();in a MySQL client):(paste TiDB cluster version here)Upstream TiKV version (execute
tikv-server --version):(paste TiKV version here)TiCDC version (execute
cdc version):(paste TiCDC version here)