Skip to content

Capture drain can complete before event broker dispatchers are removed #6316

Description

@wk989898

What did you do?

Drain a TiCDC capture before restarting it during a rolling upgrade.

The drain sequence can reach the following state:

  1. Maintainers and dispatcher tasks have been moved away from the target capture.
  2. Maintainer-reported drain progress has converged to zero.
  3. The target capture has reported STOPPING.
  4. Its local event brokers still have registered dispatchers.

Event broker registrations include both ordinary dispatchers and table trigger dispatchers. Their cleanup is not covered by maintainer-reported drain progress alone.

What did you expect to see?

Drain should remain incomplete until the target capture's event broker dispatcher count reaches zero, in addition to satisfying the existing drain completion conditions.

The coordinator should observe this count from a STOPPING heartbeat for the current capture epoch. A liveness response alone, or a delayed heartbeat from a previous process, should not authorize completion.

What did you see instead?

The drain completion check does not include event broker registrations. It can return zero remaining work after the existing liveness and maintainer progress conditions converge, even while event brokers still have registered dispatchers.

This allows a rolling upgrade controller to restart the capture prematurely, potentially causing avoidable replication interruptions and increased checkpoint lag.

This gap was identified from the drain logic. The proposed fix includes a unit test covering the state transition above; a cluster-level rolling upgrade reproduction has not been performed for this standalone change.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions