Skip to content

rebalance dispatchers by traffic when table splitting is disabled #6251

Description

@wk989898

What did you do?

With table splitting disabled, TiCDC can have equal dispatcher counts on all nodes while all replication traffic remains on one node. In a 10-minute test on master, one changefeed replicated nine tables across three TiCDC nodes, with three table dispatchers per node. We continuously wrote only to the three tables initially assigned to one node. No dispatcher migration occurred, and the other two nodes handled no DML traffic for this changefeed.

  1. Run three TiCDC nodes and create a MySQL changefeed with this configuration:
     [scheduler]
     enable-table-across-nodes = false
  1. Create 9 tables, t1 through t9, in the filtered database. Each table has a primary key, a sequence column, and a 1 KiB payload.
  2. Wait for the initial placement to reach three table dispatchers per node. Identify the three tables assigned to one node. In this run, they were t3, t5, and t7 on upstream-ticdc-0.
  3. Write only to those three tables for 10 minutes, keeping the other six tables completely idle:
    • First 180 seconds: target 1,000 row writes/s per hot table, approximately 3,000 total.
    • Next 420 seconds: target 3,000 row writes/s per hot table, approximately 9,000 total.
  4. Observe dispatcher placement and per-node sink traffic without manually moving tables or restarting nodes.

What did you expect to see?

TiCDC node Assigned tables Table dispatchers Lower-load sink throughput Higher-load sink throughput
upstream-ticdc-0 t3, t5, t7 - receiving writes 3 5.91 MiB/s 18.30 MiB/s
upstream-ticdc-1 t4, t6, t8 - idle 3 0 MiB/s 0 MiB/s
upstream-ticdc-2 t1, t2, t9 - idle 3 0 MiB/s 0 MiB/s
Image

What did you see instead?

TiCDC should detect sustained traffic imbalance and automatically migrate existing table dispatchers between nodes, even when dispatcher counts are already equal and enable-table-across-nodes = false.

For this reproduction, an expected placement would be:

TiCDC node Hot tables Idle tables Expected traffic share
upstream-ticdc-0 1 2 Approximately one third
upstream-ticdc-1 1 2 Approximately one third
upstream-ticdc-2 1 2 Approximately one third

Since the three hot tables receive equivalent workloads, this should distribute replication traffic approximately evenly while retaining three table dispatchers per node. The scheduler should account for dispatcher traffic when deciding whether a placement is balanced, preserve the no-splitting configuration, and achieve this without manual table moves. Equal dispatcher counts alone should not prevent rebalancing when their workloads differ substantially.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions