[feat](compaction) Support ADMIN COMPACT TABLE type='FULL' and enable it in cloud mode#62596
Open
Yukang-Lian wants to merge 2 commits intoapache:masterfrom
Open
[feat](compaction) Support ADMIN COMPACT TABLE type='FULL' and enable it in cloud mode#62596Yukang-Lian wants to merge 2 commits intoapache:masterfrom
Yukang-Lian wants to merge 2 commits intoapache:masterfrom
Conversation
… it in cloud mode ADMIN COMPACT TABLE previously only accepted BASE and CUMULATIVE in local mode and threw Unsupported operation in cloud mode, forcing users to curl each BE's HTTP API to run manual compaction. This change unifies the FE SQL entry so that base/cumulative/full all work on both local and cloud. FE - AdminCompactTableCommand: add FULL to CompactionType, relax analyzeWhere, map FULL->"full" in getCompactionType, drop the cloud-mode gate so the command dispatches via the normal path. - CloudEnv.compactTable: override Env.compactTable to walk partitions/ indices/tablets, resolve the primary BE via Replica.getBackendId, and raise a DdlException if nothing was dispatched (e.g. the current compute group has no available BE). BE - submit_table_compaction_callback: strictly match base/cumulative/full and reject unknown types instead of silently downgrading non-"base" to cumulative. Full goes through submit_compaction_task(force=false, eager=true, trigger_method=MANUAL) after setting last_full_compaction_schedule_time, matching the HTTP API default. - cloud_submit_table_compaction_callback: new callback mirroring the local one but calling CloudStorageEngine::submit_compaction_task; honors the cloud HTTP convention that base/cumu need sync_delete_bitmap while full does not. - agent_server::cloud_start_workers: register the previously stubbed-out SUBMIT_TABLE_COMPACTION worker (drops the plat1ko TODO). Thrift is untouched: TCompactionReq.type is already optional string, so the existing "base"/"cumulative"/"full" wire format remains compatible. This implies a release order of "all BEs upgraded first, then FE" - a new FE against an old BE would see "full" silently downgraded (local) or dropped (cloud). Tests - regression-test/suites/compaction/test_admin_compact_table.groovy: end-to-end coverage for base/cumulative/full under the local engine plus negative cases (unknown type, missing WHERE). - regression-test/suites/cloud_p0/compaction/test_cloud_admin_compact_table.groovy: same matrix under the cloud engine, with a SELECT before each poll to force BE to sync rowsets from Meta Service.
Yukang-Lian
requested review from
dataroaring,
gavinchou and
w41ter
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
…ack changes Reflow the new submit_compaction_task calls and the cloud callback's log messages to satisfy clang-format.
Collaborator
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unify the
ADMIN COMPACT TABLE ... WHERE type = '...'SQL entry so that BASE, CUMULATIVE and FULL all work on both the local and cloud (storage-compute split) engines. Before this change, FULL was rejected in FE and cloud mode was blocked outright, forcing users to curl each BE's HTTP API to run manual compaction.What changes
FE
AdminCompactTableCommand: addFULLtoCompactionType, relaxanalyzeWhere, mapFULL -> \"full\"ingetCompactionType, and drop the cloud-mode gate so the command dispatches via the normal Agent RPC path.CloudEnv.compactTable: overrideEnv.compactTableto walk partitions/indices/tablets, resolve the primary BE viaReplica.getBackendId(), skip tablets whose compute group has no available BE, and raiseDdlExceptionif nothing was dispatched.BE
submit_table_compaction_callback: strictly matchbase/cumulative/fullinstead of silently downgrading any non-\"base\"value toCUMULATIVE_COMPACTION. FULL goes throughsubmit_compaction_task(force=false, eager=true, trigger_method=MANUAL)after settinglast_full_compaction_schedule_time, matching the HTTP API default.cloud_submit_table_compaction_callback: new callback mirroring the local one but callingCloudStorageEngine::submit_compaction_task; honors the cloud HTTP convention that base/cumu needsync_delete_bitmapwhile full does not.agent_server::cloud_start_workers: register the previously stubbed-outSUBMIT_TABLE_COMPACTIONworker (drops the// TODO(plat1ko)comment).Thrift / wire compatibility
TCompactionReq.typeremainsoptional string, so no IDL change. A new FE against an old BE would see\"full\"silently downgraded (local) or dropped (cloud), so the release order is all BEs upgraded first, then FE. FE rollback can lead BE since the FE simply stops sending\"full\".Test plan
mvn -pl fe-core -am compilepasses.regression-test/suites/compaction/test_admin_compact_table.groovycovers base / cumulative / full end-to-end on the local engine, plus negative cases (unknown type, missing WHERE).regression-test/suites/cloud_p0/compaction/test_cloud_admin_compact_table.groovycovers the same matrix on the cloud engine.