[feature](mtmv) Support date_add/sub hour offset in MTMV partition expressions by hakanuzum · Pull Request #62599 · apache/doris

hakanuzum · 2026-04-18T10:59:34Z

What problem does this PR solve?

Issue Number: close #62395

Problem Summary:
MTMV partition expressions like date_trunc(date_add(col, INTERVAL N HOUR), 'day') were not supported. This pattern is essential for timezone-aware partitioning, where users need to align daily/weekly/monthly aggregations with their local timezone instead of the raw datetime values stored in the base table.

Example 1 - Positive offset (UTC+3, e.g., Istanbul):
Base table stores data in UTC. A record 2025-07-25 22:00:00 UTC is actually 2025-07-26 01:00:00 in Istanbul time and should belong to July 26, not July 25.

-- Shifts data +3 hours before truncating to day
date_trunc(date_add(col, INTERVAL 3 HOUR), 'day')
-- 2025-07-25 22:00:00 → +3h → 2025-07-26 01:00:00 → truncate → 2025-07-26 ✓

Example 2 - Negative offset (UTC-5, e.g., New York):
Base table stores data in UTC. A record 2025-07-26 03:00:00 UTC is actually 2025-07-25 22:00:00 in New York time and should belong to July 25, not July 26.

-- Shifts data -5 hours before truncating to day
date_trunc(date_sub(col, INTERVAL 5 HOUR), 'day')
-- 2025-07-26 03:00:00 → -5h → 2025-07-25 22:00:00 → truncate → 2025-07-25 ✓

Previously, these expressions failed when a single base partition spanned multiple roll-up buckets. Now it correctly maps to multiple MTMV partitions (1-to-N mapping).

What is changed and how does it work?

Core Changes:

New Partition Expression Type: MTMVPartitionExprDateTruncDateAddSub
- Handles date_trunc(date_add/sub(col, INTERVAL N HOUR), 'day/week/month/quarter/year')
- Supports both positive (date_add) and negative (date_sub) hour offsets
- Validates partition column types (DATETIME/DATETIMEV2 only)
1-to-N Partition Mapping:
- generateRollUpPartitionKeyDescs() returns List<PartitionKeyDesc>
- Iterates through all roll-up buckets between lower and upper bounds
- Each base partition can map to multiple MTMV partitions
Full Lifecycle Support:
- Union compensation: Hour-offset applied to predicates via HoursAdd/HoursSub
- Refresh: Correct partition range calculation with offset
- SQL serialization: Preserves expression structure
- Backward compatibility: Default implementation in interface

Supported Time Unit Combinations:

PARTITION BY Unit	Allowed SELECT Units
hour	hour, day, week, month, quarter, year
day	day, week, month, quarter, year
week	week, month, quarter, year
month	month, quarter, year
quarter	quarter, year
year	year

Note: SELECT unit must be equal to or coarser than PARTITION BY unit (roll-up hierarchy).

Supported Column Types:

Type	Supported	Note
DATE	❌	No time component
DATEV2	❌	No time component
DATETIME	✅	Full support
DATETIMEV2	✅	Full support
DATETIMEV2(0-6)	✅	All precision levels

Limitations:

Only RANGE partition supported (LIST not supported)
Only HOUR unit in date_add/sub (MINUTE, SECOND not supported)
Base table partition column must be DATETIME/DATETIMEV2

Future Consideration:
Hour offset could be validated to [-14, +14] range based on real-world timezone limits (UTC-12 to UTC+14).

Release note

Feature: Support date_add/sub hour offset in MTMV partition expressions for timezone-aware partitioning. Users can now create materialized views with expressions like date_trunc(date_add(col, INTERVAL 3 HOUR), 'day') to align partitions with their local timezone.

Check List (For Author)

Test: Regression test + Unit test
- Added comprehensive regression tests for hour-offset scenarios (positive and negative)
- Added unit tests for 1-to-N partition mapping
- Tested aligned and cross-boundary edge cases
- Tested all valid time unit combinations (hour→day, day→week, etc.)
- Negative scenarios (DATE type, unsupported units)
Behavior changed: Yes
- Previously failed with alignment error; now succeeds with 1-to-N mapping
Does this need documentation: Yes
- User-facing feature, should document supported expressions and limitations

…TC-midnight base partitions ### What problem does this PR solve? Issue Number: close apache#62395 Problem Summary: Implement 1-to-N partition mapping to support hour-offset MTMV partition expressions when base table partitions are aligned to UTC-midnight boundaries. ### Release note Feature: Support hour-offset MTMV partition expressions with UTC-midnight base partitions. ### Check List (For Author) - Test: Regression test + Unit test - Behavior changed: Yes - Does this need documentation: Yes

Thearas · 2026-04-18T10:59:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hakanuzum · 2026-04-18T11:04:59Z

run buildall

…scRollUpGeneratorTest

hakanuzum · 2026-04-18T13:19:03Z

/review

github-actions · 2026-04-18T15:15:13Z

OpenCode automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24605553375

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

hakanuzum · 2026-04-18T15:28:07Z

run buildall

hello-stephen · 2026-04-18T17:00:22Z

FE UT Coverage Report

Increment line coverage 18.04% (70/388) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-18T18:10:40Z

FE Regression Coverage Report

Increment line coverage 6.70% (26/388) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Problem Summary: MTMV partition expressions with CAST and hour-offset arithmetic (hours_add/hours_sub) were being rejected as invalid implicit expressions during partition increment validation. ### What is changed and how does it work? Extended SUPPORT_EXPRESSION_TYPES in PartitionIncrementMaintainer to include: - Cast.class - Allow CAST expressions in partition columns - HoursAdd.class - Support hours_add arithmetic - HoursSub.class - Support hours_sub arithmetic This enables partition expressions like: date_trunc(date_add(cast(k2 as date), INTERVAL 3 HOUR), 'day') ### Release note None - Internal fix for MTMV partition expression validation ### Check List (For Author) - Test: Unit Test - Fixed MTMVPlanUtilTest.testPartitionExprPreservesCastInHourOffset - Fixed MTMVPlanUtilTest.testPartitionExprUsesLineageForAliasHourOffset - All 18 FE unit tests now pass - Behavior changed: No (enables previously blocked functionality) - Does this need documentation: No

### What problem does this PR solve? Problem Summary: Unit test coverage for MTMVPartitionExprDateTruncDateAddSub and MTMVRelatedPartitionDescRollUpGenerator was insufficient (~60-70%). Many code paths including dateIncrement() time units, Type.DATE handling, and equals/hashCode methods were untested. ### What is changed and how does it work? Added 9 new unit tests to MTMVRelatedPartitionDescRollUpGeneratorTest: - dateIncrement() coverage: week, month, quarter, year, hour time units - dateTimeToStr() Type.DATE path coverage - equals() and hashCode() comprehensive edge case testing - generateRollUpPartitionKeyDesc() direct 1-to-1 mapping test - date_sub with UTC-midnight 1-to-N edge case test Test count: 5 → 14 tests (+180%) All 27 MTMV tests pass (MTMVPlanUtilTest 13/13 + MTMVRelatedPartitionDescRollUpGeneratorTest 14/14) ### Release note None - Test coverage improvement only ### Check List (For Author) - Test: Unit Test - Added 9 new test cases - All 27 MTMV unit tests pass (100% success rate) - Estimated coverage improvement: * MTMVPartitionExprDateTruncDateAddSub: 60-70% → 85-95% (+25-30%) * MTMVRelatedPartitionDescRollUpGenerator: 60% → 75% (+15%) - Branch coverage: * dateIncrement(): 6/6 branches (100%) * dateTimeToStr(): 2/2 paths (100%) * equals/hashCode: full coverage - Behavior changed: No - Does this need documentation: No

hakanuzum · 2026-04-18T22:30:57Z

run buildall

hello-stephen · 2026-04-19T00:04:50Z

FE UT Coverage Report

Increment line coverage 29.31% (114/389) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-19T01:14:13Z

FE Regression Coverage Report

Increment line coverage 24.42% (95/389) 🎉
Increment coverage report
Complete coverage report

…cate for hour-offset MTMV ### What problem does this PR solve? Problem Summary: 1. generateRollUpPartitionKeyDescs() included the upper-bound bucket incorrectly when the offset-shifted upper bound landed exactly on a time-unit boundary (aligned partitions like 21:00 +3h = 00:00). This caused 9/14 unit tests to fail with wrong partition counts. 2. UpdateMvByPartitionCommand.constructTableWithPredicates() applied the MV partition range directly to the base column (k2 >= '2025-07-25'), ignoring the hour offset. For date_trunc(date_add(k2, 3h), 'day'), the correct predicate is hours_add(k2, 3) >= '2025-07-25', which expands to k2 >= '2025-07-24 21:00:00'. Without this fix, MTMV refresh task failed with 'no partition for this tuple'. ### What is changed and how does it work? MTMVPartitionExprDateTruncDateAddSub.generateRollUpPartitionKeyDescs(): - Compute upperWithOffset = dateOffset(upperRaw) BEFORE truncation - includeEndBucket = !isSameTime(upperWithOffset, endBucket) - Loop: use <= when includeEndBucket (UTC-midnight 1-to-N), < otherwise (aligned) - Extract applyDateTrunc() helper; add public getOffsetHours() getter UpdateMvByPartitionCommand.constructTableWithPredicates(): - Detect MTMVPartitionExprDateTruncDateAddSub via MTMVPartitionExprFactory - Use HoursAdd/HoursSub(slot, N) as predicate target instead of raw slot - Add constructPredicates(Set<PartitionItem>, Expression) overload - Update convertListPartitionToIn / convertRangePartitionToCompare to accept Expression Tests: - Add 6 new unit tests in MTMVRelatedPartitionDescRollUpGeneratorTest: getRollUpIdentity (single/multi-same-day/different-day), dateTimeToStr error path, constructor validation, strengthened testRollUpRangeDateSubHourWithUtcMidnightBasePartitions - Fix datetime normalization (T vs space) in union compensation regression test - Fix table name length (65 > 64 limit) for utc-midnight test scenario - Remove unsupported union-compensation assertions for UTC-midnight 1-to-N case ### Release note None - internal fix for MTMV partition refresh ### Check List (For Author) - Test: Unit Test + Regression Test - MTMVRelatedPartitionDescRollUpGeneratorTest: 20/20 pass - MTMVPlanUtilTest: 13/13 pass - test_rollup_partition_mtmv_date_add: PASSED - test_union_compensation_mtmv_date_add_hour_offset: PASSED - Behavior changed: Yes (refresh now correctly uses hour-shifted predicate) - Does this need documentation: No

hakanuzum · 2026-04-19T10:39:35Z

run buildall

hello-stephen · 2026-04-19T12:14:47Z

FE UT Coverage Report

Increment line coverage 34.62% (144/416) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-19T13:28:39Z

FE Regression Coverage Report

Increment line coverage 43.75% (182/416) 🎉
Increment coverage report
Complete coverage report

hakanuzum · 2026-04-19T13:44:38Z

@zddr @seawinde could you please take a look?

hakanuzum · 2026-04-19T17:42:30Z

/review

github-actions · 2026-04-19T19:38:42Z

OpenCode automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24635179064

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

hakanuzum mentioned this pull request Apr 18, 2026

[improvement](fe) Support hour-offset date_add/date_sub in MTMV partition refresh #62535

Closed

3 tasks

[fix](test) Remove merge conflict markers from MTMVRelatedPartitionDe…

826d5d7

…scRollUpGeneratorTest

Merge branch 'master' into feat/mtmv-hour-offset-v3

81ec31d

hakanuzum added 2 commits April 19, 2026 01:29

hakanuzum changed the title ~~[feature](mtmv) Support hour-offset MTMV partition expressions with UTC-midnight base partitions~~ [feature](mtmv) Support date_add/sub hour offset in MTMV partition expressions Apr 19, 2026

Conversation

hakanuzum commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how does it work?

Release note

Check List (For Author)

Uh oh!

Thearas commented Apr 18, 2026

Uh oh!

hakanuzum commented Apr 18, 2026

Uh oh!

hakanuzum commented Apr 18, 2026

Uh oh!

github-actions bot commented Apr 18, 2026

Uh oh!

hakanuzum commented Apr 18, 2026

Uh oh!

hello-stephen commented Apr 18, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 18, 2026

FE Regression Coverage Report

Uh oh!

hakanuzum commented Apr 18, 2026

Uh oh!

hello-stephen commented Apr 19, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 19, 2026

FE Regression Coverage Report

Uh oh!

hakanuzum commented Apr 19, 2026

Uh oh!

hello-stephen commented Apr 19, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 19, 2026

FE Regression Coverage Report

Uh oh!

hakanuzum commented Apr 19, 2026

Uh oh!

hakanuzum commented Apr 19, 2026

Uh oh!

github-actions bot commented Apr 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hakanuzum commented Apr 18, 2026 •

edited

Loading