feat: add support for specifying queryable historical tiers via query context by jtuglu1 · Pull Request #19537 · apache/druid

jtuglu1 · 2026-06-01T18:14:54Z

Description

Support specifying specific historical tiers to be queried dynamically via query context. Tiers may either be physical or virtual (mapped to physical tiers via coordinator dynamic config).

Release note

Support specifying specific historical tiers to be queried dynamically via query context.

This PR has:

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 9 of 9 changed files.

This is an automated review by Codex GPT-5.5

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 9 of 9 changed files.

This is an automated review by Codex GPT-5.5

… context

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 9 of 9 changed files.

This is an automated review by Codex GPT-5.5

abhishekrb19

LGTM. Left some comments - this is on the query path, so it'll be good to remove cleanup any redundant ops - thanks.

abhishekrb19 · 2026-06-25T02:17:03Z

@@ -604,10 +611,15 @@ private SortedMap<DruidServer, List<SegmentDescriptor>> groupSegmentsByServer(
        CloneQueryMode cloneQueryMode
    )
    {
+      final Set<String> queryableHistoricalTiers = query.context().getQueryableHistoricalTiers();


Similar to cloneQueryMode, it'll be good to get the historical tiers once and pass it to both computeResultLevelCachingEtag() and pick() to avoid getting the tiers from parsing the context unnecessarily

abhishekrb19 · 2026-06-25T02:31:11Z

+    for (final int priority : queryableServers.keySet()) {
+      final Set<QueryableDruidServer> priorityServers = new HashSet<>();
+      for (final QueryableDruidServer server : queryableServers.get(priority)) {


Since this is a tree map, you could avoid the double lookup by iterating on the queryableServers.int2ObjectEntrySet() once as you go instead of getting the priority and then looking it up again with it

Would likely require a small restructure here.

    )
    {
-      Hasher hasher = Hashing.sha1().newHasher();
+      final Hasher hasher = Hashing.sha1().newHasher();


FrankChen021

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Reviewed 9 of 9 changed files.

Found one correctness issue: virtual tier aliases are compared directly with physical server tiers and can silently select no servers.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021 · 2026-07-15T12:20:06Z

+    for (final Int2ObjectMap.Entry<Set<QueryableDruidServer>> entry : queryableServers.int2ObjectEntrySet()) {
+      final Set<QueryableDruidServer> priorityServers = new HashSet<>();
+      for (final QueryableDruidServer server : entry.getValue()) {
+        if (queryableHistoricalTiers.contains(server.getServer().getTier())) {


[P1] Resolve virtual tier aliases before filtering servers

This compares requested tiers directly with each server's physical tier, but the PR description says queryableHistoricalTiers supports virtual tiers from historicalTierAliases. Alias keys are never physical tiers: Coordinator expansion loads segments only onto their mapped tiers. Therefore, with historicalTierAliases={"hot":["hot_1","hot_2"]}, querying queryableHistoricalTiers=["hot"] matches no server and silently returns empty or partial results. The Broker's coordinator-config view should expand aliases before hasQueryableHistoricalServer and getQueryableHistoricalServers perform these checks, with coverage for virtual and mixed virtual/physical tier sets.

@jtuglu1 , this seems valid. It doesn't seem that we are doing tier alias expansion in the current patch.
If not, we can do it in a follow up and update the PR description/release note.

A follow-up is reasonable if this PR is intentionally narrowed to physical tier names only. Please update the description and release note before merge, since they still promise virtual tiers; the added query-context documentation already describes physical druid.server.tier matching. Reviewed 9 of 9 changed files.

kfaraz · 2026-07-30T06:41:01Z

+|`setProcessingThreadNames`|`true`| Whether processing thread names will be set to `queryType_dataSource_intervals` while processing a query. This aids in interpreting thread dumps, and is on by default. Query overhead can be reduced slightly by setting this to `false`. This has a tiny effect in most scenarios, but can be meaningful in high-QPS, low-per-segment-processing-time scenarios. |
 |`sqlPlannerBloat`|`1000`|Calcite parameter which controls whether to merge two Project operators when inlining expressions causes complexity to increase. Implemented as a workaround to exception `There are not enough rules to produce a node with desired properties: convention=DRUID, sort=[]` thrown after rejecting the merge of two projects.|
 |`cloneQueryMode`|`excludeClones`| Indicates whether clone Historicals should be queried by brokers. Clone servers are created by the `cloneServers` Coordinator dynamic configuration. Possible values are `excludeClones`, `includeClones` and `preferClones`. `excludeClones` means that clone Historicals are not queried by the broker. `preferClones` indicates that when given a choice between the clone Historical and the original Historical which is being cloned, the broker chooses the clones. Historicals which are not involved in the cloning process will still be queried. `includeClones` means that broker queries any Historical without regarding clone status. This parameter only affects native queries. MSQ does not query Historicals directly.|
+|`queryableHistoricalTiers`|`null`|Set of Historical tier names that may be queried. When set, the Broker only queries Historical servers whose `druid.server.tier` is in this set. Segments without a replica on one of the listed tiers are skipped.|


Does this also need to mention that tier aliases may be used instead of actual tier names?

kfaraz · 2026-07-30T06:46:49Z

+    return hasQueryableServer(getQueryableHistoricalTiers(query), cloneQueryMode);
+  }
+
+  public boolean hasQueryableServer(


Please add a short javadoc.

kfaraz · 2026-07-30T06:47:57Z

  {
    synchronized (this) {
-      if (!historicalServers.isEmpty()) {
-        return historicalTierStrategy.pick(query, filter.getQueryableServers(historicalServers, cloneQueryMode), segment.get());
+      if (queryableHistoricalTiers != null && queryableHistoricalTiers.isEmpty()) {


This check can be done before the synchronized block.

kfaraz · 2026-07-30T06:50:09Z

+        final Int2ObjectRBTreeMap<Set<QueryableDruidServer>> queryableHistoricalServers =
+            getQueryableHistoricalServers(
+                filter.getQueryableServers(historicalServers, cloneQueryMode),
+                queryableHistoricalTiers


It would probably make more sense to pass the list of queryable tiers into the filter.getQueryableServers since the function seems very closely related.
This would allow us to keep the server filtration logic in a single place, and will also provide an easy way for tier alias expansion in BrokerViewOfCoordinatorConfig, which is an impl of the HistoricalFilter.

kfaraz · 2026-07-30T06:55:52Z

+  )
+  {
+    synchronized (this) {
+      if (queryableHistoricalTiers != null) {


This check can happen outside the synchronized block.

kfaraz · 2026-07-30T07:44:52Z

As discussed with @jtuglu1 offline, this PR needs some more changes. We are leaving it out of Druid 38 for now.

github-actions Bot added the Area - Documentation label Jun 1, 2026

github-advanced-security AI found potential problems Jun 1, 2026

View reviewed changes

jtuglu1 force-pushed the filter-specific-historical-tiers-in-query-context branch 2 times, most recently from f00a25a to 121ada6 Compare June 1, 2026 19:53

jtuglu1 marked this pull request as ready for review June 1, 2026 22:19

FrankChen021 reviewed Jun 2, 2026

View reviewed changes

jtuglu1 force-pushed the filter-specific-historical-tiers-in-query-context branch from 8bf1431 to dd0e846 Compare June 2, 2026 22:28

FrankChen021 reviewed Jun 3, 2026

View reviewed changes

feat: add support for specifying queryable historical tiers via query…

33bee46

… context

jtuglu1 force-pushed the filter-specific-historical-tiers-in-query-context branch from dd0e846 to 33bee46 Compare June 19, 2026 02:04

FrankChen021 reviewed Jun 19, 2026

View reviewed changes

jtuglu1 requested a review from abhishekrb19 June 19, 2026 15:50

abhishekrb19 approved these changes Jun 25, 2026

View reviewed changes

jtuglu1 added this to the 38.0.0 milestone Jul 14, 2026

github-advanced-security AI found potential problems Jul 14, 2026

View reviewed changes

Comment thread server/src/main/java/org/apache/druid/client/CachingClusteredClient.java

)

{

Hasher hasher = Hashing.sha1().newHasher();

final Hasher hasher = Hashing.sha1().newHasher();

more changes

49c83dc

jtuglu1 force-pushed the filter-specific-historical-tiers-in-query-context branch from fca81e3 to 49c83dc Compare July 15, 2026 00:09

FrankChen021 reviewed Jul 15, 2026

View reviewed changes

kfaraz reviewed Jul 30, 2026

View reviewed changes

kfaraz modified the milestones: 38.0.0, 39.0.0 Jul 30, 2026

Uh oh!

Conversation

jtuglu1 commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release note

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

abhishekrb19 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jtuglu1 commented Jun 1, 2026 •

edited

Loading

kfaraz Jul 30, 2026 •

edited

Loading