HIVE-29262: Incorrect column ordering output in case of different ordering of mutual columns in query & window function in vectorized PTF by tanishq-chugh · Pull Request #6512 · apache/hive

tanishq-chugh · 2026-05-27T15:11:46Z

What changes were proposed in this pull request?

Reorder partition columns so they match the key input column sequence during Partitioned Table Function (PTF) execution

Why are the changes needed?

To ensure, the query column ordering is maintained in case of mutual columns in query and PTF having different ordering

Does this PR introduce any user-facing change?

Yes, the column ordering in the query now matches as the input column sequence provided by the user

How was this patch tested?

Manual Testing + added Qtest

difin · 2026-06-17T14:30:14Z

+LOAD DATA LOCAL INPATH '../../data/files/web_sales_2k' OVERWRITE INTO TABLE web_sales_txt;
+select  ws_bill_customer_sk,ws_item_sk from web_sales_txt;
+
+SET hive.vectorized.execution.enabled;


Why is this line needed? hive.vectorized.execution.enabled is set to true on line 1.

Hi @difin , Thanks for checking this!
was a leftover, removed in commit: a5e3bfc

difin · 2026-06-17T14:32:11Z

+    ws_sold_date_sk,
+    ws_sales_price,
+    LAG(ws_sales_price) OVER (
+        PARTITION BY ws_item_sk,ws_bill_customer_sk


Please add a space between ws_item_sk,ws_bill_customer_sk

Added in commit: a5e3bfc

…nExpressions

difin

LGTM, pending green build.

sonarqubecloud · 2026-06-23T07:54:47Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.0% Duplication on New Code

See analysis details on SonarQube Cloud

difin · 2026-06-23T18:00:56Z

Thanks for contribution, @tanishq-chugh !

…ering of mutual columns in query & window function in vectorized PTF (apache#6512) (cherry picked from commit 46927a2)

Aggarwal-Raghav · 2026-07-10T09:17:02Z

This PR, is also fixing 1 additional issue of ClassCastException when there is overlap column in partition by and order by clause. I observed the below stacktrace on Hive 4.0.1 after debugging I found this PR is fixing it as well.

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 8)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:410)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:258)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:293)
    ... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 8)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:518)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)
    ... 19 more
Caused by: java.lang.ClassCastException: class [B cannot be cast to class java.lang.Long ([B and java.lang.Long are in module java.base of loader 'bootstrap')
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.setRepeatingColumn(VectorPTFGroupBatches.java:803)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.copyPartitionColumnToOverflow(VectorPTFGroupBatches.java:785)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.forwardSpilledBatches(VectorPTFGroupBatches.java:723)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.fillGroupResultsAndForward(VectorPTFGroupBatches.java:525)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.finishPartition(VectorPTFOperator.java:633)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.process(VectorPTFOperator.java:402)
    at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
    at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:502)
    ... 20 more

Attaching the q file for the repro without this PR. vector_ptf_spill_partition_order_overlap.sql

Adding this here for reference if someone faces this issue on older hive version :-)
CC @deniskuzZ @abstractdog

deniskuzZ · 2026-07-10T09:55:09Z

@Aggarwal-Raghav is #6579 addressing the same ?

deniskuzZ · 2026-07-10T09:56:43Z

    int[] nonKeyInputColumnMap = ArrayUtils.toPrimitive(nonKeyInputColumns.toArray(new Integer[0]));

+    if (isPartitionOrderBy && partitionKeyCount > 1) {
+      reorderPartitionColumnsToMatchOutputOrder(outputSignature, evaluatorCount, outputColumnProjectionMap,


TBH, i am not sure it's a right place for the fix.
I think the PARTITION-BY-key restoration bug should be fixed in VectorPTFOperator/VectorPTFGroupBatches — map partitionKey[i] to keyWithoutOrderColumnMap positions by column number at the point of use — rather than reordering VectorPTFInfo in the Vectorizer

Yes, that is what I was thinking, I'll check if vector_ptf_spill_partition_order_overlap.sql and vector_ptf_cols_order.q‎ are passing after that.

Subject: [PATCH] patch --- Index: ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java --- a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java (revision 1b1fa6d6a00f38ff8968bea673b5006615cbc2b3) +++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java (date 1783678424148) @@ -349,6 +349,7 @@ bufferedTypeInfos, orderColumnMap, keyWithoutOrderColumnMap, + partitionColumnMap, overflowBatch); isFirstPartition = true; Index: ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java --- a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java (revision 1b1fa6d6a00f38ff8968bea673b5006615cbc2b3) +++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java (date 1783678635446) @@ -26,6 +26,7 @@ import java.util.List; import java.util.stream.Collectors; +import org.apache.commons.lang3.ArrayUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.common.type.HiveIntervalDayTime; import org.apache.hadoop.hive.ql.exec.PTFPartition; @@ -64,6 +65,7 @@ private int[] bufferedColumnMap; private int[] orderColumnMap; private int[] keyWithoutOrderColumnMap; + private int[] partitionKeyIndices; private int spillLimitBufferedBatchCount; private String spillLocalDirs; @@ -289,13 +291,21 @@ public void init(VectorPTFEvaluatorBase[] evaluators, int[] outputProjectionColumnMap, int[] bufferedColumnMap, TypeInfo[] bufferedTypeInfos, int[] orderColumnMap, - int[] keyWithoutOrderColumnMap, VectorizedRowBatch overflowBatch) { + int[] keyWithoutOrderColumnMap, int[] partitionColumnMap, VectorizedRowBatch overflowBatch) { this.evaluators = evaluators; this.outputProjectionColumnMap = outputProjectionColumnMap; this.bufferedColumnMap = bufferedColumnMap; this.orderColumnMap = orderColumnMap; this.keyWithoutOrderColumnMap = keyWithoutOrderColumnMap; + partitionKeyIndices = new int[keyWithoutOrderColumnMap.length]; + for (int i = 0; i < keyWithoutOrderColumnMap.length; i++) { + partitionKeyIndices[i] = ArrayUtils.indexOf(partitionColumnMap, keyWithoutOrderColumnMap[i]); + Preconditions.checkState(partitionKeyIndices[i] != ArrayUtils.INDEX_NOT_FOUND, + "Key input column %s not found among partition columns %s", + keyWithoutOrderColumnMap[i], Arrays.toString(partitionColumnMap)); + } + this.overflowBatch = overflowBatch; bufferedBatches = new ArrayList<BufferedVectorizedRowBatch>(0); @@ -775,6 +785,11 @@ } } + /** + * Sets the partition key values as repeating columns in the overflowBatch. Key columns are in + * output projection order, partitionKey values in PARTITION BY order; partitionKeyIndices maps + * between the two. + */ private void copyPartitionColumnToOverflow(Object[] partitionKey) { // Set partition column in overflowBatch. // We can set by ref since our last batch is held by us. @@ -782,7 +797,7 @@ for (int i = 0; i < keyInputColumnCount; i++) { final int keyColumnNum = keyWithoutOrderColumnMap[i]; Preconditions.checkState(overflowBatch.cols[keyColumnNum] != null); - setRepeatingColumn(partitionKey[i], overflowBatch, keyColumnNum); + setRepeatingColumn(partitionKey[partitionKeyIndices[i]], overflowBatch, keyColumnNum); } } Index: ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java b/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java --- a/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java (revision 1b1fa6d6a00f38ff8968bea673b5006615cbc2b3) +++ b/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java (date 1783678472285) @@ -588,6 +588,7 @@ /* bufferedTypeInfos */ new TypeInfo[] { getTypeInfo("int"), getTypeInfo("string") }, /* orderColumnMap */ new int[] { 1 }, // p_date /* keyWithoutOrderColumnMap */ new int[] { 0 }, // p_mfgr + /* partitionColumnMap */ new int[] { 0 }, // p_mfgr getFakeOperator().setupOverflowBatch(3, new String[] { "bigint", "bigint" }, outputProjectionColumnMap, outputTypeInfos)); }

@deniskuzZ I came up with this. Will check the best of both and will raise PR.

diff --git i/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java w/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java index bd60d215ff..1288703402 100644 --- i/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java +++ w/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFGroupBatches.java @@ -64,6 +64,7 @@ public class VectorPTFGroupBatches extends PTFPartition { private int[] bufferedColumnMap; private int[] orderColumnMap; private int[] keyWithoutOrderColumnMap; + private int[] partitionColumnMap; private int spillLimitBufferedBatchCount; private String spillLocalDirs; @@ -289,12 +290,13 @@ public VectorPTFGroupBatches(Configuration hconf, int vectorizedPTFMaxMemoryBuff public void init(VectorPTFEvaluatorBase[] evaluators, int[] outputProjectionColumnMap, int[] bufferedColumnMap, TypeInfo[] bufferedTypeInfos, int[] orderColumnMap, - int[] keyWithoutOrderColumnMap, VectorizedRowBatch overflowBatch) { + int[] keyWithoutOrderColumnMap, int[] partitionColumnMap, VectorizedRowBatch overflowBatch) { this.evaluators = evaluators; this.outputProjectionColumnMap = outputProjectionColumnMap; this.bufferedColumnMap = bufferedColumnMap; this.orderColumnMap = orderColumnMap; this.keyWithoutOrderColumnMap = keyWithoutOrderColumnMap; + this.partitionColumnMap = partitionColumnMap; this.overflowBatch = overflowBatch; bufferedBatches = new ArrayList<BufferedVectorizedRowBatch>(0); @@ -772,8 +774,16 @@ private void copyPartitionColumnToOverflow(Object[] partitionKey) { final int keyInputColumnCount = keyWithoutOrderColumnMap.length; for (int i = 0; i < keyInputColumnCount; i++) { final int keyColumnNum = keyWithoutOrderColumnMap[i]; + int partitionKeyIndex = -1; + for (int j = 0; j < partitionColumnMap.length; j++) { + if (partitionColumnMap[j] == keyColumnNum) { + partitionKeyIndex = j; + break; + } + } + Preconditions.checkState(partitionKeyIndex != -1, "Could not find keyColumnNum " + keyColumnNum + " in partitionColumnMap"); Preconditions.checkState(overflowBatch.cols[keyColumnNum] != null); - setRepeatingColumn(partitionKey[i], overflowBatch, keyColumnNum); + setRepeatingColumn(partitionKey[partitionKeyIndex], overflowBatch, keyColumnNum); } } diff --git i/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java w/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java index 26987a68f4..feb12617de 100644 --- i/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java +++ w/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java @@ -348,6 +348,7 @@ protected void initializeOp(Configuration hconf) throws HiveException { bufferedTypeInfos, orderColumnMap, keyWithoutOrderColumnMap, + vectorPTFInfo.getPartitionColumnMap(), overflowBatch); isFirstPartition = true; diff --git i/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java w/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java index 3039585817..c74d407c14 100644 --- i/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java +++ w/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFGroupBatches.java @@ -588,6 +588,7 @@ private void init(VectorPTFGroupBatches groupBatches) throws HiveException { /* bufferedTypeInfos */ new TypeInfo[] { getTypeInfo("int"), getTypeInfo("string") }, /* orderColumnMap */ new int[] { 1 }, // p_date /* keyWithoutOrderColumnMap */ new int[] { 0 }, // p_mfgr + /* partitionColumnMap */ new int[] { 0 }, getFakeOperator().setupOverflowBatch(3, new String[] { "bigint", "bigint" }, outputProjectionColumnMap, outputTypeInfos)); }

@Aggarwal-Raghav looks similar to mine (i've used ArrayUtils.indexOf) :) please, go ahead. maybe reuse same jira id just add addendum in commit msg

final int partitionKeyIndex = ArrayUtils.indexOf(partitionColumnMap, keyColumnNum); Preconditions.checkState(partitionKeyIndex != ArrayUtils.INDEX_NOT_FOUND, "Could not find keyColumnNum %s in partitionColumnMap", keyColumnNum);

Aggarwal-Raghav · 2026-07-10T10:06:02Z

@Aggarwal-Raghav is #6579 addressing the same ?

No, #6579 is different issue and still open on master branch

asf-ci-hive added tests pending tests unstable and removed tests pending tests unstable labels May 27, 2026

tanishq-chugh force-pushed the HIVE-29262 branch from 4a2a93b to b895766 Compare June 1, 2026 19:07

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Jun 1, 2026

tanishq-chugh force-pushed the HIVE-29262 branch from b895766 to 62d8db8 Compare June 2, 2026 07:21

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Jun 2, 2026

tanishq-chugh force-pushed the HIVE-29262 branch from 62d8db8 to ca4ac35 Compare June 13, 2026 18:15

asf-ci-hive added tests pending tests unstable tests passed and removed tests unstable tests pending labels Jun 13, 2026

tanishq-chugh changed the title ~~[WIP] HIVE-29262: Incorrect column ordering output in case of different ord…~~ [WIP] HIVE-29262: Incorrect column ordering output in case of different ordering of mutual columns in query & window function Jun 16, 2026

tanishq-chugh changed the title ~~[WIP] HIVE-29262: Incorrect column ordering output in case of different ordering of mutual columns in query & window function~~ HIVE-29262: Incorrect column ordering output in case of different ordering of mutual columns in query & window function Jun 17, 2026

difin reviewed Jun 17, 2026

View reviewed changes

tanishq-chugh added 3 commits June 22, 2026 09:09

Update reorderPartitionColumnsToMatchOutputOrder not reorder partitio…

d3ca6a9

…nExpressions

Address Sonarqube - 1

84dac79

Address Review comments

5abfd2c

tanishq-chugh force-pushed the HIVE-29262 branch from a5e3bfc to 5abfd2c Compare June 22, 2026 03:39

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Jun 22, 2026

difin approved these changes Jun 22, 2026

View reviewed changes

asf-ci-hive added tests pending tests passed tests unstable and removed tests unstable tests pending tests passed labels Jun 22, 2026

asf-ci-hive added tests passed and removed tests pending labels Jun 23, 2026

difin merged commit 46927a2 into apache:master Jun 23, 2026
6 checks passed

Asmoday pushed a commit to arenadata/hive that referenced this pull request Jun 28, 2026

HIVE-29262: Incorrect column ordering output in case of different ord…

c992079

…ering of mutual columns in query & window function in vectorized PTF (apache#6512) (cherry picked from commit 46927a2)

deniskuzZ reviewed Jul 10, 2026

View reviewed changes

Aggarwal-Raghav mentioned this pull request Jul 10, 2026

HIVE-29262: [Addendum] Fix VectorPTFGroupBatches partition key copy when partition and order columns overlap #6597

Open

Uh oh!

Conversation

tanishq-chugh commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

difin Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

tanishq-chugh Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

difin Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

tanishq-chugh Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

difin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Jun 23, 2026

Quality Gate passed

Uh oh!

Uh oh!

difin commented Jun 23, 2026

Uh oh!

Aggarwal-Raghav commented Jul 10, 2026

Uh oh!

deniskuzZ commented Jul 10, 2026

Uh oh!

deniskuzZ Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tanishq-chugh commented May 27, 2026 •

edited

Loading

difin left a comment •

edited

Loading

deniskuzZ Jul 10, 2026 •

edited

Loading

deniskuzZ Jul 10, 2026 •

edited

Loading

Aggarwal-Raghav commented Jul 10, 2026 •

edited

Loading