Skip to content

fix(isthmus): preserve field types in sort and aggregate references - #1090

Merged
nielspardon merged 3 commits into
substrait-io:mainfrom
alexandrefimov:agent/aggregate-call-collations
Aug 13, 2026
Merged

fix(isthmus): preserve field types in sort and aggregate references#1090
nielspardon merged 3 commits into
substrait-io:mainfrom
alexandrefimov:agent/aggregate-call-collations

Conversation

@alexandrefimov

@alexandrefimov alexandrefimov commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Calcite collations were already propagated to Substrait sort fields, but those field references used the whole input struct as their type. The shared conversion is used by both aggregate sort arguments and Sort relations, so those plans were not round-trip stable. Aggregate pre-measure filters had the same problem.

Build each reference from the selected input field, preserving its exact type in Sort relations, aggregate sort arguments, and pre-measure filters.

Closes #215

Use the referenced input field type for aggregate sort expressions and cover direct conversion plus full round-trip behavior.
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Known CI dependency: the two OSV failures are from the existing HttpCore version and are fixed by #1089. No additional change is needed in this PR; I will update it from main after #1089 is merged.

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type fix is the right one, and it's worth stating that it repairs more than the title says: toSortField is shared with visit(Sort), so this also makes every Calcite-derived Sort rel round-trip-stable. I checked by reverting only the production line — assertFullRoundTrip("SELECT L_ORDERKEY FROM LINEITEM ORDER BY L_ORDERKEY") fails on main and passes with your change.

Three asks that have no line in the diff to hang off:

  • The fromAggCall Javadoc (lines 509-514) attributes the fp64 rewrite to this method and describes it as casting the argument in place. It's actually castStatisticalAggregatesToFloatingPoint, called from visit(Aggregate), and it appends fp64 columns and re-points the call. This diff removes the method's only other comment, so that paragraph is now its whole description.
  • Please add one assertFullRoundTrip with an ORDER BY to pin the Sort-rel half. Nothing currently covers it: ComplexSortTest compares Calcite explain text, SubstraitRelNodeConverterTest$Sort asserts only getRowType(), and simpleOrderByClause uses assertSqlSubstraitRelRoundTrip, which compares two POJOs both produced by SubstraitRelVisitor and so is blind to a uniformly-wrong type.
  • Since the squash message becomes the changelog entry, could the title name the real defect and the Sort scope rather than "preserve aggregate collations"? Collations have in fact been propagated since the initial commit (AggregateFunctionConverter.sortFields), which your body says correctly — the removed TODO was just stale, so #215's literal text was already satisfied and the type defect is the fix worth advertising. I'd also drop the "cover mixed direction and null-order collations…" sentence; the diff shows the tests.

Everything else I ran into is pre-existing and out of scope here, and has been filed as issues.

Comment thread isthmus/src/main/java/io/substrait/isthmus/SubstraitRelVisitor.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/SubstraitRelVisitor.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/SubstraitRelVisitor.java
Comment thread isthmus/src/test/java/io/substrait/isthmus/AggregationFunctionsTest.java Outdated
Comment thread isthmus/src/test/java/io/substrait/isthmus/AggregationFunctionsTest.java Outdated
@alexandrefimov alexandrefimov changed the title fix(isthmus): preserve aggregate collations fix(isthmus): preserve field types in sort and aggregate references Aug 13, 2026
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Addressed the requested changes in 56106e7: fixed the pre-measure filter field type, switched sort references to constructOnRoot, strengthened the null-order expectations, corrected the fromAggCall Javadoc, and added a full ORDER BY round-trip. I also updated the title and description to describe the field-type defect and Sort scope. I left NullDirection.UNSPECIFIED and the optional test move out of this PR as suggested.

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All addressed — thanks. The UNSPECIFIED null-direction case you left out is tracked in #1092.

@nielspardon
nielspardon merged commit 43da396 into substrait-io:main Aug 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AggregateCall conversion ignores collations

2 participants