Spark 4.2 (SPARK-57298) makes collect_set normalize NaN and -0.0 for floating-point types. Comet's native collect_set aggregate does not perform this normalization, so results differ from Spark.
Failing Spark SQL test (currently marked IgnoreComet in dev/diffs/4.2.0.diff):
DataFrameAggregateSuite: "SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point types"
Note: the array-expression normalization path (array_distinct / array_except) is already handled; this issue tracks the aggregate collect_set path.
Spark 4.2 (SPARK-57298) makes
collect_setnormalizeNaNand-0.0for floating-point types. Comet's nativecollect_setaggregate does not perform this normalization, so results differ from Spark.Failing Spark SQL test (currently marked
IgnoreCometindev/diffs/4.2.0.diff):DataFrameAggregateSuite: "SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point types"Note: the array-expression normalization path (
array_distinct/array_except) is already handled; this issue tracks the aggregatecollect_setpath.