fix(arrow-cast): do not truncate integers when casting to Decimal32/64 by yongster · Pull Request #10707 · apache/arrow-rs

yongster · 2026-08-16T07:28:46Z

Which issue does this PR close?

Closes cast from integer to Decimal32/64 silently truncates values that do not fit #10706.

Rationale for this change

cast / cast_with_options from an integer array to Decimal32 or
Decimal64 can rewrite the value instead of rejecting it.

cast_integer_to_decimal first used AsPrimitive (as), which wraps when
the source integer does not fit the decimal native type. The precision check
then ran on the already-truncated value. If that wrapped value happened to
fit the requested precision, it was stored as if it were the original number.

Examples on current main:

Int64(5_000_000_000) -> Decimal32(9, 0) (safe: true) becomes 705032704
UInt32(4_000_000_000) -> Decimal32(9, 0) (safe: false) becomes -294967296
UInt64::MAX -> Decimal64(18, 0) (safe: false) becomes -1

The same inputs cast to Decimal128 already return null / Err, because
i64 as i128 is lossless.

What changes are included in this PR?

Convert the source integer through a range-checked i128 path
(num_cast + DecimalCast) before applying scale and precision.
safe: true writes null when the original value does not fit; safe: false
returns Err and reports the original value.
Add regression tests for Int64 -> Decimal32, UInt32 -> Decimal32, and
UInt64::MAX -> Decimal64.

Are these changes tested?

Yes. I ran:

cargo test -p arrow-cast --lib -- cast
cargo clippy -p arrow-cast --all-targets --all-features -- -D warnings
cargo +stable fmt --all -- --check

Closes apache#10706. cast_integer_to_decimal used AsPrimitive before the precision check. That silently wraps values that do not fit the decimal native type, so Int64(5_000_000_000) became Decimal32 705032704, and UInt32(4e9) / UInt64::MAX could become negative. Convert through a range-checked i128 path first, then apply scale and precision. Safe casts become null; unsafe casts error with the original value. Add regression tests for Int64 -> Decimal32, UInt32 -> Decimal32, and UInt64::MAX -> Decimal64.

Jefffrey · 2026-08-17T05:25:38Z

+/// Convert an integer to a decimal native value without wrapping.
+///
+/// `AsPrimitive` / `as` silently truncates when the source is wider than `M`
+/// (for example `5_000_000_000i64 as i32`). All integer sources fit in `i128`,
+/// so go through that and then use [`DecimalCast`] which is range-checked.
+fn integer_to_decimal_native<I, M>(value: I) -> Option<M>
+where
+    I: NumCast,
+    M: DecimalCast,
+{
+    num_cast::<I, i128>(value).and_then(M::from_decimal)
+}


do we go through i128 because theres no easy way to go directly from the input integer type to the native decimal type?

e.g. for i64 -> decimal32, on main we do i64 as i32 then do the scaling, this PR seems to do i32::try_from(i64 as i128) if im understanding it correctly?

though maybe LLVM optimizes this away, not sure if we should rely on that

Yes, the i128 intermediate is intentional.

This helper is shared by casts to Decimal32, Decimal64, Decimal128, and
Decimal256, so the target native type M can be i32, i64, i128, or
i256. A direct generic num_cast::<I, M> is not available for all of these
targets, and DecimalCast::from_decimal provides the checked final conversion.

All integer input types dispatched to this helper fit losslessly in i128.
I agree that the current num_cast expression obscures that invariant, so I
will replace it with an explicit lossless Into<i128> conversion.

Scale negative values in the source integer type before the checked conversion to the decimal native type. This allows values that fit after scaling to be cast successfully and avoids per-element i128 division. Use an explicit lossless i128 intermediate for decimal conversion and add Criterion benchmarks for integer-to-decimal casts.

Compute negative scale factors once in the source integer type before casting to the decimal native type. This allows values that fit after scaling to be converted without moving exponentiation into the hot loop. Move integer-to-decimal benchmarks into the existing cast kernel suite and remove the standalone arrow-cast benchmark target.

Jefffrey · 2026-08-18T03:43:39Z

+/// `AsPrimitive` / `as` silently truncates when the source is wider than `M`
+/// (for example `5_000_000_000i64 as i32`). All integer sources fit in `i128`
+/// losslessly, which [`DecimalCast`] then converts to the decimal native type
+/// with a range check.


Suggested change

/// with a range check.

/// with a range check. For types that always fit (e.g. `i64` to `Decimal128`) this

/// should get optimized to being equivalent to `i64 as i128`.

Jefffrey · 2026-08-18T03:44:15Z

+            // A scale factor that overflows the source type is larger than all
+            // source values, so integer division produces zero.


Suggested change

// A scale factor that overflows the source type is larger than all

// source values, so integer division produces zero.

// A scale factor that overflows the source type is larger than all

// source values, so integer division produces zero.

//

// For a well formed decimal scale, this path should never be reachable.

Jefffrey · 2026-08-18T03:44:27Z

-    i8: num_traits::AsPrimitive<M>,
-    i16: num_traits::AsPrimitive<M>,
-    i32: num_traits::AsPrimitive<M>,
-    i64: num_traits::AsPrimitive<M>,


Jefffrey · 2026-08-18T03:46:02Z

+            err.contains("5000000000"),
+            "unsafe error should report the original value, got {err}"
+        );
+        assert!(
+            !err.contains("705032704"),


could we assert more of the error message; this isnt clearly obvious that we're getting the error we expect

Jefffrey · 2026-08-18T03:49:18Z

+    let i64_decimal32_array: ArrayRef = Arc::new(Int64Array::from_iter_values(0..512));
+    let i64_decimal32_scaled_array: ArrayRef = Arc::new(Int64Array::from_value(5_000_000_000, 512));


do we need these new arrays? can we use the existing i64_array above?

Assert full overflow error messages, reuse the existing i64 array in cast benchmarks, and clarify integer-to-decimal conversion comments.

github-actions Bot added arrow Changes to the arrow crate arrow-cast labels Aug 16, 2026

Jefffrey added the bug label Aug 16, 2026

Merge branch 'main' into fix/cast-integer-to-decimal32-truncation

c48686a

Jefffrey reviewed Aug 17, 2026

View reviewed changes

yang3.xie added 2 commits August 17, 2026 16:42

bench(arrow-cast): add integer to decimal cast benchmark

fb7ea0c

Jefffrey reviewed Aug 17, 2026

View reviewed changes

Comment thread arrow-cast/benches/integer_to_decimal.rs Outdated

Comment thread arrow-cast/src/cast/mod.rs Outdated

Jefffrey reviewed Aug 18, 2026

View reviewed changes

yongster and others added 2 commits August 18, 2026 16:54

Merge branch 'main' into fix/cast-integer-to-decimal32-truncation

ea8ad91

fix(arrow-cast): address review comments

8c95572

Assert full overflow error messages, reuse the existing i64 array in cast benchmarks, and clarify integer-to-decimal conversion comments.

Navigation Menu

Uh oh!

fix(arrow-cast): do not truncate integers when casting to Decimal32/64 - #10707

fix(arrow-cast): do not truncate integers when casting to Decimal32/64#10707
yongster wants to merge 7 commits into
apache:mainfrom
yongster:fix/cast-integer-to-decimal32-truncation

yongster commented Aug 16, 2026

Uh oh!

Jefffrey Aug 17, 2026

Uh oh!

yongster Aug 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jefffrey Aug 18, 2026

Uh oh!

Jefffrey Aug 18, 2026

Uh oh!

Jefffrey Aug 18, 2026

Uh oh!

Jefffrey Aug 18, 2026

Uh oh!

Jefffrey Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	/// with a range check.
	/// with a range check. For types that always fit (e.g. `i64` to `Decimal128`) this
	/// should get optimized to being equivalent to `i64 as i128`.

		// A scale factor that overflows the source type is larger than all
		// source values, so integer division produces zero.

		let i64_decimal32_array: ArrayRef = Arc::new(Int64Array::from_iter_values(0..512));
		let i64_decimal32_scaled_array: ArrayRef = Arc::new(Int64Array::from_value(5_000_000_000, 512));

Uh oh!

Conversation

yongster commented Aug 16, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Uh oh!

Jefffrey Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

yongster Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jefffrey Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants