Feat/enable native decimal support
Summary
Enables native Arrow decimal type support by removing unnecessary blocking condition in ScanRow and enabling native decimal support by default. The decimal128Container infrastructure was already complete - these changes allow decimal columns to return native Arrow decimal128 types instead of raw bytes.
Changes
-
Enable native decimal by default: Set
UseArrowNativeDecimal = trueinArrowConfig.WithDefaults() -
Remove ScanRow blocking: Remove decimal type check from
ScanRowerror condition - Update test expectations: Allow decimal columns to scan successfully
- Interval types remain blocked (still unsupported)
Why Native Arrow Decimal128?
This provides significant advantages over the previous raw bytes approach:
-
Native Arrow types: Arrow records now contain proper
decimal128types with precision/scale metadata -
Type safety: Schema correctly reflects
arrow.DECIMAL128instead ofarrow.STRING - No manual parsing: Direct access to decimal values without byte-level parsing
- Precision preservation: Full decimal precision maintained through Arrow's native decimal128 format
Impact
Before:
-
UseArrowNativeDecimal = falseby default - Decimal columns → raw bytes (
[]byte) in Arrow records requiring manual parsing - Schema shows
arrow.STRINGtype for decimal columns
After:
-
UseArrowNativeDecimal = trueby default - Decimal columns → native
arrow.Decimal128types in Arrow records - Schema correctly shows
arrow.DECIMAL128with precision/scale metadata - Direct conversion to
float64available viadecimal128Value.ToFloat64(scale)
For Direct Arrow Usage
Users working with Arrow records directly now get:
// Schema reflects true types
field.Type.ID() == arrow.DECIMAL128 // ✅ Instead of arrow.STRING
Configuration Note
The UseArrowNativeDecimal option is currently internal and not exposed through the public API. Users cannot currently override this setting, but the new default behavior provides proper Arrow type semantics.
Testing
✅ Updated test expectations to reflect new default behavior
✅ All tests pass locally
✅ Arrow records now contain native decimal128 types instead of strings
Breaking Change
This is a minor breaking change for users directly accessing Arrow records, as decimal columns now return arrow.Decimal128Array instead of arrow.StringArray. However, this provides the correct Arrow type semantics and eliminates the need for manual parsing.