datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

DataFusion serialization doesn't handle `ScalarValue::Dictionary, Binary, LargeBinary, Time64, IntervalMonthDayNano, Struct`

Open alamb opened this issue 3 years ago • 1 comments

Describe the bug

While updating to the latest and greatest DataFusion pin in IOx (https://github.com/influxdata/influxdb_iox/pull/5672) I got the following error:

Error encoding expr as protobuf: Dictionary(Int32, Utf8(\"B\")) is invalid as a DataFusion scalar value"

I think the issue is that now more data type casting is done as part of logical planning (https://github.com/apache/arrow-datafusion/pull/3396) the types of some constants (ScalarValues ) have been coerced to the correct type resulting in serialized ScalarValue::Dictionary that were not there before

To Reproduce Try to serialize/deserialize ScalarValue::Dictionary

Expected behavior it works without error!

Additional context https://github.com/influxdata/influxdb_iox/pull/5672

Types that need support:

  • [ ] ScalarValue::Dictionary https://github.com/apache/arrow-datafusion/pull/3532
  • [ ] ScalarValue::Binary
  • [ ] ScalarValue::LargeBinary
  • [ ] ScalarValue::Time64
  • [ ] ScalarValue::IntervalMonthDayNano
  • [ ] ScalarValue::Struct

alamb avatar Sep 19 '22 13:09 alamb

I will fix this

alamb avatar Sep 19 '22 13:09 alamb