heavydb icon indicating copy to clipboard operation
heavydb copied to clipboard

`ArrowForeignStorage` does not support importing `date32[day]` type

Open dchigarev opened this issue 3 years ago • 1 comments

Attempt to import Arrow table containing a column with date32 type results into a RuntimeError thrown from ArrowForeignStorage::getOmnisciType: https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/DataMgr/ForeignStorage/ArrowForeignStorage.cpp#L817-L818

Reproducer with pydbe
import sys
sys.setdlopenflags(1 | 256)  # RTLD_LAZY+RTLD_GLOBAL
from omniscidbe import PyDbEngine
import pyarrow as pa

at = pa.Table.from_pydict({"a": [1, 2, 3, 4]}, schema=pa.schema({"a": pa.date32()}))
server = PyDbEngine()
server.importArrowTable("test_table", at) # RuntimeError: date32[day] is not yet supported

OmniSci seems to have a type with the similar name, so possibly the required change is just to add a case for the date type to the switch where the exception is happening https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/Shared/sqltypes.h#L53 and fix this todo https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/DataMgr/ForeignStorage/ArrowForeignStorage.cpp#L924-L928

cc @Garra1980

dchigarev avatar Sep 10 '21 19:09 dchigarev

Looks like should be something like https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/QueryEngine/ArrowResultSetConverter.cpp#L972

Garra1980 avatar Sep 13 '21 07:09 Garra1980