json2parquet icon indicating copy to clipboard operation
json2parquet copied to clipboard

INT 96 timestamps giving problems.

Open saurabhdaga-merkle opened this issue 4 years ago • 0 comments

in JSON, my timestamp field looms like block_timestamp":1333413850

after conversion to pq using pa.timestamp('s', tz='UTC') and use_deprecated_int96_timestamps=True my timestamp starts looking like

block_timestamp = zg+IUgAAAACMPSUA(FYI, I used parquet-tools to check the contents)

also, I get weird dates when I import it into Athena.

What would be the best way to model this? FYI, I am using 0.028 version

saurabhdaga-merkle avatar May 25 '20 04:05 saurabhdaga-merkle