Datetime-like values errors

[grkhr]

Got some problems with datetime-like values. Tried with pandas 1.0.3 and 0.25.3, both don't working. fastavro 0.23.4.

date

Traceback (most recent call last):

  File "<ipython-input-180-724a28b4d15a>", line 1, in <module>
    pdx.to_avro('test.avro', df.drop(columns=['event_timestamp']))

  File "/opt/anaconda3/lib/python3.7/site-packages/pandavro/__init__.py", line 151, in to_avro
    records=df.to_dict('records'), codec=codec)

  File "fastavro/_write.pyx", line 628, in fastavro._write.writer

  File "fastavro/_write.pyx", line 581, in fastavro._write.Writer.write

  File "fastavro/_write.pyx", line 335, in fastavro._write.write_data

  File "fastavro/_write.pyx", line 285, in fastavro._write.write_record

  File "fastavro/_write.pyx", line 333, in fastavro._write.write_data

  File "fastavro/_write.pyx", line 249, in fastavro._write.write_union

ValueError: datetime.date(2020, 6, 10) (type <class 'datetime.date'>) do not match ['null', 'string']

NaTType

Traceback (most recent call last):

  File "<ipython-input-182-991911d54074>", line 1, in <module>
    pdx.to_avro('test.avro', df.drop(columns=['event_date','items']))

  File "/opt/anaconda3/lib/python3.7/site-packages/pandavro/__init__.py", line 151, in to_avro
    records=df.to_dict('records'), codec=codec)

  File "fastavro/_write.pyx", line 628, in fastavro._write.writer

  File "fastavro/_write.pyx", line 581, in fastavro._write.Writer.write

  File "fastavro/_write.pyx", line 335, in fastavro._write.write_data

  File "fastavro/_write.pyx", line 285, in fastavro._write.write_record

  File "fastavro/_write.pyx", line 333, in fastavro._write.write_data

  File "fastavro/_write.pyx", line 234, in fastavro._write.write_union

  File "fastavro/_validation.pyx", line 169, in fastavro._validation._validate

  File "fastavro/_validation.pyx", line 178, in fastavro._validation._validate

  File "fastavro/_logical_writers.pyx", line 72, in fastavro._logical_writers.prepare_timestamp_micros

  File "fastavro/_logical_writers.pyx", line 105, in fastavro._logical_writers.prepare_timestamp_micros

  File "pandas/_libs/tslibs/nattype.pyx", line 58, in pandas._libs.tslibs.nattype._make_error_func.f

ValueError: NaTType does not support timestamp

[ghost]

Have the same problem using date. It is supported according to the AVRO specification, see here. @grkhr I guess writing it into a string field will not be supported

So, the type casting here should be extended:

import datetime

NUMPY_TO_AVRO_TYPES = {
    ....

    datetime.date: {'type': 'int', 'logicalType': 'date'}
}

I just don't know where the date would be converted to int (starting at unix epoch 1 January 1970)

[Geoiv]

1. Regarding datetime.date:

pandas will represent any dataframe containing datetime.date values as having a dtype of object. As a result, the solution suggested by @hz-lschick won't behave as expected, because even if you put an entry for datetime.date into NUMPY_TO_AVRO_TYPES, it won't get looked at, because pandas will give that column's dtype as object instead of datetime.date.

pandavro largely works off of the dtypes given by pandas, so a simpler solution would be to convert your datetime.date columns into np.datetime64 columns, which you could accomplish like this:

df['column_name'] = df['column_name'].astype(np.datetime64)

2. Regarding pd.NaT:

This issue actually comes from fastavro, rather than pandavro. pandavro correctly generates the Avro schema for a column containing pd.NaT values. But when the fastavro library tries to actually write this to file, it tries to access the timestamp attribute, which pd.NaT does not have.

The easiest solution to this (other than trying to fix this in fastavro) would be to either replace all the NaT values with a placeholder np.datetime64, or to cast the column to a string, which can then be written by fastavro.