feast-hive
feast-hive copied to clipboard
Materialization crashes on `TIMESTAMP`-typed fields
Hi again! I've faced yet another problem = (
I've tried to use a Hive table where one of the field had TIMESTAMP
type. When I tried to run feast materialize incremental ...
it crashed with the following stacktrace.
Traceback (most recent call last):
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/bin/feast", line 8, in <module>
sys.exit(cli())
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/cli.py", line 469, in materialize_incremental_command
end_date=utils.make_tzaware(datetime.fromisoformat(end_ts)),
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/usage.py", line 280, in wrapper
raise exc.with_traceback(traceback)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
return func(*args, **kwargs)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/feature_store.py", line 884, in materialize_incremental
tqdm_builder=tqdm_builder,
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/infra/passthrough_provider.py", line 151, in materialize_single_feature_view
rows_to_write = _convert_arrow_to_proto(batch, feature_view, join_keys)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/infra/provider.py", line 317, in _convert_arrow_to_proto
for column, dtype in columns
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/infra/provider.py", line 317, in <dictcomp>
for column, dtype in columns
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/type_map.py", line 332, in python_values_to_proto_values
return _python_value_to_proto_value(value_type, values)
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/type_map.py", line 293, in _python_value_to_proto_value
return [ProtoValue(int64_val=int(value)) for value in values]
File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/type_map.py", line 293, in <listcomp>
return [ProtoValue(int64_val=int(value)) for value in values]
TypeError: int() argument must be a string, a bytes-like object or a number, not 'datetime.datetime'
My test repo and base population script to reproduce the error could be found here: https://github.com/Felix-neko/feast_sandbox/tree/timestamp_field_checkup/repos/timestamp_checkup_repo