feast-hive icon indicating copy to clipboard operation
feast-hive copied to clipboard

Materialization crashes on `TIMESTAMP`-typed fields

Open Felix-neko opened this issue 3 years ago • 0 comments

Hi again! I've faced yet another problem = (

I've tried to use a Hive table where one of the field had TIMESTAMP type. When I tried to run feast materialize incremental ... it crashed with the following stacktrace.

Traceback (most recent call last):
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/cli.py", line 469, in materialize_incremental_command
    end_date=utils.make_tzaware(datetime.fromisoformat(end_ts)),
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/usage.py", line 280, in wrapper
    raise exc.with_traceback(traceback)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/feature_store.py", line 884, in materialize_incremental
    tqdm_builder=tqdm_builder,
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/infra/passthrough_provider.py", line 151, in materialize_single_feature_view
    rows_to_write = _convert_arrow_to_proto(batch, feature_view, join_keys)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/infra/provider.py", line 317, in _convert_arrow_to_proto
    for column, dtype in columns
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/infra/provider.py", line 317, in <dictcomp>
    for column, dtype in columns
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/type_map.py", line 332, in python_values_to_proto_values
    return _python_value_to_proto_value(value_type, values)
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/type_map.py", line 293, in _python_value_to_proto_value
    return [ProtoValue(int64_val=int(value)) for value in values]
  File "/home/felix/.cache/pypoetry/virtualenvs/feast-hive-tests-NQ4GOj-a-py3.7/lib/python3.7/site-packages/feast/type_map.py", line 293, in <listcomp>
    return [ProtoValue(int64_val=int(value)) for value in values]
TypeError: int() argument must be a string, a bytes-like object or a number, not 'datetime.datetime'

My test repo and base population script to reproduce the error could be found here: https://github.com/Felix-neko/feast_sandbox/tree/timestamp_field_checkup/repos/timestamp_checkup_repo

Felix-neko avatar Feb 02 '22 12:02 Felix-neko