Not working on databricks dbfs

[vinicius-alves]

Hello

I'm trying to use feast in my databricks projects. So, I just installed the lib and followed the quickstart guide until the apply step:

%sh cd /dbfs/feature_store/feature_repo; feast apply

But when I enter this I receive the error below:

Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/usage Traceback (most recent call last): File "/databricks/python3/bin/feast", line 8, in sys.exit(cli()) File "/databricks/python3/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(*args, **kwargs) File "/databricks/python3/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/databricks/python3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/databricks/python3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/databricks/python3/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/databricks/python3/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/databricks/python3/lib/python3.8/site-packages/feast/cli.py", line 369, in apply_total_command apply_total(repo_config, repo, skip_source_validation) File "/databricks/python3/lib/python3.8/site-packages/feast/usage.py", line 269, in wrapper return func(*args, **kwargs) File "/databricks/python3/lib/python3.8/site-packages/feast/repo_operations.py", line 193, in apply_total store.apply( File "/databricks/python3/lib/python3.8/site-packages/feast/usage.py", line 269, in wrapper return func(*args, **kwargs) File "/databricks/python3/lib/python3.8/site-packages/feast/feature_store.py", line 554, in apply self._get_provider().update_infra( File "/databricks/python3/lib/python3.8/site-packages/feast/infra/passthrough_provider.py", line 49, in update_infra self.online_store.update( File "/databricks/python3/lib/python3.8/site-packages/feast/usage.py", line 280, in wrapper raise exc.with_traceback(traceback) File "/databricks/python3/lib/python3.8/site-packages/feast/usage.py", line 269, in wrapper return func(*args, **kwargs) File "/databricks/python3/lib/python3.8/site-packages/feast/infra/online_stores/sqlite.py", line 189, in update conn.execute( sqlite3.OperationalError: disk I/O error

I think this could happen because the limitations on dbfs file storage, according to documentation dbfs does not support random writes.

Anyone has a workaround on this ?

[pyalex]

Hi @vinicius-alves , thanks for reporting this. I suspect this is related to the fs locking mechanism and there's seems to be a workaround: https://stackoverflow.com/questions/66368204/sqlite-disk-i-o-error-python-databricks

But it will require some minimal update to Feast code.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vinicius-alves]

Hello! Any workaround on this ? Thanks in advance