Question: DATADIR usage and configuration

Open Aaronzinhoo opened this issue 11 months ago • 3 comments

Checklist

[ x ] I have included information about relevant versions
[ ] I have verified that the issue persists when using the master branch of Faust. (N/A)

Expected behavior

Use of DATADIR should contain metadata related to a Table is being used. The datadir in created here

Actual behavior

DATADIR is created but the tables folder is always empty. I am just curious if this is the expected behavior for the data dir folder and if/when would it be populated with any data? Is this some older feature that is no longer needed? if its not needed, is there a way to configure it such that we dont create the datadir folder if not needed?

Versions

Python version 3.11.5
Faust version 0.11.2
Operating system Ubuntu 20.04
Kafka version 3.7.0
RocksDB version N/A

Jan 09 '25 00:01 Aaronzinhoo

Do you have an example code? IMO the tables folder is populated if you define a Table app.Table("name", key_serializer="str") and also write to this table table["John"] = 100

I do use this in one service and the rocksDB is put into this directory.

Apr 15 '25 07:04 dada-engineer

@dada-engineer some of the services I use dont utilize the a table. Some services are simple and just perform a transformation on data from topic and send the transformed into another topic. Regardless the DATADIR is made though which is strange to me since if we are not using the Table functionality.

Sep 25 '25 03:09 Aaronzinhoo

It's initialized regardless. This is expected. Do you have a hard blocker due to this?

Sep 25 '25 06:09 dada-engineer