fluid
fluid copied to clipboard
[FEATURES] Add an option to skip loading metadata
What feature you'd like to add:
Add an option to skip loading metadata after the dataset is initialized.
Why is this feature needed:
We may have too many small files to load, which makes loadMetadata
in alluxio significantly slow and very likely to crash(due to OOM issues). So, in that case, fliud forcing the runtime to load metadata before use seems not a wise action.
Alluxio could serve requests if alluxio.user.file.metadata.load.type
is ONCE
What about introducing a new attribute in the alluxioRuntime such as
metadataManagementPolicy{
sync: (Never, Once, Always, Cron)
syncPeriod: 1h
}
What about introducing a new attribute in the alluxioRuntime such as
metadataManagementPolicy{ sync: (Never, Once, Always, Cron) syncPeriod: 1h }
That would work in my case.
@framlog This feature is now supported in #2591 . Feel free to have a try following the guide doc here