fluid icon indicating copy to clipboard operation
fluid copied to clipboard

[FEATURES] Add an option to skip loading metadata

Open framlog opened this issue 2 years ago • 3 comments

What feature you'd like to add:

Add an option to skip loading metadata after the dataset is initialized.

Why is this feature needed:

We may have too many small files to load, which makes loadMetadata in alluxio significantly slow and very likely to crash(due to OOM issues). So, in that case, fliud forcing the runtime to load metadata before use seems not a wise action.

Alluxio could serve requests if alluxio.user.file.metadata.load.type is ONCE

framlog avatar Feb 09 '23 08:02 framlog

What about introducing a new attribute in the alluxioRuntime such as

metadataManagementPolicy{
    sync: (Never, Once, Always, Cron)
    syncPeriod:  1h
}

cheyang avatar Feb 12 '23 08:02 cheyang

What about introducing a new attribute in the alluxioRuntime such as

metadataManagementPolicy{
    sync: (Never, Once, Always, Cron)
    syncPeriod:  1h
}

That would work in my case.

framlog avatar Feb 13 '23 01:02 framlog

@framlog This feature is now supported in #2591 . Feel free to have a try following the guide doc here

TrafalgarZZZ avatar Mar 07 '23 12:03 TrafalgarZZZ