alluxio
alluxio copied to clipboard
Alluxio
Add Invalidation Cache for metadata sync
What changes are proposed in this pull request?
This adds an invalidation timestamp to the metadata sync cache. This allows a path to be marked as invalidated, resulting in it or any of its children's metadata being synced with the UFS when accessed. This can be done through the FileSystemMaster interface method called invalidateSyncPath.
Why are the changes needed?
These changes are needed for cross cluster sync.
Does this PR introduce any user facing changes?
Adds a method invalidateSyncPath to FileSystemMaster.
a high level question, does this invalidation take effect even if metadata sync is turned off. (ie. meta sync interval = -1)
a high level question, does this invalidation take effect even if metadata sync is turned off. (ie. meta sync interval = -1)
Yes. If there is an invalidation, then that takes precedence for any type of mount. This way invalidations can be made on any mount path (even those without cross cluster enabled) https://github.com/Alluxio/alluxio/pull/16144/files#diff-6478c3ba4bfc7b522fbcedf0de8d9c59b2ca00d7be9dbcf7db746552216903a1R221
For a path in a cross cluster mount path, there will multiple checks.
- If a syncInterval is set (i.e. > -1) and the time since the latest sync is greater than the interval then a sync will happen.
- If the last invalidation time is greater than the last sync time then a sync will happen.
The idea is to allow people to use both an interval and an invalidation for cross cluster mounts, or to force a sync on a cross cluster mount by setting the time to zero. It also allows cross cluster mounts to be nested in non-cross cluster mounts and the other way around by checking if a sync is needed either by time or by invalidation.