alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Support the startSync feature in OSS

Open JiamingMai opened this issue 1 year ago • 1 comments

What changes are proposed in this pull request?

Support the startSync feature in OSS

Why are the changes needed?

By now, startSync is not supported in OSS and it throws an exception "Active Syncing is not supported on this UFS type: oss" when we execute the startSync command. The startSync feature is useful for user whose environment is Alluxio + OSS.

Does this PR introduce any user facing changes?

No, it doesn't.

Here is an example. With this startSync feature, we can use the startSync command even though the UFS is OSS. image After uploading a file test3.txt to the specified OSS, we can see clearly that the metadata is updated in the Alluxio WebUI. image

image

JiamingMai avatar Aug 18 '22 05:08 JiamingMai

is there a specific version of OSS that will work with activesync, because only hdfs >2.7 had support for this..

In the implement of HDFS, it is based on iNotify. There isn't a similar way for OSS to push event to Alluxio. But in this PR, it records the last modified time of the files under the sync point, and then check and compare the last modified time of the OSS files intervally. Once it is found that the last modified time of the file is updated, the metadata of the file is going to be updated.

JiamingMai avatar Aug 23 '22 08:08 JiamingMai