ozone icon indicating copy to clipboard operation
ozone copied to clipboard

HDDS-9448. Add RpcClient cache for READ_VOLUME and READ_BUCKET.

Open z-bb opened this issue 2 years ago • 6 comments

What changes were proposed in this pull request?

At present, we have found that when writing a large amount of data, there will be many invalid rcp requests, such as READ_VOLUME and READ_BUCKET. Because a cache can be added to reduce these two types of calls.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9448)

z-bb avatar Oct 13 '23 02:10 z-bb

Instead of adding a cache, which can lead to inconsistent reads, we should walk over the client code to see why the additional read volume and bucket calls were added. During the effort to optimize getKeyInfo, we eliminated a lot of the redundant calls.

kerneltime avatar Oct 16 '23 16:10 kerneltime

cc @tanvipenumudy can you please take a look as well?

kerneltime avatar Oct 16 '23 16:10 kerneltime

Instead of adding a cache, which can lead to inconsistent reads, we should walk over the client code to see why the additional read volume and bucket calls were added. During the effort to optimize getKeyInfo, we eliminated a lot of the redundant calls.

Ok there are a lot of getBucket calls in BasicRootedOzoneClientAdapterImpl so is this necessary

z-bb avatar Oct 17 '23 02:10 z-bb

Ok there are a lot of getBucket calls in BasicRootedOzoneClientAdapterImpl so is this necessary

@kerneltime I think this question is for you.

adoroszlai avatar Nov 28 '23 12:11 adoroszlai

... we should walk over the client code to see why the additional read volume and bucket calls were added. ...

For the FileSystem API, there is probably no way to reduce volume and bucket calls since the FileSystem client is unaware of volume and bucket.

Cache (the current change) seems a good idea.

szetszwo avatar Nov 28 '23 20:11 szetszwo

cc @tanvipenumudy

kerneltime avatar Feb 12 '24 21:02 kerneltime