alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Reuse or release BlockWorkerClient when writing files

Open ssz1997 opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. Related issues: https://github.com/Alluxio/alluxio/issues/15828

When writing files, all BlockWorkerClients will be held until the whole file is finished. If the file is large and block is small, there will be many BlockWorkerClients being held.

Describe the solution you'd like After the block is done written, release the corresponding resources.

ssz1997 avatar Jul 15 '22 19:07 ssz1997

I think both "reuse" and "release" are feasible ideas, and I'm happy to work on a patch for this issue:

  • "Release" is relatively easy to implement. There is absolutely no need to keep a BlockWorkerClient when reading or writing data, because writeBlock is a streaming rpc, and the client can be safely dropped when the stream is established. This change is quite cheap.
  • To reuse BlockWorkerClient might be better for the long term benifit as it is really state-less functional client, and there's no reason why it can't be shared. I'm also drafting a related PR https://github.com/Alluxio/alluxio/pull/15837, which could make DefaultBlockWorkerClient thread safe. But this requires changing the implementation of BlockWorkerClientPool.

YangchenYe323 avatar Jul 20 '22 03:07 YangchenYe323