alluxio
alluxio copied to clipboard
Reuse or release BlockWorkerClient when writing files
Is your feature request related to a problem? Please describe. Related issues: https://github.com/Alluxio/alluxio/issues/15828
When writing files, all BlockWorkerClients will be held until the whole file is finished. If the file is large and block is small, there will be many BlockWorkerClients being held.
Describe the solution you'd like After the block is done written, release the corresponding resources.
I think both "reuse" and "release" are feasible ideas, and I'm happy to work on a patch for this issue:
- "Release" is relatively easy to implement. There is absolutely no need to keep a
BlockWorkerClient
when reading or writing data, becausewriteBlock
is a streaming rpc, and the client can be safely dropped when the stream is established. This change is quite cheap. - To reuse
BlockWorkerClient
might be better for the long term benifit as it is really state-less functional client, and there's no reason why it can't be shared. I'm also drafting a related PR https://github.com/Alluxio/alluxio/pull/15837, which could makeDefaultBlockWorkerClient
thread safe. But this requires changing the implementation ofBlockWorkerClientPool
.