DAOS-14834 control: Enable parallel server->engine dRPCs (#14193)

[mjmac]

The idea here is to remove the bottleneck in daos_server that serializes dRPC calls, to enable daos_server to pass along multiple dRPC calls even if the first one hasn't yet returned. In the current master branch, we have a single dRPC client structure that uses RW locks to control access to its internals. dRPC calls that take a long time can potentially impede other commands.

My proposed solution is to create a new drpc.ClientConnection for each command that needs to be sent to the daos_engine. Each command is handled on its own connection. We were using a connect->send->disconnect pattern on the client connection anyway.

Required-githooks: true Change-Id: Ibe5a03d28ecc5099b5827ef22fbbace9e3d8b963 Signed-off-by: Kris Jacque [email protected]

[github-actions[bot]]

Bug-tracker data: Ticket title is 'LRZ: complete control system hang-up' Status is 'Resolved' Labels: 'LRZ,md_on_ssd,scrubbed' https://daosio.atlassian.net/browse/DAOS-14834

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14367/3/display/redirect