RPC servers aren't scalable

[ChristopherHogan]

Adding ranks and nodes to a Hermes runs also requires increasing the rpc_num_threads configuration parameter. This number can quickly get above 20 on just a few nodes, which obviously isn't scalable. The reason we need to keep increasing it is because that number represents the maximum level of nested RPCs possible before an RPC server hangs (no threads left to make progress). The reason we have such deeply nested calls is because the metadata required for a single operation (Bucket::Destroy for example) could otentially be spread out over multiple nodes. I think the solution to eliminating nested RPC calls is to keep all metadata together on the node that generated it rather than spreading it out by hashing the name.

Try the "rpc breadcrumbs" feature of margo to get a nice graphic of how calls are being nested.