Make gluster-block less of a pet

[JohnStrunk]

With our push toward GCS, we are working to remove host dependencies from the gluster containers. Today, that includes directly accessed devices and using the host's IP.

While we have a workable plan using kube services and raw block PVs to remove these dependencies for the gluster bricks and GD2, gluster-block has presented more of a challenge. Our dependency on tcmu means we have a host IP dependency for the iSCSI target.

Yesterday, I had a discussion w/ @cuppett, and he mentioned an approach that may help us get rid of our node-locked gluster-block pods...

Today, we have the iSCSI target running on a chosen few nodes, with clients connecting to them via multipath. The target nodes can then use gfapi to access the bricks. Stephen's suggestion was to move the iSCSI target to the client nodes themselves as a part of the CSI driver.

In this model, the CSI "node" portion that already runs as a DaemonSet would have the tcmu code, and it would play the role of the iSCSI target for that individual host only. Multipath would no longer be used since the client and server are on the same machine.

Open issues:

• We would still have the fencing problems when a node goes offline.
• I'm not sure how to upgrade the CSI driver w/o draining the node (same problem w/ file though)
• This requires separating GD2 and gluster-block into separate pods, potentially on separate nodes

cc: @ShyamsundarR @amarts @humblec @gluster/gluster-block @atinmu

[vbellur]

I like the overall idea. I am not sure if there are issues with having iSCSI client and server on the same node. Loopback nfs used to have deadlock problems in the kernel primarily due to a server memory allocation request being dependent on reclamation of a dirty page that the client has to flush. There have been some attempts to fix this problem but I am not very certain about the current state. It could be good to ensure that iSCSI does not have the same problem before going very far with this approach.