Execute iSCSI shell commands on multiple remote hosts (in addition to localhost)

[naved001]

Just documenting this before I forget.

Right now we execute all iscsi commands on the host where BMI is running. We need to update those methods in a way so that we can run in on any number of iSCSI servers (to support multi-pathing). We would also need to copy the tgt configuration files to the remote hosts.

Here's what I have in mind:

1. Read a list of iscsi servers from the config file. We should document that a bmi-user with the privilege to execute iscsi commands must be created on those machines.

[iscsi]
servers = remote-host, remote-host2.example.com, 10.10.1.1
username = bmi-user

2. Read this list, and then use the parallel-ssh library to copy the iscsi conf files and execute commands simultaneously on all hosts.

Another library to do the same is the paramiko library, but it executes the commands on one host at a time.

3. We need to have error handling and as such now that the BMI code is dealing with multiple machines. Probably log at warning level if any of the iscsi server is not reachable.

4. We could also recommend having a cron job running on the machines to check for the status of iscsi service (tgt, iet) and reporting it to admins in case of failures.