mon: allow using network public address range as monitor endpoint

[prazumovsky]

Is this a bug report or feature request?

• Feature Request

What should the feature do:

Add some flag, for example allowAddressRangeEndpoint, which enables ability to assign IP address from public network range if spec.network.addressRanges.public is defined and ceph public network iface is found on a node. How it should look like for user:

1. User defines allowAddressRangeEndpoint = true flag.
2. User defines spec.network.addressRanges.public section with CIDR list of assigned ifaces to cluster nodes. For example, 1.2.3.0/24 and 4.5.6.0/24 (let say this is multi-rack cluster).
3. During mon assign rook-operator looks for node ifaces with their ip addresses (for example, node has 172.10.0.0/24 network with 172.10.0.132 ip address assigned to a node, 10.10.10.0/24 with 10.10.10.130, 1.2.3.0/24 iface with 1.2.3.15 ip address).
4. After rook-operator examined node ip addresses (it could be examined with mon-X-canary pod), it assigns first entry of ip address from a node which could be found in spec.network.addressRanges.public list. In our case, mon assigns 1.2.3.15 ip address as monitor endpoint because its CIDR is included into addressRange.public list and is found as node iface endpoint.

What is use case behind this feature:

Host-based ceph with rook v1.14 finally got an ability to specify mon endpoints which are different from host network by annotation mon-ip. This ease Ceph cluster access out of the cluster and in other cases, for example, integration with OpenStack services such as Nova and Manila. While node annotation mon-ip just allows to specify static ip addresses which still have some compexness in further lifecycle management, the way which is described in this feature request enables ability to not over-manage Ceph clusters with mon-ip support in case of any lifecycle management operations. We are configuring underlying nodes with assigning all required ceph public and cluster networks to a node and then defining that monitors should place themselves on node ceph public network ip addresses.