rootless-containers
docs: note on order of starting components
@AkihiroSuda - this is a small note for the README to comment on the order of installing components. As you know, the setup uses annotation targeted at flannel to use a host external ip for a multi-node setup. The issue arises with order of operations. If we install flannel with the control plane, that means when new nodes come up, their flannel pods will be created (along with the control plane) to use the "host" discovered IP, which is the usernetes 10.x one. If these addresses that are in the private space can be routed between nodes (possible in some clouds) this is not an issue. It becomes an issue in an HPC or similar environment where the private 10.x address goes to a router and is not known, and the packets are dropped. We ran into this issue on our HPC system, and I realized it was because of the order of operations - we should make sync-external-ip first (adding the annotation) and then make install-flannel to use it. This would only be a bug for specific, multi-node environments. In summary, the current instructions describe:
bring up control plane
install flannel
bring up workers
add annotation and patches
And the order should be:
bring up control plane
bring up workers
add annotation and patches
install flannel
@AkihiroSuda I've moved it to advanced topic. I also fixed a detail that, as stated, was incomplete. Higher ports are not required for multi-tenancy. The reason we need them for some systems is if the system does not allow the lower port range. If the ports are allowed, the different nodes have no issue using the same ports.
And I think it would be unlikely for multiple users to be using the same physical node with Usernetes.
Update: I added it back, but put experimental. I suppose it could be done, but it's unlikely, and the port customization does support that.