[Question] Setting a range of peer addresses for to use unicast with docker swarm?

[Alqio]

I'm deploying a ROS2 application on two machines: one linux and one windows. I'm using docker swarm and it has lots of difficulties with multicast particularly with Windows hosts. I decided to try using unicast instead, but I can't find a guide how one should set up the communication between containers in this case.

To further complicate the problem, on the windows host, I also possibly need to have a ROS node outside of the docker network that can still communicate with other nodes. However, this is lesser priority so I haven't focused on that yet.

How should I try to achieve this? I tried something like this:

<CycloneDDS>
    <Domain id='any'>
        <General>
            <AllowMulticast>false</AllowMulticast>
        </General>
        <Discovery>
            <ParticipantIndex>auto</ParticipantIndex>
            <Peers>
                <Peer address="host.docker.internal"/>
                <Peer address="172.30.128.2"/>
                <Peer address="172.30.128.1"/>
            </Peers>
        </Discovery>
    </Domain>
</CycloneDDS>

and using docker-compose

version: '2'
services:
  eeg_simulator:
    build:
      context: .
      dockerfile: ros2_ws/src/eeg_simulator/Dockerfile
    env_file:
      - ${ENV_FILE}
    networks:
      mtmsnetwork:
        ipv4_address: 172.30.128.2


  data_batcher:
    build:
      context: .
      dockerfile: ros2_ws/src/data_batcher/Dockerfile
    env_file:
      - ${ENV_FILE}
    privileged: true
    networks:
      mtmsnetwork:
        ipv4_address: 172.30.128.1

networks:
  mtmsnetwork:
    ipam:
      config:
        - subnet: 172.30.0.0/16

The nodes can find each other when they are on the same host and in theory it works. However, I seem to be unable to ping from the containers to each other when running them in separate host machines. In addition, the ip addresses change everytime the network is created again and docker swarm does not support setting a static ipv4 address. Is it possible to set a range of ip addresses for peers? Also I'm wondering is if this is the correct approach for this? Is there a way to discover other nodes without specifying ip addresses when using unicast?

[eboasson]

I have been smart enough(🤣) to use a mac and avoid the complications caused by docker and Windows. This of course mostly joking, but it is true insofar that I don't know the intricacies of docker. The rare case I use it, I just copy-paste instructions and so any advice I can give is to be taken with a grain of salt.

It is also true that I have come to see docker (and even more so on Windows: https://github.com/eclipse-cyclonedds/cyclonedds/issues/677) as one of those things that seemingly make life easier but in reality make it more complicated. But here, I have to admit it is really easy to be dismissive of things you don't really know.

With those disclaimers out of the way ...

No, you can't currently paste in a range of IP addresses. I suppose that we could add that, but the real problem with having many addresses (let alone the 64k present in a /16 network) is that you start spamming your network with gigantic amounts of discovery data. This has been discussed before and so I don't think it is useful to go into the details here.

There is a trick we have at times considered but never actually implemented, and that is to include information on the addresses of known peers in the participant discovery messages. That way, if you have one machine with a fixed IP address, all you need to do is point all others to discover that one node and you're in business. It is not really hard to do, and I'd love to say I'll whip it up for you (and most likely making other people happy as well) but that's not possible right now.

Do you really need the docker swarm on Windows as well? Because if on Windows you can manage known IP addresses, then it seems like letting the Linux-based ones use multicast discover among each other, while listing the Windows IP addresses explicitly you should be ok. The latter works because A discovering B always also results in B discovering A, regardless of whether B has listed A as a peer.

Another possibility might be to first gather the IP addresses, write them out to a file that is a valid Cyclone configuration fragment (i.e., say peers.xml containing <CycloneDDS><Domain><Discovery><Peers><Peer a=.../><Peer a=.../></Peers></Discovery></Domain></CycloneDDS>) and set CYCLONEDDS_URI=config.xml,peers.xml. It'll combine the two files in that case.