Containers unreachable after upgrade from 3.0 to 4.0

[alsemo]

Describe the bug

Hi,

I newly upgraded my Photon 3.0 to 4.0 today. Upgrade was successful and it can boot properly. I can access to the Photon OS via SSH, so I can confirm that the OS and it's IP Address is ok. It's that all the containers are now unreachable on the network. I can see the containers up and running with the IP assigned, but I suspect the containers are somehow not connected to the network.

I connect to one of the container's shell and tried pinging to other host but it's unreachable. Then I check networkctl and the result as per below. Thanks in advance.

[ ~ ]$ networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 eth0 ether routable configured 3 br-5f993280fc82 bridge no-carrier unmanaged 4 br-61d7fa096103 bridge no-carrier unmanaged 5 br-8cf937f16283 bridge no-carrier unmanaged 6 br-e5ca7561b3a6 bridge no-carrier unmanaged 7 br-07cc005c9835 bridge no-carrier unmanaged 8 br-5241b1e0f073 bridge no-carrier unmanaged 9 br-ee224c333cc6 bridge no-carrier unmanaged 10 br-b282f9dbe180 bridge no-carrier unmanaged 11 docker0 bridge routable unmanaged 15 vethfaa779a ether degraded unmanaged 17 vethbb25be9 ether degraded unmanaged 19 veth1db171a ether degraded unmanaged 21 veth09a9a42 ether degraded unmanaged

Reproduction steps

1. Upgrade Photon OS from 3.0 to 4.0
2. Restart the OS after upgrade
3. Check running container for network connection ...

Expected behavior

All containers are reachable on the network after upgrade

Additional context

No response

[dcasota]

Hi @alsemo, Unfortunately the actual upgrade docs do not contain content for typical application deliverables such as interconnected containers, overlay network, nfs-shares, etc. and what to consider for upgrade. Luckily there were similar issues solved in the past. Have you checked e.g. https://github.com/vmware/photon/issues/1326 ? If it doesn’t help, please share the logfiles.

[alsemo]

Hi @dcasota ,

Thanks for your pointers. I'm following the troubleshooting found in #1321. So far, this is what I've been up to:

1. Copied and edited the /usr/lib/systemd/network/99-default.link and added " TransmitChecksumOffload=false ".
2. stop iptables via systemctl stop iptables ( For testing purpose ).

So far it's still no go. Taking example using my minecraft server (paperMC), I can ping to the Photon OS machine from it but not to other machines in the same network.

/papermc # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:59 errors:0 dropped:0 overruns:0 frame:0
          TX packets:984 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3466 (3.3 KiB)  TX bytes:92420 (90.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:504 (504.0 B)  TX bytes:504 (504.0 B)

Photon OS Host IP: 192.168.0.6

/papermc # ping 192.168.0.6
PING 192.168.0.6 (192.168.0.6): 56 data bytes
64 bytes from 192.168.0.6: seq=0 ttl=64 time=0.217 ms
64 bytes from 192.168.0.6: seq=1 ttl=64 time=0.179 ms

When I tried to ping my other server, IP 192.168.0.4, it's not working.

I do not use swarm but I do have quite a few containers. But many are currently off as I'm using those simple containers such as paperMC and portainer to get this network issue resolved first before I turn on the other containers.