u-bmc

u-bmc copied to clipboard

Investigate why this doesn't work:

bluecmd@provision-rojter:~$ ping -6 -Iens256 ff02::1
ping6: Warning: source address might be selected on device other than ens256.
PING ff02::1(ff02::1) from :: ens256: 56 data bytes
64 bytes from fe80::20c:29ff:feca:cf97%ens256: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from fe80::56ab:3aff:fe05:fc0a%ens256: icmp_seq=1 ttl=64 time=0.302 ms (DUP!)
64 bytes from fe80::268a:7ff:fe90:70ac%ens256: icmp_seq=1 ttl=64 time=0.317 ms (DUP!)
64 bytes from fe80::20c:29ff:fe6f:4f00%ens256: icmp_seq=1 ttl=64 time=0.454 ms (DUP!)
64 bytes from fe80::ce37:abff:fe6f:5648%ens256: icmp_seq=1 ttl=64 time=0.545 ms (DUP!)
64 bytes from fe80::268a:7ff:fe90:7066%ens256: icmp_seq=1 ttl=64 time=0.594 ms (DUP!)
64 bytes from fe80::268a:7ff:fe90:7065%ens256: icmp_seq=1 ttl=255 time=110 ms (DUP!)
64 bytes from fe80::20c:29ff:feca:cf97%ens256: icmp_seq=2 ttl=64 time=0.081 ms
64 bytes from fe80::56ab:3aff:fe05:fc0a%ens256: icmp_seq=2 ttl=64 time=0.330 ms (DUP!)
64 bytes from fe80::268a:7ff:fe90:70ac%ens256: icmp_seq=2 ttl=64 time=0.347 ms (DUP!)
64 bytes from fe80::20c:29ff:fe6f:4f00%ens256: icmp_seq=2 ttl=64 time=0.481 ms (DUP!)
64 bytes from fe80::ce37:abff:fe6f:5648%ens256: icmp_seq=2 ttl=64 time=0.558 ms (DUP!)
64 bytes from fe80::268a:7ff:fe90:7066%ens256: icmp_seq=2 ttl=64 time=0.682 ms (DUP!)
64 bytes from fe80::268a:7ff:fe90:7065%ens256: icmp_seq=2 ttl=255 time=96.4 ms (DUP!)
^C
--- ff02::1 ping statistics ---
2 packets transmitted, 2 received, +12 duplicates, 0% packet loss, time 3ms
rtt min/avg/max/mdev = 0.081/15.065/109.712/36.008 ms
bluecmd@provision-rojter:~$ ping -6 -Iens256 ff02::1 | grep 1260
ping6: Warning: source address might be selected on device other than ens256.
bluecmd@provision-rojter:~$ ping6 -I ens256 fe80::8cc8:e7ff:fe92:1260
ping6: Warning: source address might be selected on device other than ens256.
PING fe80::8cc8:e7ff:fe92:1260(fe80::8cc8:e7ff:fe92:1260) from :: ens256: 56 data bytes
64 bytes from fe80::8cc8:e7ff:fe92:1260%ens256: icmp_seq=1 ttl=64 time=0.748 ms
^C
--- fe80::8cc8:e7ff:fe92:1260 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.748/0.748/0.748/0.000 ms

Messing around with NCSI code and adding logging, it appreas the EGMF is never set up until an IP address is added. After that happens pinging ff02::1 works.

<6>[   15.517051] ncsi_dev_state_config_xx: 772
<6>[   15.517088] ncsi_dev_state_config_xx: 773
<6>[   15.517110] ncsi_dev_state_config_xx: 774
<6>[   15.517457] ncsi_dev_state_config_xx: 775
<6>[   15.517834] ncsi_dev_state_config_xx: 776
<6>[   15.518184] ncsi_dev_state_config_xx: 778
<6>[   15.518494] ncsi_dev_state_config_xx: 779
<6>[   15.518830] ncsi_dev_state_config_xx: 781
# I add an ip to eth0
<6>[  701.390943] ncsi_inet6addr_event EGMF mode
<6>[  701.390963] ncsi_inet6addr_event called

Qemu adds an fec0 address automatically somehow, so that causes the EGMF to be called almost straight away.

However, testing on prozz RAs were accepted so this might be a local thing as well.

While RA works ff02::1 does not seem to work, so still something odd with multicast

There are some NC-SI stuff in the second to last firmware release, and the cards on these particular host had a pretty old firmware. Tried upgrading the CX3 MCX342A card to 2.42.5000 and that seems to have helped. The ff02::1 ping does not appear to work however.

Doing a full power cycle seems to have brought back the state where pinging over Ipv6 does not work - on both sites. So there needs to be more debugging here.

Debugging this some more, it looks like solicited multicast does not work:

$ sudo tcpdump -i ens224 'ether host 33:33:ff:41:03:f0'
listening on ens224, link-type EN10MB (Ethernet), capture size 262144 bytes
22:35:30.746471 IP6 fc00::1 > ff02::1:ff41:3f0: ICMP6, neighbor solicitation, who has fc00::7efe:90ff:fe41:3f0, length 32

/# ./dumpcap -filter "ether host 33:33:ff:41:03:f0"
[nothing]

While running ping6 ff02::1 this is the outgoing traffic:

$ sudo tcpdump -i ens224 'ether host 33:33:00:00:00:01'
listening on ens224, link-type EN10MB (Ethernet), capture size 262144 bytes
22:36:43.982567 IP6 fe80::250:56ff:feb7:8c60 > ip6-allnodes: ICMP6, echo request, seq 2050, length 64
22:36:44.420983 IP6 fe80::250:56ff:feb7:8c60 > ip6-allnodes: ICMP6, router advertisement, length 56
22:36:44.982545 IP6 fe80::250:56ff:feb7:8c60 > ip6-allnodes: ICMP6, echo request, seq 2051, length 64

I lost the logs, but the incoming traffic was only the RAs I seem to remember.

I'm trying without the multicast filtering enabled now.

Yep, that works.

/# ./dumpcap -filter 'ether host 33:33:ff:41:03:f0'
PACKET: 90 bytes, wire length 90 cap length 90 @ 1970-01-01 00:03:09.407245 +0000 UTC
- Layer 1 (14 bytes) = Ethernet	{Contents=[..14..] Payload=[..76..] SrcMAC=00:50:56:b7:8c:60 DstMAC=33:33:ff:41:03:f0 EthernetType=IPv6 Length=0}
- Layer 2 (40 bytes) = IPv6	{Contents=[..40..] Payload=[..32..] Version=6 TrafficClass=0 FlowLabel=0 Length=32 NextHeader=ICMPv6 HopLimit=255 SrcIP=fc00::1 DstIP=ff02::1:ff41:3f0 HopByHop=nil}
- Layer 3 (04 bytes) = ICMPv6	{Contents=[135, 0, 135, 211] Payload=[..28..] TypeCode=NeighborSolicitation Checksum=34771 TypeBytes=[]}
- Layer 4 (28 bytes) = ICMPv6NeighborSolicitation	{Contents=[..28..] Payload=[] TargetAddress=fc00::7efe:90ff:fe41:3f0 Options=[ICMPv6Option(SourceAddress:00:50:56:b7:8c:60)]}

Gargh. That means that multicast filtering is broken either in the kernel or the network card itself. Having looked a small amount on the code my suspicion is the ladder, but the former is of course what we can change.

That said, I'll create a patch to disable GMF right now.

From the standard:

IPv6 Neighbor Solicitation messages are not covered by the currently defined multicast filters. When
multicast, Neighbor Solicitation messages are sent to a Solicited Node multicast address that is derived
from the target node’s IPv6 address. To enable forwarding of Solicited Node multicasts when global
multicast filtering is active, the Management Controller would configure a multicast or mixed MAC address
filter for the specific Solicited Node multicast address required, using the Set MAC Address command. 

and SMA:

MAC address filters may be configured as unicast or multicast addresses, depending on the capability of
the channel. The channel may implement three distinct types of filter:
 Unicast filters support exact matching on 48-bit unicast MAC addresses.
 Multicast filters support exact matching on 48-bit multicast MAC addresses.
 Mixed filters support exact matching on both unicast and multicast MAC addresses. 

The code in Linux, and the only one I see for SMA:

/* Use first entry in unicast filter table. Note that
 * the MAC filter table starts from entry 1 instead of
 * 0.
 */
nca.type = NCSI_PKT_CMD_SMA;
for (index = 0; index < 6; index++)
    nca.bytes[index] = dev->dev_addr[index];
nca.bytes[6] = 0x1;
nca.bytes[7] = 0x1;

So most likely the 33:33:00:00:00:01 and the solicited IPv6 multicast group needs to be explicitly enabled in the filter as well.

Re-opening to track the long term resolution of this.