How can I trace packet after forwarding with XDP?

[exec-to]

I have one network interface in my system with many VLAN. And I use static routes between them.

    enp3s0f1:
      addresses: [192.168.170.6/24]
      routes:
        - to: 1.1.1.0/24
          via: 192.168.170.6

  vlans:
    vlan302:
      id: 302
      link: enp3s0f1
      addresses: [ "10.0.30.2/32" ]
      routes:
      - to: 2.2.2.0/24
        via: 10.0.30.2

    vlan304:
      id: 304
      link: enp3s0f1
      addresses: [ "10.0.30.4/32" ]
      routes:
      - to: 3.3.3.0/24
        via: 10.0.30.4

When I use netsniff-ng (tcpdump) on interface enp3s0f1, I see incoming and outgoing packets with VLAN tags corresponding to the incoming and outgoing virtual interface.

But, when I use XDP program on inteface enp3s0f1, I see only one packet with VLAN tag corresponding to the incoming interface (before forwarding). How can I trace packet in my XDP-program after forwarding by Linux network stack?

For testing I plugged in a second network interface and configured it for outgoing traffic. In this case, I run a separate XDP-program on each network interface and process the incoming and outgoing packets. It doesn't work with one interface.

[chaudron]

You could use xdp-dump to capture the return code and modifications your program made before sending it to the kernel.

Use something like xdpdump -i <int> --rx-capture entry,exit -w capture.pcap and then open the capture in the latest Wireshark to see the result codes and compare the before/after packet.

Also, did you disable VLAN offloading on the NIC? See the link in the tutorial: https://github.com/xdp-project/xdp-tutorial/tree/master/packet01-parsing#a-note-about-vlan-offloads

[exec-to]

You could use xdp-dump to capture the return code and modifications your program made before sending it to the kernel.

Use something like xdpdump -i <int> --rx-capture entry,exit -w capture.pcap and then open the capture in the latest Wireshark to see the result codes and compare the before/after packet.

I can't start xdpdump with this parameters when XDP program is loaded. I got error message:

# ./xdpdump -i enp3s0f1 --rx-capture entry, exit -w capture.pcap
ERROR: Can't find function '' on interface!

List of my interfaces:

# ./xdpdump -D 
Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
enp6s0                 <No XDP program loaded!>
enp7s0                 <No XDP program loaded!>
enp3s0f0               <No XDP program loaded!>
enp3s0f1                                 native   41   5c947f981bb4dd1e 
vlan302                <No XDP program loaded!>
vlan304                 <No XDP program loaded!>

When I start xdpdump without loaded XDP program on interface enp3s0f1 then I get capture.pcap. I inspect packets with Wireshark. This file contains packets before forwarding and after forwarding. For example TCP packets (Ack num is repeated two times):

717	8.077333	IP_SRC	IP_DST	TCP	64	54135 → 443 [ACK] Seq=1208 Ack=3327 Win=65536 Len=0  54135
718	8.077333	IP_SRC	IP_DST	TCP	64	54135 → 443 [ACK] Seq=1208 Ack=8783 Win=65536 Len=0  54135
719	8.077346	IP_SRC	IP_DST	TCP	54	54135 → 443 [ACK] Seq=1208 Ack=3327 Win=65536 Len=0  54135
720	8.077351	IP_SRC	IP_DST	TCP	54	54135 → 443 [ACK] Seq=1208 Ack=8783 Win=65536 Len=0  54135
734	8.158893	IP_SRC	IP_DST	TCP	64	54135 → 443 [ACK] Seq=1208 Ack=11703 Win=65536 Len=0  54135
735	8.158893	IP_SRC	IP_DST	TCP	64	54135 → 443 [ACK] Seq=1208 Ack=16998 Win=65536 Len=0  54135
736	8.158905	IP_SRC	IP_DST	TCP	54	54135 → 443 [ACK] Seq=1208 Ack=11703 Win=65536 Len=0  54135
737	8.158910	IP_SRC	IP_DST	TCP	54	54135 → 443 [ACK] Seq=1208 Ack=16998 Win=65536 Len=0  54135

Ethernet header before forwarding:

Ethernet II, Src: Cisco_c9:01:c2 (00:26:51:c9:01:c2), Dst: HewlettP_35:91:95 (38:ea:a7:35:91:95)

Ethernet header after forwarding:

Ethernet II, Src: HewlettP_35:91:95 (38:ea:a7:35:91:95), Dst: VMware_08:ab:bb (00:50:56:08:ab:bb)

In my case, when i ran XDP-program on interface enp3s0f1, i got only packet with header:

Ethernet II, Src: Cisco_c9:01:c2 (00:26:51:c9:01:c2), Dst: HewlettP_35:91:95 (38:ea:a7:35:91:95)

Link to my XDP program

Also, did you disable VLAN offloading on the NIC? See the link in the tutorial: https://github.com/xdp-project/xdp-tutorial/tree/master/packet01-parsing#a-note-about-vlan-offloads

Yes, I did disable hardware offloading on the NIC. I use this modes:

    $ETHTOOL -C enp3s0f1 adaptive-rx off
    $ETHTOOL -G enp3s0f1 rx 4096
    $ETHTOOL -G enp3s0f1 tx 4096
    $ETHTOOL --offload enp3s0f1 rx off tx off
    $ETHTOOL -K enp3s0f1 tso off gso off gro off lro off
    $ETHTOOL -K enp3s0f1 rxvlan off txvlan off
    $ETHTOOL --features enp3s0f1 ntuple off

[chaudron]

If there is no XDP program loaded, xdpdump falls back to libpcap mode (you should see a warning informing you of that).

Looks like your kernel and/or XDP program does not have any BPF symbols loaded. xdpdump needs these to work.

Also, it's not sure what you want to see? XDP only captures ingress packets before they hit the Linux stack. You can not capture packets going out of the Linux stack (i.e. egress). If you want to see the packet being sent to the VM you need to apply another XDP program to the ingress side of the veth to the VM (assuming you use a veth pair).

Or do you see both packets on the ingress side of enp3s0f1? Do you have promiscuous mode enabled before you load the XDP program? If so, is it still enabled after loading?

[exec-to]

If there is no XDP program loaded, xdpdump falls back to libpcap mode (you should see a warning informing you of that).

Looks like your kernel and/or XDP program does not have any BPF symbols loaded. xdpdump needs these to work.

Also, it's not sure what you want to see? XDP only captures ingress packets before they hit the Linux stack. You can not capture packets going out of the Linux stack (i.e. egress). If you want to see the packet being sent to the VM you need to apply another XDP program to the ingress side of the veth to the VM (assuming you use a veth pair).

Or do you see both packets on the ingress side of enp3s0f1? Do you have promiscuous mode enabled before you load the XDP program? If so, is it still enabled after loading?

Thanks for the answer. Now I realized that I can only see incoming packets before they hit the network stack. I was misled by the fact that I saw incoming and outgoing packets in tсpdump, as well as in the case of using the second network interface. I saw on the second network interface only incoming packets that were forwarded from the first network interface. Enabling promiscuous mode also does not help in this case. I think this issue can be closed.

[tohojo]

Eelco Chaudron @.***> writes:

Looks like your kernel and/or XDP program does not have any BPF symbols loaded. xdpdump needs these to work.

Could we maybe improve the error message in this case? :)

[exec-to]

Eelco Chaudron @.***> writes: Looks like your kernel and/or XDP program does not have any BPF symbols loaded. xdpdump needs these to work. Could we maybe improve the error message in this case? :)

The message becomes clearer when the verbose option is enabled.

# ./xdpdump -i enp3s0f1 --rx-capture entry,exit -w capture.pcap -vv 
Current rlimit 67108864 already >= minimum 1048576
libxdp: No BTF for prog ID 11
libxdp: No BTF found for program
libxdp: Found legacy program with id 11 and 0 component progs
No BTF found for program
ERROR: Can't find function '' on interface!

[chaudron]

I've opened an issue for xdp-tools, will fix it when I get some time:

https://github.com/xdp-project/xdp-tools/issues/112