[debug] containerlab + FRR and MPLS (kernel modules?)

[barajus]

Hi, disclaimer, this is not a netlab issue per say, that's an issue related to the environment. I decided to post anyway, in case @ipspace or anyone else hit a similar issue in the past and therefore could come up with an idea or a lead.

Context: I am building a small MPLS lab, no LDP, just statically swapping labels. My end goal is to run it with clab + FRR containers. Before I start automated things with netlab (I know there is a module and some examples), I wanted first to build everything manually. The lab is made of 3 routers and 2 hosts.

• When I build the lab with Cumulus VM and KVM (no netlab), it works as expected (ICMP works between the 2 hosts, and I can see the MPLS labels with tcpdump). The only configuration required, is to enable MPLS on interfaces:

sysctl net.mpls.conf.INTERFACE.input=1
sysctl net.mpls.conf.INTERFACE.input=2

Cumulus kernel has net.mpls.platform_labels with a high value enabled by default.

• When I create this lab with containerlab, and CX containers (with ignite runtime, which is basically a micro-VM with the CL kernel, not the host's kernel), everything works as well, which is quite expected.

• My goal is use to FRR containers and clab, unfortunately, with this setup and the same FRR configs, I don't see MPLS packets between routers.

On the container's host, I need to get a recent kernel and the MPLS modules enabled

modprobe mpls_router
modprobe mpls_gso
modprobe mpls_iptunnel
sysctl -w net.mpls.platform_labels=1048575

then, in each containers, I activate MPLS via the sysctl per interfaces, like I did with CL. At this point it should see MPLS labels in tcpdump, but that's not the case.

Once the lab is up, the host has a few veth/br/docker interfaces enabled. I blindly try to enable sysctl net.mpls.conf.XX.input on each of them, but no success.

At this point, I suspect something is missing in my host's kernel, a module or a sysctl value, basically something which is enabled in CL's kernel.

FWIW, I made an extra test with namespaces, and that's works as well (cf: https://liuhangbin.netlify.app/post/mpls-on-linux).

Host kernel: Linux primary 5.15.0-101-generic

Basically, my next step is to reproduce without clab, with docker networking.

Any hint is welcome :)

[ipspace]

I had the same problem with CL containers when trying to build an MPLS-based BGP-free core. In the end, I used FRR containers. If you use mpls module with frr and enable mpls.ldp, you'll get a working configuration (run netlab initial -o to get the configuration commands into text files).

If you manage to figure it out, please post a solution somewhere. I'd love to add it to the documentation.

[barajus]

Thanks for your answer! I didn't figure it out the issue yet, however, I did another attempt yesterday with the cumulus_cvx (runtime: docker, no ignite) and surprisingly...that works!

root@r2:/# tcpdump -ni swp1 mpls
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on swp1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:33:35.323971 MPLS (label 200, exp 0, [S], ttl 62) IP 192.168.1.10 > 192.168.2.10: ICMP echo request, id 11, seq 3, length 64
08:33:35.324046 MPLS (label 300, exp 0, [S], ttl 63) IP 192.168.2.10 > 192.168.1.10: ICMP echo reply, id 11, seq 3, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
root@r2:/# uname -sr
Linux 5.15.0-101-generic
root@r2:/# exit

ubuntu@primary:~/host$ uname -a
Linux primary 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

The main difference I see here, is the FRR container being built over Alpine, and not ubuntu. I will try with an Ubuntu container. TBC

[barajus]

Okay, I rebuilt my setup and it works.. I believe things got mixed up during the troubleshooting process, it's all fine with a new instance. Thanks for the help :)