NSM VL3 over VXLAN interfaces between forwarders

[BenAgai]

I have a question regarding NSM’s VL3 support that I did not find an answer for and I would appreciate your help.

From the vL3 single cluster example I saw the below diagram:

After running the example in my cluster I found out that the tunnels, created between forwarders, are of Wireguard type.

I wanted to run the above example, but to keep the interfaces between the forwarders with VXLAN type. For that I saw that the payload type of the NSE should be ETHERNET and not IP.

I cloned the NSE repo used for the example, located in the link below, and adjusted its code so that the payload type could be provided through configuration: https://github.com/networkservicemesh/cmd-nse-vl3-vpp

I later created an image for that NSE and adjusted its yaml file to use my image. The yaml file of the NSE was taken from the below link: https://github.com/networkservicemesh/deployments-k8s/blob/main/apps/nse-vl3-vpp/nse.yaml

After applying the VL3 example with the above changes, I got the following result: -) I could ping from an NSC to the NSE it is connected to - looking at the diagram I could ping from nsc1 to vl3 NSE1 and from nsc2 to vl3 NSE2. -) I couldn’t ping from NSC to the NSE connected to the other NSC - looking at the diagram I couldn’t ping from nsc1 to vl3 NSE2 -) I couldn’t ping from NSC to the other NSC - looking at the diagram I couldn’t ping from nsc1 to nsc2 and vice versa

I thought that the interface type used for tunneling shouldn’t affect the functionality of VL3 (though there is the xconnect code that differentiate between layer 2 and layer 3 which might explain what I experience).

Is it possible to use NSM’s VL3 capability while using VXLAN interfaces between the forwarders instead of Wireguard interfaces? I would like to know what I’m missing to better understand the flow causing this situation.

Thanks!

[denis-tingaikin]

Hello @BenAgai !

Payload Ethernet is indeed needed for VXLAN remote mechanism.

But note: vl3 networks mean L3 level of OSI. So we need an IP payload.

At this moment we have only WIREGUARD mechanism that can be used for IP payloads.

We're planning to add more remote mechanisms soon such as SRV6, IPSec and so on.

Does it cover your question?

[edwarnicke]

@BenAgai I'll look more closely at your issue here in a bit (Kubecon stuff going on)... but I'm curious why you desire VXLAN as the remote mechanism instead of Wireguard. :)

[BenAgai]

Hi @denis-tingaikin and @edwarnicke Thank you for your fast response!

@denis-tingaikin, As both WIREGUARD and VXLAN can be used for tunneling, I though that VXLAN could also be used for the IP payload configuration concept-wise.

As it didn’t work for me I wanted to ask if there are any functionalities required from the tunneling protocol for it to be used in the IP payload type that VXLAN does not have?

In addition, Denis, could you link me to any source material regarding the implementation of XConnect in layer 2 and 3? (I examined the code in the sdk-vpp but it is not so indicative of what is going on there)

@edwarnicke, If you could take a look that would be great! In a project we are working on we want to use the VL3 feature of NSM. Some of our network components in the system can only support VXLAN as remote mechanism and thats why we desire VXLAN instead of Wireguard.

Thank you both for your help

[edwarnicke]

@BenAgai Got it. Immovable outside components makes sense to me.

I'll explain briefly, purely to aid understanding and not to argue against your need, why VXLAN is not used for IP payload in NSM :) Note: This reasoning is imperfect, but it can aid understanding.

'Payload' describes that thing which is guaranteed end to end. In traditional Service Mesh, the payload tends to be L7 (mostly HTTP messages). The TCP or QUIC session information is completely disposable, but the HTTP Message needs to make it to where its supposed to go.

In Network Service Mesh, a Network Service may act on either 'Ethernet' or 'IP' payloads. For vL3, clearly the only element of interest is the 'IP' part of the payload.

All of that said, it should be possible to do the hack you are doing. Let's see if we can figure out how to get that working for you.

One other question: what encaps do the devices you are working with support? I ask, because it may make more logical sense to consider whether there is benefit in expanding the mechanisms.

[BenAgai]

Hi @edwarnicke

Thanks for the explanation above, it is indeed clear to me :) As the devices we are working with support only VXLAN, I would be glad to find a way to make the VL3 support work with it as its remote mechanism.

Of course, if you have any suggestions/instructions on how to make it work, I can concurrently take it on my side as well and apply the changes to make it work with VXLAN.

Thank you