Improve UX for NSM examples

[denis-tingaikin]

Overview

1. It would be good if we could get to one approach to such common tasks, or one approach plus some additional layer (say ‘spiffe setup’ + ‘spiffe federation’ etc) … I’ve seen folks follow the wrong spiffe or the wrong nsm setup instructions and have use cases like interdomain simply fail

2. Anything NSC that uses kernel interfaces should be using the webhook… we currently have a lot of cases that are just running the NSC bare… and that sometimes confuses folks who expected a simple annotation entry point.

Tasks

1. Reduce the count of repeatable setups.
2. Re-check examples that are not using webhook.

Link to the Google Docs document

References

[denis-tingaikin]

@d-uzlov

Anything NSC that uses kernel interfaces should be using the webhook… we currently have a lot of cases that are just running the NSC bare… and that sometimes confuses folks who expected a simple annotation entry point.

Example of what is expected: https://github.com/networkservicemesh/deployments-k8s/blob/main/examples/use-cases/Kernel2Kernel/client.yaml

Example of what is not expected: https://github.com/networkservicemesh/deployments-k8s/blob/main/examples/use-cases/SmartVF2SmartVF/patch-nsc.yaml

[d-uzlov]

My thoughts on simplifying NSM setup:

What we want

Ideally we would want 4 setups, same as Spire: 1 for single-cluster examples, 3 for each cluster in multi-cluster examples. This is the minimum because in single-cluster setup we need to use the default kubectl config file, and in multi-cluster setup we need to specify kubectl config manually, a different one for each cluster.

What we currently have

basic:                  vpp + reg-k8s
floating_interdomain:   vpp +           reg-mem + reg-proxy (+ reg-k8s in floating domain)
interdomain:            vpp + reg-k8s +           reg-proxy
k8s_monolith:           vpp + reg-k8s
memory:                 vpp +           reg-mem
miltiforwarder:         vpp + reg-k8s +           sriov
ovs:                          reg-k8s +           ovs
remotevlan-vpp:         vpp + reg-k8s +                 nse-remote-vlan
remotevlan-ovs:               reg-k8s +           ovs + nse-remote-vlan
sriov:                  vpp + reg-k8s

The "Basic setup" is almost common: nsmgr, forwarder-vpp, registry-k8s.

However, other app are not so common across setups, and some setups are quite unique.

Issues that I see

• forwarder-sriov and forwarder-ovs will only work in clusters with special hardware and special k8s config.
• Some setups don't use the default forwarder-vpp settings: sriov, remotevlan-vpp. They are incompatible with the basic setup, and with each other. They will also only work in a cluster with special config.

What we can do

1. We can still unify the basic setup with k8s_monolith setup.
2. We can change floating_interdomain setup to use registry-k8s instead of current registry-memory, and then we can unify multi-cluster setups.
3. Optionally we can move all of the possible setups into examples/nsm folder, to have a complete list of them, and then just reference them in scenario setups, just like we currently reference Spire in the "Requires" section.

The result would be similar to what we have in the [currently still open] Spire setup unification pull request: https://github.com/networkservicemesh/deployments-k8s/pull/7733

Except in Spire we don't have forwarders but here we will still have miltiforwarder, ovs, remotevlan, sriov as separate setups. I don't think we can do anything useful with those setups because they use unique configs (incompatible with each other). And memory requires absence of registry-k8s, so it also needs its own setup.

What else we can do

I had an idea that NSM setup doesn't have to be all-in-one.

Technically we can split NSM setup:

• "nsmgr+webhook" setup, which will be common for everything.
• Several registry setups
• Several forwarder setups

Then we could deploy NSM as Lego, selecting different parts in different scenarios.

However, I came to the conclusion that this wouldn't really simplify anything. We would have even more setups then we have now, even though they will be small ones. It would likely be more cumbersome so deploy than a single setup, and it would be harder to maintain. It would also probably be easier to mess things up when switching between setups.

So I don't think this is a viable option.

Conclusion

We can remove separate setup for k8s_monolith. We can use unified multi-cluster setup for both interdomain and floating_interdomain scenarios (but we should change them to use the same registry app everywhere). We can't really do much for other setups.

Maybe we can just update the comments in the README files to make it more obvious which things are supported by the current setup and which aren't.

[denis-tingaikin]

Seems like done.

[denis-tingaikin]

@d-uzlov Feel free to open new ticket with https://github.com/networkservicemesh/deployments-k8s/issues/7673#issuecomment-1314291091