multus-cni Support for dynamic NIC attachment

Support for dynamic NIC attachment

Open fabiand opened this issue 3 years ago • 13 comments

What would you like to be added: We'd like to be able to attach and detach ("hot-plug" and "hot-unplug") network interfaces from a pod at runtime. I.e. by modifying the annotations of a pod at runtime.

Why is this needed: In order to dynamically create L2 domains between a group of pods, which i.e. are running Kubernetes and need to communicate.

Nov 24 '20 08:11 fabiand

Hi @fabiand ! Dynamic attachments, a classic feature request.... I think we're nearing the point where this may be more feasible.

Some of the folks from Kaloom, who are solid contributors to the NPWG have been doing this with a fork of Multus (which they call kactus) for some time, using an architecture that we refer to as a "thick plugin" -- basically a long running process that is resident in memory, as opposed to a one-shot CNI plugin, and you use a "shim CNI plugin" to handle the CNI requests proper.

Recently there's been some more discussion about converting Multus into a thick plugin -- due to a few factors, however, one of those factors being that it's sometimes difficult to get further visibility into Multus, and having a shim CNI do "very little" and then have the thick plugin do the majority of the work, and allow us to have some further logging which is kept in a pod, and gives us further visibility into the Multus process.

...This does open the door to functionality like this, as well.

Fabian, I'll try to follow up and plug you into some design discussions, as well.

Nov 24 '20 13:11 dougbtv

Yo Doug.

Some of the folks from Kaloom, who are solid contributors to the NPWG have been doing this with a fork of Multus (which they call kactus) for some time, using an architecture that we refer to as a "thick plugin" -- basically a long running process that is resident in memory, as opposed to a one-shot CNI plugin, and you use a "shim CNI plugin" to handle the CNI requests proper.

That's awesome. We were actually discussing such an approach on our side. Great to see that somebody came up with pretty much the same idea and did it!

Fabian, I'll try to follow up and plug you into some design discussions, as well.

Thanks - I'll see that I'll loop in some people from my side . Much appreciated. /cc @phoracek @AlonaKaplan

I think we're nearing the point where this may be more feasible.

Yes. I think so as well. By now we have enough experience to see how we can do this external to the core, which remains - to me - a requirement.

Nov 24 '20 13:11 fabiand

@dougbtv I'd be interested in joining this discussion also. Keep me posted! Thanks!

Nov 25 '20 10:11 martinkennelly

Consider me in the loop. This looks promising. Having a "thick" plugin would really make the hot-plug effort way easier (and shared between VMs and Pods in our case). And I can only second the logging issue.

Nov 25 '20 11:11 phoracek

@dougbtv Please keep me in the loop as well. One of the interesting and more challenging use cases for hot-plug of network attachments are SR-IOV VFs or VirtIO devices that today require a the help of a device plugin at pod creation. Can these be included in the analysis?