lxc
Handle `parent=none` for physical networks
Is there an existing issue for this?
- [x] There is no existing issue for this feature
What are you currently unable to do
When creating a physical Network into the Incus cluster, it is mandatory for every Incus node to have a set up of the corresponding bridge.
incus cluster list
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-gwnode1 | https://172.31.0.161:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node1 | https://172.31.0.151:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node2 | https://172.31.0.152:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node3 | https://172.31.0.153:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
incus network create provider_302 --type=physical parent=br302 --target inc1-gwnode1
incus network create provider_302 --type=physical parent=br302 --target inc1-node1
incus network create provider_302 --type=physical parent=br302 --target inc1-node2
incus network create provider_302 --type=physical parent=br302 --target inc1-node3
incus network create provider_302 --type=physical \
ovn.ingress_mode=routed \
ipv4.ovn.ranges=100.64.1.2-100.64.1.254 \
ipv6.ovn.ranges=2001:db8::2-2001:0db8::ffff:ffff:ffff:ffff \
ipv4.routes=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 \
ipv6.routes=2001:db8:1::/48 \
ipv4.gateway=100.64.1.1/24 \
ipv6.gateway=2001:db8::1/64 \
dns.nameservers=8.8.8.8
In the above scenario, when using OVN the user would like to have a centralized network node for all the North/South traffic and not distributed as Incus by design. The reason for this requirement could be scaling where there are thousound of chassis and stretching an L2 domain over an IP Fabric becames a huge challenge. Another reason would be number of OVS's Openflows, for every Incus network created, the OVS's Openflow flows for OVN Logical Router's NAT would be installed on all Incus nodes, causing all the nodes to deal with million of Openflows of NAT unnecessarly when dealing with high scaling clusters.
For such scaling scenario be viable for Incus, first we can use Gateway Chassis to deal with North/South traffic and centralizing this layer on a few network chassis.
Into Incus's cluster we can set a parameter for the cluster node:
incus cluster set inc1-gwnode1 scheduler.instance=manual
incus cluster role add inc1-gwnode1 ovn-chassis
incus cluster list
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-gwnode1 | https://172.31.0.161:8443 | ovn-chassis | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node1 | https://172.31.0.151:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node2 | https://172.31.0.152:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| inc1-node3 | https://172.31.0.153:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
+--------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
For a regular OVN Logical Router, the ha chassis group will have only the Gateway chassis as members:
ovn-nbctl list logical_router_port incus-net92-lr-lrp-ext
_uuid : 5c86ed08-de63-4903-b602-527204705b46
enabled : []
external_ids : {}
gateway_chassis : []
ha_chassis_group : ed63891a-326a-4dfe-8d6f-9c4ce6880421
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "00:16:3e:c4:6d:13"
name : incus-net92-lr-lrp-ext
networks : ["100.64.1.13/24", "2001:db8::5/64"]
options : {gateway_mtu="1500"}
peer : []
status : {hosting-chassis="c202f93e-a669-4f14-8db1-208901c990bf"}
ovn-nbctl ha-chassis-group-list ed63891a-326a-4dfe-8d6f-9c4ce6880421
ed63891a-326a-4dfe-8d6f-9c4ce6880421 (incus-net92)
408b3b47-5316-4333-b3be-32501ad4a9ea (c202f93e-a669-4f14-8db1-208901c990bf)
priority 2125
ovn-sbctl find chassis name=c202f93e-a669-4f14-8db1-208901c990bf
_uuid : 75ea11d5-251a-4df1-8a39-ab6d890436cf
encaps : [5e73e729-4736-488b-8951-7d8fdb0be945]
external_ids : {}
hostname : inc1-gwnode1
name : "c202f93e-a669-4f14-8db1-208901c990bf"
What do you think would need to be added
After make clear the reasons, next step would be to make optional for the user to create the physical network on all Incus nodes, when the user setup would require only for Gateway chassis.
Incus really likes having consistent configuration within a cluster, so having a network or storage pool only exist on a subset of servers would require rather massive changes.
That said we do have the special none value supported in a variety of such situation, so adding support for parent=none which would allow for the network to be defined, but not used, would likely work here and line up with other similar cases.
Would that work?
Basically we would:
- Allow the special
parent=nonevalue for physical networks - If set to this value, the network cannot be used on that server
- In the OVN case, this would prevent Incus from acting as a gateway chassis on that server
Hi,
Probably it would work, the problem would be when there are thousand of nodes and if we forget some node that would complain that does not have the bridge while creating the network. We can survive with that.
Tiago Pires
Hi, we wanted to check our understanding before diving in. From my understanding, we need to: (1) Allow setting parent=none on a per-node basis (via --target) when defining a physical or OVN network. (2) Internally, skip NIC validation and activation during startup on nodes where parent=none. (3) For OVN networks, treat parent=none as a signal to exclude that node from acting as a gateway chassis (i.e., avoid NAT OpenFlow setup).
Please let me know if there's anything wrong. Also, is it possible to get some guidance on which files are relevant to implement each of these steps? Thank you!
- Yeah, that part should be easy as it's already allowed for valid values, so it's just a matter of also allowing the special
nonevalue - Yep, should be pretty easy to handle in start/stop
- Yep. That's likely the most complex part as it's code I haven't messed with in quite a while. We have a per-network ha chassis group which contains all servers capable of handling gateway role for the network. That's the logic that needs to be changed here to skip adding ourselves if dealing with parent=none.
Awesome! I just made a pull request with all these changes. #2073 Please let me know if there are any issues you'd like me handle.
Hi, I noticed this issue hasn’t been closed and hasn’t had any updates in a while. Would it be okay if @ViniRodrig and I continue working on it?
Yeah. What happened is that a group of student started working on it but didn't complete the work so it's been staying as an open draft PR for someone to look at.
We've been a bit swamped lately so I've not been able to get to those.
If you have some time to work on this, that'd be great!
