cluster-api-provider-vsphere
cluster-api-provider-vsphere copied to clipboard
kubernetes-sigs
[IPAM] Allow network interfaces without assigned IP address and without dhcp.
/kind feature
Describe the solution you'd like
Currently the VSphere VM gets paused to get an IP assigned when you don't configure dhcp.
This behavior works pretty well in the use-case of manual or operator IP assignment and also with multiple interfaces.
When having multiple interfaces on a VM it is not always necessary to assign an IP. (e.g. Multus/MacVLAN/IPVlan) A way of an opt-out feature for the network interfaces which should ignore it and configure the interface without an ip and not hold the VM would be great.
- Cluster-api-provider-vsphere version: v0.7.8
- Kubernetes version: (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8", GitCommit:"5575935422cc1cf5169dfc8847cb587aa47bac5a", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:07Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"linux/amd64"}
- OS (e.g. from
/etc/os-release):
Ubuntu 20.04
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
^ That was me. The bot took a while, so I thought I was missing permissions and deleted the original message.
@Cellebyte do you have a working IPAM solution with CAPV or are you patching in static IPs?
We've built a small custom controller that interfaces with Infoblox which is available here.
I've also started working on a proposal for a proper ipam integration contract that different providers can interface with to retrieve IPs. If you're interested, feel free to comment here.
We've built a small custom controller that interfaces with Infoblox which is available here.
You got that working with the "wait until static IP is available" behaviour CAPV offers currently? So I guess the job of your controller is to understand the structure of and patch CAPV resources? Sorry for all thsoe questions but I'm really interested in your approach. In this approach for example they simply hardcoded Metal3io IPAM into the vSphereMachine resource which isn't very flexible from my point of view.
I've also started working on a proposal for a proper ipam integration contract that different providers can interface with to retrieve IPs. If you're interested, feel free to comment here.
Interesting. Because there's this old proposal which never was implemented as far as I can tell.
@omniproc for IPaddresses there is some kind of same CRD which we use for our general purpose CNI.
It has the following CRDs.
addressblocks
addresspools
blockrequests
a block can be any kind of /8 to /128 and a request says which size it should be.
The pools can be implemented with different third-party providers in mind.
Sorry for all thsoe questions but I'm really interested in your approach.
Our current approach is similar to the spectrocloud one, it just directly talks to Infoblox instead of interfacing with metal3-ip-address-manager. It relies on CAPV waiting for the IP and simply adds the IP Address to the VSphereMachine.
Interesting. Because there's this old proposal which never was implemented as far as I can tell.
Interesting indeed, I wasn't even aware of that proposal. But it's very similar to mine and the current metal3-ip-address-manager. The idea is to build something similar to Kubernetes' persistent volumes, but for IP Addresses. With different providers for different IPAM solutions.
But we're getting really off-topic from this issue, let's maybe continue discussion in slack @omniproc
@omniproc for IPaddresses there is some kind of same CRD which we use for our general purpose CNI.
It has the following CRDs.
addressblocks addresspools blockrequestsa block can be any kind of
/8to/128and a request says which size it should be. The pools can be implemented with different third-party providers in mind.
Thanks, that's clear. My question was rather how you then assign those IPs defined and made availabe by your IPAM to the VsphereMachine resources? All solutions I've seen so far require a custom controler for that which will patch the resources with the IP provided by the IPAM.
Our current approach is similar to the spectrocloud one, it just directly talks to Infoblox instead of interfacing with metal3-ip-address-manager. It relies on CAPV waiting for the IP and simply adds the IP Address to the VSphereMachine.
Well yes but you guys wrote a controller for that while the approach spectrocloud took in the repo i mentioned is patching the vSphereMachine controller code itself.
The missing piece seems to be the specification that you're working on so we can have a generic approach that all IPAM solutions can integrate on. I'm not sure however if this is only relevant for CAPV since it's being discussed in CAPI as well and probably the formal definition would make more sense there. Otherwise you'll end up with different IPAM implementations per CAPI provider.
@omniproc the design proposed was exactly targeting to be a capi proposal and nothing else. We are working on it to get a generic solution which can be implemented for different IPAMs.
@schrej and me @cellebyte work for the same org ^^
Our current approach is not using the design and interfaces with the existing possibilities of the different providers so adding network config in capv and interfacing with metal3s approach.
@Cellebyte according to @jzhoucliqr this spectrocloud implementation I refered to earlier is going to be merged with upstream CAPV, see #728
To come back to the main topic of this issue.
@omniproc we are not working on an CAPV specific IPAM implementation.
We are currently using the feature of CAPV which pauses the VSphereMachine in reconcilation until an IP address gets added.
We did not change any code in CAPV at all.
This Issue is only proposing a change for the current behavior of CAPV when the VSphereMachine does not have dhcp4 or dhcp6 enabled in its VSphereMachineTemplate.
It is not addressing a general IPAM approach for CAPV because that is under discussion in the CAPI community.
this spectrocloud implementation I refered to earlier is going to be merged with upstream CAPV
No, it's not. They were referring to spectrocloud/cluster-api-provider-vsphere-static-ip#26, which adds support for v1alpha4 to their implementation. There is no ongoing effort to merge spectroclouds implementation into CAPV that I can see.
CAPV is also aware of our proposal and my impression is that they currently see it as the way forward with static IP configuration. (@yastij correct me if that impression is wrong)
No, it's not. They were referring to spectrocloud/cluster-api-provider-vsphere-static-ip#26, which adds support for v1alpha4 to their implementation.
Oh sorry, you're right. I got that wrong. In that case forget what I said about it.
It is not addressing a general IPAM approach for CAPV because that is under discussion in the CAPI community.
Thanks for clearification. I can't wait to see the CAPI proposal implemented. I believe many people will be happy if they can drop dependencies on DHCP without having to patch ressources manually.
