projectcalico
unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3"
When calico-apiserver pod goes to ready state (around 15:09:34) I try to create GlobalNetworkPolicy using "kubectl apply -f security_enforcement.yml" but get an error:
[2022-09-24T15:09:52.676Z] [2022-09-24T15:09:52.676Z] unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3" [2022-09-24T15:09:52.676Z] unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3" [2022-09-24T15:09:52.676Z] unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3"
Expected Behavior
Creation of GlobalNetworkPolicy with calico-apiserver v 3.24.0 installed is successful.
Current Behavior
Creation of GlobalNetworkPolicy with calico-apiserver v 3.24.0 installed is unsuccessful. Error is visible unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3"
Possible Solution
Steps to Reproduce (for bugs)
- Kubernetes cluster is up and running
- Calico CNI is installed
- To support calico resources creation (bgpconfiguration, bgppeer, GlobalNetworkPolicy) I run calico-apiserver using instructions https://projectcalico.docs.tigera.io/maintenance/install-apiserver:
Logs from calico-apiserver: Version: v3.24.0 Build date: 2022-08-18T17:08:13+0000 Git tag ref: v3.24.0 Git commit: dd5e3a40b I0924 15:09:34.879604 1 plugins.go:158] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionWebhook. I0924 15:09:34.879664 1 plugins.go:161] Loaded 1 validating admission controller(s) successfully in the following order: ValidatingAdmissionWebhook. I0924 15:09:34.953715 1 run_server.go:69] Running the API server I0924 15:09:34.953772 1 run_server.go:58] Starting watch extension W0924 15:09:34.953800 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0924 15:09:34.971908 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0924 15:09:34.971957 1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController I0924 15:09:34.971924 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0924 15:09:34.972000 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0924 15:09:34.972029 1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0924 15:09:34.972001 1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0924 15:09:34.972361 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" I0924 15:09:34.972613 1 secure_serving.go:210] Serving securely on [::]:5443 I0924 15:09:34.972685 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0924 15:09:34.973247 1 run_server.go:80] apiserver is ready. I0924 15:09:35.072261 1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0924 15:09:35.072291 1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0924 15:09:35.072307 1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController I0924 15:10:14.432702 1 resource.go:262] GetList called with key: projectcalico.org/networksets on resource NetworkSet I0924 15:10:14.432730 1 resource.go:271] List called with key: projectcalico.org/networksets on resource NetworkSet I0924 15:10:14.432892 1 resource.go:262] GetList called with key: projectcalico.org/networkpolicies on resource NetworkPolicy I0924 15:10:14.432917 1 resource.go:271] List called with key: projectcalico.org/networkpolicies on resource NetworkPolicy I0924 15:10:14.437212 1 resource.go:206] Watch called with key: projectcalico.org/networksets on resource NetworkSet I0924 15:10:14.437604 1 resource.go:206] Watch called with key: projectcalico.org/networkpolicies on resource NetworkPolicy time="2022-09-24T15:10:14Z" level=info msg="Main client watcher loop" time="2022-09-24T15:10:14Z" level=info msg="Main client watcher loop" I0924 15:10:14.871950 1 resource.go:262] GetList called with key: projectcalico.org/ipamconfigurations on resource IPAMConfiguration I0924 15:10:14.871987 1 resource.go:271] List called with key: projectcalico.org/ipamconfigurations on resource IPAMConfiguration I0924 15:10:14.872395 1 resource.go:262] GetList called with key: projectcalico.org/clusterinformations on resource ClusterInformation I0924 15:10:14.872426 1 resource.go:271] List called with key: projectcalico.org/clusterinformations on resource ClusterInformation I0924 15:10:14.872782 1 resource.go:262] GetList called with key: projectcalico.org/felixconfigurations on resource FelixConfiguration I0924 15:10:14.872814 1 resource.go:271] List called with key: projectcalico.org/felixconfigurations on resource FelixConfiguration I0924 15:10:14.872819 1 resource.go:262] GetList called with key: projectcalico.org/bgpconfigurations on resource BGPConfiguration I0924 15:10:14.872835 1 resource.go:271] List called with key: projectcalico.org/bgpconfigurations on resource BGPConfiguration I0924 15:10:14.873271 1 resource.go:262] GetList called with key: projectcalico.org/hostendpoints on resource HostEndpoint I0924 15:10:14.873294 1 resource.go:271] List called with key: projectcalico.org/hostendpoints on resource HostEndpoint I0924 15:10:14.874124 1 resource.go:262] GetList called with key: projectcalico.org/caliconodestatuses on resource CalicoNodeStatus I0924 15:10:14.874160 1 resource.go:271] List called with key: projectcalico.org/caliconodestatuses on resource CalicoNodeStatus I0924 15:10:14.874429 1 resource.go:262] GetList called with key: projectcalico.org/globalnetworksets on resource GlobalNetworkSet I0924 15:10:14.874463 1 resource.go:271] List called with key: projectcalico.org/globalnetworksets on resource GlobalNetworkSet I0924 15:10:14.874430 1 resource.go:262] GetList called with key: projectcalico.org/bgppeers on resource BGPPeer I0924 15:10:14.874627 1 resource.go:271] List called with key: projectcalico.org/bgppeers on resource BGPPeer I0924 15:10:14.875664 1 resource.go:262] GetList called with key: projectcalico.org/kubecontrollersconfigurations on resource KubeControllersConfiguration I0924 15:10:14.875698 1 resource.go:271] List called with key: projectcalico.org/kubecontrollersconfigurations on resource KubeControllersConfiguration I0924 15:10:14.878243 1 resource.go:262] GetList called with key: projectcalico.org/blockaffinities on resource BlockAffinity I0924 15:10:14.878480 1 resource.go:271] List called with key: projectcalico.org/blockaffinities on resource BlockAffinity I0924 15:10:14.878600 1 resource.go:262] GetList called with key: projectcalico.org/ipreservations on resource IPReservation I0924 15:10:14.878666 1 resource.go:271] List called with key: projectcalico.org/ipreservations on resource IPReservation I0924 15:10:14.879087 1 resource.go:262] GetList called with key: projectcalico.org/globalnetworkpolicies on resource GlobalNetworkPolicy I0924 15:10:14.879115 1 resource.go:271] List called with key: projectcalico.org/globalnetworkpolicies on resource GlobalNetworkPolicy I0924 15:10:14.879489 1 resource.go:206] Watch called with key: projectcalico.org/ipamconfigurations on resource IPAMConfiguration I0924 15:10:14.879769 1 resource.go:206] Watch called with key: projectcalico.org/clusterinformations on resource ClusterInformation I0924 15:10:14.879897 1 resource.go:206] Watch called with key: projectcalico.org/bgpconfigurations on resource BGPConfiguration I0924 15:10:14.880217 1 resource.go:262] GetList called with key: projectcalico.org/profiles on resource Profile I0924 15:10:14.880245 1 resource.go:271] List called with key: projectcalico.org/profiles on resource Profile I0924 15:10:14.880379 1 resource.go:262] GetList called with key: projectcalico.org/ippools on resource IPPool I0924 15:10:14.880402 1 resource.go:271] List called with key: projectcalico.org/ippools on resource IPPool I0924 15:10:14.880429 1 resource.go:206] Watch called with key: projectcalico.org/globalnetworksets on resource GlobalNetworkSet I0924 15:10:14.880982 1 resource.go:206] Watch called with key: projectcalico.org/felixconfigurations on resource FelixConfiguration
When calico-apiserver pod goes to ready state (around 15:09:34) I try to create GlobalNetworkPolicy using "kubectl apply -f security_enforcement.yml" but get an error:
[2022-09-24T15:09:52.676Z] [2022-09-24T15:09:52.676Z] unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3" [2022-09-24T15:09:52.676Z] unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3" [2022-09-24T15:09:52.676Z] unable to recognize "security_enforcement.yml": no matches for kind "GlobalNetworkPolicy" in version "projectcalico.org/v3"
I noticed that GlobalNetworkPolicy creation attempt was initiated around 15:09:52 but in apiserver logs I see GetList on resource GlobalNetworkPolicy operation at 15:10:14:
I0924 15:10:14.879087 1 resource.go:262] GetList called with key: projectcalico.org/globalnetworkpolicies on resource GlobalNetworkPolicy I0924 15:10:14.879115 1 resource.go:271] List called with key: projectcalico.org/globalnetworkpolicies on resource GlobalNetworkPolicy
Is it the case that between: I0924 15:09:34.973247 1 run_server.go:80] apiserver is ready. and I0924 15:10:14.879087 1 resource.go:262] GetList called with key: projectcalico.org/globalnetworkpolicies on resource GlobalNetworkPolicy I0924 15:10:14.879115 1 resource.go:271] List called with key: projectcalico.org/globalnetworkpolicies on resource GlobalNetworkPolicy
I can not create GlobalNetworkPolicy using kubectl because this resource is not yet visible by Kubernetes API? If answer is YES why does it take so long time (40 sec) for apiserver to become availale to serve GlobalNetworkPolicy creation requests?
Any ideas why does it happen?
Context
I want to enforce GlobalNetworkPolicy on Kubernetes node immediately after cluster is up and running. Cluster installation, calico apiserver installation and Policy creation are automated using ansible. So with above circumstances whole deployment fails.
Your Environment
- Calico version 3.24.0
- Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes
- Operating System and version: Ubuntu
- Link to your project (optional):
Hey @dmytrokazantsev81, I want to clarify really quickly that the issue is that the API does not come up fast enough? Or does the API never become available for GlobalNetworkPolicy creation?
If the issue is that the custom resources are not available as soon as our API server comes up, it might be because there are some other mechanisms that need to register with the Kubernetes API before the aggregated API server is recognized. I'm not sure off the top of my head why there would be a noticeable delay in that though.
Yes this is most probably because of delay issue in calico API availability. Some time later I am able to create the same policies manually on the same cluster. The question is which conditions must be true before I am able to create GlobalNetworkPolicy using kubectl command in kubernetes cluster. I thought that using a condition that calico-apiserver pod is ready is enough to proceed with network policy installation but looks like there is some more time needed for this creation to be successful.
Unfortunately I'm not sure all of the mechanisms that go into API discovery for the kubernetes API regarding aggregated APIs (the Calico API). It could be due to certificates or API registration that I'm not sure we have much control over. Are there any logs in the your kubernetes API server or kubelet that might hint at how we could speed things up?
The Calico API runs as a separate pod. The image needs to be downloaded and started on the cluster, and register itself with the Kubernetes API. This inherently takes some time to do, and is a known limitation with the current approach. There is some discission related to this in this issue: https://github.com/projectcalico/calico/issues/6412
Right now, the option is to wait for the calico-apiserver pod to be ready or to use calicoctl, which bypasses the apiserver.
@caseydavenport Thanks for comments. I moved the same direction as you propose. I implemented waiting until "kubectl api-resources | grep '\sprojectcalico.org'" returns expected output "GlobalNetworkPolicy".