tigera-operator docs for EKS contains deprecated API versions

[barsilver]

I'm upgrading EKS from 1.21 to the 1.22 version. PodSecurityPolicy in the policy/v1beta1 API version is being deprecated in Kubernetes 1.21 and will no longer be served in v1.25. I followed the documentation here for the operator installation but the instructions aren't updated for newer Kubernetes versions which cannot deploy PodSecurityPolicy (1.22+).

Expected Behavior

Installing Calico using tigera-operator.yaml from the documentation should be compatible with newer Kubernetes versions as it is for the calico installation from manifest

Current Behavior

tigera-operator installation as mentioned in the EKS documentation of project calico installs old API versions which are deprecated in k8s v1.22+

Your Environment

• Calico version: v3.23.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): k8s v1.21

[barsilver]

I tried to run kubectl apply for this tigera-operator.yaml but it failed with the following error:

The CustomResourceDefinition "installations.operator.tigera.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

[barsilver]

kubectl replace for this YAML file was the solution

[barsilver]

Apparently, it wasn't the solution, because after it finished progressing and rebooted all the pods in calico-system tigerastatus looked like this:

NAME     AVAILABLE   PROGRESSING   DEGRADED   SINCE
calico   True        False         False      6s

Few seconds after, the pods started to reboot again and that was the tigerastatus output:

Name:         calico
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  operator.tigera.io/v1
Kind:         TigeraStatus
Metadata:
  Creation Timestamp:  2022-06-23T09:45:29Z
  Generation:          1
  Managed Fields:
    API Version:  operator.tigera.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
      f:status:
        .:
        f:conditions:
    Manager:         operator
    Operation:       Update
    Time:            2022-06-23T09:45:34Z
  Resource Version:  671188473
  UID:               b50f6c9c-1ffb-4ee2-a6ff-bf498d905453
Spec:
Status:
  Conditions:
    Last Transition Time:  2022-08-03T10:41:50Z
    Observed Generation:   5
    Reason:                Unknown
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2022-08-03T10:43:50Z
    Observed Generation:   5
    Reason:                Unknown
    Status:                False
    Type:                  Available
    Last Transition Time:  2022-08-03T10:43:50Z
    Message:               DaemonSet "calico-system/calico-node" update is rolling out (15 out of 60 updated)
    Observed Generation:   5
    Reason:                ResourceNotReady
    Status:                True
    Type:                  Progressing
Events:                    <none>

The logs from from the tigera-operator namespace:

tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523611.8609326,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523611.8609657,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523635.454624,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523635.4546576,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523642.0554605,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523642.0554945,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523672.235133,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523672.2351656,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523702.4124248,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523702.4124672,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523710.454353,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523710.454387,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523730.5845706,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523730.5846014,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523732.5866766,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523732.5867133,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523745.4569619,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523745.4569979,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523755.4577687,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523755.457799,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523762.768554,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523762.768602,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523775.4567487,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523775.4567802,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523792.9404783,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523792.940505,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523800.4550567,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523800.455081,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523815.454971,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523815.455004,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523823.119496,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523823.119522,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523835.4547129,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523835.4547586,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523845.4552584,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523845.4552994,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523853.303227,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523853.303257,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523860.4542513,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523860.4542983,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523875.451944,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523875.4519773,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523883.4882042,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523883.4882388,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523890.4543114,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523890.4543421,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523905.453304,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}

[caseydavenport]

Yes, we need to update the EKS documentation to use a newer version.

Did you install via helm originally?

[barsilver]

@caseydavenport No, I installed it as described in the documentation with the following command:

kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml

But in order to upgrade it, since I didn't use helm in the first place, I used the command:

kubectl replace -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/tigera-operator.yaml

EDIT: Installing tigera-operator via helm created the apiserver in tigerastatus and solved the errors of "APIServer config not found", but PodSecurityPolicy resources are still being deployed.

[caseydavenport]

PodSecurityPolicy has been removed in master and the upcoming v3.24 release:

• https://github.com/projectcalico/calico/pull/6270
• https://github.com/projectcalico/calico/issues/5972
• https://github.com/tigera/operator/pull/2035

Once v3.24 is out, we just need to update the EKS documentation.

[lwr20]

tigera-operator installation as mentioned in the EKS documentation of project calico installs old API versions which are deprecated in k8s v1.22+

This is deliberate. By continuing to use deprecated k8s APIs, we can ensure that the compatibility of Calico with multiple versions of k8s. (arguably this is the point of deprecation)

We typically switch to the new API just before the removal of the API from k8s - e.g. in this case Calico v3.24 was the last Calico release before k8s 1.25 came out and removed this API, so that is the release we switched from PodSecurityPolicies to PodSecurityStandards.

[caseydavenport]

v3.24 is out now, so we're clear to update the docs. I believe @coutinhop was looking at making some changes in that area already?

[viceice]

I still get The CustomResourceDefinition "installations.operator.tigera.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes error 😕

[caseydavenport]

@viceice use kubectl create and kubectl replace for CRDs, rather than using kubectl apply - that should avoid that error.

[caseydavenport]

kubectl puts annotations on objects when using apply, and the annotation it creates in this case it not valid.

We need to figure out if there's a way we can avoid this situation, but in general kubectl create and replace are better options regardless.

[shu-mutou]

@caseydavenport Thanks! kubectl create instead of apply worked fine for my local cluster.