Issue with creating a new cluster

[abssirohi90]

/kind bug

1. What kops version are you running? The command kops version, will display this information.

Version 1.23.0

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create -f cluster.yaml kops create -f ig.yaml kops update cluster kops validate cluster

5. What happened after the commands executed?

The cluster was created, new EC2 instances came up. Upon running validate cluster, we are getting: machine "i-xxx" has not yet joined cluster Machine i-xxx machine "i-xxx" has not yet joined cluster Machine i-xxx machine "i-xxx" has not yet joined cluster Machine i-xxx machine "i-xxx" has not yet joined cluster Machine i-xxx machine "i-xxx" has not yet joined cluster

6. What did you expect to happen?

Cluster to be created properly, kops validate to validate the cluster as up and running.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  generation: 22
  name: xxx
spec:
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "elasticloadbalancing:*"
          ],
          "Resource": [
            "*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:*"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "route53:ChangeResourceRecordSets"
          ],
          "Resource": [
            "arn:aws:route53:::hostedzone/*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "route53:ListHostedZones",
            "route53:ListResourceRecordSets"
          ],
          "Resource": [
            "*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
              "autoscaling:DescribeAutoScalingGroups",
              "autoscaling:DescribeAutoScalingInstances",
              "autoscaling:DescribeTags",
              "autoscaling:DescribeLaunchConfigurations",
              "autoscaling:SetDesiredCapacity",
              "autoscaling:TerminateInstanceInAutoScalingGroup"
          ],
          "Resource": [
            "*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
              "sts:AssumeRole"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
  api:
    loadBalancer:
      class: Classic
      type: Internal
  assets:
    containerProxy: xxx
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://xxx/xxx
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-ap-southeast-2a
      name: a
    - instanceGroup: master-ap-southeast-2b
      name: b
    - instanceGroup: master-ap-southeast-2c
      name: c
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-ap-southeast-2a
      name: a
    - instanceGroup: master-ap-southeast-2b
      name: b
    - instanceGroup: master-ap-southeast-2c
      name: c
    memoryRequest: 100Mi
    name: events
  fileAssets:
  - content: "apiVersion: audit.k8s.io/v1\nkind: Policy\nrules:\n# do not log requests
      to the following \n- level: None\n  nonResourceURLs:\n  - \"/healthz*\"\n  -
      \"/logs\"\n  - \"/metrics\"\n  - \"/swagger*\"\n  - \"/version\"\n\n# limit
      level to Metadata so token is not included in the spec/status\n- level: Metadata\n
      \ omitStages:\n  - RequestReceived\n  resources:\n  - group: authentication.k8s.io\n
      \   resources:\n    - tokenreviews\n\n# extended audit of auth delegation\n-
      level: RequestResponse\n  omitStages:\n  - RequestReceived\n  resources:\n  -
      group: authorization.k8s.io\n    resources:\n    - subjectaccessreviews\n\n#
      log changes to pods at RequestResponse level\n- level: RequestResponse\n  omitStages:\n
      \ - RequestReceived\n  resources:\n  - group: \"\" # core API group; add third-party
      API services and your API services if needed\n    resources: [\"pods\"]\n    verbs:
      [\"create\", \"patch\", \"update\", \"delete\"]\n\n# log everything else at
      Metadata level\n- level: Metadata\n  omitStages:\n  - RequestReceived\n"
    name: audit-policy-config
    path: /srv/kubernetes/kube-apiserver/policy-config.yaml
    roles:
    - Master
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    auditLogMaxAge: 14
    auditLogMaxBackups: 1
    auditLogMaxSize: 100
    auditLogPath: /var/log/kube-apiserver-audit.log
    auditPolicyFile: /srv/kubernetes/kube-apiserver/policy-config.yaml
    featureGates:
      ExpandPersistentVolumes: "true"
      HPAScaleToZero: "true"
    oidcClientID: kubernetes
    oidcGroupsClaim: groups
    oidcGroupsPrefix: 'keycloak:'
    oidcIssuerURL: xxx
    oidcUsernameClaim: email
    runtimeConfig:
      authentication.k8s.io/v1beta1: "true"
      autoscaling/v2beta1: "true"
  kubeControllerManager:
    featureGates:
      ExpandPersistentVolumes: "true"
  kubeProxy:
    metricsBindAddress: 0.0.0.0:10249
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
    podInfraContainerImage: k8s.gcr.io/pause:3.6    
  kubernetesApiAccess:
  - x.x.x.x/16
  - x.x.x.x/16
  - x.x.x.x/16
  kubernetesVersion: 1.23.5
  masterInternalName: xxxx
  masterPublicName: xxxx
  networkCIDR: x.x.x.x/16
  networkID: vpc-xxxx
  networking:
    calico: {}
  nonMasqueradeCIDR: xxxx
  sshAccess:
  - x.x.x.x/16
  - x.x.x.x/16
  - x.x.x.x/16
  subnets:

  - cidr: x.x.x.x/16
    egress:	nat-xxx
    id: subnet-xxx
    name: ap-southeast-2a
    type: Private
    zone: ap-southeast-2a

  - cidr: x.x.x.x/16
    egress: nat-xxx
    id: subnet-xxx
    name: ap-southeast-2b
    type: Private
    zone: ap-southeast-2b

  - cidr: x.x.x.x/16
    egress: nat-xxx
    id: 	subnet-xxx
    name: ap-southeast-2c
    type: Private
    zone: ap-southeast-2c

  - cidr: x.x.x.x/16
    id: subnet-xxx
    name: utility-ap-southeast-2a
    type: Utility
    zone: ap-southeast-2a

  - cidr: x.x.x.x/16
    id: subnet-xxx
    name: utility-ap-southeast-2b
    type: Utility
    zone: ap-southeast-2b

  - cidr: x.x.x.x/16
    id: subnet-xxx
    name: utility-ap-southeast-2c
    type: Utility
    zone: ap-southeast-2c

  topology:
    dns:
      type: Public
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

The error we are getting is: Apr 21 03:14:50 ip-10-xxx kubelet[4624]: I0421 03:14:50.094350 4624 csi_plugin.go:1063] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-xxx.ap-southeast-2.compute.internal" is forbidden: User "system:node:ip-x.x.x.x" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope: can only access CSINode with the same name as the requesting node

[olemarkus]

See https://github.com/kubernetes/kops/pull/13542

[nicolasespiau]

hello, I'm facing the same issue with all kops versions from 1.22.5.

Is there any workaround or bug fix?

[olemarkus]

Have a look at https://github.com/kubernetes/kops/issues/13934#issuecomment-1172257905

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.