kops icon indicating copy to clipboard operation
kops copied to clipboard
verified publisher icon kubernetes

Kops not creating any instances when creating a cluster on AWS

Open oceanminded opened this issue 1 year ago • 1 comments

/kind bug

1. What kops version are you running? The command kops version, will display this information. 1.29.2

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

3. What cloud provider are you using? aws

4. What commands did you run? What is the simplest way to reproduce this issue?

export BYTERAT_IRIS_AVAILABILITY_ZONES=us-east-2a,us-east-2b,us-east-2c
export NAME=cluster-xyz.my_domain.co
export KOPS_CLUSTER_NAME=$NAME
export KOPS_STATE_STORE=s3://cluster-xyz-state-store-0
export AWS_AVAILABILITY_ZONES=us-east-2a,us-east-2b,us-east-2c
export S3_BUCKET=cluster-xyz-state-store-0
export CONTROL_PLANE_SIZE="m5.8xlarge"
export NODE_SIZE="m5.8xlarge"

kops create cluster \
    --name cluster-xyz.my_domain.co \
    --ssh-public-key ~/.ssh/my_ssh_key.pub \
    --dns-zone DNS_ZONE_HOSTED_ZONEI_ID \
    --node-count 3 \
    --zones us-east-2a,us-east-2b,us-east-2c \
    --node-size $NODE_SIZE \
    --control-plane-size $CONTROL_PLANE_SIZE \
    --control-plane-zones us-east-2a,us-east-2b,us-east-2c \
    --discovery-store=s3://cluster-xyz-oidc-store-0/${NAME}/discovery

5. What happened after the commands executed? Cluster fails validation with timeouts to api servers and no ec2 instances are created.

6. What did you expect to happen? Cluster creation

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2024-07-28T01:13:58Z"
  generation: 1
  name: cluster-xyz.my_domain.co
spec:
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": [ "elasticloadbalancing:AddListenerCertificates", "elasticloadbalancing:RemoveListenerCertificates"],
          "Resource": ["*"]
        }
      ]
  api:
    loadBalancer:
      class: Network
      type: Public
  authorization:
    rbac: {}
  awsLoadBalancerController:
    enabled: true
  certManager:
    enabled: true
  channel: stable
  cloudProvider: aws
  configBase: s3://cluster-xyz-state-store-0/cluster-xyz.my_domain.co
  dnsZone: DNS_ZONE_HOSTED_ZONEI_ID
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2b
      name: b
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2c
      name: c
      volumeSize: 200
      volumeType: io1
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2b
      name: b
      volumeSize: 200
      volumeType: io1
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2c
      name: c
      volumeSize: 200
      volumeType: io1
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
    useServiceAccountExternalPermissions: true
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.29.6
  masterPublicName: api.cluster-xyz.my_domain.co
  networkCIDR: 172.20.0.0/16
  networking:
    cilium:
      enableNodePort: true
  nonMasqueradeCIDR: 100.64.0.0/10
  serviceAccountIssuerDiscovery:
    discoveryStore: s3://cluster-xyz-oidc-store-0/cluster-xyz.my_domain.co/discovery/cluster-xyz.my_domain.co
    enableAWSOIDCProvider: true
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - cidr: 172.20.0.0/18
    name: us-east-2a
    type: Public
    zone: us-east-2a
  - cidr: 172.20.64.0/18
    name: us-east-2b
    type: Public
    zone: us-east-2b
  - cidr: 172.20.128.0/18
    name: us-east-2c
    type: Public
    zone: us-east-2c
  topology:
    dns:
      type: Public

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: control-plane-us-east-2a
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: control-plane-us-east-2b
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: control-plane-us-east-2c
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:13:59Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: nodes-us-east-2a
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:14:00Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: nodes-us-east-2b
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-28T01:14:00Z"
  labels:
    kops.k8s.io/cluster: cluster-xyz.my_domain.co
  name: nodes-us-east-2c
spec:
  image: 238947058394/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240607
  machineType: m5.8xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-east-2c

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

oceanminded avatar Jul 28 '24 01:07 oceanminded

Hi @oceanminded. Just to check, was there any error when running kops update cluster --yes? Could you check the ASGs that were created to see if there is any error message (like not enough quota)?

hakman avatar Jul 28 '24 05:07 hakman

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Oct 26 '24 06:10 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Nov 25 '24 06:11 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Dec 25 '24 06:12 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Dec 25 '24 06:12 k8s-ci-robot