Failed to attach binaries ISO for VM when create kubernetes cluster

[tungntvn23]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API, Kubernetes Service

CLOUDSTACK VERSION

Cloudstack 4.14.0.0

CONFIGURATION

Advanged Networking, Vmware vSphere

OS / ENVIRONMENT

• Vmware vSphere 6.5
• CentOS 7

SUMMARY

After upgrade from 4.13 to 4.14 and enable kubernetest feature then i cannot create kubernetes cluster.

STEPS TO REPRODUCE

• Login to UI.
• Kubernetes service tab to click Add Kubernetes Cluster.
• Fill all information.
• Click OK

EXPECTED RESULTS

• Kubernetes cluster create successfully.
• State is OK

ACTUAL RESULTS

• Kubernetes cluster create failed.
• State is error.
• Error message: "Failed to attach binaries ISO for VM: test-master in the Kubernetes cluster name: test"

Note: i try manual attach binaries iso to vm test-master and get successfully.

[tungntvn23]

Attach management log

Message: Unable to find VM in vSphere to execute AttachIsoCommand, vmName: i-2-1465-VM
 Also check if your guest os is a supported version
2020-09-09 14:50:46,473 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-14:ctx-014abef7 job-114062 ctx-78218c74) (logid:528413cf) Failed to attach binaries ISO for VM: test-master in the Kubernetes cluster name: test
com.cloud.utils.exception.CloudRuntimeException: Failed to attach iso
	at com.cloud.template.TemplateManagerImpl.attachIso(TemplateManagerImpl.java:1217)
	at jdk.internal.reflect.GeneratedMethodAccessor956.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy195.attachIso(Unknown Source)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.attachIsoKubernetesVMs(KubernetesClusterActionWorker.java:320)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.attachIsoKubernetesVMs(KubernetesClusterActionWorker.java:331)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:530)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1057)
	at org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd.execute(CreateKubernetesClusterCmd.java:273)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[shwstppr]

@tungntvn23 Can you please try deploying cluster with Root disk controller=pvscsi and NIC adapter type=Vmxnet3 for CoreOS template? We have had some issue with ISOs and CoreOS VM https://github.com/apache/cloudstack-documentation/pull/151

[tungntvn23]

@shwstppr thank you, but issue happen randomly. Sometime it's OK but sometime not OK.

[shwstppr]

@tungntvn23 I'll have a look. Is it possible for you to test out a particular PR?

[tungntvn23]

@shwstppr I setting rootDiskController and NIC but it still failed attach iso. I try disable DRS from VMware cluster so it worked fine.

[shwstppr]

@tungntvn23 Will have to check how DRS is affecting attaching ISO. If you have time can you please check if attaching ISO working fine for you everytime for a normal CoreOS VM (outside cks cluster) with DRS set to enabled? Also, ACS 4.16 could see a new template k8s cluster node VMs which should prevent such issue.

[tungntvn23]

@shwstppr yes, i try many time to attach iso for normal CoreOS VM with DRS enabled and get result OK.

[rohityadavcloud]

@tungntvn23 can you try 4.15.1?

[nvazquez]

Closing this issue as no longer reproduceable cc. @davidjumani

[tungntvn23]

@rohityadavcloud this issue occurs with version 4.16.0. Sometime i can create cluster OK but when i scale cluster so it fail with error "Failed to attach binaries ISO for VM : test01-node-17e53b9a750 in the Kubernetes cluster name: test01"

[rohityadavcloud]

cc @Pearl1594 @sureshanaparti @davidjumani @shwstppr

[PaulAngus]

issue still occurring in 4.16.1

Failed to attach binaries ISO for VM : k8s-1215-2-node-181488f782a in the Kubernetes cluster name: k8s-1215-2

iso attaches 1 in 10 times

something is attached to VM

[harikrishna-patnala]

@PaulAngus any idea what is that attached to VM under CD/DVD Media ?

[rohityadavcloud]

Closing this based on inactivity and we aren't able to reproduce this, pl test latest 4.16 or 4.17 and re-open if you can share steps of reproduction cc @tungntvn23 @PaulAngus

[tungntvn23]

Hi @rohityadavcloud ,

Issue still happen. My step:

1. Login to UI.
2. Kubernetes service tab to click Add Kubernetes Cluster.
3. Fill all information.
4. Click OK

Error message: Failed to attach binaries ISO for VM : cluster01-node-18282cc1dbf in the Kubernetes cluster name: cluster01

[rohityadavcloud]

@tungntvn23 can you share your CloudStack version, hypervisor and storage you're using and CKS/kubernetes iso/version? cc @shwstppr @davidjumani

[shwstppr]

@tungntvn23 From your original logs what I understand is that ISO is failing to get attached because ACS is not being able to find VM on the given host to execute AttachIsoCommand. This must be because VM has been moved to a different host due to DRS. I can check and try to add an improvement for ACS to search on other hosts while attaching ISO if it is not on the original host. But if the fix works it will go into 4.17 branch. @PaulAngus is the issue similar for you? Is your env also having DRS enabled? Although you mentioned you see an ISO attached to the k8s VM. Can you please check if it is the k8s ISO or something different

[tungntvn23]

@tungntvn23 can you share your CloudStack version, hypervisor and storage you're using and CKS/kubernetes iso/version? cc @shwstppr @davidjumani

Hi @rohityadavcloud ,

My env:

• ACS 4.16.0
• Vmware 6.7
• Storage vSAN
• K8S version 1.22.6

[tungntvn23]

@tungntvn23 From your original logs what I understand is that ISO is failing to get attached because ACS is not being able to find VM on the given host to execute AttachIsoCommand. This must be because VM has been moved to a different host due to DRS. I can check and try to add an improvement for ACS to search on other hosts while attaching ISO if it is not on the original host. But if the fix works it will go into 4.17 branch. @PaulAngus is the issue similar for you? Is your env also having DRS enabled? Although you mentioned you see an ISO attached to the k8s VM. Can you please check if it is the k8s ISO or something different

Thank @shwstppr , i will test with ACS 4.17 after the issue is fixed.

[rohityadavcloud]

Fixed in https://github.com/apache/cloudstack/pull/6625