[3.3.1 vcd_vapp_vm BUG] Windows/Linux VM created does not customize ip, admin password with vCloud 10.2.1.17444397 + vCenter 7.0 U2b

[vfuyong]

This problem seems to occur after vCenter is upgraded to 7.0 U2b. So I think the problem is with the provider. Manual creation of VM worked fine with customization.

Terraform Version

0.13 latest

Affected Resource(s)

Please list the resources as a list, for example:

• vcd_vapp_vm

Terraform Configuration Files

resource "vcd_vapp_vm" "test-vm" { vapp_name = "test-vapp" org = "test-org" vdc = "test-vdc" name = "test-vm" computer_name = "test-vm" catalog_name = "test-catalog" template_name = "test-template" memory = 8192 cpus = 4 cpu_cores = 1

network { type = "org" name = "test-network" ip_allocation_mode = "POOL" is_primary = true adapter_type = "VMXNET3" }

customization { enabled = true force = false change_sid = true allow_local_admin_password = true auto_generate_password = false admin_password = "somePasswordWithoutWeirdSymbolForTesting" }

}

On Windows the command instead of export is set.

Expected Behavior

Windows customization should configure IP and admin password etc.

Actual Behavior

IP is not configured, and admin password not set.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

[Didainius]

Hello @vfuyong , The first thing I'd ask is - does it work using UI? Is your VCD version compatible with with new vCenter?

From your statement to me it looks that is not directly problem with Terraform provider, but with VCD/vCenter integration.

[vfuyong]

Hello @vfuyong , The first thing I'd ask is - does it work using UI? Is your VCD version compatible with with new vCenter?

From your statement to me it looks that is not directly problem with Terraform provider, but with VCD/vCenter integration.

Yes it works with GUI with vCloud 10.2.1.17444397 and vCenter 7.0 U2b. Just that Terraform provider doesnt work.

[Didainius]

Yes it works with GUI with vCloud 10.2.1.17444397 and vCenter 7.0 U2b. Just that Terraform provider doesnt work.

Thanks. Double checking the situation.

• guest customization worked using terraform provider with 10.2.1.17444397 but before vCenter was upgraded to 7.0 U2b? (which version of Terraform provider was that)
• customization stopped working after upgrading vCenter to 7.0 U2b (but having the same VCD version 10.2.1.17444397 and the same terraform provider version .

[vfuyong]

Yes it works with GUI with vCloud 10.2.1.17444397 and vCenter 7.0 U2b. Just that Terraform provider doesnt work.

Thanks. Double checking the situation.

• guest customization worked using terraform provider with 10.2.1.17444397 but before vCenter was upgraded to 7.0 U2b? (which version of Terraform provider was that)
• customization stopped working after upgrading vCenter to 7.0 U2b (but having the same VCD version 10.2.1.17444397 and the same terraform provider version .

Yes to the above question according to my knowledge now.

[vfuyong]

@Didainius Is there any update on this issue?

[Didainius]

Hello, In fact not really at the moment. I have not been able to validate and it is very odd, but lets look just thinking out more now:

• Does this happen on a single environemnt or on multiple ones?

And also my last thoughts if you know browser developer tools - would you be able to capture API calls and payloads it does in exactly your case using UI and then a log what it sends using terraform (you can enable using this https://registry.terraform.io/providers/vmware/vcd/latest/docs#logging).

Need to compare payloads as in fact we should be pretty much sending the same thing.

I wouldn't be surprised that for example a vCenter upgrade could break integration, but I am really surprised it works using UI and does not using Terraform (API)

[vfuyong]

@Didainius I am not sure whehter I can perform the above. But could you confirm with me whether Terraform provider makes API call to vCenter/ESXi directly in addition to vCloud? As long as I have this answer, I can try troubleshoot more.

[Didainius]

@Didainius I am not sure whehter I can perform the above. But could you confirm with me whether Terraform provider makes API call to vCenter/ESXi directly in addition to vCloud? As long as I have this answer, I can try troubleshoot more.

No. Terraform provider for VCD does not make calls anywhere else except VCD. VCD connects to related systems (vSphere). That is why I am so surprised that it works for you in UI, but not does work with Terraform anymore.

[vfuyong]

@Didainius I also feel this is a weird issue. I have been working with vCloud for a few years. Is it possible that Terraform is calling the older vCloud API version which still is supported but not completely working? I know that each vCloud release will introduce a newer API version.

[Didainius]

@Didainius I also feel this is a weird issue. I have been working with vCloud for a few years. Is it possible that Terraform is calling the older vCloud API version which still is supported but not completely working? I know that each vCloud release will introduce a newer API version.

Yes. Our provider currently uses V33.0 to perform most API calls (it elevates in some cases to consume new features). 10.2.1 UI uses V35.0. We do have automated tests with guest customization (but Linux, not Windows) and they passed for all non EOL versions (10.2.1 included).

I could make an experimental fork and force the default to be V35.0 - but just for trying it out as there is a big risk something will be broken. That is why we keep on the stable version - to be sure we have tested as much as we could before releasing.

[vfuyong]

@Didainius I will be grateful if you can run a test over these specs with vCenter 7.0 U2b and vCloud 10.2.1, or you can let me know whether you have such case. I know vCloud customization is quite tricky, even from UI point of view.

network { type = "org" name = "test-network" ip_allocation_mode = "POOL" is_primary = true adapter_type = "VMXNET3" }

customization { enabled = true force = false change_sid = true allow_local_admin_password = true auto_generate_password = false admin_password = "somePasswordLongEnough" }

Please connect to System org (not tenant org) with admin rights to conduct the test thanks.

[Didainius]

I am not sure I can get that exact vSphere version for testing (need to check on that). However - is your env safe enough to just test VM creation with elevated API if I provide such a forked version of terraform provider?

[vfuyong]

I am not sure I can get that exact vSphere version for testing (need to check on that). However - is your env safe enough to just test VM creation with elevated API if I provide such a forked version of terraform provider?

Yes I can run that for you. Please let me know how I can use that provider. It will be nice to have binary files (without compilation) from you.

[Didainius]

Yes I can run that for you. Please let me know how I can use that provider. It will be nice to have binary files (without compilation) from you.

Which OS are you using?

[vfuyong]

Yes I can run that for you. Please let me know how I can use that provider. It will be nice to have binary files (without compilation) from you.

Which OS are you using?

Ubuntu 20.04

Could you also tell me the exact changes in this newer version?

[Didainius]

I will just try to switch API version to V35.0 by default (it uses V33.0 currently).

[Didainius]

Here is the link for Linux 64bit binary https://onevmw-my.sharepoint.com/:f:/g/personal/dserplis_vmware_com/Enb7P1rw4K5Nr82zeBI-N5kBGH_duwkLhi87EUEJDuz8EQ?e=Zr5CXP

It is the Linux one and is using API V35.0 as a baseline for all API calls instead of V33.0 I have ran an automated test for vApp/VM creation and it was able to, so this should be runnable.

Do not use this everywhere - this is just an experimental build to test hypothesis - if we find out this is the reason - then we might investigate further.

[vfuyong]

Here is the link for Linux 64bit binary https://onevmw-my.sharepoint.com/:f:/g/personal/dserplis_vmware_com/Enb7P1rw4K5Nr82zeBI-N5kBGH_duwkLhi87EUEJDuz8EQ?e=Zr5CXP

It is the Linux one and is using API V35.0 as a baseline for all API calls instead of V33.0 I have ran an automated test for vApp/VM creation and it was able to, so this should be runnable.

Do not use this everywhere - this is just an experimental build to test hypothesis - if we find out this is the reason - then we might investigate further.

I have tested with the provider in the url, it is still producing the same result - Windows2016 VM is not customized with IP, hostname etc.

[vfuyong]

@Didainius It appears to be likely a vCenter server issue. But provider does not throw any error for customization. I know that if customization timed out or failed, provider is able to detect. Just that when vCenter server isn't healthy, no error was returned?

[Didainius]

@vfuyong, I am confused now. Was it that it worked for you using VCD UI? I am not sure VCD UI shows anywhere if customization fails, does it?

[vfuyong]

@vfuyong, I am confused now. Was it that it worked for you using VCD UI? I am not sure VCD UI shows anywhere if customization fails, does it?

VCD UI has always been working. Just that when provisioned using provider, customization doesnt work. We have ruled out the issue from vCloud/vCenter version, as we have 2 environments running the same version, only 1 environment has such issue.

[vfuyong]

@vfuyong, I am confused now. Was it that it worked for you using VCD UI? I am not sure VCD UI shows anywhere if customization fails, does it?

If customization failed, VCD UI will show in the task status. But this behavior..seems changed in the last few releases. In earlier releases, I remembered it did show customization failure. But newer ones don't show any more?

[Didainius]

@vfuyong , Have you ever resolved it? We have also recently made improvements to VM creation (v3.8.0) so would be interesting if you have any newer feedback.

I will close this issue now, but feel free to reopen if this issue is still actual.