Set static network settings by VM customization

[naveenkarunad]

I am deploying a vm on a vsphere environment using packer-builder-vsphere . here the challenge is vm gets cloned ,powered on and stucks at " waiting for ip " . Is there any option to skip this as my environment doesnt have any dhcp server . Can we skip this or assign a static ip .Please guide on any options on the concern

[mkuzmin]

There is a mandatory requirement for VMware guest tools (open-vm-tools package) installed in a guest OS. Packer is a tool for automated VM template creation, not for deployment. I don't see a way how it could work without DHCP.

If your goal is to create new VM instances, it's more a task for Terraform.

[naveenkarunad]

Hi Mkuzmin, Thanks for your response. Yes, we are using packer for creating an image template, which we will use for deployment using Terraform. So, wile creating an image, is it mandatory to have DHCP configured in our environment ? Is there a way we pass the config separately (for example like cloud-init).

[mkuzmin]

You are right.

So we have two ways to configure a static IP address inside a VM:

1. vSphere VM customization specs
2. cloud-init

I still consider your use case very exotic, and want to hear a feedback from other customers before implementing special features in the tool.

VM customization is a complex technology. There is definitely a place for it in Terraform, but in Packer it adds a risk of additional bugs and support burden.

Cloud-init is more elegant, but it requires additional agent inside your VMs to read and apply this configuration. Also, there are multiple ways to pass cloud-config inside a VM. Will your implementation be able to read guestinfo properties like in Ubuntu and CoreOS?

[AranVinkItility]

Second this, it would really be helpful in cases where DHCP is not available.

If vmware-tools is installed in the base template the customization specs should take care of this. Simply adding it to the clone spec in govmomi should suffice as far as I can see. Maybe some code can be borrowed from the terraform-provider-vsphere? https://github.com/terraform-providers/terraform-provider-vsphere/blob/1d292f1f879f289f80a6bd0fc8c47691d93ac14a/vsphere/internal/vmworkflow/virtual_machine_customize_subresource.go#L503

[exbane]

adding in a section for "vm_customization_spec:" would be a huge help in a vSphere environment.

[tpoindessous]

Hi

yesterday, I had to turn down Packer for a customer because he was using fixed IP.

I would love to have an option to specify IP configuration, just like Terraform.

Thanks for your work anyway.

[mkuzmin]

Thank you for the feedback. I think I better understand the use case now. We at JetBrains are not going to work on this feature anytime soon, but a PR would be welcomed.

[sudomateo]

I think this is an invalid use case for the vsphere-iso builder. Template creation should be using a kickstart or preseed file for any installation customization.

For the vsphere-clone builder I can see a use case here, but just two or more clones of the VM template can lead to an IP conflict.

[ebal]

Hi,

I have also stepped upon this problem aka feature request , in a non dhcp enviroment (working mostly with static IPv4) and vmware clones . I was also thinking a cloud-init approach as I think is the "de-facto" approach with clone images.

In our lab, the vmware/govmomi work-around, was to use the 'guest.upload' https://github.com/vmware/govmomi/blob/master/govc/USAGE.md#guestupload to upload a file inside the VM and then using 'guest.start' to start it as a shell script.

Both (guest.upload & guest.start) doesnt need any extra package in the vmware template. And (in our case) solved the problem with network settings so that Ansible can provision the entire VM to fit our purposes.

I am not familiar with golang, so I cant help with the code. But I was thinking perhaps this approach is a lot easiest than the above terraform link.

(sorry for the long post)

[neduma]

@ebal Can you give more details about what files you passed to guest.upload and guest.start to setup the network/ip for the given vm clone so that it will have access to internet?

[mkuzmin]

I like the idea above using CentOS kikstart, Debian preseed, or Windows Autounattend files for this task.

[ebal]

@neduma a pre-configured file with shell commands that set:

• ip address
• gateway
• dns

Then guest.run executes the file, thus the commands inside it.

[michaelarichard]

I am working in a highly regulated/vendor managed environment that will not allow me to create a new vm and boot from preseed/iso. I am however, allowed to clone from a whitelisted template. This ability would allow me to layer onto this existing template which requires a static ip in our assigned network.

I am currently working around it with the above mentioned govc guest.upload steps and feel that adding this capability along with the ability to assign the attached networkname to vsphere-clone would be a much more elegant solution.

[mkuzmin]

Well, in clone-builder this makes more sense.

[exbane]

i agree - its better suited for clone-builder instead of iso-builder..

my use case is that that i use jenkins with packer to build our vsphere templates from iso.. then once they are complete the jenkins job will kick off another job that uses the clone-builder to build a VM off of the newly created template and runs inspec checks against the deployed VM.

Adding in the option for customization would be beneficial from this perspective as then i can have the system added to the domain automatically.. have all run_once options committed to the guest that are configured in the customization file.. have the system name changed based on the vm name that is called in the packer config file.

I'm in a 6.7 environment so this all would work even faster if the linked_clone option worked with 6.7 but there is a bug with that too.. but if both of these features were available then creating test vms off of the templates initially created by vsphere-iso will be much faster and easier.

food for thought.

[88lexd]

Hi,

I have also stepped upon this problem aka feature request , in a non dhcp enviroment (working mostly with static IPv4) and vmware clones . I was also thinking a cloud-init approach as I think is the "de-facto" approach with clone images.

In our lab, the vmware/govmomi work-around, was to use the 'guest.upload' https://github.com/vmware/govmomi/blob/master/govc/USAGE.md#guestupload to upload a file inside the VM and then using 'guest.start' to start it as a shell script.

Both (guest.upload & guest.start) doesnt need any extra package in the vmware template. And (in our case) solved the problem with network settings so that Ansible can provision the entire VM to fit our purposes.

I am not familiar with golang, so I cant help with the code. But I was thinking perhaps this approach is a lot easiest than the above terraform link.

(sorry for the long post)

How did you do this? I am having the same problem at my work place now, our environment does not have any DHCP available.. I am also trying to clone a base VM (includes VM tools and other agent software)...

"govc" seems like a separate cli tool and how can I use this along with Packer build? Is there a syntax which I can use within the "builder" section of the packer JSON file?

[mommys-little-hacker]

I would like to add that PCI-DSS (informational security standard for financial organizations) implies that every VM must have complete spec including static IP before setting up. This means that we have set up DHCP exclusively for creating templates, which is a waste of time. Adding ability to specify static ip for VM creation would benefit greatly.

[kclinden]

+1 for static ip capability.

[fatred]

I too have the problem described here, and for the benefit of others, my "workaround" with Ubuntu1804 has been to not use the clone module, but to use the jetbrains vsphere-iso for multiple templates, and just split my requirements between preseed.cfg for common things via a floppy mount, and the boot_command for extremely local things (so they can be variablised at packer runtime). I understand that there is a limitation to the amount of boot commands you can post, but I figured I needed maybe 10 elements being variable driven, and the rest can live in the preseed.cfg as static things. Thus far this has been ok for us.

The optimum process is to do the site specific customisation in that method to create a minimal base template, and then use a separate process to clone that base template and then ansible apps/features in before pushing back a more specific template. End result being maybe a base template, a postgres template and a kubernetes node template, all using that base template as a starting point. My deployment processes then pick up the appropriate template, and ansible just has to apply config to them.

What I have done instead is use the same vsphere-iso process to build all 3 templates from scratch using the iso/preseed/boot_command method above. This is undoubtely time consuming, but its still better than any alternative I can find.

preseed.cfg has all the things I want in all images (filesystem layout, localisation, packages I want etc). boot_command passes in preseed formatted things like the network settings, ntp, dns and critically for us, proxy settings.

To ensure no conflicts, the last step before shutdown/template is to remove the netplan yaml file.

I can then use terraform with the vm_customisation_spec to ensure each clone of that template has a static IP in, and other terraform modules handle things like DNS registration. So long as I don't try to run two packer iso build processes at the same time, I never have to worry about IP conflicts as I assigned an IPAM address for the packer build VM.

Example workflow here: https://github.com/fatred/packer-build-staticip

[youngd24]

Another +1 for static IP, in a heavily secured private cloud where DHCP is just not allowed but, despite trying to feed in a kickstart preseed it still waits on an IP (and giving it an IP in the preseed ks.cfg as well).

[fewknow]

+1 this is needed as there is no simple way to remove the static IP after the packer build is completed. I want to later deploy with terraform and use cloud init and pass in the static IP of my choice.

[jlarfors]

Wondering if this is helpful to anyone... but I got this working using vsphere-clone with the ubuntu 20.04 cloud images.

Note: i am using HCL instead of JSON. Documentation: https://www.packer.io/docs/builders/vmware/vsphere-clone#optional-2 Snippet:

  cd_files     = ["${path.root}/meta-data", "${path.root}/user-data", "${path.root}/network-config"]
  cd_label     = "cidata"
  remove_cdrom = true

The key file here is the network-config file, which took me many hours to find any documentation on... https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html

version: 2
ethernets:
  ens192:
    addresses: ....
    gateway4: ....
    nameservers:
      addresses: ....

With the original preseed approach I had this in my preseed when using vsphere-iso builder:

# Networking
d-i netcfg/disable_autoconfig boolean true
# IPv4 Setup
d-i netcfg/get_ipaddress string YOUR_IP_ADDRESS
d-i netcfg/get_netmask string 255.255.255.0
d-i netcfg/get_gateway string YOUR_IP_ADDRESS
d-i netcfg/get_nameservers string YOUR_IP_ADDRESS
d-i netcfg/confirm_static boolean true

The preseed was mounted via floppy which only works for Ubuntu 18.04 and earlier, hence why we moved to cloud-init and specifically nocloud using the cdrom with the label cidata as described above.

And then we created a guest customization specification for vsphere-clone: Documentation: https://www.packer.io/docs/builders/vmware/vsphere-clone#customization

UPDATE:

And I just solved the last piece of my puzzle which was to use the VMWare cloud-init data source for guestinfo thanks to the lovely person who wrote this comment: https://github.com/hashicorp/packer/issues/9310#issuecomment-635738095

This time using the metadata to specify the network configurations, NOT the network-config.

  configuration_parameters = {
    "guestinfo.metadata": base64encode(file("${path.root}/meta-data")),
    "guestinfo.metadata.encoding": "base64",
    "guestinfo.userdata": base64encode(file("${path.root}/user-data")),
    "guestinfo.userdata.encoding": "base64",
  }

meta-data:

#cloud-config
instance-id: some-id
local-hostname: some-host
network:
  version: 2
  ethernets:
    ens192:
      addresses: ...
      gateway4: ...
      nameservers:
        addresses: [...]