[BUG] Duplicate Keys when deploying Windows 10 Enterprise with Salt-cloud

[j-gerding]

Description I am attempting to automate the deployment of a Windows 10 Enterprise machine using Salt-Cloud on my (virtualized) Proxmox instance. The Proxmox VM and salt-minion will succesfully deploy and no error is returned.

However, when attempting to contact the new minion, it never returns an answer.

It does however register two keys, one of which automatically gets denied by the salt-master since the minion already exists.

Manually removing this first key and accepting the second key fixes the problem, but is not a viable solution (since I'm trying to automate this process).

Another workaround is using version 3003.3 for only my Windows 10 Enterprise machines in which the issue does not present itself. Although a decent workaround, it would be preferable to use 3004 or whatever latest version is available for all of my salt-minions.

If there's any other information that could help, please do let me know.

Setup Proxmox Version: 7.1-7 Salt-Version Master: 3004 PywinRM Version: 0.4.2 SMBProtocol Version: 1.8.3 Windows installer found here

• [ ] on-prem machine
• [X] VM (In Proxmox, on local machine using QEMU)
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD

Steps to Reproduce the behavior Using the following setup with salt-cloud -m /etc/salt/cloud.maps.d/inframap.conf windows.example.com The machine is cloned from a template created with Packer. In this template I preinstall Proxmox/Virtio related drivers, the Windows 10 OS, enabling WinRM/SMB, their ports and setting a static ip.

salt/cloud.providers.d/proxmox.conf

---
proxmox:
  minion:
    master: 192.168.122.250
  user: exampleuser@pam
  password: examplepass
  url: 192.168.122.3
  driver: proxmox
  verify_ssl: False

salt/cloud.profiles.d/proxmox.conf

#Removed other profiles for brevity
proxmox-windows:
  provider: proxmox
  image: local:910/base-910-disk-0.raw

  clone: True

  clone_description: "Windows 10 Enterprise Machine"

  clone_format: qcow2

  clone_full: 1

  clone_from: 910

  technology: qemu

  host: proxmox

  ip_address: 192.168.122.240

  win_username: Administrator
  win_password: examplepass
 #This .exe is not available by default on the salt-master.
  win_installer: /root/Salt-Minion-3004-2-Py3-AMD64-Setup.exe
  use_winrm: True
  winrm_verify_ssl: False

  onboot: 1
  agent: 1
  ostype: win10

This inframap isn't strictly neccesary but included for completion's sake. Simply using the profile with salt-cloud generates the same issue.

salt/cloud.maps.d/inframap.conf

#Removed other machines for brevity
proxmox-windows:
  - window.example.com:
      clone_name: window.example.com

Expected behavior The Windows machine deploys and registers the correct key with the designated salt-master.

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

Salt Version:
          Salt: 3004
 
Dependency Versions:
          cffi: 1.11.5
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: 0.26.8
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 0.26.4
        Python: 3.6.8 (default, Nov  9 2021, 14:44:26)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 8.5 Green Obsidian
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-348.2.1.el8_5.x86_64
        system: Linux
       version: Rocky Linux 8.5 Green Obsidian

Additional context When using version 3003.3 for the Windows machine, the issue is not present. When attempting the above setup using pypsexec and smbprotocol, only the faulty key is registered to the salt-master. Manually removing this key and restarting the salt-minion does create a functioning environment. However, this is also not a viable solution when attempting to automate this process.

I'm attempting to find the code handling this, however since there is no error being thrown I am having a hard time tracking it down.

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[doesitblend]

@j-gerding Can you tell us more about the state that the Salt minion is in when this issue is happening? I believe I am seeing this same issue. Specifically, is the minion stuck in a "paused" state? Also, if there is any minion log that may also be helpful.

I believe there should be an installer log generated as well, but the location may be specified at installer execution time. Or, depending on which installer you are using, you can can have an installer log created with the /i option.

[garethgreenaway]

Closing this one out due to lack of activity.

[shadoutmapes]

I have this issue with windows 2022 as well alt Version: Salt: 3005.1

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.6 gitpython: 3.0.7 Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: 1.4.4 pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: 3.6.1 pygit2: Not Installed Python: 3.8.10 (default, Jun 22 2022, 20:18:18) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 20.0.0 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2

System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-125-generic system: Linux version: Ubuntu 20.04 focal

[shadoutmapes]

Using win_installer: /srv/salt/dist/minion/Salt-Minion-3004.2-1-Py3-AMD64-Setup.exe in my cloud.profiles file, with winrm presetup in the windows vm. I'm using salt-cloud to vcenter 7

[shadoutmapes]

Full Debug of deployment attached debug.pdf

[shadoutmapes]

to add looks like the key is denied during the customization of the cloned vm. It boots as a generic name winraw and the customization kicks in to set it to the right host name and install the minion, while its waiting for the system to reboot to deploy the mininon the key is already denied

[shadoutmapes]

Seems like the process order is initiate clone of vm template, this pre-creates the key and has it accepted in the saltmaster, then the machine clones, the key is fine, the machine comes up and is customized, the problem doesn't seem to happen til the end when the minion is actually deployed via the profile when the minion starts it isn't using the same cert I'm guessing that is generated in the beginning.

For anyone with this issue, I'm solving it out of band by wrapping the deploy command in a script and deleting the pre made key and accepting the key when the cloud task finishes. Not ideal but its working. Still would like to know how to make this work without the work around.

[Ch3LL]

@shadoutmapes just a clarification question, do you have the minion key pre-generated on the image you are cloning?

[Ch3LL]

Also @shadoutmapes can you share your provider/profile files with the important information obfuscated?

[garethgreenaway]

Possibly fixed by https://github.com/saltstack/salt/pull/63030

[twangboy]

Starting in Salt version 3004 we allowed the user to install salt anywhere and defaulted to the Program Files directory. This also moved config data to the ProgramData directory. The salt-cloud code was still dropping config, keys, etc in the old location in C:\salt. This was causing the duplicate key issue. The above PR now places config, keys, etc. from salt-cloud in the ProgramData directory. It should fix the issue.

[shadoutmapes]

No the image has no existing minion. I do have a ticket open with vmware and they are able to recreate this in the lab. The salt-cloud pre makes a key pair for the minion but when it does the install via the win_installer tag in the profile it doesn't set the previously made key/pair the same for the minion. one other note i'm using the .exe instead of the msi (its not sighned and we have security requirments I would have to get around to use an unsighed msi)

---

From: Megan Wilhite @.> Sent: Tuesday, November 15, 2022 12:03 PM To: saltstack/salt @.> Cc: Briney, Chris @.>; Mention @.> Subject: [EXTERNAL] Re: [saltstack/salt] [BUG] Duplicate Keys when deploying Windows 10 Enterprise with Salt-cloud (Issue #61328)

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

@shadoutmapeshttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fshadoutmapes&data=05%7C01%7Cchris.briney%40eku.edu%7C8f930ec87c8444a685e408dac72b5b61%7Ce23043271af04dee83fbc1b2fd6db0bb%7C0%7C0%7C638041286274356580%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jBae11dW3RsGdUirT9AEcu5QhfZcDeMWzQdraP0lTeo%3D&reserved=0 just a clarification question, do you have the minion key pre-generated on the image you are cloning?

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsaltstack%2Fsalt%2Fissues%2F61328%23issuecomment-1315611766&data=05%7C01%7Cchris.briney%40eku.edu%7C8f930ec87c8444a685e408dac72b5b61%7Ce23043271af04dee83fbc1b2fd6db0bb%7C0%7C0%7C638041286274356580%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LhfcW2MQRlI8TkvssMNTGPwtuFS8sviCWRYvd2iAtQg%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAWXXGEIS7YBPM4M57UAHO7TWIO67DANCNFSM5JO3NPYA&data=05%7C01%7Cchris.briney%40eku.edu%7C8f930ec87c8444a685e408dac72b5b61%7Ce23043271af04dee83fbc1b2fd6db0bb%7C0%7C0%7C638041286274356580%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f3M9h49UefdHbmPl3e9Le29EqMA7EU0cOLEaYsePfo0%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[Ch3LL]

Thanks for the additional information. I verified with the individual in vmware that was able to replicate the issue that the fix in #63030 did resolve the issue for them. This will be available in 3006.