packer-plugin-tart icon indicating copy to clipboard operation
packer-plugin-tart copied to clipboard

Published 20 hours ago • verified publisher icon cirruslabs

Occasional Timeout waiting for SSH

Open sanieldalib opened this issue 1 year ago • 21 comments

Hey there! Thanks for all your great work on this!

I sometimes get the following error when building an image from IPSW. My ssh_timeout is set to 180s, and I'm killing the tart process before beginning the build to ensure its not a result of too many VMs running (>2).

==> tart-cli.base: Waiting for SSH to become available...
==> tart-cli.base: Timeout waiting for SSH.

Here is what my source block looks like, its essentially identical to the templates provided in https://github.com/cirruslabs/macos-image-templates/blob/master/templates/vanilla-ventura.pkr.hcl

source "tart-cli" "base" {
  # You can find macOS IPSW URLs on various websites like https://ipsw.me/
  # and https://www.theiphonewiki.com/wiki/Beta_Firmware/Mac/13.x
  from_ipsw    = "${var.home}/macOS/UniversalMac_13.3_22E252_Restore.ipsw"
  vm_name      = var.vm_name
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 60
  ssh_username = "admin"
  ssh_password = "admin"
  ssh_timeout  = "180s"
  boot_command = [
    # hello, hola, bonjour, etc.
    "<wait60s><spacebar>",
    # Language
    "<wait30s>english<enter>",
    # Select Your Country and Region
    "<wait30s>united states<leftShiftOn><tab><leftShiftOff><spacebar>",
    # Written and Spoken Languages
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Accessibility
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Data & Privacy
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Migration Assistant
    "<wait10s><tab><tab><tab><spacebar>",
    # Sign In with Your Apple ID
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you want to skip signing in with an Apple ID?
    "<wait10s><tab><spacebar>",
    # Terms and Conditions
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # I have read and agree to the macOS Software License Agreement
    "<wait10s><tab><spacebar>",
    # Create a Computer Account
    "<wait10s>admin<tab><tab>admin<tab>admin<tab><tab><tab><spacebar>",
    # Enable Location Services
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you don't want to use Location Services?
    "<wait10s><tab><spacebar>",
    # Select Your Time Zone
    "<wait10s><tab>UTC<enter><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Analytics
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Screen Time
    "<wait10s><tab><spacebar>",
    # Siri
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Choose Your Look
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Enable Voice Over
    "<wait10s><leftAltOn><f5><leftAltOff><wait5s>v",
    # Now that the installation is done, open "System Settings"
    "<wait10s><leftAltOn><spacebar><leftAltOff>System Settings<enter>",
    # Navigate to "Sharing"
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",
    # Navigate to "Screen Sharing" and enable it
    "<wait10s><tab><down><spacebar>",
    # Navigate to "Remote Login" and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><spacebar>",
    # Open "Remote Login" details
    "<wait10s><tab><spacebar>",
    # Enable "Full Disk Access"
    "<wait10s><tab><spacebar>",
    # Click "Done"
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Disable Voice Over
    "<leftAltOn><f5><leftAltOff>",
  ]

  // A (hopefully) temporary workaround for Virtualization.Framework's
  // installation process not fully finishing in a timely manner
  create_grace_time = "30s"

sanieldalib avatar Jun 29 '23 20:06 sanieldalib

Do you observe any failures during the installation process of the macOS? This might be related to boot_command not successfully doing the installation so your VM ends up without a user configured to be SSHed in via.

fkorotkov avatar Jun 30 '23 15:06 fkorotkov

My vanilla script seems to be timing out as well.

    sonoma-vanilla.tart-cli.tart: 85%
    sonoma-vanilla.tart-cli.tart: 86%
    sonoma-vanilla.tart-cli.tart: 87%
    sonoma-vanilla.tart-cli.tart: 90%
    sonoma-vanilla.tart-cli.tart: 100%
==> sonoma-vanilla.tart-cli.tart: Waiting 30s to let the Virtualization.Framework's installation process to finish correctly...
==> sonoma-vanilla.tart-cli.tart: Updating virtual machine resources...
==> sonoma-vanilla.tart-cli.tart: Inspecting machine disk image...
==> sonoma-vanilla.tart-cli.tart: Getting partition table...
==> sonoma-vanilla.tart-cli.tart: Found a recovery partition at GPT entry 3, let's remove it to save space and allow for resizing the main partition...
==> sonoma-vanilla.tart-cli.tart: Successfully updated partitions!
==> sonoma-vanilla.tart-cli.tart: Starting the virtual machine...
==> sonoma-vanilla.tart-cli.tart: Waiting for the VNC server credentials from Tart...
==> sonoma-vanilla.tart-cli.tart: Retrieved VNC credentials, connecting...
    sonoma-vanilla.tart-cli.tart: If you want to view the screen of the VM, connect via VNC with the password "echo-family-amount-copy" to
    sonoma-vanilla.tart-cli.tart: vnc://127.0.0.1:62266
==> sonoma-vanilla.tart-cli.tart: Connected to the VNC!
==> sonoma-vanilla.tart-cli.tart: Typing the commands over VNC...
==> sonoma-vanilla.tart-cli.tart: Successfully started the virtual machine...
==> sonoma-vanilla.tart-cli.tart: Using SSH communicator to connect: 192.168.64.2
==> sonoma-vanilla.tart-cli.tart: Waiting for SSH to become available...
==> sonoma-vanilla.tart-cli.tart: Timeout waiting for SSH.
==> sonoma-vanilla.tart-cli.tart: Waiting for the tart process to exit..

No errors that I can see.

sridhar-vadlamani-ck avatar Jul 01 '24 16:07 sridhar-vadlamani-ck

@sridhar-vadlamani-ck how are you invoking the Packer, from CI, or from a user-interactive Terminal?

Also this might be related to the DHCP shortage, please check out https://tart.run/faq/#changing-the-default-dhcp-lease-time.

edigaryev avatar Jul 01 '24 16:07 edigaryev

It's being invoked in CI, from a non-interactive terminal. $TERM is set to xterm-256color but I haven't passed in any additional flags to the packer script. Are there any additional settings that would be needed to streamline building vanilla vms in CI?

This is my source:


source "tart-cli" "tart" {
  from_ipsw    = "${var.ipsw}"
  vm_name      = "vanilla"
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 40
  ssh_password = "admin"
  ssh_username = "admin"
  ssh_timeout  = "120s"
  boot_command = [
    # hello, hola, bonjour, etc.
    "<wait60s><spacebar>",
    # Language: most of the times we have a list of "English"[1], "English (UK)", etc. with
    # "English" language already selected. If we type "english", it'll cause us to switch
    # to the "English (UK)", which is not what we want. To solve this, we switch to some other
    # language first, e.g. "Italiano" and then switch back to "English". We'll then jump to the
    # first entry in a list of "english"-prefixed items, which will be "English".
    #
    # [1]: should be named "English (US)", but oh well 🤷
    "<wait30s>italiano<esc>english<enter>",
    # Select Your Country and Region
    "<wait30s>united states<leftShiftOn><tab><leftShiftOff><spacebar>",
    # Written and Spoken Languages
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Accessibility
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Data & Privacy
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Migration Assistant
    "<wait10s><tab><tab><tab><spacebar>",
    # Sign In with Your Apple ID
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you want to skip signing in with an Apple ID?
    "<wait10s><tab><spacebar>",
    # Terms and Conditions
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # I have read and agree to the macOS Software License Agreement
    "<wait10s><tab><spacebar>",
    # Create a Computer Account
    "<wait10s>admin<tab><tab>admin<tab>admin<tab><tab><tab><spacebar>",
    # Enable Location Services
    "<wait30s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you don't want to use Location Services?
    "<wait10s><tab><spacebar>",
    # Select Your Time Zone
    "<wait10s><tab>UTC<enter><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Analytics
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Screen Time
    "<wait10s><tab><spacebar>",
    # Siri
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Choose Your Look
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Enable Voice Over
    "<wait10s><leftAltOn><f5><leftAltOff><wait5s>v",
    # Now that the installation is done, open "System Settings"
    "<wait10s><leftAltOn><spacebar><leftAltOff>System Settings<enter>",
    # Navigate to "Sharing"
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",
    # Navigate to "Screen Sharing" and enable it
    "<wait10s><tab><tab><tab><tab><tab><spacebar>",
    # Navigate to "Remote Login" and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><spacebar>",
    # Disable Voice Over
    "<leftAltOn><f5><leftAltOff>",
  ]

  // A (hopefully) temporary workaround for Virtualization.Framework's
  // installation process not fully finishing in a timely manner
  create_grace_time = "30s"
}

Also, it's not the DHCP shortage. Timeout is set to 10mins and the vanilla vm build was basically the first one on these machines. I'm still building it out.

sridhar-vadlamani-ck avatar Jul 01 '24 18:07 sridhar-vadlamani-ck

@sridhar-vadlamani-ck are you trying macOS Sequia Beta 2 host by any chance? We've see this issue on Sequia hosts.

fkorotkov avatar Jul 01 '24 19:07 fkorotkov

@fkorotkov I'm building from a sonoma ipsw

sridhar-vadlamani-ck avatar Jul 01 '24 19:07 sridhar-vadlamani-ck

But what about the host? Just checking this. But please also check @edigaryev's tip about DHCP.

fkorotkov avatar Jul 01 '24 20:07 fkorotkov

The host is also a sonoma instance.

% sw_vers
ProductName:            macOS
ProductVersion:         14.5
BuildVersion:           23F79

These are freshly launched instances with the DHCP lease time set to 10mins per the FAQ already.

sridhar-vadlamani-ck avatar Jul 01 '24 20:07 sridhar-vadlamani-ck

I re-ran it a bunch of times, and of about 12 runs, one has succeeded. Everything else seems to fail with the SSH connection issue.

sridhar-vadlamani-ck avatar Jul 01 '24 20:07 sridhar-vadlamani-ck

Is there anything else you'd like me to rule out?

sridhar-vadlamani-ck avatar Jul 03 '24 08:07 sridhar-vadlamani-ck

Is there anything else you'd like me to rule out?

Can you connect to the VM using the ssh command after Packer emits the Using SSH communicator to connect: <IP> message?

edigaryev avatar Jul 03 '24 08:07 edigaryev 

% ssh -v [email protected]
OpenSSH_9.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.64.12 [192.168.64.12] port 22.
debug1: connect to address 192.168.64.12 port 22: Connection refused
ssh: connect to host 192.168.64.12 port 22: Connection refused

==> sonoma-vanilla.tart-cli.ipsw: Connected to the VNC!
==> sonoma-vanilla.tart-cli.ipsw: Typing the commands over VNC...
==> sonoma-vanilla.tart-cli.ipsw: Successfully started the virtual machine...
==> sonoma-vanilla.tart-cli.ipsw: Using SSH communicator to connect: 192.168.64.12
==> sonoma-vanilla.tart-cli.ipsw: Waiting for SSH to become available...

sridhar-vadlamani-ck avatar Jul 04 '24 08:07 sridhar-vadlamani-ck

The SSH issue is only happening when I build the vanilla image. If I build from another image directly, there are no issues.

sridhar-vadlamani-ck avatar Jul 05 '24 18:07 sridhar-vadlamani-ck

The SSH issue is only happening when I build the vanilla image. If I build from another image directly, there are no issues.

This most likely has to do with Remote Login not being enabled in System Settings → Sharing from some reason in the guest.

Make sure that your boot_command sequence is correct for the IPSW you're using by observing how it interacts with the installer/macOS UI.

edigaryev avatar Jul 07 '24 08:07 edigaryev