packer-builder-arm
packer-builder-arm copied to clipboard
Published
20 hours ago •
mkaczanowski
mkaczanowski
Ansible provisioning never returns
Overview of the Issue
I'm creating a ubuntu 21.10 raspberry pi image using ansible 2.9.6. My ansible playbook is very simple (so far), but it hangs when it tries to run echo ~root && sleep 0, which, AFAICT, ansible runs to test the connection. When I try to connect via ssh to the packer ssh communicator and run a command, the command output returns but the connection never does, which is what I think ansible is waiting on.
Reproduction Steps
% sudo PACKER_DEBUG=1 packer build packer/ubuntu_server_21.10_arm64.json.pkr.hcl
Packer version
v1.7.8
Simplified Packer Buildfile
source "arm" "raspberry_pi_k8s" {
file_checksum_type = "sha256"
file_checksum_url = "http://cdimage.ubuntu.com/releases/21.10/release/SHA256SUMS"
file_target_extension = "xz"
file_unarchive_cmd = ["xz", "--decompress", "$ARCHIVE_PATH"]
file_urls = ["http://cdimage.ubuntu.com/releases/21.10/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz"]
image_build_method = "reuse"
image_chroot_env = ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"]
image_partitions {
filesystem = "fat"
mountpoint = "/boot/firmware"
name = "boot"
size = "256M"
start_sector = "2048"
type = "c"
}
image_partitions {
filesystem = "ext4"
mountpoint = "/"
name = "root"
size = "2.8G"
start_sector = "526336"
type = "83"
}
image_path = "ubuntu-21.10.img"
image_size = "3.1G"
image_type = "dos"
qemu_binary_destination_path = "/usr/bin/qemu-aarch64-static"
qemu_binary_source_path = "/usr/bin/qemu-aarch64-static"
}
build {
sources = ["source.arm.raspberry_pi_k8s"]
provisioner "ansible" {
playbook_file = "ansible/playbook.yml"
extra_arguments = [ "-vvvv" ]
}
}
Ansible Playbook
---
- name: 'Provision image'
hosts: default
become: true
gather_facts: no
vars_files:
- vars.yml
tasks:
- name: Debug
debug:
msg: "Testing 1 2 3"
- name: Create user group
group:
name: "{{ username }}"
state: present
Operating system and Environment details
Ubuntu 20.04 on x86_64
Log Fragments and crash.log files
Output of manually running ssh command
% sudo ssh -v 127.0.0.1 -o Port=34669 -o 'IdentityFile="/tmp/ansible-key544648103"' 'pwd'
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 34669.
debug1: Connection established.
debug1: identity file /tmp/ansible-key544648103 type -1
debug1: identity file /tmp/ansible-key544648103-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug1: Authenticating to 127.0.0.1:34669 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:9ZU07UhwYA8vXs1g6QILOxwnzOYM4peD+AX1ZoB1tks
debug1: Host '[127.0.0.1]:34669' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:16
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /tmp/ansible-key544648103 explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/ansible-key544648103
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:34669).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: pwd
/
<Command hangs here>
^Cdebug1: channel 0: free: client-session, nchannels 1
Killed by signal 2.
Same here, here is the Packer output:
==> arm.recovery: Retrieving rootfs_archive
==> arm.recovery: Trying https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip?archive=false
==> arm.recovery: Trying https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspios_lite_arm64%2Fimages%2Fraspios_lite_arm64-2022-01-28%2F2022-01-28-raspios-bullseye-arm64-lite.zip.sha256
==> arm.recovery: https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspios_lite_arm64%2Fimages%2Fraspios_lite_arm64-2022-01-28%2F2022-01-28-raspios-bullseye-arm64-lite.zip.sha256 => /root/.cache/packer/258574cfe2c5e2a35c3bbc9affc3e113d9d98e63.zip
arm.recovery: unpacking /root/.cache/packer/258574cfe2c5e2a35c3bbc9affc3e113d9d98e63.zip to recovery.img
arm.recovery: resizing the image file recovery.img to 6G
arm.recovery: expanding partition no. 2 on the image recovery.img
arm.recovery: mapping image recovery.img to free loopback device
arm.recovery: image recovery.img mapped to /dev/loop2
arm.recovery: running resize2fs on /dev/loop2p2
arm.recovery: mounting /dev/loop2p2 to /tmp/3188529091
arm.recovery: mounting /dev/loop2p1 to /tmp/3188529091/boot
arm.recovery: running extra setup
arm.recovery: mounting /dev with: [mount --bind /dev /tmp/3188529091/dev]
arm.recovery: mounting /devpts with: [mount -t devpts /devpts /tmp/3188529091/dev/pts]
arm.recovery: mounting proc with: [mount -t proc proc /tmp/3188529091/proc]
arm.recovery: mounting binfmt_misc with: [mount -t binfmt_misc binfmt_misc /tmp/3188529091/proc/sys/fs/binfmt_misc]
arm.recovery: mounting sysfs with: [mount -t sysfs sysfs /tmp/3188529091/sys]
arm.recovery: running the provision hook
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell576595168
arm.recovery: [all]
arm.recovery: dtoverlay=vc4-kms-v3d
arm.recovery: max_framebuffers=1
arm.recovery: arm_64bit=1
arm.recovery: over_voltage=8
arm.recovery: arm_freq=2100
arm.recovery: gpu_freq=650
arm.recovery: enable_uart=1
arm.recovery: start_x=1
arm.recovery: gpu_mem=256
arm.recovery: initial_turbo=60
arm.recovery: gpio=14=op,dl
arm.recovery: dtoverlay=disable-wifi
arm.recovery: dtoverlay=gpio-fan,gpiopin=21,temp=65000
arm.recovery: force_turbo=1
arm.recovery: initial_turbo=60
arm.recovery: avoid_warnings=2
arm.recovery: dtparam=i2c_arm=on
arm.recovery: dtparam=spi=on
arm.recovery: dtoverlay=gpio-ir,gpio_pin=26
arm.recovery: dtoverlay=gpio-ir-tx,gpio_pin=19
arm.recovery: avoid_warnings=2
arm.recovery: disable_splash=1
arm.recovery:
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell440055694
arm.recovery: console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell292882634
arm.recovery: en_US.UTF-8 UTF-8
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell857059794
arm.recovery: deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian bullseye test
arm.recovery:
==> arm.recovery: Provisioning with Ansible...
arm.recovery: Setting up proxy adapter for Ansible....
==> arm.recovery: Executing Ansible: ansible-playbook -e packer_build_name="recovery" -e packer_builder_type=arm -vvvv -e ansible_ssh_private_key_file=/tmp/ansible-key042511905 -i /tmp/packer-provisioner-ansible556372748 /build/ansible/recovery.yml
arm.recovery: ansible-playbook 2.9.6
arm.recovery: config file = /etc/ansible/ansible.cfg
arm.recovery: configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
arm.recovery: ansible python module location = /usr/lib/python3/dist-packages/ansible
arm.recovery: executable location = /usr/bin/ansible-playbook
arm.recovery: python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
arm.recovery: Using /etc/ansible/ansible.cfg as config file
arm.recovery: setting up inventory plugins
arm.recovery: host_list declined parsing /tmp/packer-provisioner-ansible556372748 as it did not pass its verify_file() method
arm.recovery: script declined parsing /tmp/packer-provisioner-ansible556372748 as it did not pass its verify_file() method
arm.recovery: auto declined parsing /tmp/packer-provisioner-ansible556372748 as it did not pass its verify_file() method
arm.recovery: Parsed /tmp/packer-provisioner-ansible556372748 inventory source with ini plugin
arm.recovery: Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3/dist-packages/ansible/plugins/callback/default.py
arm.recovery:
arm.recovery: PLAYBOOK: recovery.yml *********************************************************
arm.recovery: Positional arguments: /build/ansible/recovery.yml
arm.recovery: verbosity: 4
arm.recovery: connection: smart
arm.recovery: timeout: 10
arm.recovery: become_method: sudo
arm.recovery: tags: ('all',)
arm.recovery: inventory: ('/tmp/packer-provisioner-ansible556372748',)
arm.recovery: extra_vars: ('packer_build_name="recovery"', 'packer_builder_type=arm', 'ansible_ssh_private_key_file=/tmp/ansible-key042511905')
arm.recovery: forks: 5
arm.recovery: 1 plays in /build/ansible/recovery.yml
arm.recovery:
arm.recovery: PLAY [default] *****************************************************************
arm.recovery: META: ran handlers
arm.recovery:
arm.recovery: TASK [Add repositories] ********************************************************
arm.recovery: task path: /build/ansible/recovery.yml:6
arm.recovery: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
arm.recovery: <127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=45941 -o 'IdentityFile="/tmp/ansible-key042511905"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d901886dc9 127.0.0.1 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
And same conclusion as @jacklund, when running the SSH command from the container, the command returns the output but I have to hit a key to exit.
root@46c5f7adc081:/tmp# ssh -v 127.0.0.1 -o Port=45941 -o 'IdentityFile="/tmp/ansible-key042511905"' 'ip a'
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 45941.
debug1: Connection established.
debug1: identity file /tmp/ansible-key042511905 type -1
debug1: identity file /tmp/ansible-key042511905-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug1: Authenticating to 127.0.0.1:45941 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:mSYV+OZHs6abq3PkdEQdsZbqyfDCvFAzPtGnTOdvRSA
debug1: Host '[127.0.0.1]:45941' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /tmp/ansible-key042511905 explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/ansible-key042511905
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:45941).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending command: ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
32: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2424, received 1892 bytes, in 3.5 seconds
Bytes per second: sent 696.0, received 543.3
debug1: Exit status 0
