image-builder failed to run packer

[dborysenko]

What happened: Trying to build ubuntu image as described in https://anywhere.eks.amazonaws.com/docs/osmgmt/artifacts/#build-bare-metal-node-images

What you expected to happen: successful image build

How to reproduce it (as minimally and precisely as possible): Follow instruction to build image as per documentation: https://anywhere.eks.amazonaws.com/docs/osmgmt/artifacts/#build-bare-metal-node-images

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 (Ootpa)

$ image-builder build --os ubuntu --hypervisor baremetal --release-channel 1-27
Creating builder config
2023/10/12 19:42:53 No EKS-A release version provided, defaulting to EKS-A version configured at build time: v0.17.3
2023/10/12 19:42:53 Fetching git commit from bundle manifest: https://anywhere-assets.eks.amazonaws.com/releases/bundles/48/manifest.yaml
2023/10/12 19:42:53 Cloning eks-anywhere-build-tooling...
2023/10/12 19:42:53 Executing command: /usr/bin/bash -c git clone https://github.com/aws/eks-anywhere-build-tooling.git /home/image-builder/eks-anywhere-build-tooling
Cloning into '/home/image-builder/eks-anywhere-build-tooling'...

[....SKIPPED....]

Starting galaxy collection install process
Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using `--force`.
hack/ensure-packer.sh
+ _version=1.8.7
++ dirname hack/ensure-packer.sh
+ cd hack/..
+ source hack/utils.sh
++ case "${OSTYPE}" in
++ HOSTOS=linux
+++ uname -m
++ _hostarch=x86_64
++ case "${_hostarch}" in
++ HOSTARCH=amd64
+ exit 0
hack/ensure-goss.sh
+ source hack/utils.sh
++ case "${OSTYPE}" in
++ HOSTOS=linux
+++ uname -m
++ _hostarch=x86_64
++ case "${_hostarch}" in
++ HOSTARCH=amd64
+ _version=3.1.4
+ darwin_sha256=ddb663a3e4208639d90b89ebdb69dc240ec16d6b01877ccbf968f76a58a89f99
+ linux_sha256=9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9
+ _bin_url=https://github.com/YaleUniversity/packer-provisioner-goss/releases/download/v3.1.4/packer-provisioner-goss-v3.1.4-linux-amd64.tar.gz
+ _tarfile=/home/image-builder/.packer.d/plugins/packer-provisioner-goss.tar.gz
+ _binfile=/home/image-builder/.packer.d/plugins/packer-provisioner-goss
+ case "${HOSTOS}" in
+ _sha256=9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9
+ '[' -f /home/image-builder/.packer.d/plugins/packer-provisioner-goss ']'
++ get_shasum /home/image-builder/.packer.d/plugins/packer-provisioner-goss
++ local present_shasum=
++ command -v shasum
+++ shasum -a 256 /home/image-builder/.packer.d/plugins/packer-provisioner-goss
+++ awk '-F ' '{print $1}'
++ present_shasum=9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9
++ echo 9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9
+ current_shasum=9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9
+ '[' 9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9 '!=' 9084877c2eea7e41fae60aaa6cdf7a7dde4e5de5e3d1f693ec8e812419ac54e9 ']'
+ echo 'Right version of binary present'
Right version of binary present
+ '[' -x /home/image-builder/.packer.d/plugins/packer-provisioner-goss ']'
+ exit 0
/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/.local/bin/packer build -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/cni.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/containerd.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/wasm-shims.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/common.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/additional_components.json"  -force -color=true -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/raw/raw-ubuntu-2004-efi.json" -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2004/config/kubernetes.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2004/config/common.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2004/config/cni.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2004/config/additional_components.json"  packer/raw/packer.json
bash: /home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/.local/bin/packer: No such file or directory
make[1]: *** [Makefile:524: build-raw-ubuntu-2004-efi] Error 127
make[1]: Leaving directory '/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi'
make: *** [Makefile:353: local-build-raw-ubuntu-2004] Error 2
make: Leaving directory '/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder'
2023/10/12 19:43:14 Error executing image-builder for raw hypervisor: failed to run command: exit status 2

Anything else we need to know?:

Environment:

• EKS Anywhere Release: v0.17.3
• EKS Distro Release:

[davidcarrera]

Hi, I've hit exactly the same issue building images for a SBE device in this case when running from a rocky linux. Not sure why packer is not properly installed. Retried exactly the same process and configs from a Amazon Linux AMI and worked without problems.

[image-builder@f8f6d4a1-220e-4189-9ce4-488b83a1a221 ~]$ cat /etc/rocky-release Rocky Linux release 8.8 (Green Obsidian)

[image-builder@f8f6d4a1-220e-4189-9ce4-488b83a1a221 ~]$ cat build.sh image-builder build --os ubuntu --hypervisor ami --release-channel 1-26 --ami-config ami.json -v99

... /home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/.local/bin/packer init packer/config.pkr.hcl bash: /home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/.local/bin/packer: No such file or directory make[1]: *** [Makefile:68: deps-ami] Error 127 make[1]: Leaving directory '/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi' make: *** [Makefile:364: local-build-ami-ubuntu-2004] Error 2 make: Leaving directory '/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder' 2023/11/30 09:58:35 Error executing image-builder for AMI hypervisor: failed to run command: exit status 2

[wasifahmed-sudo]

I am facing the same issue as well. I am trying to build a node image on Nutanix and face the same issue. Have your issue been resolved?

[abhay-krishna]

@dborysenko @wasifahmed-sudo @davidcarrera What version of image-builder are you using?

$ image-builder version

[wasifahmed-sudo]

$ image-builder version v0.3.1

[rimaulana]

Hi you might have a default binary named packer in you host machine PATH that is not a real packer from hashicorp. This is an important excerpt from hack/ensure-packer.sh that if it is success it will echo something.

# Some Linux distributions such as Fedora, RHEL, CentOS have a tool
# called packer installed by default at /usr/sbin, which will pass the
# command check, but is not the Packer we need for image builds. So we
# need to check if the Packer executable present on the machine is not
# that one. The default packer tool provided by cracklib does not have a
# version command and hangs indefinitely when the version command is
# invoked, so we are timeboxing it to 10 seconds. This shouldn't be the
# case with Packer installed from Hashicorp releases, which should give
# us a version number. This helps us distinguish the two Packer executables.
if (command -v packer && timeout 10 packer version) >/dev/null 2>&1; then exit 0; fi

Make sure to not have other packer in your host machine PATH for image builder to execute properly.

[darthVikes]

I worked around this issue, I downloaded packer manually and put it into /usr/local/bin. The latest version of image builder looks for 1.9.5 for the packer version and will complain if it's newer. The hack above didn't also work for me and did hang there per this issue.