heredoc difference buildkit vs buildah - variable retention

[shoffmeister]

Description When using a variable in a heredoc, buildkit and buildah have different behaviour:

buildkit will retain the variable, buildah will not.

Steps to reproduce the issue:

Use the following Containerfile

FROM bash

ARG MY_ARG=wombat

RUN <<EOF
set -euo pipefail

tempvar=$MY_ARG

if [ $tempvar == "wombat" ]; then
  exit 0
else
  exit 99
fi
EOF

and observe the following results:

• docker build . -f Containerfile -> build succeeds
• buildah build . -> build fails with error

sh: wombat: unknown operand
Error: building at STEP "RUN <<EOF": while running runtime: exit status 99

Describe the results you expected:

Matching behaviour across docker build and buildah build

In particular for a heredoc, the ability to compute a temporary result once and then reuse it later is very useful - for instance downloading a specific file, then validating the signature and/or the checksum. docker build allows that.

Docker version in use:

❯ docker --version
Docker version 25.0.4, build 1a576c50a9

(which is larger than 23.0, i.e. buildx / buildkit is in use)

Output of rpm -q buildah or apt list buildah:

❯ pacman -Ss buildah
extra/buildah 1.35.0-1 [installed]
    A tool which facilitates building OCI images

Output of buildah version:

Version:         1.35.0
Go Version:      go1.22.1
Image Spec:      1.1.0
Runtime Spec:    1.1.0
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.30.0
Git Commit:      fedbd79676e8aa5dc49d9434f6d9361b5e00ba62
Built:           Wed Mar 13 16:50:38 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of cat /etc/*release:

DISTRIB_ID="EndeavourOS"
DISTRIB_RELEASE="rolling"
DISTRIB_DESCRIPTION="EndeavourOS Linux"
DISTRIB_CODENAME="rolling"
NAME="EndeavourOS"
PRETTY_NAME="EndeavourOS"
ID="endeavouros"
ID_LIKE="arch"
BUILD_ID="2023.08.05"
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://endeavouros.com"
DOCUMENTATION_URL="https://discovery.endeavouros.com"
SUPPORT_URL="https://forum.endeavouros.com"
BUG_REPORT_URL="https://forum.endeavouros.com/c/arch-based-related-questions/bug-reports"
PRIVACY_POLICY_URL="https://endeavouros.com/privacy-policy-2"
LOGO="endeavouros"

Output of uname -a:

Linux arch 6.8.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 16 Mar 2024 17:15:35 +0000 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

n/a

[shoffmeister]

For reference I have pasted a more useful heredoc below. This hopefully makes the point that temporary variables - as supported by buildx / buildkit - are quite a useful thing to have.

Note that this example abuses an ARG to create a temporary variable, and to work around the problem described above. I'd love to simply compute this inside the heredoc.

ARG __ORIGINAL_DOWNLOAD_FILENAME=kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz

RUN <<EOF
set -euo pipefail

mkdir -p /opt

curl -fL https://downloads.apache.org/kafka/$KAFKA_VERSION/$__ORIGINAL_DOWNLOAD_FILENAME \
  -o $__ORIGINAL_DOWNLOAD_FILENAME

#
# NOTE:
# Deliberately breaking in case we switch to a different version of Kafka
#
cat > $__ORIGINAL_DOWNLOAD_FILENAME.sha512 <<SHA512
kafka_2.13-3.7.0.tgz: B8679283 A2D8DAB8 6E7C636B 2C688FE9 D9E64AC4 37241F65
                      EF7A1733 F4D26A2B D415EEFA 04F09F19 11373BCD 2A5DBC38
                      38C76347 F6865642 5C09202C D290CE91
SHA512

GNUPGHOME=$(mktemp -d) gpg --print-md SHA512 $__ORIGINAL_DOWNLOAD_FILENAME \
  | diff - $__ORIGINAL_DOWNLOAD_FILENAME.sha512

tar --directory /opt --extract --gzip --file $__ORIGINAL_DOWNLOAD_FILENAME
EOF

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.