matlab-dockerfile
matlab-dockerfile copied to clipboard
Published
20 hours ago •
mathworks-ref-arch
mathworks-ref-arch
issue passing iso file using source argument using kaniko approach
Just like in #95 we try to create a matlab docker image with some toolboxes on a corporate GitLab instance without privileged rights. Thus we use kaniko. This time, we try to reference the installation media as iso file, as downloaded from MathWorks.com for R2022b.
We try to reference the media as indicated by the manual using the --source flag for mpm.
Unfortunately, installation fails with Error: Unable to find the source folder. Specify a valid source folder.
We tried passing the path both with --source=/tmp/matlab.iso as well as --source /tmp/matlab.iso, as documentation is fully not clear about how to provide arguments. (release is passed with = in the example of section "Install Products" but not in the section "Product Installation Options")
We ensure that the file matlab.iso is present at /tmp/matlab.iso by issuing command ls -la within the pipeline
According to the docu,mentation for R2021b and later releases a ISO image is fine.
.gitlab-ci.yml
stages:
- build
build:
stage: build
when: manual
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- cp /R2022b_Update_7_Linux.iso "${CI_PROJECT_DIR}"
- cat /etc/gitlab-runner/certs/<redacted>.crt >> /kaniko/ssl/certs/ca-certificates.crt
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--build-arg http_proxy="${http_proxy}"
--build-arg https_proxy="${https_proxy}"
--build-arg HTTP_PROXY="${http_proxy}"
--build-arg HTTPS_PROXY="${https_proxy}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
Dockerfile
# Copyright 2019 - 2023 The MathWorks, Inc.
# This Dockerfile allows you to build a Docker® image with MATLAB® installed using the MATLAB Package
# Manager. Use the optional build arguments to customize the version of MATLAB, list of products to
# install, and the location at which to install MATLAB.
# Here is an example docker build command with the optional build arguments.
# docker build --build-arg MATLAB_RELEASE=r2023b
# --build-arg MATLAB_PRODUCT_LIST="MATLAB Deep_Learning_Toolbox Symbolic_Math_Toolbox"
# --build-arg MATLAB_INSTALL_LOCATION="/opt/matlab/R2023b"
# --build-arg [email protected]
# -t my_matlab_image_name .
# To specify which MATLAB release to install in the container, edit the value of the MATLAB_RELEASE argument.
# Use lowercase to specify the release, for example: ARG MATLAB_RELEASE=r2021b
ARG MATLAB_RELEASE=r2022b
# Specify the list of products to install into MATLAB.
ARG MATLAB_PRODUCT_LIST="MATLAB"
# We will use an input file with a list of products instead
ARG MPM_INPUT_FILE="mpm-input-files/R2022b/mpm_input_r2022b.txt"
# Specify MATLAB Install Location.
# Has to match the location in the input file
ARG MATLAB_INSTALL_LOCATION="/opt/matlab/${MATLAB_RELEASE}"
ARG MATLAB_ISO="/R2022b_Update_7_Linux.iso"
# Specify license server information using the format: port@hostname
# ARG LICENSE_SERVER
ARG LICENSE_FILE="license/20231207_license.lic"
# When you start the build stage, this Dockerfile by default uses the Ubuntu-based matlab-deps image.
# To check the available matlab-deps images, see: https://hub.docker.com/r/mathworks/matlab-deps
FROM mathworks/matlab-deps:${MATLAB_RELEASE}
# Declare build arguments to use at the current build stage.
ARG MATLAB_RELEASE
ARG MATLAB_PRODUCT_LIST
ARG MATLAB_INSTALL_LOCATION
#ARG LICENSE_SERVER
ARG MPM_INPUT_FILE
ARG MATLAB_ISO
# Install mpm dependencies.
RUN export DEBIAN_FRONTEND=noninteractive \
&& apt-get update \
&& apt-get install --no-install-recommends --yes \
wget \
unzip \
ca-certificates \
&& apt-get clean \
&& apt-get autoremove \
&& rm -rf /var/lib/apt/lists/*
# Add "matlab" user and grant sudo permission.
RUN adduser --shell /bin/bash --disabled-password --gecos "" matlab \
&& echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab \
&& chmod 0440 /etc/sudoers.d/matlab
# Set user and work directory.
USER matlab
WORKDIR /home/matlab
COPY /R2022b_Update_7_Linux.iso /tmp/matlab.iso
RUN ls -la /tmp/
RUN env
# Run mpm to install MATLAB in the target location and delete the mpm installation afterwards.
# If mpm fails to install successfully, then print the logfile in the terminal, otherwise clean up.
# Pass in $HOME variable to install support packages into the user's HOME folder.
#COPY ${MPM_INPUT_FILE} .
#RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm \
# && chmod +x mpm \
# && sudo HOME=${HOME} ./mpm install \
# --inputfile=${MPM_INPUT_FILE} \
# || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) \
# && sudo rm -f mpm /tmp/mathworks_root.log \
# && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab
RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm \
&& chmod +x mpm \
&& sudo HOME=${HOME} ./mpm install \
--source=/tmp/matlab.iso \
--destination=${MATLAB_INSTALL_LOCATION} \
--products ${MATLAB_PRODUCT_LIST} \
|| (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) \
&& sudo rm -f mpm /tmp/mathworks_root.log \
&& sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab
# Note: Uncomment one of the following two ways to configure the license server.
# Option 1. Specify the host and port of the machine that serves the network licenses
# if you want to store the license information in an environment variable. This
# is the preferred option. You can either use a build variable, like this:
# --build-arg LICENSE_SERVER=27000@MyServerName or you can specify the license server
# directly using: ENV MLM_LICENSE_FILE=27000@flexlm-server-name
#ENV MLM_LICENSE_FILE=$LICENSE_SERVER
# Option 2. Alternatively, you can put a license file into the container.
# Enter the details of the license server in this file and uncomment the following line.
COPY ${LICENSE_FILE} ${MATLAB_INSTALL_LOCATION}/licenses/
# The following environment variables allow MathWorks to understand how this MathWorks
# product (MATLAB Dockerfile) is being used. This information helps us make MATLAB even better.
# Your content, and information about the content within your files, is not shared with MathWorks.
# To opt out of this service, delete the environment variables defined in the following line.
# To learn more, see the Help Make MATLAB Even Better section in the accompanying README:
# https://github.com/mathworks-ref-arch/matlab-dockerfile#help-make-matlab-even-better
# ENV MW_DDUX_FORCE_ENABLE=true MW_CONTEXT_TAGS=MATLAB:DOCKERFILE:V1
ENTRYPOINT ["matlab"]
CMD [""]
full log of gitlab runner
Running with gitlab-runner 16.6.1 (f5da3c5a)
on <redacted>
Preparing the "docker" executor
00:02
Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
Preparing environment
00:00
Updating CA certificates...
Running on runner-jrgxsfjp3-project-263-concurrent-0 via c7c7f1401705...
Getting source from Git repository
00:01
$ git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY;
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/<redacted>/docker-matlab/.git/
Checking out 3e1a6c08 as detached HEAD (ref is <redacted>)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
08:28
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
$ cp /R2022b_Update_7_Linux.iso "${CI_PROJECT_DIR}"
$ cat /etc/gitlab-runner/certs/<redacted>.crt >> /kaniko/ssl/certs/ca-certificates.crt
$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --context "${CI_PROJECT_DIR}" --build-arg http_proxy="${http_proxy}" --build-arg https_proxy="${https_proxy}" --build-arg HTTP_PROXY="${http_proxy}" --build-arg HTTPS_PROXY="${https_proxy}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
INFO[0000] Retrieving image manifest mathworks/matlab-deps:r2022b
INFO[0000] Retrieving image mathworks/matlab-deps:r2022b from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest mathworks/matlab-deps:r2022b
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'mathworks/matlab-deps:r2022b' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/* requires it.
INFO[0014] ARG MATLAB_RELEASE
INFO[0014] ARG MATLAB_PRODUCT_LIST
INFO[0014] ARG MATLAB_INSTALL_LOCATION
INFO[0014] ARG MPM_INPUT_FILE
INFO[0014] ARG MATLAB_ISO
INFO[0014] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*
INFO[0014] Initializing snapshotter ...
INFO[0014] Taking snapshot of full filesystem...
INFO[0019] Cmd: /bin/sh
INFO[0019] Args: [-c export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*]
INFO[0019] Running: [/bin/sh -c export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*]
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3367 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3454 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.7 kB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1192 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3952 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [32.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3549 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1487 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Fetched 30.6 MB in 4s (7312 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
unzip is already the newest version (6.0-25ubuntu1.2).
wget is already the newest version (1.20.3-1ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
INFO[0028] Taking snapshot of full filesystem...
INFO[0029] RUN adduser --shell /bin/bash --disabled-password --gecos "" matlab && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab && chmod 0440 /etc/sudoers.d/matlab
INFO[0029] Cmd: /bin/sh
INFO[0029] Args: [-c adduser --shell /bin/bash --disabled-password --gecos "" matlab && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab && chmod 0440 /etc/sudoers.d/matlab]
INFO[0029] Running: [/bin/sh -c adduser --shell /bin/bash --disabled-password --gecos "" matlab && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab && chmod 0440 /etc/sudoers.d/matlab]
Adding user `matlab' ...
Adding new group `matlab' (1000) ...
Adding new user `matlab' (1000) with group `matlab' ...
Creating home directory `/home/matlab' ...
Copying files from `/etc/skel' ...
INFO[0029] Taking snapshot of full filesystem...
INFO[0030] USER matlab
INFO[0030] Cmd: USER
INFO[0030] WORKDIR /home/matlab
INFO[0030] Cmd: workdir
INFO[0030] Changed working directory to /home/matlab
INFO[0030] No files changed in this command, skipping snapshotting.
INFO[0030] COPY /R2022b_Update_7_Linux.iso /tmp/matlab.iso
INFO[0074] Taking snapshot of files...
INFO[0330] RUN ls -la /tmp/
INFO[0330] Cmd: /bin/sh
INFO[0330] Args: [-c ls -la /tmp/]
INFO[0330] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab}
INFO[0330] Performing slow lookup of group ids for matlab
INFO[0330] Running: [/bin/sh -c ls -la /tmp/]
total 19399164
drwxrwxrwt 2 root root 53 Mar 18 13:19 .
drwxr-xr-x 1 root root 4096 Mar 18 13:19 ..
-rw-r--r-- 1 root root 544 Mar 18 00:11 base-dependencies.txt
-rw-r--r-- 1 root root 19864733696 Mar 18 13:20 matlab.iso
INFO[0330] Taking snapshot of full filesystem...
INFO[0389] No files were changed, appending empty layer to config. No layer added to image.
INFO[0389] RUN env
INFO[0389] Cmd: /bin/sh
INFO[0389] Args: [-c env]
INFO[0389] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab}
INFO[0389] Performing slow lookup of group ids for matlab
INFO[0389] Running: [/bin/sh -c env]
HTTPS_PROXY=http://<redacted>:8080
MPM_INPUT_FILE=mpm-input-files/R2022b/mpm_input_r2022b.txt
MATLAB_ISO=/R2022b_Update_7_Linux.iso
HOME=/home/matlab
MATLAB_RELEASE=r2022b
https_proxy=http://<redacted>:8080
http_proxy=http://<redacted>:8080
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MATLAB_PRODUCT_LIST=MATLAB
DEBIAN_FRONTEND=noninteractive
MATLAB_INSTALL_LOCATION=/opt/matlab/r2022b
PWD=/home/matlab
HTTP_PROXY=http://<redacted>:8080
TZ=Etc/UTC
INFO[0389] Taking snapshot of full filesystem...
INFO[0435] No files were changed, appending empty layer to config. No layer added to image.
INFO[0435] RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm && chmod +x mpm && sudo HOME=${HOME} ./mpm install --source=/tmp/matlab.iso --destination=${MATLAB_INSTALL_LOCATION} --products ${MATLAB_PRODUCT_LIST} || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) && sudo rm -f mpm /tmp/mathworks_root.log && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab
INFO[0435] Cmd: /bin/sh
INFO[0435] Args: [-c wget -q https://www.mathworks.com/mpm/glnxa64/mpm && chmod +x mpm && sudo HOME=${HOME} ./mpm install --source=/tmp/matlab.iso --destination=${MATLAB_INSTALL_LOCATION} --products ${MATLAB_PRODUCT_LIST} || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) && sudo rm -f mpm /tmp/mathworks_root.log && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab]
INFO[0435] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab}
INFO[0435] Performing slow lookup of group ids for matlab
INFO[0435] Running: [/bin/sh -c wget -q https://www.mathworks.com/mpm/glnxa64/mpm && chmod +x mpm && sudo HOME=${HOME} ./mpm install --source=/tmp/matlab.iso --destination=${MATLAB_INSTALL_LOCATION} --products ${MATLAB_PRODUCT_LIST} || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) && sudo rm -f mpm /tmp/mathworks_root.log && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab]
Error: Unable to find the source folder. Specify a valid source folder.
MPM Installation Failure. See below for more information:
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
