matlab-dockerfile
matlab-dockerfile copied to clipboard
Published
20 hours ago •
mathworks-ref-arch
mathworks-ref-arch
issue with network based install from kaniko approach
We try to setup a customized docker image with a set of toolboxes of R2022b from within a corporate GitLab instance. As it does not have privileged mode enabled, we try to prepare the matlab docker image using kaniko. We pass proxy settings as environment variables, but the mpm executable is not able to obtain content and fails with a timeout.
Please advise on what we could do to avoid the following error: HTTP request failed, retry # 1
As I read the log, network access via the proxy is fine in general as both installing dependencies from apt as well as downloading mpm from mathworks.com works just fine. We were not able to identify a verbose switch to obtain a more detailed log. I could not identify helpful answers searching this GitHub repositorie's issues as well as the internet.
.gitlab-ci.yml
stages:
- build
build:
stage: build
when: manual
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
#- cp /R2022b_Update_7_Linux.iso "${CI_PROJECT_DIR}"
- cat /etc/gitlab-runner/certs/<redacted>.crt >> /kaniko/ssl/certs/ca-certificates.crt
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--build-arg http_proxy="${http_proxy}"
--build-arg https_proxy="${https_proxy}"
--build-arg HTTP_PROXY="${http_proxy}"
--build-arg HTTPS_PROXY="${https_proxy}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
Full output of the gitlab runner:
Running with gitlab-runner 16.6.1 (f5da3c5a)
on <redacted>
Preparing the "docker" executor
00:02
Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
Preparing environment
00:00
Updating CA certificates...
Running on runner-jrgxsfjp3-project-263-concurrent-0 via c7c7f1401705...
Getting source from Git repository
00:01
$ git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY;
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/<redacted>/docker-matlab/.git/
Checking out a20af1c5 as detached HEAD (ref is <redacted>)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
02:39
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
$ cat /etc/gitlab-runner/certs/<redacted>.crt >> /kaniko/ssl/certs/ca-certificates.crt
$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --context "${CI_PROJECT_DIR}" --build-arg http_proxy="${http_proxy}" --build-arg https_proxy="${https_proxy}" --build-arg HTTP_PROXY="${http_proxy}" --build-arg HTTPS_PROXY="${https_proxy}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
INFO[0000] Retrieving image manifest mathworks/matlab-deps:r2022b
INFO[0000] Retrieving image mathworks/matlab-deps:r2022b from registry index.docker.io
INFO[0003] Built cross stage deps: map[]
INFO[0003] Retrieving image manifest mathworks/matlab-deps:r2022b
INFO[0003] Returning cached image manifest
INFO[0003] Executing 0 build triggers
INFO[0003] Building stage 'mathworks/matlab-deps:r2022b' [idx: '0', base-idx: '-1']
INFO[0003] Unpacking rootfs as cmd RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/* requires it.
INFO[0016] ARG MATLAB_RELEASE
INFO[0016] ARG MATLAB_PRODUCT_LIST
INFO[0016] ARG MATLAB_INSTALL_LOCATION
INFO[0016] ARG MPM_INPUT_FILE
INFO[0016] ARG MATLAB_ISO
INFO[0016] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*
INFO[0016] Initializing snapshotter ...
INFO[0016] Taking snapshot of full filesystem...
INFO[0021] Cmd: /bin/sh
INFO[0021] Args: [-c export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*]
INFO[0021] Running: [/bin/sh -c export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install --no-install-recommends --yes wget unzip ca-certificates && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*]
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1192 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3367 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3454 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.7 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3952 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1487 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3549 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [32.4 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Fetched 30.6 MB in 7s (4650 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
unzip is already the newest version (6.0-25ubuntu1.2).
wget is already the newest version (1.20.3-1ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
INFO[0031] Taking snapshot of full filesystem...
INFO[0033] RUN adduser --shell /bin/bash --disabled-password --gecos "" matlab && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab && chmod 0440 /etc/sudoers.d/matlab
INFO[0033] Cmd: /bin/sh
INFO[0033] Args: [-c adduser --shell /bin/bash --disabled-password --gecos "" matlab && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab && chmod 0440 /etc/sudoers.d/matlab]
INFO[0033] Running: [/bin/sh -c adduser --shell /bin/bash --disabled-password --gecos "" matlab && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab && chmod 0440 /etc/sudoers.d/matlab]
Adding user `matlab' ...
Adding new group `matlab' (1000) ...
Adding new user `matlab' (1000) with group `matlab' ...
Creating home directory `/home/matlab' ...
Copying files from `/etc/skel' ...
INFO[0033] Taking snapshot of full filesystem...
INFO[0034] USER matlab
INFO[0034] Cmd: USER
INFO[0034] WORKDIR /home/matlab
INFO[0034] Cmd: workdir
INFO[0034] Changed working directory to /home/matlab
INFO[0034] No files changed in this command, skipping snapshotting.
INFO[0034] RUN echo env
INFO[0034] Cmd: /bin/sh
INFO[0034] Args: [-c echo env]
INFO[0034] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab}
INFO[0034] Performing slow lookup of group ids for matlab
INFO[0034] Running: [/bin/sh -c echo env]
env
INFO[0034] Taking snapshot of full filesystem...
INFO[0035] No files were changed, appending empty layer to config. No layer added to image.
INFO[0035] RUN echo ${HTTP_PROXY}
INFO[0035] Cmd: /bin/sh
INFO[0035] Args: [-c echo ${HTTP_PROXY}]
INFO[0035] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab}
INFO[0035] Performing slow lookup of group ids for matlab
INFO[0035] Running: [/bin/sh -c echo ${HTTP_PROXY}]
http://<redacted>.dmz:8080
INFO[0035] Taking snapshot of full filesystem...
INFO[0036] No files were changed, appending empty layer to config. No layer added to image.
INFO[0036] RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm && chmod +x mpm && sudo HOME=${HOME} ./mpm install --release=${MATLAB_RELEASE} --destination=${MATLAB_INSTALL_LOCATION} --products ${MATLAB_PRODUCT_LIST} || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) && sudo rm -f mpm /tmp/mathworks_root.log && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab
INFO[0036] Cmd: /bin/sh
INFO[0036] Args: [-c wget -q https://www.mathworks.com/mpm/glnxa64/mpm && chmod +x mpm && sudo HOME=${HOME} ./mpm install --release=${MATLAB_RELEASE} --destination=${MATLAB_INSTALL_LOCATION} --products ${MATLAB_PRODUCT_LIST} || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) && sudo rm -f mpm /tmp/mathworks_root.log && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab]
INFO[0036] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab}
INFO[0036] Performing slow lookup of group ids for matlab
INFO[0036] Running: [/bin/sh -c wget -q https://www.mathworks.com/mpm/glnxa64/mpm && chmod +x mpm && sudo HOME=${HOME} ./mpm install --release=${MATLAB_RELEASE} --destination=${MATLAB_INSTALL_LOCATION} --products ${MATLAB_PRODUCT_LIST} || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) && sudo rm -f mpm /tmp/mathworks_root.log && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab]
Installing with the following parameters:
--destination=/opt/matlab/r2022b
--doc=false
--release=R2022b
--products=MATLAB
---------------------------------------------
Error: Download failed. Check the network connection and retry.
MPM Installation Failure. See below for more information:
(Mar 18, 2024 12:58:21) HTTP request failed, retry # 1
(Mar 18, 2024 12:58:21) Request timed out.
(Mar 18, 2024 12:58:43) HTTP request failed, retry # 2
(Mar 18, 2024 12:58:43) Request timed out.
(Mar 18, 2024 12:59:07) HTTP request failed, retry # 3
(Mar 18, 2024 12:59:07) Request timed out.
(Mar 18, 2024 12:59:35) HTTP request failed, retry # 4
(Mar 18, 2024 12:59:35) Request timed out.
(Mar 18, 2024 12:59:35) Fatal RepoAdapter Web Service Exception: HTTP request timed out.
(Mar 18, 2024 12:59:35) Error Message: DWSClient Http Request Timeout
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
