[BUG] Faild to build the model from the docker file: Errors were encountered while processing: openjdk-11-jre-headless:amd64

[ltrojan]

Issues Policy acknowledgement

• [X] I have read and agree to submit bug reports in accordance with the issues policy

Where did you encounter this bug?

Local machine

Willingness to contribute

Yes. I can contribute a fix for this bug independently.

MLflow version

• Client: N/A
• Tracking server: 2.11.1

System information

• system: N/A (Dockerfile: FROM ubuntu:20.04):
• python: N/A (Dockerfile: : FROM ubuntu:20.04)

Describe the problem

the docker build . fails to run

steps

A tar file is created using the following command:

from mlflow.models.flavor_backend_registry import get_flavor_backend

backend = get_flavor_backend(
    tracker_uri,
    docker_build=True,
    env_manager=_env_manager)
backend.generate_dockerfile(
    tracker_uri,
    output_directory,
    mlflow_home=None,
    install_mlflow=False,
    enable_mlserver=False)

the commands generate a folder with the following files:

ltrojan@:tmp/$ ls -l
total 20
-rw-r--r-- 1 ltrojan ltrojan 2434 Apr 22 14:19 Dockerfile
drwxr-xr-x 3 ltrojan ltrojan 4096 Apr 22 14:19 model_dir/

and the Dockerfile:

ltrojan@:tmp/$ head Dockerfile

# Build an image that can serve mlflow models.
FROM ubuntu:20.04

RUN apt-get -y update
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends          wget          curl  
        nginx          ca-certificates          bzip2          build-essential          cmake          openjdk-8-jdk  
        git-core          maven     && rm -rf /var/lib/apt/lists/*


# Setup pyenv
RUN apt -y update

the build command is issued:

docker build .

this run for a min or two untill if fails with the following error:

94.93
95.24 done.
95.24 done.
95.26 Setting up openjdk-8-jre-headless:amd64 (8u402-ga-2ubuntu1~20.04) ...
95.30 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
95.30 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in auto mode
95.30 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
95.30 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
95.30 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
95.31 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
95.31 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
95.31 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode
95.31 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
95.31 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
95.62 Setting up openjdk-11-jre-headless:amd64 (11.0.22+7-0ubuntu2~20.04.1) ...
95.66 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/java to provide /usr/bin/java (java) in auto mode
95.66 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
95.66 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
95.67 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
95.67 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
95.67 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
95.67 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
95.67 update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
95.67 update-alternatives: error: error creating symbolic link '/usr/share/binfmts/jar.dpkg-tmp': No such file or directory
95.67 dpkg: error processing package openjdk-11-jre-headless:amd64 (--configure):
95.67  installed openjdk-11-jre-headless:amd64 package post-installation script subprocess returned error exit status 2
95.67 Setting up openjdk-8-jre:amd64 (8u402-ga-2ubuntu1~20.04) ...
95.68 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode
95.68 Setting up openjdk-8-jdk-headless:amd64 (8u402-ga-2ubuntu1~20.04) ...
95.69 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jdeps to provide /usr/bin/jdeps (jdeps) in auto mode
95.69 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode
95.69 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/rmic to provide /usr/bin/rmic (rmic) in auto mode
95.70 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode
95.70 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode
95.70 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode
95.71 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
95.71 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javac to provide /usr/bin/javac (javac) in auto mode
95.71 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode
95.71 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode
95.71 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode
95.72 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode
95.72 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
95.72 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
95.72 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
95.73 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode
95.73 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javap to provide /usr/bin/javap (javap) in auto mode
95.73 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jar to provide /usr/bin/jar (jar) in auto mode
95.73 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode
95.74 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode
95.74 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jps to provide /usr/bin/jps (jps) in auto mode
95.75 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode
95.75 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode
95.75 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode
95.75 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
95.76 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode
95.76 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode
95.77 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jfr to provide /usr/bin/jfr (jfr) in auto mode
95.77 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode
95.77 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jcmd to provide /usr/bin/jcmd (jcmd) in auto mode
95.78 Setting up openjdk-8-jdk:amd64 (8u402-ga-2ubuntu1~20.04) ...
95.78 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode
95.79 update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode
95.79 Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.40.0+dfsg-3ubuntu0.4) ...
95.81 Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
95.83 Errors were encountered while processing:
95.83  openjdk-11-jre-headless:amd64
95.86 E: Sub-process /usr/bin/dpkg returned an error code (1)
------
Dockerfile:6
--------------------
   4 |
   5 |     RUN apt-get -y update
   6 | >>> RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends          wget          curl          nginx          ca-certificates          bzip2          build-essential          cmake          openjdk-8-jdk          git-core          maven     && rm -rf /var/lib/apt/lists/*
   7 |
   8 |
--------------------
ERROR: failed to solve: process "/bin/sh -c DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends          wget          curl          nginx          ca-certificates          bzip2          build-essential          cmake          openjdk-8-jdk          git-core          maven     && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

What component(s) does this bug affect?

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [X] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• [ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [X] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

What language(s) does this bug affect?

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

[ltrojan]

I did some additional investigation and it looks like the problem is resolved when installing openjdk-8-jdk separately: i replaced this:

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends wget curl nginx ca-certificates bzip2 build-essential cmake openjdk-8-jdk git-core maven && rm -rf /var/lib/apt/lists/*

with these two:

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends wget curl nginx ca-certificates bzip2 build-essential cmake git-core maven

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends openjdk-8-jdk && rm -rf /var/lib/apt/lists/*

and that seems to fix the problem...

this however is not a feasible solution as editing the Dockerfile before being able to build the image seems very overkill...

[daniellok-db]

@ltrojan interesting, do you have any idea why it succeeds when you install openjdk-8-jdk after everything else? does it also succeed if you put openjdk-8-jdk at the end of the installs, like this?

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends wget curl nginx ca-certificates bzip2 build-essential cmake git-core maven openjdk-8-jdk && rm -rf /var/lib/apt/lists/

cc @harupy do you think the migration to the JDK 11 image has anything to do with it?

[ltrojan]

no, it's very weird...

however, i just realised those dockerfiles are created using a slightly old version of mlflow before PR #10954 where the following change was added to docker_utils.py:

RUN apt-get -y update
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends \
         wget \
         curl \
         nginx \
         ca-certificates \
         bzip2 \
         build-essential \
         cmake \
         openjdk-8-jdk \
         git-core \
         maven \
    && rm -rf /var/lib/apt/lists/*

# ...

for this further down the code (in generate_dockerfile())

        setup_java_steps = (
            "# Setup Java\n"
            "RUN apt-get install -y --no-install-recommends openjdk-8-jdk maven\n"
            "ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64"
        )

that looks like the packages openjdk-8-jdk and maven should be installed separately (there's a bunch of examples in the tests/resources/dockerfile folders in the same PR...

this is still spitting out the following:

root@c076165269b2:/# apt-get install -y --no-install-recommends openjdk-8-jdk maven

...

Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Setting up openjdk-11-jre-headless:amd64 (11.0.22+7-0ubuntu2~20.04.1) ...
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool to provide /usr/bin/keytool (keytool) in aut
o mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregi
stry) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/pack200 to provide /usr/bin/pack200 (pack200) in aut
o mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) 
in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-amd64/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
update-alternatives: error: error creating symbolic link '/usr/share/binfmts/jar.dpkg-tmp': No such file or directory
dpkg: error processing package openjdk-11-jre-headless:amd64 (--configure):
 installed openjdk-11-jre-headless:amd64 package post-installation script subprocess returned error exit status 2
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.40.0+dfsg-3ubuntu0.4) ...
Errors were encountered while processing:
 openjdk-11-jre-headless:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

so, it seems the problem is related to installing openjdk-8-jdk, maven them together...

[Beramos]

Same issue here:

sysinfo: Linux P14SGen2 5.14.0-1059-oem #67-Ubuntu SMP Mon Mar 13 14:22:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[Beramos]

I did some additional investigation and it looks like the problem is resolved when installing openjdk-8-jdk separately: i replaced this:

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends wget curl nginx ca-certificates bzip2 build-essential cmake openjdk-8-jdk git-core maven && rm -rf /var/lib/apt/lists/*

with these two:

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends wget curl nginx ca-certificates bzip2 build-essential cmake git-core maven

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y --no-install-recommends openjdk-8-jdk && rm -rf /var/lib/apt/lists/*

and that seems to fix the problem...

this however is not a feasible solution as editing the Dockerfile before being able to build the image seems very overkill...

This fix worked for me.

[github-actions[bot]]

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.