sbt-native-packager
sbt-native-packager copied to clipboard
sbt
`exec: line 1: /opt/docker/jre/bin/java: not found` when trying to use docker image build with jlink
Expected behaviour
Generate a docker image with the stripped down jre built by jlink. This is useful as slim images can be used as the base image such as alpine, rather than large jdk docker images.
Actual behaviour
When building the docker image with the jlink plugin, I get this error
Information
-
What sbt-native-packager are you using 1.9.9
-
What sbt version 1.6.2
-
What is your build system (e.g. Ubuntu, MacOS, Windows, Debian ) linux
-
What package are you building (e.g. docker, rpm, ...) docker image with jlink
As a side note, #1437 would be very useful to have for this build too.
Here is my branch where I am building this from. You can build the image locally with sbt appServer/docker:publishLocal
https://github.com/Christewart/bitcoin-s-core/tree/2022-05-22-alpine-base-image
I was having the same kind of issue, maybe this would help you: https://github.com/sbt/sbt-native-packager/issues/1449#issuecomment-926299638
I was having the same kind of issue, maybe this would help you: #1449 (comment)
In my case I am building on linux and running on linux :thinking: . The java version is
openjdk version "18" 2022-03-22
OpenJDK Runtime Environment (build 18+36-2087)
OpenJDK 64-Bit Server VM (build 18+36-2087, mixed mode, sharing)
although i don't think that should matter?
Looking closer, it does seem like my error is fundamentally different:
here is the text version rather than the image i linked above
docker run -p 9999:9999 -p 19999:19999 -e BITCOIN_S_SERVER_RPC_PASSWORD='topsecret' bitcoinscala/bitcoin-s-server:latest
/opt/docker/bin/bitcoin-s-server: line 106: /opt/docker/lib/org.bitcoin-s.bitcoin-s-server-1.9.1-66-a9997d5d-20220523-0843-SNAPSHOT-launcher.jar: Permission denied
/opt/docker/bin/bitcoin-s-server: exec: line 1: /opt/docker/jre/bin/java: not found
@Christewart Can you maybe give us the content of your oracleServer/target/docker/stage/Dockerfile file, please?
@Christewart Can you maybe give us the content of your
oracleServer/target/docker/stage/Dockerfilefile, please?
FROM alpine:latest as stage0
LABEL snp-multi-stage="intermediate"
LABEL snp-multi-stage-id="68d23a07-a1ab-4ea2-8eb4-1ea041c09d56"
WORKDIR /opt/docker
COPY 2/opt /2/opt
COPY 3/opt /3/opt
COPY 4/opt /4/opt
COPY opt /opt
USER root
RUN ["chmod", "-R", "u=rX,g=rX", "/2/opt/docker"]
RUN ["chmod", "-R", "u=rX,g=rX", "/3/opt/docker"]
RUN ["chmod", "-R", "u=rX,g=rX", "/4/opt/docker"]
RUN ["chmod", "-R", "u=rX,g=rX", "/opt/docker"]
RUN ["chmod", "u+x,g+x", "/opt/docker/wallet-server-extra-startup-script.sh"]
RUN ["chmod", "u+x,g+x", "/4/opt/docker/bin/bitcoin-s-server"]
RUN ["chmod", "u+x,g+x", "/4/opt/docker/bin/bitcoin-s-server-main"]
RUN ["apk", "add", "--no-cache", "bash"]
FROM alpine:latest as mainstage
LABEL MAINTAINER="Chris Stewart <[email protected]>"
USER root
RUN id -u bitcoin-s 1>/dev/null 2>&1 || (( getent group 0 1>/dev/null 2>&1 || ( type groupadd 1>/dev/null 2>&1 && groupadd -g 0 root || addgroup -g 0 -S root )) && ( type useradd 1>/dev/null 2>&1 && useradd --system --create-home --uid 1000 --gid 0 bitcoin-s || adduser -S -u 1000 -G root bitcoin-s ))
WORKDIR /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /2/opt/docker /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /3/opt/docker /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /4/opt/docker /opt/docker
COPY --from=stage0 --chown=bitcoin-s:root /opt/docker /opt/docker
EXPOSE 9999 19999
USER 1000:0
ENTRYPOINT ["/opt/docker/bin/bitcoin-s-server"]
CMD ["--conf", "/opt/docker/docker-application.conf"]
The JVM is living in /3/opt/docker and seems to be correctly copied 🤔
Do you see these files in oracleServer/target/docker/stage/3:
Can you copy the content of /opt/docker/bin/bitcoin-s-server maybe, please?
Do you see these files in oracleServer/target/docker/stage/3:
Here is tree 3
3
└── opt
└── docker
└── jre
├── bin
│ ├── java
│ └── keytool
├── conf
│ ├── logging.properties
│ ├── net.properties
│ ├── sdp
│ │ └── sdp.conf.template
│ └── security
│ ├── java.policy
│ ├── java.security
│ └── policy
│ ├── limited
│ │ ├── default_local.policy
│ │ ├── default_US_export.policy
│ │ └── exempt_local.policy
│ ├── README.txt
│ └── unlimited
│ ├── default_local.policy
│ └── default_US_export.policy
├── legal
│ ├── java.base
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── aes.md
│ │ ├── asm.md
│ │ ├── ASSEMBLY_EXCEPTION
│ │ ├── cldr.md
│ │ ├── c-libutl.md
│ │ ├── icu.md
│ │ ├── LICENSE
│ │ ├── public_suffix.md
│ │ └── unicode.md
│ ├── java.datatransfer
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.logging
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.management
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.naming
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.security.jgss
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.security.sasl
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.sql
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.transaction.xa
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── java.xml
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ ├── bcel.md
│ │ ├── dom.md
│ │ ├── jcup.md
│ │ ├── LICENSE
│ │ ├── xalan.md
│ │ └── xerces.md
│ ├── jdk.crypto.ec
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ ├── jdk.management
│ │ ├── ADDITIONAL_LICENSE_INFO
│ │ ├── ASSEMBLY_EXCEPTION
│ │ └── LICENSE
│ └── jdk.unsupported
│ ├── ADDITIONAL_LICENSE_INFO
│ ├── ASSEMBLY_EXCEPTION
│ └── LICENSE
├── lib
│ ├── classlist
│ ├── jexec
│ ├── jrt-fs.jar
│ ├── jspawnhelper
│ ├── jvm.cfg
│ ├── libj2gss.so
│ ├── libjava.so
│ ├── libjimage.so
│ ├── libjli.so
│ ├── libjsig.so
│ ├── libmanagement_ext.so
│ ├── libmanagement.so
│ ├── libnet.so
│ ├── libnio.so
│ ├── libverify.so
│ ├── libzip.so
│ ├── modules
│ ├── security
│ │ ├── blocked.certs
│ │ ├── cacerts
│ │ ├── default.policy
│ │ └── public_suffix_list.dat
│ ├── server
│ │ ├── libjsig.so
│ │ └── libjvm.so
│ └── tzdb.dat
└── release
27 directories, 89 files
Can you copy the content of /opt/docker/bin/bitcoin-s-server maybe, please?
I don't understand where to find this in the staged directory, or else maybe it doesn't exist?
I don't understand where to find this in the staged directory, or else maybe it doesn't exist?
It's a bash script that you can find in this directory: oracleServer/target/docker/stage/4/opt/docker/bin
#!/bin/sh
realpath () {
(
TARGET_FILE="$1"
cd "$(dirname "$TARGET_FILE")"
TARGET_FILE=$(basename "$TARGET_FILE")
COUNT=0
while [ -L "$TARGET_FILE" -a $COUNT -lt 100 ]
do
TARGET_FILE=$(readlink "$TARGET_FILE")
cd "$(dirname "$TARGET_FILE")"
TARGET_FILE=$(basename "$TARGET_FILE")
COUNT=$(($COUNT + 1))
done
if [ "$TARGET_FILE" = "." -o "$TARGET_FILE" = ".." ]; then
cd "$TARGET_FILE"
fi
TARGET_DIR="$(pwd -P)"
if [ "$TARGET_DIR" = "/" ]; then
TARGET_FILE="/$TARGET_FILE"
else
TARGET_FILE="$TARGET_DIR/$TARGET_FILE"
fi
echo "$TARGET_FILE"
)
}
# Allow user and template_declares (see below) to add java options.
addJava () {
java_opts="$java_opts $1"
}
addApp () {
app_commands="$app_commands $1"
}
addResidual () {
residual_args="$residual_args \"$1\""
}
# Allow user to specify java options. These get listed first per bash-template.
if [ -n "$JAVA_OPTS" ]
then
addJava "$JAVA_OPTS"
fi
# Loads a configuration file full of default command line options for this script.
loadConfigFile() {
cat "$1" | sed '/^\#/d;s/\r$//' | sed 's/^-J-X/-X/' | tr '\r\n' ' '
}
# Detect which JVM we should use.
get_java_cmd() {
# High-priority override for Jlink images
if [ -n "$bundled_jvm" ]; then
echo "$bundled_jvm/bin/java"
elif [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/bin/java" ]; then
echo "$JAVA_HOME/bin/java"
else
echo "java"
fi
}
# Processes incoming arguments and places them in appropriate global variables. called by the run method.
process_args () {
local no_more_snp_opts=0
while [ $# -gt 0 ]; do
case "$1" in
--) shift && no_more_snp_opts=1 && break ;;
-h|-help) usage; exit 1 ;;
-v|-verbose) verbose=1 && shift ;;
-d|-debug) debug=1 && shift ;;
-no-version-check) no_version_check=1 && shift ;;
-mem) echo "!! WARNING !! -mem option is ignored. Please use -J-Xmx and -J-Xms" && shift 2 ;;
-jvm-debug) require_arg port "$1" "$2" && addDebugger $2 && shift 2 ;;
-main) custom_mainclass="$2" && shift 2 ;;
-java-home) require_arg path "$1" "$2" && jre=`eval echo $2` && java_cmd="$jre/bin/java" && shift 2 ;;
-D*|-agentlib*|-XX*) addJava "$1" && shift ;;
-J*) addJava "${1:2}" && shift ;;
*) addResidual "$1" && shift ;;
esac
done
if [ $no_more_snp_opts ]; then
while [ $# -gt 0 ]; do
addResidual "$1" && shift
done
fi
}
app_commands=""
residual_args=""
real_script_path="$(realpath "$0")"
app_home="$(realpath "$(dirname "$real_script_path")")"
lib_dir="$(realpath "${app_home}/../lib")"
app_mainclass=-jar "$lib_dir/org.bitcoin-s.bitcoin-s-server-1.9.1-67-55506f28-SNAPSHOT-launcher.jar"
script_conf_file="${app_home}/../conf/application.ini"
app_classpath=""
bundled_jvm="$(realpath "${app_home}/../jre")"
if [[ "$OS" == "OSX" ]]; then
#mac doesn't allow random binaries to be executable
#remove the quarantine attribute so java is executable on mac
xattr -d com.apple.quarantine jre/bin/java
fi
chmod +x jre/bin/java #make sure java is executable
process_args "$@"
java_cmd="$(get_java_cmd)"
# If a configuration file exist, read the contents to $opts
[ -f "$script_conf_file" ] && opts=$(loadConfigFile "$script_conf_file")
eval "exec $java_cmd $java_opts -classpath $app_classpath $opts $app_mainclass $app_commands $residual_args"
This part looks weird to me:
if [[ "$OS" == "OSX" ]]; then
#mac doesn't allow random binaries to be executable
#remove the quarantine attribute so java is executable on mac
xattr -d com.apple.quarantine jre/bin/java
fi
chmod +x jre/bin/java #make sure java is executable
It's not using the get_java_cmd function
This part looks weird to me:
if [[ "$OS" == "OSX" ]]; then #mac doesn't allow random binaries to be executable #remove the quarantine attribute so java is executable on mac xattr -d com.apple.quarantine jre/bin/java fi chmod +x jre/bin/java #make sure java is executableIt's not using the
get_java_cmdfunction
That part is a custom bash script addition needed to get the desktop app working on mac.
This script
- Makes sure that
jre/bin/javais executable (if I didn't do this it would not be executable on certain platforms when packaging with github workflows) - On mac, remove the quarintine attribute for the java binary
Here is a link to the script.
https://github.com/bitcoin-s/bitcoin-s/blob/master/app/server/src/universal/wallet-server-extra-startup-script.sh
If I replace the references to java with the get_java_cmd this will work?
the desktop app working on mac.
Are you not creating a Docker image? What is this a concern in the Docker world?
If I replace the references to java with the get_java_cmd this will work?
Well at least, you script is incorrect right now. The chmod +x is not chmoding the packaged/jlinked JRE
the desktop app working on mac. Are you not creating a Docker image? What is this a concern in the Docker world?
We need to deliver the software in the form of docker images and desktop applications. The desktop applications require the usage of bash script additions due to the problems I detail above. Is there a way to omit bash script additions with the docker plugin?
We need to deliver the software in the form of docker images and desktop applications.
Make two "build modules": One with the constraints/configuration of your desktop app, the other one with constraints/configuration of a Docker image
lazy val myApp = ...
lazy val myDesktopApp =
project
.enablePlugins(JavaAppPackaging, JlinkPlugin)
.settings(jlinkSettings: _*)
.settings(desktopSettings: _*)
.settings(Compile / mainClass := Some("my.app.Main"))
.dependsOn(myApp)
lazy val myDockerApp =
project
.enablePlugins(JavaAppPackaging, JlinkPlugin, DockerPlugin)
.settings(jlinkSettings: _*)
.settings(dockerSettings: _*)
.settings(Compile / mainClass := Some("my.app.Main"))
.dependsOn(myApp)
I just removed the extra bash script stuff for now, I get the exact same error after removing it. Here is the new bash script after disabling the bash script additions.
The error
docker run -p 9999:9999 -p 19999:19999 -e BITCOIN_S_SERVER_RPC_PASSWORD='topsecret' bitcoinscala/bitcoin-s-server:latest
/opt/docker/bin/bitcoin-s-server: line 106: /opt/docker/lib/org.bitcoin-s.bitcoin-s-server-1.9.1-67-55506f28-20220523-1201-SNAPSHOT-launcher.jar: Permission denied
/opt/docker/bin/bitcoin-s-server: exec: line 1: /opt/docker/jre/bin/java: Permission denied
The new bash script without the extra additions.
#!/bin/sh
realpath () {
(
TARGET_FILE="$1"
cd "$(dirname "$TARGET_FILE")"
TARGET_FILE=$(basename "$TARGET_FILE")
COUNT=0
while [ -L "$TARGET_FILE" -a $COUNT -lt 100 ]
do
TARGET_FILE=$(readlink "$TARGET_FILE")
cd "$(dirname "$TARGET_FILE")"
TARGET_FILE=$(basename "$TARGET_FILE")
COUNT=$(($COUNT + 1))
done
if [ "$TARGET_FILE" = "." -o "$TARGET_FILE" = ".." ]; then
cd "$TARGET_FILE"
fi
TARGET_DIR="$(pwd -P)"
if [ "$TARGET_DIR" = "/" ]; then
TARGET_FILE="/$TARGET_FILE"
else
TARGET_FILE="$TARGET_DIR/$TARGET_FILE"
fi
echo "$TARGET_FILE"
)
}
# Allow user and template_declares (see below) to add java options.
addJava () {
java_opts="$java_opts $1"
}
addApp () {
app_commands="$app_commands $1"
}
addResidual () {
residual_args="$residual_args \"$1\""
}
# Allow user to specify java options. These get listed first per bash-template.
if [ -n "$JAVA_OPTS" ]
then
addJava "$JAVA_OPTS"
fi
# Loads a configuration file full of default command line options for this script.
loadConfigFile() {
cat "$1" | sed '/^\#/d;s/\r$//' | sed 's/^-J-X/-X/' | tr '\r\n' ' '
}
# Detect which JVM we should use.
get_java_cmd() {
# High-priority override for Jlink images
if [ -n "$bundled_jvm" ]; then
echo "$bundled_jvm/bin/java"
elif [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/bin/java" ]; then
echo "$JAVA_HOME/bin/java"
else
echo "java"
fi
}
# Processes incoming arguments and places them in appropriate global variables. called by the run method.
process_args () {
local no_more_snp_opts=0
while [ $# -gt 0 ]; do
case "$1" in
--) shift && no_more_snp_opts=1 && break ;;
-h|-help) usage; exit 1 ;;
-v|-verbose) verbose=1 && shift ;;
-d|-debug) debug=1 && shift ;;
-no-version-check) no_version_check=1 && shift ;;
-mem) echo "!! WARNING !! -mem option is ignored. Please use -J-Xmx and -J-Xms" && shift 2 ;;
-jvm-debug) require_arg port "$1" "$2" && addDebugger $2 && shift 2 ;;
-main) custom_mainclass="$2" && shift 2 ;;
-java-home) require_arg path "$1" "$2" && jre=`eval echo $2` && java_cmd="$jre/bin/java" && shift 2 ;;
-D*|-agentlib*|-XX*) addJava "$1" && shift ;;
-J*) addJava "${1:2}" && shift ;;
*) addResidual "$1" && shift ;;
esac
done
if [ $no_more_snp_opts ]; then
while [ $# -gt 0 ]; do
addResidual "$1" && shift
done
fi
}
app_commands=""
residual_args=""
real_script_path="$(realpath "$0")"
app_home="$(realpath "$(dirname "$real_script_path")")"
lib_dir="$(realpath "${app_home}/../lib")"
app_mainclass=-jar "$lib_dir/org.bitcoin-s.bitcoin-s-server-1.9.1-67-55506f28-20220523-1201-SNAPSHOT-launcher.jar"
script_conf_file="${app_home}/../conf/application.ini"
app_classpath=""
bundled_jvm="$(realpath "${app_home}/../jre")"
process_args "$@"
java_cmd="$(get_java_cmd)"
# If a configuration file exist, read the contents to $opts
[ -f "$script_conf_file" ] && opts=$(loadConfigFile "$script_conf_file")
eval "exec $java_cmd $java_opts -classpath $app_classpath $opts $app_mainclass $app_commands $residual_args"
Well, now it's Permission denied and not not found anymore. It looks like a progress to me.
Are we allowed to have such a user name in Linux: bitcoin-s? What if you remove the -?
Also, I'd recommand you to avoid Alpine and prefer something like Ubuntu especially if you want to use bash (I see RUN ["apk", "add", "--no-cache", "bash"])
Are we allowed to have such a user name in Linux:
bitcoin-s? What if you remove the-?
For context, all of these docker images worked perfectly fine when using openjdk:17-slim. I don't think there is any issues with users.
@guizmaii or anyone else that is curious:
I finally got to the bottom of this. It appears that builds cannot be automated to produce jlink'd jres on github actions. This is because github actions does not support arm64 runners as far as I know. My jre/bin/java fails on arm64 due to the linker it expects to find. The jre produced by github actions is linked against /lib64/ld-linux-x86-64.so.2 however on arm platforms it expects a non x86 linker in my case ld-linux-aarch64.so.1 i believe.
https://github.com/bitcoin-s/bitcoin-s/issues/4369#issuecomment-1149949091
Here is a stackoverflow question that really helped me get to the root cause: https://stackoverflow.com/questions/63544874/jlink-does-not-produce-redistributable-image/63595415#63595415
A request:
It would be nice to be able to disable the jlink build based on an environment variable or predefined sbt native packager setting. This would allow my docker containers targeting arm64 to not use the jlink'd jre. I can just revert to using openjdk:17-slim as I mentioned above.
Here is my suggested workaround for our specific project if it is of interest of anyone else.
https://github.com/bitcoin-s/bitcoin-s/issues/4369#issuecomment-1149978653
@Christewart
I finally got to the bottom of this. It appears that builds cannot be automated to produce jlink'd jres on github actions. This is because github actions does not support arm64 runners as far as I know
Good to hear :) We've automated it by having a runner running on an M1 machine we rent in the cloud ;)
@Christewart
I finally got to the bottom of this. It appears that builds cannot be automated to produce jlink'd jres on github actions. This is because github actions does not support arm64 runners as far as I know
Good to hear :) We've automated it by having a runner running on an M1 machine we rent in the cloud ;)
Can you link to the guides you followed to set this up? Would be much appreciated :-)
@Christewart I didn't do it. One of my Ops did it. But here's the documentation https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners
Our workaround in bitcoin-s: https://github.com/bitcoin-s/bitcoin-s/pull/4377