scylla installation will force to switch to openjdk-8 even SUPPORTED openjdk-11 exists

[amoskong]

Description

• In clean machine, openjdk-8 will be installed as scylla-tools-java dependency. even openjdk-11 is available. It's acceptable.
  • we assume that the openjdk-8 is more stable?
  • or it's good to run our production on same java env on all distros?
• But I tested in a machine, which has default openjdk-11, we won't try to reuse it, but force to switch it back to openjdk-8. it has problem.

install default openjdk on Ubuntu 18.04

• openjdk 11 is installed by default.

scylla-test@amos-ubuntu18-tmp:~$ sudo apt install default-jre-headless
....
The following additional packages will be installed:
  ca-certificates-java fontconfig-config fonts-dejavu-core java-common libasound2 libasound2-data libavahi-client3 libavahi-common-data libavahi-common3 libcups2 libfontconfig1
  libjpeg-turbo8 libjpeg8 liblcms2-2 libpcsclite1 libxi6 libxrender1 libxtst6 openjdk-11-jre-headless x11-common
Suggested packages:
  default-jre libasound2-plugins alsa-utils cups-common liblcms2-utils pcscd libnss-mdns fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei | fonts-wqy-zenhei
  fonts-indic
The following NEW packages will be installed:
  ca-certificates-java default-jre-headless fontconfig-config fonts-dejavu-core java-common libasound2 libasound2-data libavahi-client3 libavahi-common-data libavahi-common3 libcups2
  libfontconfig1 libjpeg-turbo8 libjpeg8 liblcms2-2 libpcsclite1 libxi6 libxrender1 libxtst6 openjdk-11-jre-headless x11-common
0 upgraded, 21 newly installed, 0 to remove and 14 not upgraded.
Need to get 39.8 MB of archives.
After this operation, 180 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
....
....

scylla-test@amos-ubuntu18-tmp:~$ java -version
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04, mixed mode, sharing)

Problem:

openjdk-11 had been supported, but the scylla installation will force to switch to openjdk-8.

scylla-test@amos-ubuntu18-tmp:~$ sudo apt install scylla
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  grub-pc-bin libasound2 libasound2-data libnuma1
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib openjdk-8-jre-headless python python-minimal python-yaml python2.7 python2.7-minimal scylla-conf scylla-jmx scylla-kernel-conf
  scylla-python3 scylla-server scylla-tools scylla-tools-core
Suggested packages:
  libnss-mdns fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei fonts-indic python-doc python-tk python2.7-doc binutils binfmt-support
The following packages will be REMOVED:
  default-jre-headless openjdk-11-jre-headless
The following NEW packages will be installed:
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib openjdk-8-jre-headless python python-minimal python-yaml python2.7 python2.7-minimal scylla scylla-conf scylla-jmx
  scylla-kernel-conf scylla-python3 scylla-server scylla-tools scylla-tools-core
0 upgraded, 17 newly installed, 2 to remove and 14 not upgraded.
Need to get 123 MB of archives.
After this operation, 228 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u252-b09-1~18.04 [27.5 MB]
Get:2 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-conf amd64 4.1.0-0.20200618.b443b2574a8-1 [21.8 kB]
Get:3 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-python3 amd64 3.7.7-0.20200618.b443b2574a8-1 [20.0 MB]
Get:4 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython2.7-minimal amd64 2.7.17-1~18.04ubuntu1 [335 kB]
Get:5 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python2.7-minimal amd64 2.7.17-1~18.04ubuntu1 [1,294 kB]
Get:6 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic/main amd64 python-minimal amd64 2.7.15~rc1-1 [28.1 kB]
Get:7 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython2.7-stdlib amd64 2.7.17-1~18.04ubuntu1 [1,915 kB]
Get:8 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python2.7 amd64 2.7.17-1~18.04ubuntu1 [248 kB]
Get:9 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic/main amd64 libpython-stdlib amd64 2.7.15~rc1-1 [7,620 B]
Get:10 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic/main amd64 python amd64 2.7.15~rc1-1 [140 kB]
Get:11 http://us-east1.gce.archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB]
Get:12 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-server amd64 4.1.0-0.20200618.b443b2574a8-1 [33.9 MB]
Get:13 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-jmx all 4.1.0-20200618.9f366b16fba-1 [7,601 kB]
Get:14 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-tools-core all 4.1.0-20200618.51737edf123-1 [30.1 MB]
Get:15 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-tools all 4.1.0-20200618.51737edf123-1 [216 kB]
Get:16 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla-kernel-conf amd64 4.1.0-0.20200618.b443b2574a8-1 [15.0 kB]
Get:17 http://downloads.scylladb.com/deb/unstable/unified/branch-4.1/2020-06-18T14:49:56Z/scylladb-4.1 stable/main amd64 scylla amd64 4.1.0-0.20200618.b443b2574a8-1 [14.2 kB]
Fetched 123 MB in 3s (45.7 MB/s)
Selecting previously unselected package openjdk-8-jre-headless:amd64.
(Reading database ... 66230 files and directories currently installed.)
Preparing to unpack .../openjdk-8-jre-headless_8u252-b09-1~18.04_amd64.deb ...
Unpacking openjdk-8-jre-headless:amd64 (8u252-b09-1~18.04) ...
(Reading database ... 66461 files and directories currently installed.)
Removing default-jre-headless (2:1.11-68ubuntu1~18.04.1) ...
dpkg: openjdk-11-jre-headless:amd64: dependency problems, but removing anyway as you requested:
 ca-certificates-java depends on openjdk-11-jre-headless | java8-runtime-headless; however:
  Package openjdk-11-jre-headless:amd64 is to be removed.
  Package java8-runtime-headless is not installed.
  Package openjdk-8-jre-headless:amd64 which provides java8-runtime-headless is not configured yet.
  Package default-jre-headless which provides java8-runtime-headless is not installed.
  Package openjdk-11-jre-headless:amd64 which provides java8-runtime-headless is to be removed.
 ca-certificates-java depends on openjdk-11-jre-headless | java8-runtime-headless; however:
  Package openjdk-11-jre-headless:amd64 is to be removed.
  Package java8-runtime-headless is not installed.
  Package openjdk-8-jre-headless:amd64 which provides java8-runtime-headless is not configured yet.
  Package default-jre-headless which provides java8-runtime-headless is not installed.
  Package openjdk-11-jre-headless:amd64 which provides java8-runtime-headless is to be removed.

Removing openjdk-11-jre-headless:amd64 (11.0.7+10-2ubuntu2~18.04) ...
Selecting previously unselected package libpython2.7-minimal:amd64.
(Reading database ... 66152 files and directories currently installed.)
Preparing to unpack .../0-libpython2.7-minimal_2.7.17-1~18.04ubuntu1_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../1-python2.7-minimal_2.7.17-1~18.04ubuntu1_amd64.deb ...
Unpacking python2.7-minimal (2.7.17-1~18.04ubuntu1) ...
Selecting previously unselected package python-minimal.
Preparing to unpack .../2-python-minimal_2.7.15~rc1-1_amd64.deb ...
Unpacking python-minimal (2.7.15~rc1-1) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../3-libpython2.7-stdlib_2.7.17-1~18.04ubuntu1_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.17-1~18.04ubuntu1) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../4-python2.7_2.7.17-1~18.04ubuntu1_amd64.deb ...
Unpacking python2.7 (2.7.17-1~18.04ubuntu1) ...
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack .../5-libpython-stdlib_2.7.15~rc1-1_amd64.deb ...
Unpacking libpython-stdlib:amd64 (2.7.15~rc1-1) ...
Setting up libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1) ...
Setting up python2.7-minimal (2.7.17-1~18.04ubuntu1) ...
Linking and byte-compiling packages for runtime python2.7...
Setting up python-minimal (2.7.15~rc1-1) ...
Selecting previously unselected package python.
(Reading database ... 66900 files and directories currently installed.)
Preparing to unpack .../0-python_2.7.15~rc1-1_amd64.deb ...
Unpacking python (2.7.15~rc1-1) ...
Selecting previously unselected package python-yaml.
Preparing to unpack .../1-python-yaml_3.12-1build2_amd64.deb ...
Unpacking python-yaml (3.12-1build2) ...
Selecting previously unselected package scylla-conf.
Preparing to unpack .../2-scylla-conf_4.1.0-0.20200618.b443b2574a8-1_amd64.deb ...
Unpacking scylla-conf (4.1.0-0.20200618.b443b2574a8-1) ...
Selecting previously unselected package scylla-python3.
Preparing to unpack .../3-scylla-python3_3.7.7-0.20200618.b443b2574a8-1_amd64.deb ...
Unpacking scylla-python3 (3.7.7-0.20200618.b443b2574a8-1) ...
Selecting previously unselected package scylla-server.
Preparing to unpack .../4-scylla-server_4.1.0-0.20200618.b443b2574a8-1_amd64.deb ...
Unpacking scylla-server (4.1.0-0.20200618.b443b2574a8-1) ...
Selecting previously unselected package scylla-jmx.
Preparing to unpack .../5-scylla-jmx_4.1.0-20200618.9f366b16fba-1_all.deb ...
Unpacking scylla-jmx (4.1.0-20200618.9f366b16fba-1) ...
Selecting previously unselected package scylla-tools-core.
Preparing to unpack .../6-scylla-tools-core_4.1.0-20200618.51737edf123-1_all.deb ...
Unpacking scylla-tools-core (4.1.0-20200618.51737edf123-1) ...
Selecting previously unselected package scylla-tools.
Preparing to unpack .../7-scylla-tools_4.1.0-20200618.51737edf123-1_all.deb ...
Unpacking scylla-tools (4.1.0-20200618.51737edf123-1) ...
Selecting previously unselected package scylla-kernel-conf.
Preparing to unpack .../8-scylla-kernel-conf_4.1.0-0.20200618.b443b2574a8-1_amd64.deb ...
Unpacking scylla-kernel-conf (4.1.0-0.20200618.b443b2574a8-1) ...
Selecting previously unselected package scylla.
Preparing to unpack .../9-scylla_4.1.0-0.20200618.b443b2574a8-1_amd64.deb ...
Unpacking scylla (4.1.0-0.20200618.b443b2574a8-1) ...
Setting up scylla-conf (4.1.0-0.20200618.b443b2574a8-1) ...
Setting up scylla-python3 (3.7.7-0.20200618.b443b2574a8-1) ...
Setting up scylla-server (4.1.0-0.20200618.b443b2574a8-1) ...
Setting up scylla-kernel-conf (4.1.0-0.20200618.b443b2574a8-1) ...
kernel.sched_tunable_scaling = 0
kernel.sched_min_granularity_ns = 500000
kernel.sched_wakeup_granularity_ns = 450000
kernel.sched_latency_ns = 1000000
kernel.sched_autogroup_enabled = 0
kernel.numa_balancing = 0
fs.aio-max-nr = 1048576
vm.swappiness = 1
Setting up libpython2.7-stdlib:amd64 (2.7.17-1~18.04ubuntu1) ...
Setting up openjdk-8-jre-headless:amd64 (8u252-b09-1~18.04) ...
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up python2.7 (2.7.17-1~18.04ubuntu1) ...
Setting up libpython-stdlib:amd64 (2.7.15~rc1-1) ...
Setting up scylla-jmx (4.1.0-20200618.9f366b16fba-1) ...
Setting up python (2.7.15~rc1-1) ...
Setting up python-yaml (3.12-1build2) ...
Setting up scylla-tools-core (4.1.0-20200618.51737edf123-1) ...
Setting up scylla-tools (4.1.0-20200618.51737edf123-1) ...
Setting up scylla (4.1.0-0.20200618.b443b2574a8-1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for mime-support (3.60ubuntu1) ...


scylla-test@amos-ubuntu18-tmp:~$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~18.04-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

[amoskong]

$ git grep openjdk-
dist/debian/control.mustache:Depends: openjdk-8-jre-headless | openjdk-8-jre | oracle-java8-set-default | adoptopenjdk-8-hotspot-jre | openjdk-11-jre-headless | openjdk-11-jre | oracle-java11-set-default, python (>= 2.7), python-yaml, procps
dist/redhat/scylla-tools.spec.mustache:Requires:       java-1.8.0-openjdk-headless python2
install-dependencies.sh:    apt -y install openjdk-8-jdk-headless ant ant-optional python
install-dependencies.sh:    yum install -y ant java-1.8.0-openjdk-devel python ant-junit fakeroot

We can fix the problem by adjusting the depends order.

But we need to consider our AIM before make a decision:

• do we expected to reuse the existing supported openjdk unless it's latest or older?
  • if yes, it's not enough to just adjust the order
• do we expect to use openjdk-8 if it exists or it can be installed? do we accept to switch openjdk from newer (eg: 11) to openjdk-8 (it might effect other application/service)?
  • this is the currently logic
• do we support all openjdk between 8 and 11 ( 8 <= x <= 11)?
  • currently only openjdk-8 and openjdk-11 are supported, other version can't be installed (eg: openjdk-9)

/CC @penberg @tzach @slivne

[amoskong]

Hi @tzach ,

The dependency change was introduced from 4.1 by https://github.com/scylladb/scylla-tools-java/commit/6d76e51472ae3350cbe9b9adc2134eec616d9318

I think we need to mention the effect (existing openjdk-11 will be switched to openjdk-8 in scylla installation) in release Note. Or consider to fix it if you think it's a serious problem and users can't accept this.

[tzach]

Best will be to find a solution, even if we need 4.1.1 for it, or else all of Ubuntu users (~50%) will fail to upgrade. If docs updated is needed, we should update both release notes and upgrade guide, as many read one (most do not read either)

[slivne]

I don;t understand whats the issue and if its a regression compared to Scylla 4.0

Scylla 4.0 forced installation of Java8,

Scylla 4.1 still forces installation of Java8 although we tried to fix that - we haven't, its ugly I agree - but does it regress compared to Scylla 4.0 ?

Scylla runs on dedicated machines.

[tzach]

@slivne there is a change from 4.0 (see above)

We need to validate:

1. Installation works, as documented, for Ubuntu (16/18) and Debain
2. Upgrade works, as documented

AFAIU, this is not the case. If you suggest to update the docs, we need a "patch" for both install and upgrade.

[roydahan]

We didn't change anything in the procedure of installing /upgrading the cluster. We still install openjdk-8 for ububntu18 & 16.

The artifacts tests are passing and also upgrades.

This is our install procedure:

export DEBIAN_FRONTEND=noninteractive
                    apt-get install software-properties-common -y
                    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 6B2BFD3660EF3F5B
                    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 17723034C56D4B19
                    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5e08fbd8b5d6ec9c
                    add-apt-repository -y ppa:scylladb/ppa
                    apt-get update
                    apt-get install -y openjdk-8-jre-headless
                    update-java-alternatives --jre-headless -s java-1.8.0-openjdk-amd64

[tzach]

The documented procedure is slightly different (one key, no ppa). I do not think it should effect https://www.scylladb.com/download/?platform=ubuntu-18.04&version=scylla-4.1

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 5e08fbd8b5d6ec9c
sudo curl -o /etc/apt/sources.list.d/scylla.list -L http://repositories.scylladb.com/scylla/repo/6f5f052a-53c9-4bbd-9572-4996aa359869/ubuntu/scylladb-4.1-bionic.list
sudo apt-get update
sudo apt-get install scylla
sudo apt-get update
sudo apt-get install -y openjdk-8-jre-headless
sudo update-java-alternatives --jre-headless -s java-1.8.0-openjdk-amd64

[tzach]

For Ubuntu 16 docs are:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 5e08fbd8b5d6ec9c
sudo curl -o /etc/apt/sources.list.d/scylla.list -L http://repositories.scylladb.com/scylla/repo/6f5f052a-53c9-4bbd-9572-4996aa359869/ubuntu/scylladb-4.1-xenial.list
sudo apt-get update
sudo apt-get install scylla

[tzach]

For Debian 9

sudo apt-get install apt-transport-https wget gnupg2 dirmngr
sudo apt-key adv --fetch-keys https://download.opensuse.org/repositories/home:/scylladb:/scylla-3rdparty-stretch/Debian_9.0/Release.key
sudo apt-get update
sudo wget -O /etc/apt/sources.list.d/scylla.list http://repositories.scylladb.com/scylla/repo/6f5f052a-53c9-4bbd-9572-4996aa359869/debian/scylladb-4.1-stretch.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 17723034C56D4B19
sudo apt-get update
sudo apt-get install scylla

For Debian 10

sudo apt-get install apt-transport-https wget gnupg2 dirmngr
sudo apt-key adv --fetch-keys https://download.opensuse.org/repositories/home:/scylladb:/scylla-3rdparty-stretch/Debian_9.0/Release.key
sudo apt-get update
sudo wget -O /etc/apt/sources.list.d/scylla.list http://repositories.scylladb.com/scylla/repo/6f5f052a-53c9-4bbd-9572-4996aa359869/deb/debian/scylla-4.1-buster.list
sudo apt-get update
sudo apt-get install scylla

Note the mistake in the second line. i need to fix it

[amoskong]

I don;t understand whats the issue and if its a regression compared to Scylla 4.0

Scylla 4.0 forced installation of Java8,

Scylla 4.1 still forces installation of Java8 although we tried to fix that - we haven't, its ugly I agree - but does it regress compared to Scylla 4.0 ?

It's not a regression. We only support Java8 for Scylla 4.0, but we start to support Java11 from Scylla 4.1

Scylla runs on dedicated machines.