Upgrade Breedbase from Debian 9 to Debian 11

[lukasmueller]

Expected Behavior

Debian 9 will run out of support in April 2022 Also, servers (vm host, file servers) may need to be upgraded inidividual development machines Docker system We are skipping 10. (upgrades are stepwise, so 10 will be there briefly :-)

For Bugs:

Environment

Steps to Reproduce

[lukasmueller]

perl libraries can be upgraded (recompiled) using cpan upgrade.

[lukasmueller]

R packages can be updated usign the command: update.packages(checkBuilt=TRUE, ask=FALSE)

[lukasmueller]

The above methods do not complete the upgrade process and error out.

[lukasmueller]

NOTE: An attempt to fix the issue described below was made in #4030.

# To fix these issues in a xen vm, usually the directories in 
# the js/ dir need to be chowned to production:
#
sudo chown -R production js/node_modules
sudo chown -R production js/build
sudo chown -R production js/package-lock.json

the docker image builds now but when I start the docker I get issues with npm. I replaced the npm install script with npm from apt. Now I get the following errors when starting the breedbase app:

818. [DEP_WEBPACK_DEPRECATION_ARRAY_TO_SET_LENGTH] DeprecationWarning: chunk.files was changed from Array to Set (using Array property 'length' is deprecated) (node:9818) [DEP_WEBPACK_DEPRECATION_ARRAY_TO_SET_INDEXER] DeprecationWarning: chunk.files was changed from Array to Set (indexing Array is deprecated) assets by status 4.72 KiB [cached] 7 assets runtime modules 2.47 KiB 3 modules modules with errors 234 bytes [errors] ./source/entries/boxplotter.js 39 bytes [built] [code generated] [1 error] ./source/entries/fieldmap.js 39 bytes [built] [code generated] [1 error] ./source/entries/heritability.js 39 bytes [built] [code generated] [1 error] ./source/entries/mixedmodels.js 39 bytes [built] [code generated] [1 error] ./source/entries/sequenced_accessions.js 39 bytes [built] [code generated] [1 error] ./source/entries/wizard.js 39 bytes [built] [code generated] [1 error]

ERROR in ./source/entries/boxplotter.js Module build failed (from ./webpack_util/jsan-preprocess-loader.js): TypeError: loaderUtils.getOptions is not a function at Object.module.exports (/home/production/cxgn/sgn/js/webpack_util/jsan-preprocess-loader.js:16:30)

ERROR in ./source/entries/fieldmap.js Module build failed (from ./webpack_util/jsan-preprocess-loader.js): TypeError: loaderUtils.getOptions is not a function at Object.module.exports (/home/production/cxgn/sgn/js/webpack_util/jsan-preprocess-loader.js:16:30)

ERROR in ./source/entries/heritability.js Module build failed (from ./webpack_util/jsan-preprocess-loader.js): TypeError: loaderUtils.getOptions is not a function at Object.module.exports (/home/production/cxgn/sgn/js/webpack_util/jsan-preprocess-loader.js:16:30)

ERROR in ./source/entries/mixedmodels.js Module build failed (from ./webpack_util/jsan-preprocess-loader.js): TypeError: loaderUtils.getOptions is not a function at Object.module.exports (/home/production/cxgn/sgn/js/webpack_util/jsan-preprocess-loader.js:16:30)

ERROR in ./source/entries/sequenced_accessions.js Module build failed (from ./webpack_util/jsan-preprocess-loader.js): TypeError: loaderUtils.getOptions is not a function at Object.module.exports (/home/production/cxgn/sgn/js/webpack_util/jsan-preprocess-loader.js:16:30)

ERROR in ./source/entries/wizard.js Module build failed (from ./webpack_util/jsan-preprocess-loader.js): TypeError: loaderUtils.getOptions is not a function at Object.module.exports (/home/production/cxgn/sgn/js/webpack_util/jsan-preprocess-loader.js:16:30)

webpack 5.70.0 compiled with 6 errors in 704 ms npm ERR! code 1 npm ERR! path /home/production/cxgn/sgn/js npm ERR! command failed npm ERR! command sh -c npm run reset && webpack --config build.webpack.config.js

[lukasmueller]

When using a virtual machine, Perl dependencies may be missing because many Perl dependencies are installed from apt in the docker, they are not available in the perl local-lib git repo anymore. Use this command line to install the missing libraries in a virtual machine:

aptitude install npm libterm-readline-zoid-perl nginx starman emacs gedit vim less sudo htop git dkms  perl-doc ack make xutils-dev nfs-common lynx xvfb ncbi-blast+ libmunge-dev libmunge2 munge slurm-wlm slurmctld slurmd libslurm-perl libssl-dev graphviz lsof imagemagick mrbayes muscle bowtie bowtie2 postfix mailutils libcupsimage2  libglib2.0-dev libglib2.0-bin screen apt-transport-https libgdal-dev libproj-dev libudunits2-dev locales locales-all rsyslog cron  libnlopt0 libcatalyst-controller-html-formfu-perl libmoosex-runnable-perl

[lukasmueller]

Also if working with virtual machines, to get slurm working, move the /etc/slurm-llnl/slurm.conf file to /etc/slurm/

Also, if not yet added, add to /etc/slurm/slurm.conf (remove any other SelectTypes)

SelectType=select/cons_res
SelectTypeParameters=CR_CORE

Remove the option FastSchedule=1 if present

the old slurm state may also have to be removed: sudo rm -Rf /var/spool/slurmstate/*

[lukasmueller]

On virtual machines, run apt install linux-image-5.10.0-15-amd64 (or similar version - must match version in /etc/xen/...cfg file) in case the files in /boot in the vm went missing in the upgrade process which seems to happen from time to time

[lukasmueller]

To get npm to work, the following commands may have to be run on a vm:

# remove the node_modules and .npm directories
rm -Rf js/node_modules
rm -Rf ~/.npm

# remove manually installed npm and node executables from old OS versions
rm /usr/local/bin/npm   
rm /usr/local/bin/node 

[lukasmueller]

Note: sometimes $PERL5LIB needs to be unset, as in unset PERL5LIB. Also, the contents of $PERL5LIB may have to be adjusted on older servers.

[lukasmueller]

ALWAYS use screen to run an upgrade:

sudo screen apt upgrade

so that nothing bad happens when an ssh connection hangs.

[lukasmueller]

To start docker-compose on boot using systemd on the host, create a systemd service file, for example at /etc/systemd/system/breedbase_docker.service, with the following contents:

[Unit]
Description=Starts the Breedbase docker-compose server

[Service]
ExecStart=/home/production/docker/breedbase_docker.sh

[Install]
WantedBy=multi-user.target

the breedbase_docker.sh file contains:

#!/bin/bash

export HOME=/home/production

echo "cd into docker dir...";
cd /home/production/docker;

echo "start docker-compose..."
docker-compose up -d;

echo "done."

Use

systemctl enable breedbase_docker 

to enable it at startup.

[lukasmueller]

Another note: Some systems upgraded from Debian 9 to 10 to 11 seem to have problem with network connectivity. Network drops after a few seconds after turning it on. The only way to fix this is to do a fresh install of the operating system... :-(

[lukasmueller]

For systems that run docker, if after the host upgrade docker refuses to run, with an error message such as

docker.service: Failed with result 'exit-code'.

try the following:

rm -Rf /var/log/docker/aufs

Be careful not to blow away the entire docker dir...

[lukasmueller]

for Debian 11, the only way more loop devices could be generated was to put the word loop in the file /etc/modules and create a file called /etc/modprobe.d/etc-modules-parameters.conf with the contents: options loop max_loop=64

[lukasmueller]

default /etc/apt/sources.list file for vms and hosts:

deb http://deb.debian.org/debian bullseye main contrib non-free
deb-src http://deb.debian.org/debian bullseye main contrib non-free

deb http://deb.debian.org/debian-security bullseye/updates main contrib non-free
deb-src http://deb.debian.org/debian-security bullseye/updates main contrib non-free

deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb-src http://deb.debian.org/debian bullseye-updates main contrib non-free

deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free

deb http://cloud.r-project.org/bin/linux/debian bullseye-cran40/

Also, run:

apt-key adv --keyserver keyserver.ubuntu.com --recv-key '95C0FAF38DB3CCAD0C080A7BDC78B2DDEABC47B7'

[lukasmueller]

All Breedbase servers and VMs have been updated! :-)