Add LSF docker files to CI

[raybellwaves]

See examples for PBS, SGE and SLURM here https://github.com/dask/dask-jobqueue/tree/master/ci

I don't have any docker experience. However, people are welcome to point me to resources.

[guillaumeeb]

One good place to start to install LSF : https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_offering/lsfce10.1_quick_start.html. The community edition would be perfect for our need.

https://hub.docker.com/r/ibmcom/lsfce

http://www.sachinpbuzz.com/2017/01/spectrum-lsf-101-installation-and-job.html

[lesteve]

@mrocklin not sure how likely this is, but it would help a lot if you could get someone with LSF install skills involved (maybe someone from Oak Ridge National Labs?).

I think what we need for our CI is a script that installs LSF Community Edition from scratch. I am not sure whether this is feasible to be perfecly honest (it seems like you need to log in with an IBMid to download some files at the beginning of the install). I think for our CI purposes it is fine to install LSF on a single node as described in https://www.ibm.com/support/knowledgecenter/en/SSZUDP_10.2.0/suite_install/lsf_suite_install_poc.html (my understanding is that the LSF workers and the LSF master are on the same node).

I had a look at this for a few hours and here is what I found:

Install LSF Community Edition from scratch

You need to download some files to boostrap the installation. These files need an IBMid account. The download links are very hard to find for some reason (the IBM documentation is extremely hard to navigate and you are never sure you what you are looking at applies to the Community Edition).

The download link is http://ibm.biz/Bd4vxR that redirects to https://www.ibm.com/account/reg/us-en/login?formid=urx-34770&target=https://www-01.ibm.com/marketing/iwm/iwm/web/dispatcher.do?source=swerpzsw-lsf-3.

You can then download lsfsce10.2.0.6-x86_64.tar.gz and are in a better position to follow this blog post or this one.

I have not continued this road further but it feels like this is the most promising option. Not sure how we would add this to our CI. Would there be some licensing issues if we want to add lsfsce10.2.0.6-x86_64.tar.gz to a docker image, no idea ...

Full disclosure: the only way I was able to find the download link from a picture (with a QR code) inside this blog post: https://www.ibm.com/developerworks/community/blogs/hpcgoulash/entry/Standing_up_IBM_Spectrum_LSF_Community_Edition_on_ARMV8?lang=en

Existing docker images

https://hub.docker.com/r/ibmcom/lsfce says:

This component is not intended for separate use. It is a part of the IBM Cloud Private product

Maybe there is a way to reuse this docker image somehow, I am not sure, my feeling is that this image is for LSF workers. When trying to run it you get:

$ docker run -it ibmcom/lsfce:10.2.0
tee: '': No such file or directory
Tue Aug 27 09:26:17 UTC 2019 INFO: CMD parameter: ROLE=  MYSQL_PASSWORD=  LSF_MASTER_LIST=  LDAP_ENABLE=
Tue Aug 27 09:26:17 UTC 2019 INFO: CMD parameter: ROLE=  MYSQL_PASSWORD=  LSF_MASTER_LIST=  LDAP_ENABLE=
Tue Aug 27 09:26:17 UTC 2019 INFO: Running:  config_ldap()
Tue Aug 27 09:26:17 UTC 2019 INFO: Running:  init_share_dir()
Tue Aug 27 09:26:17 UTC 2019 INFO: ROLE=, MYHOST=4a1b2165b399, LSF_MASTER_LIST=
cp: omitting directory '/etc/skel/.'
cp: omitting directory '/etc/skel/..'
Tue Aug 27 09:26:19 UTC 2019 INFO: waiting for lsf master service startup ...
Tue Aug 27 09:26:21 UTC 2019 INFO: waiting for lsf master service startup ...

and the same line every two seconds.

I looked at IBM Cloud Private product and it seems overkill for what we want to do. At what point I naively thought, maybe this can be run locally (maybe using minikube). The only thing I found is: https://github.com/IBM/deploy-ibm-cloud-private/blob/master/docs/deploy-vagrant.md

Even if deploying IBM Cloud Private locally is possible, you still have to configure it. For example: https://developer.ibm.com/storage/2018/04/06/running-lsf-suite-10-2-enterprise-edition-ibm-cloud-private/ with this phrase at the end:

While it is possible to run LSF Suite inside of IBM Cloud Private, the process to make that a useful configuration takes an investment of time and effort.

[mrocklin]

but it would help a lot if you could get someone with LSF install skills involved

Good idea. I've pushed that request upstream.