easybuild-easyblocks icon indicating copy to clipboard operation
easybuild-easyblocks copied to clipboard

Published 20 hours ago • verified publisher icon easybuilders

patch Java binaries/libraries when using alternate sysroot to ensure correct glibc & co are picked up + add custom sanity check (WIP)

Open boegel opened this issue 4 years ago • 3 comments
trafficstars

Enhancement for custom easyblock for Java to fix problems that occur when using Java in the context of an alternate sysroot and using RPATH (see https://github.com/EESSI/software-layer/issues/123)

Also adds basic custom sanity check for Java, including running java -help and javac -help, which work for Java 1.7.x, 1.8.x, 11.x, 13.x

This is a draft PR because this should be cleaned up a bit, by adding support in EasyBuild framework for leveraging patchelf, so it's a lot cleaner to patch a binary installation like this (we may also need this for other stuff, like CUDA, Intel tools, etc.)

boegel avatar Aug 24 '21 19:08 boegel

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS Java-11.0.8-aarch64.eb
  • SUCCESS Java-1.8_265-b01-OpenJDK-aarch64.eb
  • SUCCESS Java-11.eb
  • SUCCESS Java-1.8.eb

Build succeeded for 4 out of 4 (2 easyconfigs in total) select-pika-c6gd-2xlarge-0001 - Linux centos linux 8.3.2011, AArch64, ARM UNKNOWN (graviton2), Python 3.6.8 See https://gist.github.com/1edb00fae023ea8c1b16cb9531945a89 for a full test report.

edit: no --sysroot or --rpath used, just verifying the custom sanity check on aarch64

boegel avatar Aug 24 '21 19:08 boegel

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS Java-1.6.0_24.eb
  • SUCCESS Java-1.7.0_10.eb
  • SUCCESS Java-1.7.0_15.eb
  • SUCCESS Java-1.7.0_21.eb
  • SUCCESS Java-1.7.0_40.eb
  • SUCCESS Java-1.7.0_45.eb
  • SUCCESS Java-1.7.0_60.eb
  • SUCCESS Java-1.7.0_75.eb
  • SUCCESS Java-1.7.0_76.eb
  • SUCCESS Java-1.7.0_79.eb
  • SUCCESS Java-1.7.0_80.eb
  • SUCCESS Java-1.8.0_112.eb
  • SUCCESS Java-1.8.0_121.eb
  • SUCCESS Java-1.8.0_131.eb
  • SUCCESS Java-1.8.0_141.eb
  • SUCCESS Java-1.8.0_144.eb
  • SUCCESS Java-1.8.0_152.eb
  • SUCCESS Java-1.8.0_162.eb
  • SUCCESS Java-1.8.0_172.eb
  • SUCCESS Java-1.8.0_181.eb
  • SUCCESS Java-1.8.0_192.eb
  • SUCCESS Java-1.8.0_20.eb
  • SUCCESS Java-1.8.0_202.eb
  • SUCCESS Java-1.8.0_212.eb
  • SUCCESS Java-1.8.0_221.eb
  • SUCCESS Java-1.8.0_231.eb
  • SUCCESS Java-1.8.0_241.eb
  • SUCCESS Java-1.8.0_25.eb
  • SUCCESS Java-1.8.0_271.eb
  • SUCCESS Java-1.8.0_281.eb
  • SUCCESS Java-1.8.0_31.eb
  • SUCCESS Java-1.8.0_40.eb
  • SUCCESS Java-1.8.0_45.eb
  • SUCCESS Java-1.8.0_60.eb
  • SUCCESS Java-1.8.0_65.eb
  • SUCCESS Java-1.8.0_66.eb
  • SUCCESS Java-1.8.0_72.eb
  • SUCCESS Java-1.8.0_74.eb
  • SUCCESS Java-1.8.0_77.eb
  • SUCCESS Java-1.8.0_92.eb
  • SUCCESS Java-1.8.eb
  • SUCCESS Java-1.9.0.4.eb
  • SUCCESS Java-11.0.2.eb
  • SUCCESS Java-11.eb
  • SUCCESS Java-13.0.2.eb
  • SUCCESS Java-13.eb
  • SUCCESS Java-16.0.1.eb
  • SUCCESS Java-16.eb

Build succeeded for 48 out of 48 (48 easyconfigs in total) node2639.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8 See https://gist.github.com/446147fb667daab212342adbd426ace4 for a full test report.

edit: no --sysroot or --rpath used, just verifying the custom sanity check on x86_64

boegel avatar Aug 24 '21 19:08 boegel

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS Java-11.0.6-ppc64le.eb
  • SUCCESS Java-1.8_191-b26-OpenJDK.eb
  • SUCCESS Java-11.eb
  • SUCCESS Java-1.8.eb

Build succeeded for 4 out of 4 (2 easyconfigs in total) easybuild2.novalocal - Linux centos linux 8.3.2011, POWER, IBM pSeries (emulated by qemu) (power9le), Python 3.6.8 See https://gist.github.com/9a45cdbbe706e6ce41ec32000a15de59 for a full test report.

edit: no --sysroot or --rpath used, just verifying the custom sanity check on ppc64le

boegel avatar Aug 25 '21 07:08 boegel

@ocaisa I'm fine with leveraging some of the code being added here to framework, but I feel we can still do that later, maybe only once we have a second use case for using patchelf like we do here.

I would really like to see this PR merged as is, since it has been used to install Java in EESSI for a while now. Once similar functionality is supported in framework, we can update the Java accordingly to use that rather than implement it's own approach?

boegel avatar Sep 04 '23 15:09 boegel

Sure we can move it to framework later, but I think it is already good practice to leverage the additions to PATH and LIBRARY_PATH when we decide what we are going to run patchelf on (rather than work off of bin and lib assumptions).

EDIT: PR for this in https://github.com/boegel/easybuild-easyblocks/pull/41

ocaisa avatar Sep 04 '23 15:09 ocaisa

Retested in EESSI pilot 2021.12, still works like a charm.

boegel avatar Sep 05 '23 16:09 boegel

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS Java-1.8.0_311.eb
  • SUCCESS Java-1.8.eb
  • SUCCESS Java-8.362.eb
  • SUCCESS Java-8.eb
  • SUCCESS Java-11.0.20.eb
  • SUCCESS Java-11.eb
  • SUCCESS Java-15.0.1.eb
  • SUCCESS Java-17.0.6.eb

Build succeeded for 8 out of 8 (8 easyconfigs in total) node3116.skitty.os - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8 See https://gist.github.com/boegel/363f693d378f247bee9607ffd4bd1b14 for a full test report.

boegel avatar Sep 05 '23 17:09 boegel