LLNL
Cannot build on theta via spack
System information
| Type | Version/Name |
|---|---|
| Operating System | |
| OS Version | |
| Architecture | |
| UnifyFS Version |
ALCF Theta CNL7
Describe the problem you're observing
spack install unifyfs fails to build flatcc
flatcc appears to not recognize the cray cross compile environment. It's build fails with "Illegal instruction" errors.
% spack spec unifyfs
Input spec
--------------------------------
unifyfs
Concretized
--------------------------------
[email protected]%[email protected]+auto-mount~fortran~hdf5~pmi~pmix patches=1b691bbf34d1178642cba579d4e4eb2b5a091505613392bec797468c34d332f6 arch=cray-cnl7-mic_knl
^[email protected]%[email protected] arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=cray-cnl7-mic_knl
^[email protected]%[email protected] arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+cpanm+shared+threads patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac arch=cray-cnl7-mic_knl
^[email protected]%[email protected] arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+shared build_type=RelWithDebInfo arch=cray-cnl7-mic_knl
^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt patches=1c540040c7e203dd8e27aa20345ecb07fe06570d56410a24a266ae570b1c4c39 arch=cray-cnl7-mic_knl
^[email protected]%[email protected]~test build_type=RelWithDebInfo arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+shared build_type=RelWithDebInfo arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+pic+shared build_type=RelWithDebInfo patches=c9cfecb1f7a623418590cf4e00ae7d308d1c3faeb15046c2e5090e38221da7cd arch=cray-cnl7-mic_knl
^[email protected]%[email protected] arch=cray-cnl7-mic_knl
^[email protected]%[email protected] arch=cray-cnl7-mic_knl
^[email protected]%[email protected]~debug~valgrind arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+bmi~boostsys~cci~mpi+ofi+selfforward+shared+sm+udreg+verbose build_type=RelWithDebInfo arch=cray-cnl7-mic_knl
^bmi@develop%[email protected] arch=cray-cnl7-mic_knl
^[email protected]%[email protected]~kdreg fabrics=gni arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+internal_glib arch=cray-cnl7-mic_knl
^[email protected]%[email protected]~benvolio+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch3 netmod=tcp pmi=pmi arch=cray-cnl7-mic_knl
^[email protected]%[email protected]+systemcerts arch=cray-cnl7-mic_knl
Errors are of the form
10 errors found in build log:
252 cd /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/test/emit_test && /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/bin/flatcc -a -o /tmp/robl/spack-stage/spack-stage-flatcc-0.
5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/test/emit_test/generated /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/test/emit_test/emit_test.fbs
253 cd /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/test/reflection_test && /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/bin/flatcc --schema -o /tmp/robl/spack-stage/spack-sta
ge-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/test/reflection_test/generated /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/test/monster_test/monster_test.fbs
254 cd /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/samples/monster && /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/bin/flatcc -a -o /tmp/robl/spack-stage/spack-stage-flatcc-0
.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/samples/monster/generated /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/samples/monster/monster.fbs
255 cd /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/samples/reflection && /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/bin/flatcc --schema -o /tmp/robl/spack-stage/spack-stage
-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/samples/reflection/generated /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/samples/monster/monster.fbs
256 make[2]: Leaving directory '/tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build'
257 /bin/sh: line 1: 8087 Illegal instruction (core dumped) /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-src/bin/flatcc -a --prefix=zzz_ --stdout /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqif
j3p2/spack-src/test/monster_test/monster_test.fbs > /tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build/test/monster_test_prefix/generated/zzz_monster_test.h
>> 258 make[2]: *** [test/monster_test_prefix/CMakeFiles/gen_monster_test_prefix.dir/build.make:61: gen_monster_test_prefix] Error 132
259 make[2]: Leaving directory '/tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build'
>> 260 make[1]: *** [CMakeFiles/Makefile2:1656: test/monster_test_prefix/CMakeFiles/gen_monster_test_prefix.dir/all] Error 2
261 make[1]: *** Waiting for unfinished jobs....
262 make[2]: *** [test/monster_test_concat/CMakeFiles/gen_monster_test_concat.dir/build.make:61: gen_monster_test_concat] Illegal instruction (core dumped)
263 make[2]: Leaving directory '/tmp/robl/spack-stage/spack-stage-flatcc-0.5.3-xsdgfnjvb5ozykt2izzwi6zngqifj3p2/spack-build'
If possible, could you do be a quick favor and try this again with any version 8 of gcc and see if it still fails? We've seen an issue before where flattcc wouldn't build with gcc@9.
On a similar note, I just noticed there is a newer version of flatcc available that the Spack doesn't know about. I'll get the Spack package updated and see if that helps.
Alternatively, this has brought up the question again that we may no longer need FlatCC as a dependency (#550). So we may just remove this sooner rather than later.
Thanks for the suggestion. That's very strange but yes, gcc-8 builds flatcc on theta. Now the problem is that unify cannot find an mpi compiler:
checking for /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc... /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc
Checking whether /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc responds to '-showme:compile'... no
Checking whether /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc responds to '-showme'... no
Checking whether /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc responds to '-compile-info'... no
Checking whether /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc responds to '-show'... no
Unable to find suitable MPI Compiler. Try setting MPICC.
configure: error: "Couldn't find MPI"
On cray, there are no mpicc wrapers. 'cc' brings in the right headers and libraries . spack tries to to the same thing, i guess, because here's the config.log output:
configure:17754: checking for /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc
configure:17784: result: /gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc
configure:18255: error: "Couldn't find MPI"
(I do not like that the LX_FIND_MPI macro assumes the mpi compiler is called mpicc but this is not a defect tracker for the LX macros...)
I have tried updating the spack package to set either CC or MPICC but still get this error. Spack tells me it is configured like so:
spack.util.executable.ProcessError: Command exited with status 1:
'/tmp/robl/spack-stage/spack-stage-unifyfs-0.9.0-hqt7nr6wju3mqrwugo7w4t6sh6jkraxq/spack-src/configure' '--prefix=/gpfs/mira-home/robl/src/spack/opt/spack/cray-cnl7-mic_knl/gcc-8.3.0/unifyfs-0.9.0-hqt7nr6wju3mqrwugo7w4t6sh6jkraxq' 'MPICC=/gpfs/mira-home/robl/src/spack/lib/spack/env/gcc/gcc' '--without-hdf5' '--enable-mpi-mount' '--enable-silent-rules'
A couple updates/things to try to see if we can figure out what's happening:
-
Try installing with our dev branch;
spack install unifyfs@develop. We've made a lot of changes since our 0.9.0 release, including MPI related stuff. So possibly the issue has been resolved (fingers crossed). We're probably about due for a new release.- Note that FlatCC did just get removed (#553), so the gcc version might not matter anymore when installing the develop branch. You'll need to pull the latest Spack develop branch changes to get this though, as it was just merged (https://github.com/spack/spack/pull/17913).
- This update also added an optional dependency for spath as well, but you can turn that off
spack install unifyfs@develop ~spath
- This update also added an optional dependency for spath as well, but you can turn that off
- Note that FlatCC did just get removed (#553), so the gcc version might not matter anymore when installing the develop branch. You'll need to pull the latest Spack develop branch changes to get this though, as it was just merged (https://github.com/spack/spack/pull/17913).
-
As you've probably saw in updating our Spack package, we didn't do anything special with MPI aside from depending on it as our configure is set up to find it automatically. Could try installing using our boostrap.sh script that installs all the dependencies for you without using Spack. Then try installing UnifyFS manually. Something along the lines of:
$ export PKG_CONFIG_PATH=path/to/mercury/lib/pkgconfig:path/to/argobots/lib/pkgconfig:path/to/margo/lib/pkgconfig
$ ./autogen.sh
$ ./configure --prefix=/path/to/install --with-gotcha=/path/to/gotcha --with-leveldb=/path/to/leveldb --enable-mpi-mount
$ make
$ make install
If this still fails (my guess is it will if installing from the develop branch didn't work), then we can be sure it's us and not Spack, which might make it easier to debug.
Going through older issues. UnifyFS has successfully been built and run on cray systems since this was an issue. Is this still relevant or a concern for Theta?