ROOT 6.30.01 unable to find module.modulemap when packaged via fpm

[ktf]

Check duplicate issues.

• [ ] Checked for duplicates

Description

We have our own packaging of root via fpm which relocates our builds which happen in some folder in /tmp to and rpm which installs in /opt/o2.

This was working fine in v6.28.00 however when switching to v6-30-01 we have a bunch of errors along the lines of:

Module Net not found.
Error in <TCling::LoadPCM>: ROOT PCM /opt/o2/lib/libNet_rdict.pcm file does not exist
Module MultiProc not found.
Module Imt not found.
Module MathCore not found.
Error in <TCling::LoadPCM>: ROOT PCM /opt/o2/lib/libMathCore_rdict.pcm file does not exist
Error in <TClass::LoadClassInfo>: no interpreter information for class TSignalHandler is available even though it has a TClass initialization routine.
Error in <TClass::LoadClassInfo>: no interpreter information for class TFileHandler is available even though it has a TClass initialization routine.

when starting root on the command line.

By doing a strace, we figured out that this derives from root not being able to find module.modulemap which is present in /opt/o2/include, yet it is not considered. We are able to workaround the issue by simply copying module.modulemap in the current directory, which is apparently in the search path, however that is clearly not a viable solution in a deployed environment.

What mechanism is used by default to lookup module.modulemap? We would be fine if we could work it around via some environment variable or systemrc tweak.

Reproducer

Build root in one folder and relocate it to a different one.

ROOT version

v6-30-01. v6-28-01 works fine.

Installation method

alibuild, fpm

Operating system

el8

Additional context

This is preventing ALICE DAQ to use the latest ROOT.

[vgvassilev]

Do you have the pcm files in $ROOTSYS/lib? What does it expand to? /opt/o2/lib/?

[ktf]

We do not have ROOTSYS set explicitly, however even if I set it by hand, it produces the same result.

In /opt/o2/lib we do not have /opt/o2/lib/libNet_rdict.pcm, however we do have /opt/o2/lib/Net.pcm, not sure if that's the issue.

[Barthelemy]

And to add on top of that, I would still be interested to understand where the modulemap files are searched and if there is another environment variable we could set to force the search in /opt/o2/include.

[ktf]

Any idea of what might be going on here? To a first loop, we just need a some hint on how to force the lookup, and then we can add a workaround in our packaging instructions.

[vgvassilev]

I do not have time to look at this now. Maybe in a week?

[ktf]

That would be great. Anything we can try on our own?

[Barthelemy]

Are there any news ? we are kind of stuck

[vgvassilev]

Is there a way for me to reproduce your setup? Can you create some reproducing environment on lxplus with ROOT build in debug mode (-DCMAKE_BUILD_TYPE=Debug -DLLVM_BUILD_TYPE=Debug)?

[Barthelemy]

that will be a challenge. We see this after we package ROOT in an RPM. I can try to setup something but not on lxplus and it will take time.

Meanwhile, could you tell me if there is a way to force ROOT to look for the modulemap.module in a given place ? Why would ROOT not find the modulemap installed in an item of ROOT_INCLUDE_PATH ?

I can reproduce the issue on a local setup by simply deleting the module.modulemap. So it is really just ROOT not finding the module.modulemap.

[vgvassilev]

Yes, you can use CLING_MODULEMAP_FILES as a path delimiter separated entries.

[pcanal]

The module map used by ROOT was renamed recently. Do you also see the problem with v6.32.00?

[Barthelemy]

so, if the file is /opt/o2/include/module.modulemap, I should do export CLING_MODULEMAP_FILES=/opt/o2/include/module.modulemap before calling root ?

[vgvassilev]

Yes.

[Barthelemy]

Hi @pcanal, Thank you for chiming in. I don't know, I can give it a try probably. I have to check with @ktf .

[Barthelemy]

Yes.

Strangely, it does not work, root still complains and fails. Copying /opt/o2/include/module.modulemap to . still works. 🤔

[vgvassilev]

Would EXTRA_CLING_ARGS=" -v " produce some useful output that can hint us where the modulemap is being searched for?

[Barthelemy]

Nice option, I must remember that !

The long output is below, but let me highlight this :

'/opt/o2/include/module.modulemap' does not exist. Mounting '/opt/o2/etc/cling/vc.modulemap' as '/opt/o2/include/module.modulemap'

I swear /opt/o2/include/module.modulemap exists !

long output:

[root@ansible-ci-cs8-6 ~]# root
cling version ROOT_1.0~dev
Looking for C++ headers with:
  LC_ALL=C c++   -xc++ -E -v /dev/null 2>&1 | sed -n -e '/^.include/,${' -e '/^ \/.*++/p' -e '}'
Looking for C++ headers in "/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12:/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux:/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward"
Found:
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: 
Found candidate GCC installation: /../lib/gcc/x86_64-redhat-linux/8
Selected GCC installation: /../lib/gcc/x86_64-redhat-linux/8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
ignoring nonexistent directory "/../lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/o2/include"
ignoring duplicate directory "/opt/o2/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/o2/include/vmc
 /opt/o2/include/QualityControl
 /opt/o2/include/fairmq
 /opt/o2/include/GPU
 /opt/o2/include
 /opt/o2/etc
 /opt/o2/etc//cling
 /opt/o2/etc//cling/plugins/include
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward
 /opt/o2/etc//cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
'/usr/include/module.modulemap' does not exist. Mounting '/opt/o2/etc/cling/libc.modulemap' as '/usr/include/module.modulemap'
'/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/module.modulemap' does not exist. Mounting '/opt/o2/etc/cling/std.modulemap' as '/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/module.modulemap'
'/opt/o2/include/module.modulemap' does not exist. Mounting '/opt/o2/etc/cling/vc.modulemap' as '/opt/o2/include/module.modulemap'
'/opt/o2/include/boost/module.modulemap' does not exist. Mounting '/opt/o2/etc/cling/boost.modulemap' as '/opt/o2/include/boost/module.modulemap'
modulemap.overlay.yaml
{
 'version': 0,
 'roots': [
{ 'name': '/usr/include', 'type': 'directory',
'contents': [
   { 'name': 'libc.modulemap', 'type': 'file',
  'external-contents': '/opt/o2/etc/cling/libc.modulemap'
}
 ]
 },
{ 'name': '/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12', 'type': 'directory',
'contents': [
   { 'name': 'std.modulemap', 'type': 'file',
  'external-contents': '/opt/o2/etc/cling/std.modulemap'
}
 ]
 },
{ 'name': '/opt/o2/include', 'type': 'directory',
'contents': [
   { 'name': 'module.modulemap', 'type': 'file',
  'external-contents': '/opt/o2/etc/cling/vc.modulemap'
}
 ]
 },
{ 'name': '/opt/o2/include/boost', 'type': 'directory',
'contents': [
   { 'name': 'boost.modulemap', 'type': 'file',
  'external-contents': '/opt/o2/etc/cling/boost.modulemap'
}
 ]
 }]
 }
 ]
 }
Adding runtime include paths:
  ""
Added include paths:
Added include paths:
Setting up system headers with clang:
ignoring nonexistent directory "/../lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/o2/include"
ignoring duplicate directory "/opt/o2/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/o2/include/vmc
 /opt/o2/include/QualityControl
 /opt/o2/include/fairmq
 /opt/o2/include/GPU
 /opt/o2/include
 /opt/o2/etc
 /opt/o2/etc//cling
 /opt/o2/etc//cling/plugins/include
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward
 /opt/o2/etc//cling/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include
End of search list.
#include <cling/Interpreter/RuntimeUniverse.h>
namespace cling { class Interpreter; namespace runtime { Interpreter* gCling=(Interpreter*)0x23f75e0;
RuntimeOptions* gClingOpts=(RuntimeOptions*)0x23f7760;}}
extern "C" int __cxa_atexit(void (*f)(void*), void*, void*)  noexcept;
#define __dso_handle ((void*)0x23f75e0)
extern "C" int atexit(void(*f)())  throw ()  { return __cxa_atexit((void(*)(void*))f, 0, __dso_handle); }
extern "C++" int at_quick_exit(void(*f)())  throw ()  { return __cxa_atexit((void(*)(void*))f, 0, __dso_handle); }
Module ROOT_Foundation_C not found.

[vgvassilev]

What are the permissions of the existing but not found modulemap?

[vgvassilev]

Would anything change if you disable Vc?

[Barthelemy]

What are the permissions of the existing but not found modulemap?

-rw-r--r--. 1 root root 168987 Jun 6 10:17 /opt/o2/include/module.modulemap

I tried to chmod 777 and it does not change anything.

Would anything change if you disable Vc?

Vc ? sorry , I don't understand. @ktf ?

[ktf]

I think he is implying there is something wrong with the Vc support in root. I guess we could try moving /opt/o2/etc/cling/vc.modulemap out of the way?

[vgvassilev]

Yes, are you using system installed Vc? If so, maybe use -Dbuiltin_vc=On.

[ktf]

We have Vc switched off in ROOT actually, but then we build it ourself and depend on it in O2. Could that be the issue?

[Barthelemy]

Hi, I have tried removing /opt/o2/etc/cling/vc.modulemap and it works. Giulio, I have tried with your PR that makes ROOT use our VC and it does not work.

Cheers,

[Barthelemy]

Another thing I noticed is that the extra logs show this when I run and it fails :

{ 'name': '/opt/o2/include', 'type': 'directory',
'contents': [
   { 'name': 'module.modulemap', 'type': 'file',
  'external-contents': '/opt/o2/etc/cling/vc.modulemap'
}

whereas it shows something different before the packaging :

{ 'name': '/home/alibuild/sw/slc8_x86-64/Vc/1.4.1-local7/include', 'type': 'directory',
'contents': [
   { 'name': 'vc.modulemap', 'type': 'file',
  'external-contents': '/data/sw/slc8_x86-64/ROOT/v6-30-01-alice5-local4/etc/cling/vc.modulemap'
}

Could the name be the issue ? where it it coming from ?

[Barthelemy]

By the way I have a machine setup if you want to see by yourselves. Send me your public key in private if you want.

[vgvassilev]

Do you have this change: https://github.com/root-project/root/pull/14974 ?

[ktf]

No.

[Barthelemy]

The change you mention did not help.

However, bumping to 6.32.00 "fixed" the issue.

I will see with @ktf if and when we can upgrade.

[vgvassilev]

We have Vc switched off in ROOT actually, but then we build it ourself and depend on it in O2. Could that be the issue?

I suspect that the headers that Vc installs and the location of the headers of installed ROOT overlap and that makes it confused.