Turn soversion ON by default

[lmoureaux]

Explain what you would like to see improved and how.

ROOT binary distributions don't always enable the soversion option even though it has always been recommended. https://github.com/root-project/root/blob/15f0275211ba50d36449d0be82d8f470dba834a1/cmake/modules/RootBuildOptions.cmake#L161

For instance, Conda enables it explicitly and LCG doesn't.

Since this option is recommended and can help prevent subtle bugs caused by version mismatches, it should be enabled by default. Versioning libraries is in any case good practice.

ROOT version

ROOT 6.34.04 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.34.04-2bf94/x86_64-el9-gcc14-opt/. ROOT 6.32.02 from conda-forge. Own build with last shared commit at cd805272f42e2992da10a1e204afafbfa64cd411.

Installation method

conda, LCG, source build

Operating system

Linux (all)

Additional context

No response

[dpiparo]

thanks @lmoureaux for this report.

[dpiparo]

@andresailer for LCG Releases (the path above is controlled by ROOT, of course)

[lmoureaux]

Thanks for the quick action! Wouldn't it be better to change the default in CMake though?

[guitargeek]

Yes, that would be better. Enabling the soversion flag in the CI was important to see if things work, but to address this issue we'll have to turn soversion ON by default in the RootBuildOptions.cmake:

https://github.com/root-project/root/blob/master/cmake/modules/RootBuildOptions.cmake#L161

[guitargeek]

Also, I'm curious in general: what is the advantage of building without soversion? Why don't we just always do it, and deprecate that option?

[ferdymercury]

Maybe related: 09501f794eda08886db9b48c4353f42a36d3aade

[dpiparo]

Let's discuss this this week and converge. I did not appreciate the fact that the cvmfs path is indeed the one of LCG Releases and not the CVMFS area we control as ROOT Project (/cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.36.00/). That's why I closed the issue (too fast), apologies for that!

[guitargeek]

Since this option is recommended and can help prevent subtle bugs caused by version mismatches, it should be enabled by default. Versioning libraries is in any case good practice.

Thanks to discussing with @bellenot, I understand now that the "recommendation" was misleading, because as soversion=ON is implemented now, version mismatches would still happen in case on multiple installed ROOT versions, because the rootmap files don't follow the same versioning pattern. So soversion=ON can actually result in more intransparent errors in case of version mismatches, that can be even less helpful to the user than duplicate filenames. We therefore suggest to address this issue by un-recommending this flag:

https://github.com/root-project/root/pull/18876

Maybe we can one day recommend it if the rootmap files etc. follow the same versioning convention too as well, ensuring consistency. But for now there is not a usecase that motivates working on this.

[bellenot]

Right, and AFAIK, the .pcm files are also not versioned (if that matters at all)...

[pcanal]

It probably matters but also the module based autoloading of library is also not restricting itself to versioning libraries (this is an alternative to the rootmap mechanism)

[lmoureaux]

Sad. I guess this is the same class of issues that forces the use of LD_LIBRARY_PATH over RUNPATH everywhere.

[guitargeek]

See this nice post by @cholmcc to see what still needs to be done before we can consider turning soversion ON by default: https://github.com/root-project/root/pull/18876#discussion_r2132445473

[cholmcc]

Note, the .pcm and .rootmap files are intricately tied to the corresponding .so. For that reason, ROOT should - to be as compatible with best-practices on Linux and as robust as possible - insist on loading these files versioned. That is, if ROOT loads libFoo.so.x.y.z, then libFoo.pcm.x.y.z and libFoo.rootmap.x.y.z is loaded. In a sense, that's relatively easily done in the code that loads these files - simply look for the versioned files (first, and then, possibly, for backward compatibility, load un-versioned files).

But, ROOT should really take this opportunity to rethink a couple of things. ROOT has two kinds of loadable "modules"

• Libraries, in the sense that they are explicitly referenced at runtime, and
• Plugins that are loaded on specific requests and accessed through an abstract interface

Per best-practices and standard, the first kind should live in <prefix>/lib, while there's no real standard for the second kind. However, many similar systems will place the plugins in something like <prefix>/lib/root/<version>, and they should not be named libFoo.so, but rather Foo.so to make clear that these are not libraries to link against.

This is actually doable already now with the current code base. I did that distinction when I made the first go at ROOT Debian Packages (for ROOT 5.35 I think). The point was to add <prefix>/lib/root/<version> as a search path for plugins.

There's a few instances where the code isn't really ready for that - for example libMinuit.so (which is really a plugin), is referenced a few places in the core code explicitly.

This approach actually has a number of benefits, in that it promotes separation between implementation and interface, makes it far easier to make true plugins, and makes piecemeal installation of ROOT for more easy and likely to work.

Back in the day, I had many discussions with Fons about this, and I think we agreed in the end. However, the approach was not enforced due to concern over backward compatibility, and with ROOT 6 and CMake (sigh!) those things disappeared out of the build system. A shame, because it would have made ROOT even better.

Yours, Christian

[dpiparo]

Thanks for the instructive input @cholmcc ! I think here we may be confronted with 2 distinct timelines. Something along the lines @cholmcc proposes is certainly very appealing, but cannot happen on a short timescale - it requires thinking and dedicated effort.

However, I wonder if simple versioning of libraries is what @lmoureaux is after. If the problem to be solved on a short time scale is to avoid silently using at runtime "the wrong" libraries (e.g. framework validated for ROOT version X, running against ROOT version Y), enabling soversion can be a good solution. @lmoureaux would you mind commenting?

As matter of fact, the central ROOT installation on lxplus nodes done via el9 rpms have soversion enabled already today...

[lmoureaux]

The use case that triggered writing this issue was actually slightly different. I was trying to deduce the version without loading the libraries. It's something I'd expect to be in the SONAME (well, technically the ABI version, but for ROOT it's the same). Dictionaries don't matter for this specific case.

Since I suffer from version mismatches from time to time and it sounds like a more widespread issue, I wrote the story along these lines instead. It's relatively easy to achieve if I forget to source the LCG environment my software is compiled in before running it. (LCG doesn't enable soversion.)

The discussion you are having reminds me that ROOT cannot locate dictionaries for libraries loaded through RPATH nor for static libraries. In general the way dictionaries work makes pretty strong assumptions on how things are built and only supports dynamic linking options and the most basic features of ld-linux.so. I wish ROOT could load all classes within a given address namespace, e.g. dlopen(nullptr), without relying on external files...

[dpiparo]

Thanks for the additional input. I think I understand that if soversion is turned on, e.g. in LCG, your use case would be addressed. Can you confirm that if LCG enables soversion the problem we are discussing ("deduce the version without loading the libraries") would be addressed?

[lmoureaux]

Yes it would. I understand that there are good arguments against it, but if official distributions through conda and RPM already enable the flag, it should either be disabled there or made the default.

[dpiparo]

Thanks a lot. Transferred to https://its.cern.ch/jira/browse/SPI-2440

[andresailer]

Thanks a lot. Transferred to https://its.cern.ch/jira/browse/SPI-2440

😕 ???

[andresailer]

It's relatively easy to achieve if I forget to source the LCG environment my software is compiled in before running it. (LCG doesn't enable soversion.)

I am probably missing something, but I am not sure how enabling SO versions is going to help you not forget sourcing the environment?

also it is now "not recommended" to enable SO-versions

https://github.com/root-project/root/blob/2a6eaa17d8cdb82ea548d1c45deedf2759260b90/cmake/modules/RootBuildOptions.cmake#L161

[lmoureaux]

Thanks a lot. Transferred to https://its.cern.ch/jira/browse/SPI-2440

I'm very confused, that issue is years old and doesn't say anything about soversion?

[lmoureaux]

I am probably missing something, but I am not sure how enabling SO versions is going to help you not forget sourcing the environment?

It will fail earlier. Instead of loading the system-wide ROOT and crashing at some later point, it will simply refuse to start.

also it is now "not recommended" to enable SO-versions

If this is really the policy, [ROOT] should apply it consistently in the (official, AFAIK) RPM and Conda distributions. Do you know where I can open issues for those?

[ferdymercury]

I'm very confused, that issue is years old and doesn't say anything about soversion?

I think it was a typo and the right one is https://its.cern.ch/jira/browse/SPI-2839

[andresailer]

We discussed this with our largest clients (i.e., ATLAS and LHCb), and the general feeling is that we rather leave things in the LCG stacks as they currently are. We might re-evaluate this once ROOT treats libraries, pcm, and other files consistently.

By the way: why not use rpath to fix the link to the root libraries?

[cholmcc]

Hi all,

OK, so this post will be a bit of a rant, but trust me when I say this is with the deepest respect for your work and a genuine wish for improving ROOT and its adaption.

We discussed this with our largest clients (i.e., ATLAS and LHCb),

Surely your customer base is must larger than ATLAS and LHCb. There's ALICE to name one (the first CERN adopter of ROOT), and in the wider experimental HEP community also FAIR, and BNL (Phobos was the first experiment to use ROOT in production). In an even wider HEP community, there's a large user base amongst model developers and so on.

This attitude that the CERN computing environment is prioritised above all else, is unfortunately very typical of projects that come out of CERN and very navel-gazing that generally makes adoption of such software much harder for a wider audience.

CERN has one model for its computing needs which is by no means the only model and is certainly not applicable in all areas where ROOT could be used.

Instead of focusing on particular customers and their needs, I think you would be far better off if you tried to adhere to well-established standards such as the FreeDesktop specifications or Filesystem Hierarchy Standard. Another great source of considerations for best deploying software on in large scale and over a diverse set of environments is the Debian Policy which lays out what is meant to go where and gives rational for it.

You really do not need to re-invent the wheel here. You can learn a lot by simply observing what other people have done before and understand what best-practices are and how they have evolved. Frankly, that CERN software projects generally are so poorly structured is surprising - given the long tradition of software development CERN. I think it can be explained by a generally arrogant attitude of CERN which has no eye for other people's and organisations' expertise or needs. It is not a new thing, and not unique to ROOT, but ROOT has made very little efforts to improve the situation.

We might re-evaluate this once ROOT treats libraries, pcm, and other files consistently.

As I pointed out several times above, you can already now to a lot with very simple measures to help improve the situation. Sure, it will be better to have the code also take versioning into account, but you do not need to wait for that to progress.

Also, ROOT is not "special" in any relevant way that would justify deviating from standards and specifications. In fact, ROOT is pretty straight forward in that sense.

Perhaps ask yourself why ROOT has not been integrated into major Linux distributions - the answer is quite simple - the expected environment - CERN - is very different from what is standard and standardised. It is surprising - and not in any good way - that after more than 30 years that ROOT isn't in any major Linux distribution (not that it has n't been tried only to be "sabotaged" by ROOT developments).

By the way: why not use rpath to fix the link to the root libraries?

That is a really bad idea. rpath is set at compile-time and then points to a fixed directory for all eternity. That means you can not reliably install such libraries and execuptables on target systems, and it potentially opens up the door for security breaches. For those, and other reasons, rpath is deprecated in most production environments.

Anyways, my 2¢.

Yours, Christian

[andresailer]

We discussed this with our largest clients (i.e., ATLAS and LHCb),

Apologies for the confusion I am speaking on behalf of the LCG stacks.

[andresailer]

That is a really bad idea. rpath is set at compile-time and then points to a fixed directory for all eternity.

CVMFS installed ROOT releases are there for all eternity though.

[lmoureaux]

By the way: why not use rpath to fix the link to the root libraries?

This would be my preferred option as LD_LIBRARY_PATH is even worse than RPATH. Unfortunately, ROOT doesn't understand RPATH for loading libraries and dictionaries (and IIRC, even dlopens itself during initialization). It wants LD_LIBRARY_PATH, even on macOS where this variable is otherwise meaningless.

rpath is set at compile-time and then points to a fixed directory for all eternity.

This is wrong, $ORIGIN allows the creation of relative paths.

rpath is deprecated in most production environments.

As far as I know, this is because DT_RPATH comes before LD_LIBRARY_PATH in the ld.so search order and applies recursively to other loaded objects. DT_RUNPATH doesn't have these drawbacks and is the preferred alternative. It is successfully used by Conda instead of the environment variable hell we typically face in our HEP environments.

[andresailer]

Unfortunately, ROOT doesn't understand RPATH for loading libraries and dictionaries (and IIRC, even dlopens itself during initialization). It wants LD_LIBRARY_PATH, even on macOS where this variable is otherwise meaningless.

Ok, I just don't know enough about how you are running your application.

[guitargeek]

Hi @lmoureaux,

thanks a lot for your post! You make valid points, and these are things that I also thought about a lot.

It is surprising - and not in any good way - that after more than 30 years that ROOT isn't in any major Linux distribution (not that it has n't been tried only to be "sabotaged" by ROOT developments).

Well, the situation is improving, and we try to actively help with this. On Fedora/RHEL for example, we have the packages maintained by @ellert, on NixOS we have a package maintained by me and @veprbl, and there is also a well maintained Arch package that I have used myself for many years. For Debian/Ubuntu, we have had some discussions some years ago, but in the end we missed the final push to make it happen. I should move the relevant JIRA discussions to GitHub, so they can be more easily followed by the interested parties and again be picked up by someone.

What we need is someone to drive e.g. the Debian packaging effort, either from the ROOT CERN team or an external contributor. I can only encourage people to step up there, and I can also help with discussions and reviewing PRs that make ROOT better comply the FHS or Debian packaging guidelines.

ROOT packaging has many issues, and the soversion RPATH stuff is only one of them. We will continue to improve the ROOT packaging experience, but the core team focuses now more on multiplatform/Python package managers like pip and Coda (where we can hopefully share good news soon). So more external help to also cover the Linux distribution package manager site would be extremely appreciated!