Impossible to build for Ubuntu Jammy 22.04 / Humble (probably because of OpenVDB / TBB)

[doisyg]

Hello, FYI and to track the issue: In preparation for the Humble release I tried to build STVL under Humble beta with Ubuntu 22.04 and the blocking point seems to be, surprise, the version of the OpenVDB package released in Ubuntu. My quick analysis of the cause:

• OpenVDB was upgraded from v6.2 in Ubuntu 20.04/focal to v8.1 in Ubuntu 22.04/jammy
• TBB was upgraded from 2020.1 in Ubuntu 20.04/focal to 2021.5 in Ubuntu 22.04/jammy
• OpenVDB 8.1 uses headers from TBB 2020 which were deprecated a while ago and now are completely gone in TBB 2021.5. For instance #include <tbb/task_scheduler_init.h> https://github.com/AcademySoftwareFoundation/openvdb/blob/ea786c46b7a1b5158789293d9b148b379fc9914c/openvdb/openvdb/tools/VolumeToMesh.h#L21 See https://www.intel.com/content/www/us/en/developer/articles/release-notes/intel-oneapi-threading-building-blocks-release-notes.html and https://github.com/DrTimothyAldenDavis/SuiteSparse/issues/72

OpenVDB seems to already have a version (v8.2) compatible with TBB 2021.5 since november : https://github.com/AcademySoftwareFoundation/openvdb/releases/tag/v8.2.0. My guess is that releasing v8.2 instead of v8.1 to Ubuntu 22.04/jammy would solve the issue. I will try to file a bug report on launchpad hoping that it will be faster than https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1882998 (remember https://github.com/SteveMacenski/spatio_temporal_voxel_layer/issues/167).

(Side note, looking at OpenVDB release page: https://github.com/AcademySoftwareFoundation/openvdb/releases, I noticed the release of v9.0 with

Official release of NanoVDB, which for the first time offers GPU support for static sparse volumes in OpenVDB.

It is GPU agnotic, more info: https://academysoftwarefoundation.github.io/openvdb/NanoVDB_FAQ.html @SteveMacenski, did you follow ? Maybe that's interesting to look at for STVL and voxels in nav2)

[SteveMacenski]

Thanks for bringing this up / investigating, this is very frustrating. I think you've taken all of the reasonable steps, let me know if there's anything I can do to help! This seems like a crazy oversight and I really hope its handled quickly. Who's deciding on these openVDB versions? Is the problem that it's being released by someone else hap-haphazardly?

I think there's nothing wrong with multiple released versions, so if 8.2 handles, we can target that and update the rosdistro keys to use that one explicitly. How is 8.1 built / released without TBB?

I have not followed the nanoVDB work. I'm happy to see some of that Nvidia GVDB work getting back into the main repository vs trying to consume openVDB with their own project. I would agree that sounds indeed interesting - would that contain some of these same issues as we see with OpenVDB / TBB on versioning? I haven't found a good list of "where" the GPU is used most other than raycasting, which we don't use here, but it claims random access is faster, which we use alot of. So either way, its a good bet.

I'd love to use NanoVDB as the basis of a new nav2 environmental model, but all these versioning issues are killing us and if this was in the core nav2 stack, that could be critical to release and put us back months on each OS release. The docs say NanoVDB has few external dependencies, is that true that it would solve this issue for us? If that is, and could stop future versioning issues, it might be a good idea to do it here as a trial and use our experience there to inform our future decisions. Having GPU enablable raycasting could really help save me a ton of time down the road on a voxel-grid replacement in Nav2.

[nachovizzo]

Hello there, just throwing in my 2 cents.

In case you are interested, I managed to build OpenVDB from the source within the project build. The build script is here. Feel free to copy-paste it, just acknowledge the source :) With this "superbuild" approach I managed to also build the application on different Ubuntu, Fedora, CentOS distributions, macOS, and so on.

I quickly tried to build vdbfusion on Ubuntu 22.04 and the OpenVDB build was working and linking correctly to the application. There was a new compiler error (GCC 11 on ubuntu 22.04) for the TBB build but I managed to disable it from the compiler flags.

I hope this helps, if you need extra support, or if you would like me to help migrate the build system just let me know.

NOTE: The drawback of this approach is that you need to build OpenVDB from source(slow), but using ccache basically turns this into a one-time problem and never more.

Best wishes!

[doisyg]

Thanks for bringing this up / investigating, this is very frustrating. I think you've taken all of the reasonable steps, let me know if there's anything I can do to help! This seems like a crazy oversight and I really hope its handled quickly. Who's deciding on these openVDB versions? Is the problem that it's being released by someone else hap-haphazardly?

I am not familiar with the Ubuntu release process so I don't really now, but it is certainly not triggered by the OpenVdb team.

I think there's nothing wrong with multiple released versions, so if 8.2 handles, we can target that and update the rosdistro keys to use that one explicitly. How is 8.1 built / released without TBB?

No idea... maybe I miss something obvious

I have not followed the nanoVDB work. I'm happy to see some of that Nvidia GVDB work getting back into the main repository vs trying to consume openVDB with their own project. I would agree that sounds indeed interesting - would that contain some of these same issues as we see with OpenVDB / TBB on versioning? I haven't found a good list of "where" the GPU is used most other than raycasting, which we don't use here, but it claims random access is faster, which we use alot of. So either way, its a good bet.

I'd love to use NanoVDB as the basis of a new nav2 environmental model, but all these versioning issues are killing us and if this was in the core nav2 stack, that could be critical to release and put us back months on each OS release. The docs say NanoVDB has few external dependencies, is that true that it would solve this issue for us? If that is, and could stop future versioning issues, it might be a good idea to do it here as a trial and use our experience there to inform our future decisions. Having GPU enablable raycasting could really help save me a ton of time down the road on a voxel-grid replacement in Nav2.

Well, I believe the versioning is properly done by the OpenVDB team: https://github.com/AcademySoftwareFoundation/openvdb/releases. Then it is just a matter of proper deb packaging. If we fear that the Ubuntu releases are going to be an issue, we can decide on doing our own ros openvdb release, for instance ros-humble-openvdb8.2 and ros-humble-openvdb9.0 plus according rosdep keys.

And I would love to see the result in nav2!

[doisyg]

Hello there, just throwing in my 2 cents.

In case you are interested, I managed to build OpenVDB from the source within the project build. The build script is here. Feel free to copy-paste it, just acknowledge the source :) With this "superbuild" approach I managed to also build the application on different Ubuntu, Fedora, CentOS distributions, macOS, and so on.

I quickly tried to build vdbfusion on Ubuntu 22.04 and the OpenVDB build was working and linking correctly to the application. There was a new compiler error (GCC 11 on ubuntu 22.04) for the TBB build but I managed to disable it from the compiler flags.

I hope this helps, if you need extra support, or if you would like me to help migrate the build system just let me know.

NOTE: The drawback of this approach is that you need to build OpenVDB from source(slow), but using ccache basically turns this into a one-time problem and never more.

Best wishes!

Good to know that it is easy to build and link to from source, at least for trying it out. Can you confirm that you have an issue too building a software depending on OpenVDB on Ubuntu 22.04 with the Ubuntu provided package ?

[nachovizzo]

Hello there, just throwing in my 2 cents. In case you are interested, I managed to build OpenVDB from the source within the project build. The build script is here. Feel free to copy-paste it, just acknowledge the source :) With this "superbuild" approach I managed to also build the application on different Ubuntu, Fedora, CentOS distributions, macOS, and so on. I quickly tried to build vdbfusion on Ubuntu 22.04 and the OpenVDB build was working and linking correctly to the application. There was a new compiler error (GCC 11 on ubuntu 22.04) for the TBB build but I managed to disable it from the compiler flags. I hope this helps, if you need extra support, or if you would like me to help migrate the build system just let me know. NOTE: The drawback of this approach is that you need to build OpenVDB from source(slow), but using ccache basically turns this into a one-time problem and never more. Best wishes!

Good to know that it is easy to build and link to from source, at least for trying it out. Can you confirm that you have an issue too building a software depending on OpenVDB on Ubuntu 22.04 with the Ubuntu provided package ?

Not really, since I'm always building OpenVDB from source... to avoid exactly this sort of problems. I can't rely on Ubuntu and their extremely slow and complicated packaging process (I don't blame them though, it's a big jungle anyways). That's why I've decided to include it in my own build, I can control it and I know it's working (almost) everywhere. As an example, I'm using some docker images built on top of CentOS (manylinux) where I can't control which OpenVDB version will be packaged for that distro, therefore It made much more sense to just build it :)

If you think it will be valuable for the community I can simple open a PR here with the updated build system (shouldn't take long)

Let me know!

[doisyg]

All right, that leave us the following options:

A. Wait for a package fix in Ubuntu.

• Advantages: nothing to do.
• Drawbacks: don't know when it will happen and will not fix similar issue in future distributions.

B. Do an OpenVDB ros release.

• Advantages: we control which OpenVDB version is used which would have avoided the mess of the Galactic STVL release, potentially solve our current issue, and also can already let us choose to go for v9.0 (with NanoVDB).
• Drawbacks: ros build farm setup cost. I never did so I don't know how long/how complicated it is but we know where to ask for help (@clalancette )

C. OpenVDB source build as part of the package that needs it (here STVL), i.e @nachovizzo solution

• Advantages: same as B
• Drawbacks: not as clean as an installed deb package. Compilation and maintenance cost. Not commonality and potential version mess if multiple ros packages need it (think STVL + nav2 + some custom packages)

I am personally in favor of B.

If you think it will be valuable for the community I can simple open a PR here with the updated build system (shouldn't take long)

Thx, I ll take it and it will be helpful for testing STVL with Humble beta though I am not sure it is ideal in the long term

[SteveMacenski]

I think B is the most reasonable, but I would give A a few weeks / a month to see if we can't get an updated release out and have that just be part of the package manager without our doing. If we add our own buildfarm setup, we'll have to be owning that process and version controlling for ROS and having to explain to folks why we need it. The issue we bring up is in the same theme as the other openVDB issues we've had in the past that did eventually get resolved.

Want to start that process? It might be easier than we think if we have this versioning information / issues documented like above, seems like a no brainer to get v8.2 out -- maybe in addition to v8.1 if we can't replace

[clalancette]

I'm going to preface this by saying that this, unfortunately, is going to be tricky to fix. (as a side note, this is one of the reasons that we aggressively change Ubuntu distributions to the latest LTS as soon as packages are available; if you find these issues before the release, they are easier to fix in the distribution itself)

A. Wait for a package fix in Ubuntu.

This is, by far, the best option. If we can convince the Ubuntu maintainers to upgrade, or at least take a patch to fix this particular issue, that will fix it for this package and for the rest of the distribution. It will also make it so that binaries that link both against code here and against anything else that uses OpenVDB will continue to work without ABI issues.

B. Do an OpenVDB ros release.

This can be done, but it has to be done extremely carefully. In particular, you need to make sure that nothing else in the system will accidentally find this package instead of the system one (as that can lead to the ABI issues I mentioned above). For instance, when we vendor OGRE for Rviz, we install it to /opt/ros/rolling/opt/rviz_ogre_vendor , which is outside of the normal hierarchy, and won't be found by CMake or pkg-config by default. Then you need to install an "extras" file so that that path is exposed to things looking for the vendor package. And finally, you need to be careful to do find_package(rviz_ogre_vendor) as the very first thing in the CMakeLists.txt, to make sure nothing else finds the underlying package first.

That makes it work, but it is still not great. If a third-party package wants to depend both on something that uses libogre-dev (from the system), and something that uses rviz_ogre_vendor, they can't. Actually, it is worse than that; it will all compile just fine, and randomly crash because of ABI issues.

So in short, I'd say this is a last-ditch effort, because it is tricky to setup properly and tricky for downstream users to use properly.

C. OpenVDB source build as part of the package that needs it (here STVL), i.e @nachovizzo solution

If you can't do A, I'll suggest this one. However, you also need to be careful here. The absolute best way to do this is to build OpenVDB as a static library, and then link it into your main executable with private symbols. If you do that, then anything downstream that wants to depend on both your library and something else that uses OpenVDB should work.

[doisyg]

Thanks for the clarification Chris! Well then I guess we should push for A.

I investigated a bit more and managed to build STVL (with few code adaptation https://github.com/SteveMacenski/spatio_temporal_voxel_layer/pull/233) by uninstalling libopenvdb-dev and building/installing OpenVDB v8.2 (also tried successfully v9.0) from source with these instructions https://github.com/AcademySoftwareFoundation/openvdb#building-openvdb This tend to confirm my quick diagnostic of an incompatibility between OpenVDB v8.1 and TBB 2021.5 due to deprecated headers (<tbb/task_scheduler_init.h>) OpenVDB v8.1 to v8.2 changes were mainly about solving this issue I believe.

Launchpad bug report is here: https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1970108 I guess it will need confirmation from other users to start to be considered

[xouillet]

In the meantime waiting for a proper integration by Ubuntu packaging team, I've ported debian packaging on the v8.2.0 tag of source and built .deb. They are delivered without any warranty but we are currently using it without issue on our dev branches.

You can find them here: https://github.com/wyca-robotics/openvdb/releases/tag/v8.2.0-debian

[Myzhar]

@SteveMacenski @clalancette is there an official update on this? We tested STVL on Foxy and it's really promising, so we would like to use it on Humble too. @xouillet solution is good, but it's not good to suggest users install "unofficial" packages to make it work

[doisyg]

The resolution is in Ubuntu's hand. You can put pressure by confirmed the issue on this ticket: https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1970108

[tonynajjar]

In the meantime waiting for a proper integration by Ubuntu packaging team, I've ported debian packaging on the v8.2.0 tag of source and built .deb. They are delivered without any warranty but we are currently using it without issue on our dev branches.

You can find them here: https://github.com/wyca-robotics/openvdb/releases/tag/v8.2.0-debian

Thank you @xouillet for the current workaround, I can confirm that STVL builds and works after installing these debians.

[doisyg]

An alternative idea: replacing OpenVDB with Bonxai https://github.com/facontidavide/Bonxai . But for this we would need a bit of visibility on Bonxai maintenance and maturity. What do you think @facontidavide ?

[SteveMacenski]

https://github.com/facontidavide/Bonxai/issues/1 We've had this discussion - it doesn't seem to be suitable given Davide's interest in it as more an exercise than something to support long-term.

[facontidavide]

Said that... Bonxai just works. Even without further development, it might be sufficient for your application

[facontidavide]

I really think you should try bonxai, if I remember what you do, you already have everything you need

[SteveMacenski]

Definitely not a judgement on valuable proposition :smile: Its more a matter of risk, if you're yourself not a user of it and not going to maintain it long term, that introduces a bit more there. If you were using it actively in another project (or another major project was using it and could give the thumbs up) or committing to maintaining it for the foreseeable future, that'd be a different story. Since STVL is a relatively "historical" package, I'm a bit risk averse to major backbone changes since I'm not currently in a position to do extensive hardware testing to make sure this still works for the spanning set of user applications. I simply don't have the time or the hardware to make that kind of change comfortably or to contribute to Bonxai should there be issues or future features needed. This is very much a hardware-first driven idea and I don't have robots right now with a satisfactory number of sensors for me to test and feel totally comfortable that I'm not going to break people processing 10+ RGBD sensors (e.g. scales) for long-duration uses (e.g. verify no memory leaks or growth).

OpenVDB works well enough (when you can install the damn thing). Its definitely not ideal, but its the beast we know after you get the install complete. Its my plan in the next ~2 years to fully revamp costmaps and this work so in the medium-term future this project may be deprecated by being ingested into the main navigation stack itself in a new form.

With that said, I wouldn't block the use of it if someone opened a PR and was happy with it after some relatively extensive testing with a number of sensors. I don't know how I would deal with it at that time off the top of my head, but I'd find a compromise that works for everyone (maybe a bonxai branch of stvl released under a slightly different name & removing OpenVDB's version after enough users give the +1).

[facontidavide]

Perfectly valid reasoning 😊 The last thing I want is to unintentionally insert a regression in your software 😅

[SteveMacenski]

If I had 2 months I could make this my primary project, 10/10 I would rip out OpenVDB from this + vet Bonxai so we can be forever done with it. However even if I was time rich, I'm still 10+ sensor robot poor. The systems that motivated this work I left when I left Simbe.

But, temporal-ness is a key requirement I'm establishing from the on-set when I do my environmental modeling kick. Its unclear if I'll get to that in 2023 or 2024, but its high up on the list next to localization and my current project in trajectory planning. Never a boring year...

[doisyg]

Thank you both for your comments. I am not using any RGB-D camera at the moment and hence I am only following from far the maintenance of STVL. But I may have to jump back early 2023, and if by that time OpenVDB is still not fixed in Ubuntu 22.04, I may just do that: create a Bonxai based branch of STVL and test its robustness on a continuously running robot. If I have good results I will happily share them and then we can debate if this is a desirable direction. I feel committed to help maintaining this beautiful package with which I had so good results in the past.

[SteveMacenski]

Agreed, building OpenVDB isn't too bad but yes, it is rather annoying that we cannot release STVL binaries for users to install more naturally. Your release though makes that easier

[agoeckner]

It seems that there hasn't been any progress on this. Anything that I can do to move it along?

[SteveMacenski]

@doisyg I believe was making some headway but I know he's busy right now so it might be some time before he responds

[nachovizzo]

Sorry for adding noise once more, but I still can provide support for managing OpenVDB as a static library. I think it was option C while ago.

While this will impact build times + binary size... It can help getting rid of annoying packaging problems from Debian/Ubuntu.

Your call @SteveMacenski. Even if you need a draft I'm happy to do it.

Best!

[SteveMacenski]

Can you clarify what you mean / intend to do? A static .so file probably isn't going to be architecture indepedent, but a submodule to a version of openvdb that works could be a good stop gap

[clalancette]

Can you clarify what you mean / intend to do? A static .so file probably isn't going to be architecture indepedent, but a submodule to a version of openvdb that works could be a good stop gap

Yes, to be clear, my option C above wasn't "commit a binary .so file to the package". It was "set things up so that during compilation of this package, you compile OpenVDB as a .a and link it in".

[agoeckner]

Can you clarify what you mean / intend to do? A static .so file probably isn't going to be architecture indepedent, but a submodule to a version of openvdb that works could be a good stop gap

Yes, to be clear, my option C above wasn't "commit a binary .so file to the package". It was "set things up so that during compilation of this package, you compile OpenVDB as a .a and link it in".

That certainly sounds like a good option that would allow for STVL binaries to be created & released. I'm all for it!

[nachovizzo]

Hello there, just throwing in my 2 cents.

In case you are interested, I managed to build OpenVDB from the source within the project build. The build script is here. Feel free to copy-paste it, just acknowledge the source :) With this "superbuild" approach I managed to also build the application on different Ubuntu, Fedora, CentOS distributions, macOS, and so on.

I quickly tried to build vdbfusion on Ubuntu 22.04 and the OpenVDB build was working and linking correctly to the application. There was a new compiler error (GCC 11 on ubuntu 22.04) for the TBB build but I managed to disable it from the compiler flags.

I hope this helps, if you need extra support, or if you would like me to help migrate the build system just let me know.

NOTE: The drawback of this approach is that you need to build OpenVDB from source(slow), but using ccache basically turns this into a one-time problem and never more.

Best wishes!

@SteveMacenski as mentioned here the idea is to solve the problem via cmake.

I've done this quote some time ago and has been working so far, check vdbfusion as an example. Another similar one would be KISS-ICP

[SteveMacenski]

Apologies, I was a bit quick on the draw there and re-read the thread to familiarize myself. I think Option C is the best at this point as well. It puts some additional compiling time on the users, but hopefully its a 1-and-done kind of thing after its cached in the workspace.

I'd definitely support adding that into STVL so we can get binaries to turn over as well potentially again. It will have very long build times in the farm, but it should be finally available again.