randombit
Botan moving very fast with language requirements
Hello. Just informing you of our issues, and wondering if you have any thoughts.
Thunderbird uses the OpenPGP library RNP, which depends on Botan.
Thunderbird/Firefox are slower in migrating to modern C++ features, because they wants to remain compatible with older environments across multiple operating systems, while Botan seems to be very quick in wanting to migrate to modern features.
On the systems we currently support we don't have C++20 and libstdc++10 available yet. We'd like to migrate to Botan 3, because Botan 2 is EOL, but we have trouble migrating, because of Botan's dependencies.
We're looking for a way to use a non-EOL branch of Botan (one that still gets security fixes) that builds successfully in our older environments.
Failing that, we'll have to stay with EOL version 2 branch for another while.
Thanks in advance for any thoughts you might have.
I wonder whether it would be a lot of work to undo the use of c++20 concept. https://github.com/randombit/botan/commit/7d8ccb7ffcdac144d7d40e6f432a8c0b7530f865 (in a local patch)
Do you have any intuition of the time frame before you could adopt C++20?
It's still undecided. Based on project schedules, we might have more clarity by the end of May.
If the Firefox base platform (that Thunderbird depends on), remains with the old libstdc++ for the yearly Firefox ESR version 140 release, worst case, if we don't find another solution, it could mean that we'd be stuck with Botan 2 until end of 2026.
because they wants to remain compatible with older environments across multiple operating systems
I think I am missing some context here. Is there something about the binaries that GCC 11 or MSVC 2022 create that prevents them from running on older systems?
On the systems we currently support we don't have C++20 and libstdc++10 available yet.
Could you be specific about what versions you do have? FWIW it seems like it is possible to build most of the library with GCC 10 with a couple of small patches. (Exceptions being TLS, Classic McEliece and SPHINCS+; only the last is used by RNP.)
Botan seems to be very quick in wanting to migrate to modern features.
Main motivator on the choice of C++20 was span tbh. We could have done without the rest, and the MISRA people would have been a lot happier had we stayed on C++17, but span was worth it in my view. Botan will probably stay at C++20 for some time.
I wonder whether it would be a lot of work to undo the use of c++20 concept.
Removing 90% of the concepts usage would be easy, just make it a plain unrestricted template. Removing the last ~10% is likely highly nontrivial, I'm thinking especially of #3707 and later related work, and in the ECC code like #4209 though I guess there you could just ignore the specialized routines.
It's not just concepts though - in various places we rely quite a bit on extended constexpr/consteval, and of course std::span was added in GCC 10 and at this point very little of the library is buildable without span. Potentially an old libstdc++ plus a standin span could get close.
Is it possible to use Botan3 in cases where your compiler is sufficiently new? Or is this something set across the entire project for all platforms, and updated in lockstep?
Failing that, we'll have to stay with EOL version 2 branch for another while.
The good news here is that RNP uses only a relatively small subset of the library and most of that is pretty well studied at this point. If there was a serious bug (eg memory corruption) or major side channel discovered that affected RNPs usage it would also affect a lot of other users and we'd probably do a patch. (Botan2 is EOL and certainly I'd like to encourage people to not use it but I'm not trying to be mean about it.) The main danger here though is that a lot of the implementations have changed, and nobody is necessarily looking for bugs in an EOL version. Typically researchers, auditors, etc are looking at the latest release, since it's less fun to report a juicy security bug and then be told it was already fixed 5 versions ago. So if Botan2 has a bug that has been missed so far, and Botan3 accidentally removed that bug due to something being refactored or optimized, we might well never find out about it.
Well, I, for one, don't think it was worth moving away from C++17, span or not - but what's done is done
FWIW: @pstanisz did try some back ports (up to Botan 3.2) to C++17. I'm not sure how far he go with it, but I believe CI was passing for him. Obviously we moved deeper into the C++20 rabbit hole since then, but maybe its still worth a look: https://github.com/pstanisz/botan/tree/3.2.0_cpp17
I think I am missing some context here. Is there something about the binaries that GCC 11 or MSVC 2022 create that prevents them from running on older systems?
Firefox/Thunderbird don't bundle libstdc++. They ship Linux binaries that users can download, and it's desired that those will run on systems as old as possible/reasonable.
I'm not a build engineer, I cannot give the full explanation. I remember from the past, that older compilers were used to build, as that was an easy way to ensure no dependencies on newer components sneak in. I don't know how it's done these days.
FWIW: @pstanisz did try some back ports (up to Botan 3.2) to C++17. I'm not sure how far he go with it, but I believe CI was passing for him. Obviously we moved deeper into the C++20 rabbit hole since then, but maybe its still worth a look: https://github.com/pstanisz/botan/tree/3.2.0_cpp17
I managed to port versions 3.0, 3.1 and 3.2 to C++17. The CI builds, including tests were passing, however I have never used the ported version for real-life use-cases. Gave up with next versions (>3.2) as too many C++20 features appeared here and there, making it harder and harder to maintain.
Have you considered maintaining a stable branch in parallel to new development, and backport security fixes to it?
Have you considered maintaining a stable branch in parallel to new development, and backport security fixes to it?
Well, you did maintain the 2.x branch for a while.
A better question is: Given you want to move very fast with new language features, would you consider supporting a previous branch for a longer amount of time? E.g. backport all vulnerability fixes to the 2.x branch for another 1-2 years?
From a user perspective, I'm very excited for Botan to adopt C++20, as it makes designing APIs / library contracts a lot nicer.
Firefox/Thunderbird don't bundle libstdc++. They ship Linux binaries that users can download, and it's desired that those will run on systems as old as possible/reasonable.
So ultimately, this is a self fabricated problem. You can statically link & ship your application with libstdc++ without any change in licensing.
For example, RedHat always offers the newest gcc even on RHEL 8 (which is built with gcc 8). When you want to provide a universal executable, the general expectation is to build against the oldest glibc feasible and link the rest statically. This is also how e.g. python wheels with C extensions are usually built.
Of course I don't expect Mozilla to "just" migrate the linux release binaries based on this, but it always leaves a sour taste in my mouths when projects file issues that boil down to "we don't know when to use -static-libstdc++"
Perhaps Botan could consider a LTS release before every language standard change?
Perhaps Botan could consider a LTS release before every language standard change?
That's a great idea.
Regarding your static linking suggestion, the complexity on our side is rather with the requirement to use multiple build toolchains in parallel, as I understand it.
A better question is: Given you want to move very fast with new language features, would you consider supporting a previous branch for a longer amount of time? E.g. backport all vulnerability fixes to the 2.x branch for another 1-2 years?
I guess it's just a question of - how long is enough? Botan2 got ~20 months of support after Botan 3.0 was released. Keeping in mind that while there are a number of commercial and governmental interests pushing Botan forward in various ways, there is no dedicated commercial enterprise that extracts funding (eg from support contracts) to have full time dedicated engineers. Some of the backports to 2.x were really quite hackish compared to the full fixes on mainline, the divergence between the two branches is quite high and often a patch needed to be rewritten from scratch.
Anyway its something I'll consider but this policy change would add to my workload which is not so fun for me.
Perhaps Botan could consider a LTS release before every language standard change?
I'm not sure if that make sense, at least with how we've done language version changes in the past. These have coincided with also a new major version (Botan 2: C++11, Botan 3: C++20) and then we keep the language version (and minimum supported compiler versions) fixed for the whole lifecycle of that major version. This is out of a desire to not cause problems for users where version 3.N works fine for them and then 3.{N+1} will not compile with their toolchains. Obviously you here are encountering a similar situation but at least it is confined to one jump, 2->3, and you can feel confident with this approach that you won't encounter it again once on Botan3. [FWIW Botan4, with a rough ETA of 2027, very likely will also be C++20]
Given that, a 'LTS before a language standard change' in practice would again just mean extending the support overlap between two major releases from the currently practiced ~ 20-24 months to something longer.