build: Add CMake-based build system

[hebasto]

This PR adds a CMake-based build system.

Added build instructions and examples to the README.md file.

Added a few toolchain files for easy cross compiling.

Discussions on IRC:

• https://gnusha.org/secp256k1/2022-06-23.log
• https://gnusha.org/secp256k1/2022-06-24.log
• https://gnusha.org/secp256k1/2022-06-27.log

---

Related PRs:

• #315
• #549
• #761

---

Implementation notes.

Minimum required CMake version is 3.1. This was required to provide C_STANDARD property.

In turn, this choice of CMake version implies it is not possible to build with CMake on Debian 8, which has CMake v3.0.2 only.

---

What is not implemented:

• valgrind support
• experimental features

Autotools -- CMake Feature Parity Tables

1. Configuration options

Autotool-based build system features being listed according to the ./configure --help output.

Autotools	CMake
--prefix	-DCMAKE_INSTALL_PREFIX
--enable-shared	-DSECP_BUILD_SHARED
--enable-static	-DSECP_BUILD_STATIC
--enable-benchmark	-DSECP_BUILD_BENCHMARK
--enable-coverage	-DCMAKE_BUILD_TYPE=Coverage
--enable-tests	-DSECP_BUILD_TESTS
--enable-experimental	-DSECP_ENABLE_EXPERIMENTAL
--enable-exhaustive-tests	-DSECP_BUILD_EXHAUSTIVE_TESTS
--enable-examples	-DSECP_BUILD_EXAMPLES
--enable-module-ecdh	-DSECP_ENABLE_MODULE_ECDH
--enable-module-recovery	-DSECP_ENABLE_MODULE_RECOVERY
--enable-module-extrakeys	-DSECP_ENABLE_MODULE_EXTRAKEYS
--enable-module-schnorrsig	-DSECP_ENABLE_MODULE_SCHNORRSIG
--enable-external-default-callbacks	-DSECP_USE_EXTERNAL_DEFAULT_CALLBACKS
--with-asm	TBD
--with-ecmult-window	-DSECP_ECMULT_WINDOW_SIZE
--with-ecmult-gen-precision	-DSECP_ECMULT_GEN_PREC_BITS
--with-valgrind	TBD
--enable-dev-mode hidden	see https://github.com/bitcoin-core/secp256k1/pull/1113#discussion_r916979117

2. make targets

Autotools	CMake
make	make
make check	make test
make install	make install *

• Installation of lib/pkgconfig/libsecp256k1.pc to be done.

[real-or-random]

Weak Concept ACK. I think there's user demand for this, and if we get some occasional help from the Core build people, maintenance should be doable.

[hebasto]

Updated be3bb3072cf52202b3f334e340bf4992e19b8b5c -> 697107c06e0837014ae24124dd6db81d6cb87485 (pr1113.01 -> pr1113.02, diff):

• addressed @fanquake's comment:

Tested using the suggested instructions, on a macos box, but the build fails to link

[hebasto]

Updated 697107c06e0837014ae24124dd6db81d6cb87485 -> dba933ac9ffaea9fedf62a9f324891f440c2ca76 (pr1113.02 -> pr1113.03, diff):

• make install works as expected now :tiger2:

[hebasto]

Updated dba933ac9ffaea9fedf62a9f324891f440c2ca76 -> ec372dc0b8a95053a771614931982e29291b6dd8 (pr1113.03 -> pr1113.04, diff):

• added an example of cross compiling for Android
• added an example of native compiling on Windows with Visual Studio

[hebasto]

Updated ec372dc0b8a95053a771614931982e29291b6dd8 -> d98be43d08fcfaef756309bb3048a57274b0ccb6 (pr1113.04 -> pr1113.05, diff):

• addressed @real-or-random's comments

[real-or-random]

It's correct to add the config header template libsecp256k1-config.h.in because it replicates what autoconf does but I think maintaining this file will be annoying, I guess. It will mean we'll have the duplicate the help strings.

I'm not exactly sure what's the best way forward.

We could either try to get rid of the config header entirely, also in autoconf. That simplifies things (generated source files are always annoying in some way. The autoconf manual gives the following rationale for preferring a config header over -D:

When a package contains more than a few tests that define C preprocessor symbols, the command lines to pass -D options to the compiler can get quite long. This causes two problems. One is that the make output is hard to visually scan for errors. More seriously, the command lines can exceed the length limits of some operating systems. [...] The fact that the symbols are documented is important in order to check that config.h makes sense. The fact that there is a well-defined list of symbols that should be defined (or not) is also important for people who are porting packages to environments where configure cannot be run: they just have to fill in the blanks.

This first part is not entirely convincing for us. The path limitation seems to a problem only on ancient system and anyway, the number of options is limited: With #929 solved, we'd have preprocessor defaults for the two numeric options (-DECMULT_GEN_PREC_BITS=4 -D ECMULT_WINDOW_SIZE=15), and so the values a typical user needs to set are of type ENABLE_MODULE_FOO and -DUSE_EXTERNAL_ASM.

On the other hand, one can argue that a config header is actually a good thing for people who build without a build system because it lists and documents all the options. Unfortunately the .in formats for autotools and cmake are not really compatible, so we can't use the same file for both.

Maybe it doesn't matter all that much. In the end, we need a least one place (and as few as possible) places where the options are listed and documented, and some duplication will be unavoidable if we want to support more than one of building even if it's just "autotools" and "manual". But I still I tend to think that we should get rid of the config header entirely. The only real advantage is that it helps people without a build system to fill in the gaps but that's a rare case, and we should simply document the options somewhere else. Maybe we could have a single table in the README with multiple columns: one for the autotool argument, one for manual build, and (if this PR here finds acceptance), for cmake:

Option	./configure argument	cmake argument	manual argument
Enable the Schnorrsig module	--enable-module-schnorrsig	-DENABLE_MODULE_SCHNORRSIG=ON	-DENABLE_MODULE_SCHNORRSIG

There may even be a way to unify the last two columns.

[hebasto]

Updated ff836b6199d81e63f61afee1b8385e815ca25c47 -> f587a8cfe16704d29ad0d4409a90807ef8320147 (pr1113.06 -> pr1113.07, diff):

• addressed @fanquake's and @real-or-random's comments
• moved toolchain fails into cmake directory

[real-or-random]

Hm, this is currently very confusing... so /src/util.h (where / is the project root) has this: https://github.com/bitcoin-core/secp256k1/blob/43756da819a235d813e7ecd53eae6df073b53247/src/util.h#L10-L12

Though this is implementation-defined, basically all compilers look in the current directory first, so they look for /src/libsecp256k1-config.h, which exists on my system because autotools generated it.

Only if this file doesn't exist, the compiler will try the -I paths, which -- when set correctly -- could include /src/build, so then it would find the /src/build/libsecp256k1-config.h, which cmake generated in its out-of-tree build directory...

That's pretty annoying. So we could change the include to #include <src/libsecp256k1-config.h> and then let the build system set -I accordingly. That's okayish, but getting rid of the config header (see above) is probably even easier.

[hebasto]

Updated f587a8cfe16704d29ad0d4409a90807ef8320147 -> 5dda6460657778e52b89a7920a8b63a032f5183f (pr1113.07 -> pr1113.08):

• rebased on top of the #1116
• addressed @real-or-random's comment

[sipa]

I tried the following:

mkdir build
cd build
cmake .. -DENABLE_MODULE_SCHNORRSIG=ON
make
./src/bench

The result seems to indicate that:

• ECDH & recovery modules were built (despite configure telling me they weren't)
• schnorrsig being built (despite being experimental, and experimental modules not enabled)
• performance is very low (compiler optimizations off?)

EDIT: seems -DCMAKE_BUILD_TYPE=Release is needed to get optimizations enabled. This should perhaps be documented? (I guess this is standard, but I'm not familiar with this style of configuring things)

[real-or-random]

@sipa

ECDH & recovery modules were built (despite configure telling me they weren't)

This is probably the issue I already discovered (https://github.com/bitcoin-core/secp256k1/pull/1113#issuecomment-1171407985) ... can you try removing src/libsecp256k1-config.h, which is probably still around in your source tree from the autotools build, before trying this PR?

[sipa]

@real-or-random

can you try removing src/libsecp256k1-config.h, which is probably still around in your source tree from the autotools build, before trying this PR?

Done, that seems to help, though now I get errors because the extrakeys module doesn't seem enabled (which schnorrsig depends on). If I enable extrakeys explicitly, it works, and with -DCMAKE_BUILD_TYPE=Release added, I get believable bench output.

[real-or-random]

Oh module dependencies...

[sipa]

Those are not serious issues of course, and probably easy to resolve. It's nice to see that the hard part (it building/running in the first place, apparently including for MSVC) already seems to work.

[hebasto]

Updated 6b65314195baeee982c7d5213377d493d073f119 -> 13c63e79b56ab00bffb0ff40d8a6a0f4b03ed620 (pr1113.09 -> pr1113.10):

• rebased
• added examples/CMakeLists.txt
• option docstrings style now follows CMake's docstring style
• addressed @real-or-random's and @sipa's comments
• fixed module dependecies

[hebasto]

EDIT: seems -DCMAKE_BUILD_TYPE=Release is needed to get optimizations enabled.

By default (at least on Linux), -DCMAKE_BUILD_TYPE=Release adds -O3 -DNDEBUG flags, and -DCMAKE_BUILD_TYPE=Debug does -g.

Therefore, I'd suggest to do not use this option at all. Instead, Autotools' behavior has been re-imlemented, i.e., -O2, if no COVERAGE requested. And -O0 otherwise.

[real-or-random]

Thanks for constantly making updates, I think we're almost there. Though we may want to simplify the config system first and only then add CMake.

Therefore, I'd suggest to do not use this option at all.

I'm not sure if this is the way to go. CMAKE_BUILD_TYPE seems to be pretty idiomatic in CMake, so maybe that's what people would expect. But I need to do more reading, I haven't really checked.

So Autotools always enable -g -O2 and so far this was pretty useful for us... So as I understand it, RelWithDebInfo does this. Could we just set this as a default?

And expressing Coverage as build type in CMake seems to use build types in the right way.

edit: removed leftover draft text

[hebasto]

Updated 13c63e79b56ab00bffb0ff40d8a6a0f4b03ed620 -> e9cebaaadfa3ba1a0f233d125fe2eed50f3993cb (pr1113.10 -> pr1113.11, diff):

• improved documentation
• removed the ENABLE_DEV_MODE option

[hebasto]

@real-or-random

So Autotools always enable -g -O2 and so far this was pretty useful for us... So as I understand it, RelWithDebInfo does this. Could we just set this as a default?

The default value of the CMAKE_C_FLAGS_RELWITHDEBINFO variable is -O2 -g -DNDEBUG. I think, we want to strip out -DNDEBUG. Or to define our own custom configuration, along with Coverage?

[real-or-random]

Sorry, I guess my suggestion wasn't clear. I meant

• make RelWithDebInfo the default build type (is this even possible?)
• and independently, introduce a separate build type Coverage.

The default value of the CMAKE_C_FLAGS_RELWITHDEBINFO variable is -O2 -g -DNDEBUG. I think, we want to strip out -DNDEBUG.

Yes, we probably would want to strip it. (Though it won't hurt, we never check for NDEBUG it in the code.)

[hebasto]

Though it won't hurt, we never check for NDEBUG it in the code.

But we use assert.

[hebasto]

Updated e9cebaaadfa3ba1a0f233d125fe2eed50f3993cb -> 2ecf0dc437db7c38d4428eca9d17b0168287983a (pr1113.11 -> pr1113.12):

• rebased

CMAKE_BUILD_TYPE seems to be pretty idiomatic in CMake

• introduced two new build types: "RelWithAsserts" and "Coverage"; the latter should be used instead of the COVERAGE option
• removed the COVERAGE option

All default build types, i.e., "Debug", "Release", "MinSizeRel" and "RelWithDebInfo", are still available.

The default build type is "RelWithAsserts" which is equivalent to the Autotools' default.

[sipa]

There are only asserts in the API examples. For those it's probably better to by default leave the asserts in.

[hebasto]

@sipa

• schnorrsig being built (despite being experimental, and experimental modules not enabled)

schnorrsig has not been treated as experimental since #995.

[hebasto]

From the recent meeting on IRC:

<real_or_random> hebasto: so you think we'd still have a config file for cmake? <hebasto> if build, in general, will support config-less routine, cmake won't require it

I was wrong. The cmake/libsecp256k1-config.h.in has been dropped in the latest push.

[hebasto]

https://github.com/bitcoin-core/secp256k1/pull/1113#discussion_r910845410:

You should probably add your name here and everywhere else. (I think that's an annoyance here in the library. It would be good to switch to "libsecp256k1 contributors" but last time I brought that up, it wasn't clear if this is legally okay.)

Wrt the recent discussion on IRC, should I switch to "libsecp256k1 contributors" in this PR?

[hebasto]

According to yesterday's IRC meeting, copyright headers have been removed from the new added files.

[hebasto]

Updated dc9124b2b23c5cfe4f3524b1e7defb528b62b839 -> 8d019748b599193510a61975371805c2841d7990 (pr1113.14 -> pr1113.15):

• rebased
• improved building with MSVC
• improved hygiene of user-provided CFLAGS

[hebasto]

Updated 8d019748b599193510a61975371805c2841d7990 -> d77e4c89de78c2b2a523e9e56a931db80104a569 (pr1113.15 -> pr1113.16, diff):

• added make test which is CMake's analogue of Autotools' make check
• various improvements based on feedback from bitcoin/bitcoin#25797

The PR description has been updated with detailed Autotools -- CMake Feature Parity Tables.

[hebasto]

Updated d77e4c89de78c2b2a523e9e56a931db80104a569 -> 3dc4de05299835aa4ff39275507858ebd2ecb8f9 (pr1113.16 -> pr1113.17, diff):

• fixed accidentally broken compatibility with CMake 3.1

Tested on Debian 8 with non-system CMake installation:

$ cmake --version
cmake version 3.1.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).