Fetch content

[flagarde]

Base on the conversation on #437. This PR is using the "standard" CMake feature with find_apckage etc.. But the CPM.cmake people need to change the version in command project before releasing.

Should be a better solution than #437

This is an example with the new features provided by this PR in action:

cmake_minimum_required(VERSION 3.14)

project(MyWonderfulProject)

find_package(CPM)
if(NOT CPM_FOUND)
include(FetchContent)

FetchContent_Declare(
  CPM
  GIT_REPOSITORY https://github.com/cpm-cmake/CPM.cmake.git
  GIT_TAG        v0.37.0
)
FetchContent_MakeAvailable(CPM)
else()
message(STATUS "I'm using the installed version")
endif()

## CPM automatically included

# add executable
add_executable(main main.cpp)

CPMAddPackage("gh:fmtlib/fmt#7.1.3")
CPMAddPackage("gh:nlohmann/[email protected]")
CPMAddPackage("gh:catchorg/[email protected]")

# link dependencies
target_link_libraries(main fmt::fmt nlohmann_json::nlohmann_json Catch2::Catch2WithMain)

The other feature is that the PR allow CPM to be installed on user system (for example)

git clone https://github.com/cpm-cmake/CPM.cmake.git
cd CPM.cmake
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=CPMInstallPath ..
make install

then user can use previous cmake code and let CMake find CPM in a la CMake way :

cmake -DCPM_ROOT=CPMInstallPath ..

As you can see this PR makes CPM followinf CMake standards for package

[flagarde]

Concerning the Style it seems there is a clash with my Clion and the cmake-format file in the repo. The file gets regenerated with the wrong formatting. Maybe a PR with a pre-commit config files would help

[LecrisUT]

Concerning the Style it seems there is a clash with my Clion and the cmake-format file in the repo. The file gets regenerated with the wrong formatting. Maybe a PR with a pre-commit config files would help

Yeah, I prefer CLion's formatting in some parts as well. I've opened an issue about that on cmake-format. The pre-commit file for that would be straightforward but would need the owner to add pre-commit-ci to automatically cmake-format

[flagarde]

Concerning the Style it seems there is a clash with my Clion and the cmake-format file in the repo. The file gets regenerated with the wrong formatting. Maybe a PR with a pre-commit config files would help

Yeah, I prefer CLion's formatting in some parts as well. I've opened an issue about that on cmake-format. The pre-commit file for that would be straightforward but would need the owner to add pre-commit-ci to automatically cmake-format

At least having a pre-commit config could simplify the process for developper. It can be easy to forget to run cmake-format. With pre-commit it would be at least automatic on developper computers. pre-commit.ci for the repo would be the best option of course

[flagarde]

@LecrisUT Does everything looks fine now?

[flagarde]

I don't know how to test this case

[LecrisUT]

lgtm

Here's a snippet you can run:

cmake_minimum_required(VERSION 3.24)
project(TestCPMFetchContent)

include(FetchContent)
FetchContent_Declare(CPM
		GIT_REPOSITORY https://github.com/flagarde/CPM.cmake
		GIT_TAG FetchContent
		OVERRIDE_FIND_PACKAGE)
find_package(CPM)

[LecrisUT]

Hmm my tests on this do not succeed yet. I've added a message(WARNING) inside CPMConfig.cmake.in and it did not print out

[flagarde]

cmake_minimum_required(VERSION 3.24)
project(TestCPMFetchContent)

include(FetchContent)
FetchContent_Declare(CPM
		GIT_REPOSITORY https://github.com/flagarde/CPM.cmake
		GIT_TAG FetchContent
		)


FetchContent_MakeAvailable(CPM)
find_package(CPM)
# the install option has to be explicitly set to allow installation
CPMAddPackage(
  GITHUB_REPOSITORY jarro2783/cxxopts
  VERSION 2.2.1
  OPTIONS "CXXOPTS_BUILD_EXAMPLES NO" "CXXOPTS_BUILD_TESTS NO" "CXXOPTS_ENABLE_INSTALL YES"
)

this one create a warning but still working

[LecrisUT]

Of course, but that's the intended behaviour :smile: The warning I get there is:

CMake Warning at cmake-build-debug/_deps/cpm-src/cmake/CPM.cmake:80 (message):
  CPM: Your project is using an unstable development version of CPM.cmake.
  Please update to a recent release if possible.  See
  https://github.com/cpm-cmake/CPM.cmake for details.

Which is ok because it is not a tagged version :)

This one won't show the error even though it is not a release ;)

FetchContent_Declare(CPM
		URL https://github.com/flagarde/CPM.cmake/archive/refs/heads/FetchContent.zip
		OVERRIDE_FIND_PACKAGE)

[flagarde]

Of course, but that's the intended behaviour smile The warning I get there is:

CMake Warning at cmake-build-debug/_deps/cpm-src/cmake/CPM.cmake:80 (message):
  CPM: Your project is using an unstable development version of CPM.cmake.
  Please update to a recent release if possible.  See
  https://github.com/cpm-cmake/CPM.cmake for details.

Which is ok because it is not a tagged version :)

This one won't show the error even though it is not a release ;)

FetchContent_Declare(CPM
		URL https://github.com/flagarde/CPM.cmake/archive/refs/heads/FetchContent.zip
		OVERRIDE_FIND_PACKAGE)

I'm talking about this warning :

CMake Warning at CMakeLists.txt:13 (find_package):
  By not providing "FindCPM.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "CPM", but
  CMake did not find one.

  Could not find a package configuration file provided by "CPM" with any of
  the following names:

    CPMConfig.cmake
    cpm-config.cmake

  Add the installation prefix of "CPM" to CMAKE_PREFIX_PATH or set "CPM_DIR"
  to a directory containing one of the above files.  If "CPM" provides a
  separate development package or SDK, be sure it has been installed.

I tested we need PARENT_SCOPE for the variable and it works. We can try with PARENT_SCOPE and move to a CACHE variable if necessary

[LecrisUT]

cmake_minimum_required(VERSION 3.24)
project(TestCPMFetchContent)

include(FetchContent)
FetchContent_Declare(CPM
		GIT_REPOSITORY https://github.com/flagarde/CPM.cmake
		GIT_TAG FetchContent
		)


FetchContent_MakeAvailable(CPM)
find_package(CPM)
# the install option has to be explicitly set to allow installation
CPMAddPackage(
  GITHUB_REPOSITORY jarro2783/cxxopts
  VERSION 2.2.1
  OPTIONS "CXXOPTS_BUILD_EXAMPLES NO" "CXXOPTS_BUILD_TESTS NO" "CXXOPTS_ENABLE_INSTALL YES"
)

this one create a warning but still working

You have an error there. Without OVERRIDE_FIND_PACKAGE you must not use find_package, only FetchContent_MakeAvailable. These two are mutually exclusive!

CPM_VERSION and everything is handled by FetchContent_MakeAvailable

[flagarde]

cmake_minimum_required(VERSION 3.24)
project(TestCPMFetchContent)

include(FetchContent)
FetchContent_Declare(CPM
		GIT_REPOSITORY https://github.com/flagarde/CPM.cmake
		GIT_TAG FetchContent
		)


FetchContent_MakeAvailable(CPM)
find_package(CPM)
# the install option has to be explicitly set to allow installation
CPMAddPackage(
  GITHUB_REPOSITORY jarro2783/cxxopts
  VERSION 2.2.1
  OPTIONS "CXXOPTS_BUILD_EXAMPLES NO" "CXXOPTS_BUILD_TESTS NO" "CXXOPTS_ENABLE_INSTALL YES"
)

this one create a warning but still working

You have an error there. Without OVERRIDE_FIND_PACKAGE you must not use find_package, only FetchContent_MakeAvailable. These two are mutually exclusive!

CPM_VERSION and everything is handled by FetchContent_MakeAvailable

I khow but how we know how CPM as been included upstream as now it could have been installed (so you need find_package) ot by fetch_content so you need nothing ... That's why i want to try using fetchcontented one but refinded (uselessly but unknwowingly) by find_package ... Like this consumer just have to do find_package(CPM) whatever the way CPM is included.

To say it in other way I would like to be independent on the way CPM is included... Maybe we know how maybe not...

[flagarde]

maybe a NO_POLICY_SCOPE in include(cmake/CPM.cmake) would be safer as we set some policy

[LecrisUT]

maybe a NO_POLICY_SCOPE in include(cmake/CPM.cmake) would be safer as we set some policy

I don't know about this. I defer to others on this.

I khow but how we know how CPM as been included upstream as now it could have been installed (so you need find_package)

Oh I see, this is the example that you were referring to where B might now be calling find_package(CPM). Hmm I think it should be up to the maintainers of B to properly detect CPM is loaded or not. E.g. they can expose an option to use either system or downloaded version. In principle C should use OVERRIDE_FIND_PACKAGE if they know B looks for CPM as well so that they share the same detected version.

In other words we should not interfere in how they want to organize and have FetchContent do the heavylifting.

[flagarde]

maybe a NO_POLICY_SCOPE in include(cmake/CPM.cmake) would be safer as we set some policy

I don't know about this. I defer to others on this.

I khow but how we know how CPM as been included upstream as now it could have been installed (so you need find_package)

Oh I see, this is the example that you were referring to where B might now be calling find_package(CPM). Hmm I think it should be up to the maintainers of B to properly detect CPM is loaded or not. E.g. they can expose an option to use either system or downloaded version. In principle C should use OVERRIDE_FIND_PACKAGE if they know B looks for CPM as well so that they share the same detected version.

In other words we should not interfere in how they want to organize and have FetchContent do the heavylifting.

maybe a NO_POLICY_SCOPE in include(cmake/CPM.cmake) would be safer as we set some policy

I don't know about this. I defer to others on this.

I khow but how we know how CPM as been included upstream as now it could have been installed (so you need find_package)

Oh I see, this is the example that you were referring to where B might now be calling find_package(CPM). Hmm I think it should be up to the maintainers of B to properly detect CPM is loaded or not. E.g. they can expose an option to use either system or downloaded version. In principle C should use OVERRIDE_FIND_PACKAGE if they know B looks for CPM as well so that they share the same detected version.

In other words we should not interfere in how they want to organize and have FetchContent do the heavylifting.

C uses B and we don't know if B is using CPM (it could even depends on option you give to B fetching it like optinal libraries etc). If it uses it and has fetched it we don't need to include CPM by ourself but there is no way to know it.. with this trick (providing a generated CPMConfig.cmake) C can do :

find_package(CPM)
if(CPM_NOTFOUND)
### fetch it
endif()

I mean you probably could know it checking if CPM variables are defined etc but it's a bit ugly

If we don't do the trick C could only do this if B as included it by find_package ... An other way could be to define CPM_FOUND in CPM.cmake but i'm not sure

[LecrisUT]

It's a bit tricky because find_package(B) will not propagate find_package(CPM) unless it is OVERRIDE_FIND_PACKAGE. C is the top level so it should be deciding if it wants a specific version of CPM or the system installed one. If C does not use CPM altogether, then it should not touch it unless A or B expose the relevant option to it.

Maybe the only ambiguity can come from A and B. But scopes will not propagate all around from A.CPM to B unless A specifically specifies each scoped variable, e.g. set(CPM_DIR PARENT_SCOPE). That would be too much to ask. We could make a special exception to cache variabe CPM_DIR for this specific cases (don't FORCE cache it though).

I think the recent version have an intelligent way to design this. Need to read the docs more

[LecrisUT]

Found what we were looking for: https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_PACKAGE_REDIRECTS_DIR.html#variable:CMAKE_FIND_PACKAGE_REDIRECTS_DIR

[flagarde]

Oh. Interesting but this is a very new feature. Maybe you can try wil the actual code and check of people have problem.. A cache variable could be a solution.

Anyway I think we are fine because the ability to download CPM by fetchcontent os new so we could say to user try it but still don't trust i will work all the time... And it can be polishing while more people test it.

There is always corner cases unexpected or people finding fancy or nice bahaviour

I seems the actual PR is doing what people should expect in most case. Would you agree to merge in this way?

[LecrisUT]

Yes 👍 from me. These stuff are just QOL improvements we can do later.

You still need a proper reviewer to accept and cmake-format it.

[flagarde]

Yes +1 from me. These stuff are just QOL improvements we can do later.

You still need a proper reviewer to accept and cmake-format it.

Wonderful, let's wait for the reviewing and if it's ok I can do a cmake-format after

[TheLartians]

Hey thanks for the PR! To be sure I understand, the idea of this is to allow using FetchContent to shorten the required code for integrating CPM.cmake in a project?

I have some questions regarding the approach.

• One big concern while designing CPM was to support fast and offline configurations. Would projects using this still be able to configure when offline assuming CPM and all dependencies are locally cached?
• From my understanding FetchContent currently doesn't work as it expects a project directory and not single CMake files. Could we not simply add a zipped directory containing CPM inside a CMakeLists.txt to the releases and pointing FetchContent to that URL to support it?

[LecrisUT]

Could we not simply add a zipped directory containing CPM inside a CMakeLists.txt to the releases and pointing FetchContent to that URL to support it?

This would needlessly overcomplicate it. It is best to stick to the built-in supported format. We only need 2-3 files to get it working CmakeLists.txt, CPMConfig.cmake (if youwant to enable find_package), and CPM.cmake. It works with tar balls, local git repos, submodules, you name it.

Would projects using this still be able to configure when offline assuming CPM and all dependencies are locally cached?

From my testing, once downloaded, you can freely change the source in _deps so it doesn't re-download or anything. Also note: FetchContent, including MakeAvailable only execute at configuration stage, so it shouldn't be any concern.

[flagarde]

Hey thanks for the PR! To be sure I understand, the idea of this is to allow using FetchContent to shorten the required code for integrating CPM.cmake in a project?

I have some questions regarding the approach.

* One big concern while designing CPM was to support fast and offline configurations. Would projects using this still be able to configure when offline assuming CPM and all dependencies are locally cached?

* From my understanding `FetchContent` currently doesn't work as it expects a project directory and not single CMake files. Could we not simply add a zipped directory containing `CPM` inside a `CMakeLists.txt` to the releases and pointing `FetchContent` to that URL to support it?

Yes, the idea is to use FetchContent to download CPM (not using get_cpm at all).

When CPM is Fetched I thimk they will be any problem going online because CMake will know CPM as been download and is available on the binar directory.

FetchContent just need a CMakeLists to know how to configure the FetchContent (I think it can be even skipped but we need it)..

[flagarde]

Could we not simply add a zipped directory containing CPM inside a CMakeLists.txt to the releases and pointing FetchContent to that URL to support it?

This would needlessly overcomplicate it. It is best to stick to the built-in supported format. We only need 2-3 files to get it working CmakeLists.txt, CPMConfig.cmake (if youwant to enable find_package), and CPM.cmake. It works with tar balls, local git repos, submodules, you name it.

Would projects using this still be able to configure when offline assuming CPM and all dependencies are locally cached?

From my testing, once downloaded, you can freely change the source in _deps so it doesn't re-download or anything. Also note: FetchContent, including MakeAvailable only execute at configuration stage, so it shouldn't be any concern.

Yes, this PR add an other feature. It could allow system manager to install CPM as package on the system. So the 2 added files can be very useful I think and they are standard CMAke Config files generated for CPM.

It can makes CPM closer to the CMake standard of find_package etc... A user could install CPM in system or in his home directory (whereever he wants) and then do in other project :

find_package(CPM)

and launch

cmake -DCPM_ROOT=cpm_path ...

which is a standard way post CMake3.14 to let CMake find some package in paths different than the standard ones.

[CraigHutchinson]

Firstly, CPM being structured as a normal package is awesome.

Some general observation is to please update the PR description with example usage?

• Trolling the long thread is not simple
• Some overview of the discussed changes benefits, known issues, etc = general changes for the end users ;)

Some thoughts on edge-cases:

• Does this handle the version being provided to find_package()?
  • I will have to try this PR sometime soon, but if you have an old CPM installed on the system, can/will, CPM dog-food itself to fetch the latest version! 🚀
• With get_cpm.cmake we had the confidence the download would be put under CPM_SOURCE_CACHE outside the build artifacts folder.
  • Could we print a hint to developers if the CPM version they are using is not installed? Most likely only when CPM_SOURCE_CACHE has been set I guess. -- EDIT: Devs could just provide SOURCE_DIR ${CPM_SOURCE_CACHE} on the fetch I think to place the fetch outside of the source

[flagarde]

@CraigHutchinson Yes sorry for the long PR comments etc. I will had some example and usages.

The CPM package autocreate a version check file but I use the keywork SameMajorVersion as compatibility (mainly because it seems minor versions add features but don't change so much behaviours, and because this is mainly a CPM developer decision, maybe a SameMinorVersion would be better).

The user can find_package and require a version and if not installed, FetchContent the new version. (The version provided by the CPM developper in the project command is used to generate a versionfile by CMake). I imagine CPM could be used to download himself but in this case,we should unset the CACHE variable the find_package has set. But the CPM is loaded by include so it should work

If we don't unset the cache variable, the next find_package(CPM) would use them (while the newest CPM version file would be included and use). So it should requires a bit of polishing but is feasible and should works now but in a messy-unclean way.

For the hint message I think it should just be included in the CMakeLists.txt in CPM, just before we include the CPM.cmake file.

-- EDIT: Devs could just provide SOURCE_DIR ${CPM_SOURCE_CACHE} on the fetch I think to place the fetch outside of the source. YES, all would be managed in CMake classical way :)

[flagarde]

@CraigHutchinson I have provided some examples this PR allows to do in my first message.

[CraigHutchinson]

A couple of extra suggestions:

• Update document so people know about this.
• Maybe Mark as experimental or flag the old get_cpm as deprecated?
• Update get_cpm to use the FetchContent approach 🚀

[flagarde]

@CraigHutchinson Documents should be updated of course. I have focused on the technical side to have feedback from CPM people.

This PR could be documented as experimental and as potential successor of get_cpm.

For your last point I think it would be better to say that get_cpm is the legacy way to obtain CPM and people should try the new approach. I fear that if we use FetchContent inside get_cpm people will continue to use it and this will create one more case to try :

1. Old get_cpm
2. get_cpm using FechtContent
3. This PR

One of the goal of this PR is to rid of get_cpm

[flagarde]

@CraigHutchinson Do you require document changes in this PR?