prevent downloads during build

[christianrauch]

The build process uses FetchContent_Declare to download additional sources during the build process, instead of git submodules. Some packaging processes, such as snap and flatpak, run builds in a sandbox and prevent; and some (Debian) discourage network access during build time and expect all sources to be available before the build.

On master, the add_subdirectory(Geo) tries to fetch https://github.com/geographiclib/geographiclib.git during the build, which causes build issues on flatpak:

[1/9] Creating directories for 'geographiclib-populate'
[1/9] Performing download step (git clone) for 'geographiclib-populate'
Cloning into 'geographiclib-src'...
fatal: unable to access 'https://github.com/geographiclib/geographiclib.git/': Could not resolve host: github.com
Cloning into 'geographiclib-src'...
fatal: unable to access 'https://github.com/geographiclib/geographiclib.git/': Could not resolve host: github.com
Cloning into 'geographiclib-src'...
fatal: unable to access 'https://github.com/geographiclib/geographiclib.git/': Could not resolve host: github.com
-- Had to git clone more than once: 3 times.
CMake Error at geographiclib-subbuild/geographiclib-populate-prefix/tmp/geographiclib-populate-gitclone.cmake:39 (message):
  Failed to clone repository:
  'https://github.com/geographiclib/geographiclib.git'

Would it be possible to add geographiclib as a submodule to the git project in order to allow builds in sandboxed/isolated environments?

[HTRamsey]

I'm largely unfamiliar with building via flatpaks, but I do understand they are generally preferred over appimages for security & versioning/updating reasons. Do other repos that build with flatpak just not let cmake handle dependencies at all? I believe @DonLakeFlyer generally prefers to use submodules too.

One of the reasons I was in favor of using cmake to handle dependencies was because it's really easy to separate dependencies based on platform (which is nice because we started running out of space in the github CI builds). Also, how would you handle the parts where you get dependencies from somewhere other than github? One example being that cmake downloads gstreamer stuff during the build. I did intend on changing the FetchContent command to use FIND_PACKAGE_ARGS to find installed versions of dependencies at some point, so that could be a solution to geographiclib at least.

[christianrauch]

Do other repos that build with flatpak just not let cmake handle dependencies at all?

Yes. At least for flathub, all builds run in a sandbox that prevents network access etc. during the build. All sources have to be specified before the build.

Also, how would you handle the parts where you get dependencies from somewhere other than github?

I am not entirely sure what you mean. If you add subprojects as submodules, you can add any remote repo, not just those from github.com. Other dependencies can usually be installed with platform-specific package managers (apt, dnf, brew, vcpkg, ...). Finally, you can always install dependencies from source. The latter is what is usually done for flatpaks.

[heeplr]

This also breaks gentoo live builds. Network access during configure phase is always disabled for security reasons.

I believe @DonLakeFlyer generally prefers to use submodules too.

Submodules would be the preferred way.

One of the reasons I was in favor of using cmake to handle dependencies was because it's really easy to separate dependencies based on platform (which is nice because we started running out of space in the github CI builds).

That could be a problem. Maybe different branches for each platform could solve this. Not sure.

Also, how would you handle the parts where you get dependencies from somewhere other than github?

As long as dependencies use git, other sources besides github should be no problem.

[heeplr]

btw. this affects all sources fetched by cmake.

FetchContent shouldn't attempt to download, if the submodule has already been fetched. It should (silently) succeed. That way it would work without git fetch'ing all submodules (i.e. non-recursive clone).

To avoid a cluttered CI environment, maybe submodules could then be git fetch'd manually or filtered with some approach like: https://stackoverflow.com/questions/56745097/exclude-submodule-of-a-submodule

[HTRamsey]

I will look into this once the custom build stuff is done. I believe we only have one submodule that doesn't use git but it's for Android only (this was actually my intended question rather than GitHub, but I realized it doesn't matter if it's not Linux). Maybe there's some setup where we can leave all the other build platforms alone besides linux

[christianrauch]

Is there news on preventing automatic downloads during the build via CPMAddPackage or similar? There are a lot of packaging tools (Debian, flatpak, snapcraft) that rely on having the full source available before the build.

[HTRamsey]

I did add CPM so if that'll work then yes

[christianrauch]

I did add CPM so if that'll work then yes

Sorry, I do not understand that. The CPMAddPackage will still download sources from the internet during the build, right? So this will still not work unless all sources are obtained before the build starts.

[HTRamsey]

Ah okay I thought that's why you mentioned CPMAddPackage in the first place. Then no.

[DonLakeFlyer]

I'm dubious on the current state of dependency management. It continues to seem overly complex. To me submodules were simpler and provided for a system which did not require downloads at configure time. I can see myself sitting in the bush in Zimbabwe and deleting my build by mistake and totally hosing myself without an internet connection. You could theoretically alleviate a bit of this by moving the cpm module cache location outside the build structure. But that still doesn't solve the problem described here. I know a lot of effort has gone into the current cmake dependency stuff but my opinion is we should move back to submodules.

[HTRamsey]

Actually when it's finished that's the point of using stuff built off of CMake's Find_Package. It's literally a single parameter change whether you pull dependencies from a folder of git submodules, CPM cached dependencies, or just download them. You don't have to download them during configure time so I don't see the issue personally. It's a trivial setup when it's actually fully implemented

[DonLakeFlyer]

That's kind of what I figured. I think we should go back to submodules and then find package into there for building. We shouldn't have the option to do it any other way.

[HTRamsey]

Up to you. People have preferences which is where I see the value in providing the option. It's not like it actually changes the complexity of the CMake scripts. Git submodules have fallen out of favor and FetchContent/FindPackage have become the closest thing to an industry standard CPP dependency management has for good reason in my opinion.

Dealing with git submodules when switching branches is extremely frustrating too. Some of the older git submodules from v3.X don't even exist anymore so it's actually pretty difficult to even go back to older branches.

[DonLakeFlyer]

I've lived through optional build things in QGC over the course of many, many years. And although they may be nice to provide, my experience with that tells me that from an open source standpoint it's a nightmare of support. The discussion goes like this:

• Someone reports there build doesn't compile or when I run it this specific feature doesn't work
• I then spend a bunch of time trying to figure out why that is
• I spend more time trying to figure out what is wrong
• Then suddenly the person says "Oh wait, I'm using this not that"
• Which then leads to the fact that using "that" no longer works since it's not verified in CI
• To which I say, don't do "that"

I just don't have the time to deal with those sorts of problems. QGC build used to be like that. Over the course of 10+ years I locked everything down to specific versions and specific usage. That way everyone has the ability to simply replicate what happens when a build is created and dropped from CI. And I don't have to deal with random crap.

As far as submodules versus FetchContent:

• I kinda don't care about ancient history of QGC builds past last stable. FetchContent would have similar problems if you are looking for specific versions.
• How do you solve the problem of Snap/Flatpak builds if you are doing FetchContent? QGC is already in Flatpak store.

[DonLakeFlyer]

So I tried using CPM_SOURCE_CACHE to move the cache outside the build directory. This way I can do a clean build without having to download the sources again. It sort of works. It specifically doesn't work with this pattern:

if(NOT QGC_BUILD_DEPENDENCIES)
    if(LINUX)
        set(EXIV2_RELEASE_URL "https://github.com/Exiv2/exiv2/releases/download/v${EXIV2_VERSION}/exiv2-${EXIV2_VERSION}-Linux64.tar.gz")
    elseif(MACOS)
        set(EXIV2_RELEASE_URL "https://github.com/Exiv2/exiv2/releases/download/v${EXIV2_VERSION}/exiv2-${EXIV2_VERSION}-Darwin.tar.gz")
    endif()
endif()

if(EXIV2_RELEASE_URL)
    include(CPM)
    CPMAddPackage(
        NAME exiv2
        URL ${EXIV2_RELEASE_URL}
        VERSION ${EXIV2_VERSION}
        DOWNLOAD_ONLY
    )

    if(exiv2_ADDED)
        target_include_directories(AnalyzeView PRIVATE ${exiv2_SOURCE_DIR}/include)
        target_link_directories(AnalyzeView PUBLIC ${exiv2_SOURCE_DIR}/lib)
        target_link_libraries(AnalyzeView PRIVATE exiv2)
        return()
    endif()
endif()

It downloads from that url every time. That said exiv2 is the only example of this pattern. Is there some way to make this work with the cache as well? If so, that would be close to making me happy.

But that still leaves the Flatpak problem.

[HTRamsey]

If I remember correctly QGC_BUILD_DEPENDENCIES isn't set completely correctly at the moment. I'd just delete the EXIV2_RELEASE_URL stuff I guess as always build from source and rely on the cache from there on. Honestly that should be almost identical to git submodules. My plan was to allow setting of the CPM parameters (like CPM_SOURCE_CACHE, CPM_DOWNLOAD_ALL, CPM_USE_LOCAL_PACKAGES, CPM_DONT_UPDATE_MODULE_PATH, etc) to determine where the dependencies are. You could just point to git submodules instead of downloading them using these.

As far as switching to older branches, git has issues even letting you switch to a branch with an invalid submodule pointer. At least with FetchContent you can still easily switch and just change the URL to download from. Also, FetchContent has a FindPackage override now, so it is basically a generic download/local package finder. I know it sounds like more to maintain, but if I set it up all correctly it really should just be a single option that says "look for dependencies in this folder, if they don't exist then download them" which then works for git submodules, CPM cache, just straight up locally downloaded sources, etc. This is useful because then you can pull dependencies from things that aren't git repos. Namely the Android GStreamer sources and such.

My question about Flatpak is when are the submodules downloaded then? How does that work differently than downloading during configure time?

[DonLakeFlyer]

I'd just delete the EXIV2_RELEASE_URL stuff I guess as always build from source and rely on the cache from there on.

I'll do that. Then we can be clean on using the cache.

[christianrauch]

For the last release (4.4.4.) this issue seem to have disappeared. But for the latest tagged release (5.0.0), the "download" issue is back:

-- CPM: Adding package ulog_cpp@ (main to /run/build/QGroundControl/_flatpak_build/cpm_modules/ulog_cpp/7a69708ff89747582667f268eeaee7cd22ba9f41)
[1/9] Creating directories for 'ulog_cpp-populate'
[1/9] Performing download step (git clone) for 'ulog_cpp-populate'
Cloning into '7a69708ff89747582667f268eeaee7cd22ba9f41'...
fatal: unable to access 'https://github.com/PX4/ulog_cpp.git/': Could not resolve host: github.com

Why is this so inconsistent? Did you try to prevent these downloads and then someone added a dependency download for ulog_cpp again?

[HTRamsey]

as far as I know v5 shouldn't have ever worked, it's always downloaded stuff during config

[christianrauch]

as far as I know v5 shouldn't have ever worked, it's always downloaded stuff during config

I don't get what you want to say. Do you mean version 4.4.4 should have "downloaded stuff during config" or that v5 should not have been tagged yet?

[HTRamsey]

You said the download issue was back for v5, I don't understand how it ever went away because the daily builds leading up to v5 have always downloaded dependencies during configuration. So I don't see how it could be inconsistent, it should have always been the same. And v4.4 has never downloaded anything, so I don't see how the issue could have disappeared when there shouldn't have been a problem to begin with