GitHub release tarballs do not include submodules

[ZedThree]

Apologies if you're already aware of this, but the GitHub-created tarballs on releases don't include any git submodules, so they fail to build. And because they're also not git repos, you can't just run git submodule update --init --recursive to check them out.

Our solution was to add git-archive-all as a submodule, and a make target (dist, for tradition) that calls it to produce a tarball with all the submodules, and upload that manually to the GitHub release. Our wrapper script is here if that's helpful for you.

[tclune]

I was not aware of the tarball issue, but it does not surprise me. And I certainly had not anticipated the complication you mention.

I intentionally designed the build system so that if the submodules are not already cloned by a recursive, the cmake will automatically download the submodule during the configuration process. This will of course fail if one attempts to build in an environment that cannot access the internet. But, this of course assumes that the ./.git directory is present. Sigh.

Given a choice, I would rather have GitHub not produce the tarball at all and instead require users to use git clone. But of course, it appears that I cannot even delete the existing artifacts manually.

I'll now walk around depressed for a little while, then get over it and then probably incorporate your solution. :-)

[ZedThree]

Yes, we went through the exact same process and thinking! It's very annoying. At least it is possible to upload tarballs to previous releases, even if it is a bit tiresome.

There is some built-in functionality to CMake 3.11 that might be useful: https://cliutils.gitlab.io/modern-cmake/chapters/projects/fetch.html

[tclune]

Yes - that might be preferable. Currently the strategy is to try find_package(). If not FOUND, then it uses submodules. I could add a 2nd check so that if submodules fails it fails over to FetchContent(). This still may not satisfy some users that want to build in an environment without access to GitHub. (E.g., the batch nodes in our computing environment.)

Probably should do both. But if I procrastinate, maybe Git or CMake will have solved the problem. Happens delightfully often with these two tools.

[kurtsansom]

I believe this is still an issue, I am creating a Dockerfile that builds pfunit from the release package. and fails on cmake. but it sounds like it is prefered to do a clone anyways.

cmake error:

CMake Error at cmake/build_submodule.cmake:30 (add_subdirectory):
  The source directory

    /heat/external/pfunit/extern/gFTL

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  extern/CMakeLists.txt:2 (build_submodule)


CMake Error at cmake/build_submodule.cmake:30 (add_subdirectory):
  The source directory

    /heat/external/pfunit/extern/gFTL-shared

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  extern/CMakeLists.txt:3 (build_submodule)


CMake Error at cmake/build_submodule.cmake:30 (add_subdirectory):
  The source directory

    /heat/external/pfunit/extern/fArgParse

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  extern/CMakeLists.txt:4 (build_submodule)

[tclune]

Yes I just discovered this myself. And spent the day working through the issue. I've found the culprit, but am waiting to hear back from the contributor in case he knows a better workaround.

Hopefully fixed in 1-2 days.

[ZedThree]

@tclune Just to clarify, I'm pretty sure @kurtsansom 's issue above is unrelated to my recent changes, and is because the release tarball here: https://github.com/Goddard-Fortran-Ecosystem/pFUnit/releases/tag/v4.1.15 is not a git repo (i.e. it doesn't include the .git directory) and so doesn't include the submodules (and because it's not a git repo, there's no automatic way to get them)

Two possible solutions:

1. Use a tool like git-archive-all to bundle all the submodules into the tarball and manually upload it, rather than use GitHub's own source bundle. This could be automated with GitHub actions
2. Use CMake's FetchContent to download the dependencies, probably instead of having git submodules.

I could probably whip up a PR demonstrating the latter option

[tclune]

Yes - I use a tool to generate the extra tar balls. But I sometimes forget. Lately I've been a bit better.

But since I cannot build the latest version in both modes that I intend to support, I'd rather just get the fix and then delete the broken release. (And add the full tar at that point.)

[sizmailov]

Just faced same issue when tried to install latest .zip from github release v4.2.1. Do you mind to add installable .tar asset?

[tclune]

Sorry about that. Proper tar is now uploaded. I hope to get this automated soon.