[Doc] Capturing Git scm information when `conanfile.py` is in subfolder

[aagor]

Hi,

we are using the Capturing Git scm information workflow to package our own Conan packages. However, the use case as described in the documentation only works for the simple cases, where the conanfile.py is in the root of the git repository.

If not, it's difficult to get both the local development workflow via conan build and the cache workflow via conan create working simultaneously. The solution we come up with looks like this:

def layout(self):
    # Needed for local developement workflow.
    self.folders.root = "../../<that many times as needed to get to repo root>"
    # But build folder etc. in the correct subfolder.
    self.folders.subproject = "path/to/subfolder/in/repo"
    cmake_layout(self)

    # Set source to the git root, as the checkout via `source()` always clones the full repo.
    self.folders.source = "."

def build(self):
    cmake = CMake(self)
    # As `self.source_folder` is the git root, we need to tell CMake where our "real" folder is.
    cmake.configure(build_script_folder=self.folders.subproject)
    ...

• Is there a better way to achieve this?
• If not, I think this should be documented.

I feel like this case is not uncommon. If you are having multiple Conan packages in a single git repository or don't have the Conan package at the root folder, this special handling is needed.

If you need to demonstrate the problem, you can use:

$ git clone https://github.com/conan-io/examples2.git
$ mkdir conan_capture_scm
$ cp -r examples2/examples/tools/scm/git/capture_scm conan_capture_scm/
$ cd conan_capture_scm/
$ git init
$ git add .
$ git commit -m WIP
$ cd capture_scm/
$ conan create .
$ conan build .

[KUGA2]

Additional question: What is the default for self.folders.subproject and could cmake.configure(build_script_folder=self.folders.subproject) be the default?

[memsharded]

I agree this is not straightforward.

Conan has a helper function called move_folder_contents() that we use in some tests, to help implementing this flow. The main challenge is that the checkout of a git repo that is a super-project containing multiple Conan packages, is that in order to mimic the layout in the cache, the only way is to do some re-arrangements of the files, and that is where the move_folder_contents() helps.

The problem is that move_folder_contents() has not been documented yet, I'll try to add this to the docs, together with some full example of this layout.

[aagor]

But isn't moving folders around etc. more complicated than the solution above? The solution above works locally and in the cache and seems just simpler.

No moving involved, just defined the layout in a (hopefully) clever way.

[aagor]

First of all, if move_folder_contents() is a viable solution depends if the subtree is completelety independent from everything outside the subdirectory. This may be the case sometimes, sometimes not. For example, some dotfiles (for linters etc.) or generic config/stuff may still be used.

If the whole git repository should be visible, I don't think setting the source folder as described above is nice.

It would be better to have the source_folder on the correct path, this will simplify a few things:

• No specific source folder settings necessary inside layout() (self.folders.source = '.').
• No specific stuff in build() necessary (cmake.configure(build_script_folder=self.folders.subproject)).
• self.cpp.source inside layout() (and other similar stuff) won't need special care.
• It's more intuitive and straight forward.

This could be achieved via git = Git(self, self.export_sources_folder), however this doesn't work with the way conan currently does the clone, as the directory isn't empty, due to the source_folder gets created before source(). Removing this folder manually seems error-prone and with exported sources you have the same problem.

I suggest that instead of git clone, git init + remote add + fetch + checkout should be used. This would basically be identical, but ignore already existing directories/files.

Furthermore, there are multiple ways how to clone a git repository. E.g. one of our (big) repositories has the following sizes:

• Normal clone: 633 MB
• Only single branch clone: 554 MB
• Shallow clone with depth=1: 291 MB

The last one is currently possible in conan via git.fetch_commit(self.conan_data["scm"]["url"], self.conan_data["scm"]["commit"]). However that depends on the internas of the coordinates_to_conandata()/checkout_from_conandata_coordinates(). Is that considered stable?

We may be able to create a PR for the checkout_from_conandata_coordinates(), where you can select these two/three modes (with the first one as the default like currently).

[aagor]

Hi @memsharded,

I've just seen no one is assigned on this issue, so I'm mentioning you just in case this gets missed.

In the meantime I played a little with move_folder_contents and it seem to work, however, there are some issues in the implementation especially at edge cases. But I will open a seperate issue for that in the future.