Conan 2.0 Git Authentication and Credentials

[MikeyH-84]

What is your question?

Hi,

Our company manages our projects via GitLab. They are internal and hence private, requiring credentials for authentication.

We are following the Conan 2.0 migration guide here. This document does not talk about credentials and how to authenticate. Conan Git 2.0 does not appear to have any built-in mechanisms for handling authentication.

The only related issue/question I can find is here. This discussion mentions that it is bad practice to merge credentials directly into the URL. I am also not sure this is possible with GitLab. If I am interpreting the discussion correctly, the only solution is to write a custom wrapper around the Conan tools Git object? To create a custom Git credentials manager?

Before now we have used the Conan exports attributes to handle copying source code to the local Conan cache. But this results in uploading the source with the package so the recipe can be stand-alone. We do not want our source code uploaded hence the migration to Conan SCM/Git.

The error we are getting is as follows:

returned non-zero exit status 128.
Cloning into '.'...
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://xxx.xxx.xxx.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://xxx/xxx/xxx/xxx.git/'

Any help would be greatful,

Thanks

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @MikeyH-84

The only related issue/question I can find is https://github.com/conan-io/conan/issues/10777. This discussion mentions that it is bad practice to merge credentials directly into the URL. I am also not sure this is possible with GitLab. If I am interpreting the discussion correctly, the only solution is to write a custom wrapper around the Conan tools Git object? To create a custom Git credentials manager?

No, we are willing to re-learn the needs for authentication, and try to provide built-in solutions. We were concerned about the complexity of the scm, and its hiding of the auth (and the risks of managing it), so we would definitely like something that lies more on the user side (recipes) than on Conan internals.

The current Git object implementation is very thin, just a wrapper around the git command line. How are you managing the git auth for developers? and for CI machines? Is it possible that it is something that should be managed directly by git (it has some built-in mechanisms) instead of relying on Conan?

[MikeyH-84]

Hi @memsharded,

Authentication for developers is just HTTPS (username/password) and/or public/private SSH keys uploaded to Gitlab. All developers really do is git clone, do some local work, and push to CI/CD pipelines.

For the CI/CD, we create and upload "service" SSH keys, using them as GitLab deploy keys here. The private SSH key is stored in the GitLab project as a secret variable.

So does the Conan SCM or newer Conan Git 2.0 work with SSH keys?

Thanks

[MikeyH-84]

Another recent issue we're facing, is that we have an infrastructure bash script (stored in each project) for dynamically creating the versioning for some software. Requires Git as part of the versioning i.e SEMVER-LATEST_TAG-BRANCH_NAME-BRANCH_COMMITS. Obviously for this script to work we need access to the projects Git. As far as I can see, exports does not upload the Git folder. Hence another reason reason for migrating to Conan SCM/Git

Thanks

[memsharded]

So does the Conan SCM or newer Conan Git 2.0 work with SSH keys?

Yes, all Git.clone() does is to execute git clone ... as a subprocess. If git requires ssh keys and they are there in the developer machine, it will take them, I have just tried and it works. This is exactly what I meant with decoupling the git auth from Conan, it is much better to rely on the Git auth than messing with credentials in Conan recipes.

[memsharded]

Another recent issue we're facing, is that we have an infrastructure bash script (stored in each project) for dynamically creating the versioning for some software. Requires Git as part of the versioning i.e SEMVER-LATEST_TAG-BRANCH_NAME-BRANCH_COMMITS. Obviously for this script to work we need access to the projects Git. As far as I can see, exports does not upload the Git folder. Hence another reason reason for migrating to Conan SCM/Git

Yes, in general exporting the git folder is unnecessary and unwanted, it will increase the size of the recipes, making it slower and also cluttering the recipe with the whole history.

The approach for extracting the version is using the set_version() recipe method, that executes at "export" time, directly obtaining and executing in the user folder, and once the version is already defined in the cache, access to the git folder is not necessary at all. Please check https://docs.conan.io/en/latest/reference/conanfile/methods.html#set-name-set-version and let me know if this makes sense.

If further information has to be extracted from the git repo rather than just the version, the recommended approach is to extract that information during the export() recipe method and store it in a json file in the recipe. The conandata.yml might be a good standard location to do it, but not mandatory, it could be other file too.

[jsallay]

Just a note that I use the dunamai pip package for this purpose. This is the library used by poetry to dynamically get version information from git tags.

[MikeyH-84]

@memsharded OK so we kept the SSH authentication really simple. Basically for downstream projects to fetch our packages, they must have an SSH key associated with the GitLab instance to authenticate. Our CI/CD pipelines have GitLab secret variables containing the SSH keys. Like you said, Git automatically handles the authentication under the hood. Our simple workflow is as follows:

class MyProject(ConanFile):
    git_ssh_url = "[email protected]:xxx/xxx/xxx.git"

    def export(self):
        git = Git(self, self.recipe_folder)
        git_url, git_commit = git.get_url_and_commit()

        if "https://" in git_url:
            git_url = self.git_ssh_url

        update_conandata(self, {"sources": {"commit": git_commit, "url": git_url}})

    def layout(self):
        self.folders.source = "."

    def source(self):
        git = Git(self)
        sources = self.conan_data["sources"]

        git.clone(url= sources["url"], target=".")
        git.checkout(commit=sources["commit"])

Works for local developer builds and/or Conan cache builds. If building in the Conan cache, replaces the default Conan Git HTTPS URL with the recipes SSH URL. As building downstream in the Conan cache will require fetching the source from the Git remote.

We can later add further logic for using HTTPS (environment variables or something), but this suites our needs and keeps the logs in the CI/CD clean (credentials, etc).

[MikeyH-84]

Maybe in the user guide for migrating to Conan 2.0, it's worth noting Git authentication is not part of the Conan tools? Possibly some hints for what people might have to do i.e. manipulating the Git URL, etc.

[MikeyH-84]

Just a note that I use the dunamai pip package for this purpose. This is the library used by poetry to dynamically get version information from git tags. @jsallay thanks for the tip, i'll look into this package.

[memsharded]

Thanks very much @MikeyH-84 for following up and sharing your solution.

I am very interested in that replacement of the URL. Why doing it in the source() method, and not replacing it in the export() method when captured?

Works for local developer builds and/or Conan cache builds. Replaces the URL if building in the Conan cache (will require fetching the source from the Git remote).

Do you (our your users) use the conan source command? do you expect that to do the replacement as well?

[MikeyH-84]

@jsallay Actually you make a good point. Moved the URL logic into the export(). Makes sense. Updated example.

We use the Conan build command for local development and the Conan create and test commands to produce the final Conan package. Me placing the URL logic the source() may have confused matters.

[memsharded]

I guess you wanted to mention me and not @jsallay ?

But in any case, yes, then everything looking good, more clear now, reads simple and effective, thanks for updating the example!

Then maybe this is now mostly an issue of improving the docs for 2.0 to explicitly document this? I can move this ticket to the "docs" repo.

[MikeyH-84]

Sorry @memsharded meant to refer to yourself. Yes I am happy with this and sure, move it to "docs".

Thanks again