Unable to clone Azure DevOps repo

[4x0v7]

Expected behavior

Repository is cloned

Current behavior

"error": "failed to create local workspace: failed to create workspace, unable to enlist in git repo: unable to clone repo: empty git-upload-pack given"

{
    "level": "error",
    "ts": 1615785960.4068184,
    "logger": "controller-runtime.controller",
    "msg": "Reconciler error",
    "controller": "stack-controller",
    "name": "test-stack",
    "namespace": "default",
    "error": "failed to create local workspace: failed to create workspace, unable to enlist in git repo: unable to clone repo: empty git-upload-pack given",
    "errorVerbose": "empty git-upload-pack given\nunable to clone repo\ngithub.com/pulumi/pulumi/sdk/v2/go/x/auto.setupGitRepo\n\t/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/x/auto/git.go:87\ngithub.com/pulumi/pulumi/sdk/v2/go/x/auto.NewLocalWorkspace\n\t/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/x/auto/local_workspace.go:516\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*reconcileStackSession).SetupPulumiWorkdir\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:495\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/runtime/asm_amd64.s:1373\nfailed to create workspace, unable to enlist in git repo\ngithub.com/pulumi/pulumi/sdk/v2/go/x/auto.NewLocalWorkspace\n\t/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/x/auto/local_workspace.go:518\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*reconcileStackSession).SetupPulumiWorkdir\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:495\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/runtime/asm_amd64.s:1373\nfailed to create local workspace\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*reconcileStackSession).SetupPulumiWorkdir\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:497\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/runtime/asm_amd64.s:1373",
    "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"
}

Steps to reproduce

1. Create an Azure DevOps Repo
2. Generate a SSH key and upload the public key
3. Create yaml like:

apiVersion: v1
kind: Secret
metadata:
  name: azdops-sshkey
type: Opaque
stringData:
  sshPrivateKey: |-
    -----BEGIN RSA PRIVATE KEY-----
    MII...==
    -----END RSA PRIVATE KEY-----

---
apiVersion: pulumi.com/v1alpha1
kind: Stack
metadata:
  name: test-stack
spec:
  gitAuthSecret: azdops-sshkey
  stack: dev
  projectRepo: [email protected]:v3/<ORG>/<PROJECT>/<REPO-NAME>
  commit: 4eecd3a9c16d0de6d6de45c908cd361971ff6111

4. kubectl apply
5. check logs of controller

Context (Environment)

Using Image pulumi/pulumi-kubernetes-operator:v0.0.8 Kubernetes v1.19.7

Affected feature

References

Looks like an issue in the git library

Related issues: https://github.com/fluxcd/flux/issues/1224 https://github.com/src-d/go-git/issues/1058 https://github.com/go-git/go-git/pull/200 https://github.com/fluxcd/source-controller/pull/213 https://github.com/argoproj/argo-cd/issues/1067

[viveklak]

This seems to be an issue with go-git. We should consider wrapping go-git with an optional flag to shell out to git in such situations?

[jancespivo]

Hi, we are also struggling with the same issue. It seems there is no progress in git-go (see https://github.com/go-git/go-git/issues/64).

[lukehoban]

We should consider wrapping go-git with an optional flag to shell out to git in such situations?

Or falling back to, or allowing configuration of, an alternative like libgit2 similar to how Flux "solved" this? https://fluxcd.io/docs/components/source/gitrepositories/#git-implementation

[squaremo]

If you can rely on git being present, I would wrap an exec. Our (Flux project) experience with both go-git and libgit2 is that they are often missing things that just work with git. Most of the time, you want porcelain (e.g., git switch) but the libraries give you plumbing, and there's lots of ways to get it subtly wrong.

If you do decide to incorporate libgit2, you may benefit from examining the work Hidde has done recently to make builds more repeatable: https://github.com/fluxcd/source-controller/pull/437 and links therein.

[dswartz-altana]

This would be super-helpful for us at the moment!

[afa3fe]

Is this issue going to be fixed? I have the same issue on my setup with pulumi kubernetes operator and azure devops

[afa3fe]

@viveklak after the issue is fixed in zarf is it possible to update the dependency and fix this issue from your side? That would be super helpful. Thank you very much in advance.

[azzab80]

We would also love to use the Kubernetes Operator with Azure DevOps repos so any update would be great. Thanks

[lukehoban]

Quite likely that this should be addressed via implementing #158 so that we can more deeply support scenarios already handled by Flux (including this one, but also many more).

[mikhailshilkov]

@lukehoban That makes sense. I'll remove it from the Platform board then, as I understand we have other folks looking into it. Let me know if I'm wrong.