Connection closed and repo files gone

[kainlite]

Hi,

I've been trying to clone a bitbucket repo and I've hit a few issues, first of all it closes the connection and deletes all local files, I've tried different options for CloneOptions with no change at all, like SingleBranch: true, or Tags: git.NoTags,

$ go run .
Counting objects: 212302, done.
Compressing objects: 100% (64830/64830), done.
Total 212302 (delta 149473), reused 204905 (delta 144107)
close tcp 192.168.1.254:40626->18.205.93.1:22: use of closed network connection

$ go version
go version go1.11.2 linux/amd64

package main

import (
        "fmt"
        "os"

        git "gopkg.in/src-d/go-git.v4"
)

func main() {
        _, err := git.PlainClone("./foo", false, &git.CloneOptions{
                URL:      "[email protected]:org/repo.git",
                Progress: os.Stdout,
        })
        fmt.Println(err)
}

I don't have a public repo to reproduce the issue, but it also does not happen with other repos that I'm using, is there something I can do or check to make this work?

Thanks.

[kainlite]

For future references or for others having the same issue I got it working by using Clone not PlainClone and memory storer.

        // Filesystem abstraction based on memory  
        fs := memfs.New()  
        // Git objects storer based on memory  
        storer := memory.NewStorage() 
  
        repo, err := git.Clone(storer, fs, &git.CloneOptions{
                URL:           repoUrl,
                Depth:         200, 
                Progress:      os.Stdout,
                ReferenceName: plumbing.ReferenceName("refs/heads/master"), 
                SingleBranch:  true, 
                Tags:          git.NoTags, 
        }) 

In any case I would expect PlainClone to work.

[smola]

@kainlite Indeed, cleanup on PlainClone errors is a behavior we introduced in go-git v4.8.0 (https://github.com/src-d/go-git/issues/741) to better match git's behavior for clone.

I'm not sure what's git behavior if the connection closes half-way during a clone though.

Is the repository you're getting after the closed connection error valid? Or is it corrupted? Because probably the problem we should try to solve is the network error, not the cleanup...

[kainlite]

The files get removed right away, so I didn't had a chance to look at the repo state after the connection error, interestingly enough with Clone I didn't had that problem at all. Also it only happened with one big repo, I tried with other smaller repos and everything was working fine.

[smola]

@kainlite Did you try your working Clone approach without Depth: 200, SingleBranch: true and Tags: git.NoTags?

[kainlite]

@smola Yes, I started with the most basic approach, only the repo, then started adding parameters but there was no difference, my first attempt was something like this:

  repo, err := git.PlainClone(directory, false, &git.CloneOptions{
                URL:           repoUrl,
                Progress:      os.Stdout,
  }) 

[smola]

@kainlite I mean with Clone, not PlainClone. PlainClone does cleanup, Clone does not. So it would be good to know if you get a non-corrupted repository when doing a full clone with Clone (without depth, single branch...) and get this network error.

[kainlite]

Yes, the repo looks fine with Clone, I even generate a branch with the code and push it to bitbucket.

[smola]

Thanks @kainlite, we'll look into the issue.

[dink10]

Hi. I have the same bug only for big repositories. Do you have any solutions for fixing it?

[jfontan]

I've tested downloading a Linux kernel and works fine with https but fails using ssh. It may be a problem of the transport that expects the connection open after the packfile finishes downloading. After downloading there's an index creation process that can be long for big repos. After packfile download the connection is no longer needed AFAIK.

[dink10]

I've tested downloading a Linux kernel and works fine with https but fails using ssh. It may be a problem of the transport that expects the connection open after the packfile finishes downloading. After downloading there's an index creation process that can be long for big repos. After packfile download the connection is no longer needed AFAIK.

Nice research, seems you're right

[paveq]

I managed to replicate exact same issue where cloning of medium sized repository (70 megs) works fine with https, but does not work with ssh. It works fine for smaller repos, but for larger repos it seems they take too long and connection gets closed in-between.