git backend fails in Windows with `perceval.errors.RepositoryError`

[penyuan]

Hello,

We're trying to set up and use perceval in a Windows environment. The setup we tested has perceval installed with pip in a Python 3.6.8 virtual environment (called "pyVirtualEnv") created with venv.

The Github backend seems to work fine, but the git backend consistently fails with an error that ends with this:

...
perceval.errors.RepositoryError: git command - Cloning into bare repository 'testDir'...
fatal: unable to access 'https://github.com/grimoirelab/perceval.git/': Could not resolve host: github.com

We were attempting to reproduce the tutorial example on the Grimoirelab website. The exact command run was python test.py https://github.com/grimoirelab/perceval.git testDir and here is the full output:

(pyVirtualEnv) C:\Users\[username]\Documents\L_GitRepos\wp2.2_dev\src>test.py https://github.com/grimoirelab/perceval.git testDir
Traceback (most recent call last):
File "C:\Users\[username]\Documents\L_GitRepos\wp2.2_dev\src\test.py", line 12, in <module>
for commit in repo.fetch():
File "C:\Users\[username]\Documents\L_Travail\Open!Next\WP2.2\pyVirtualEnv\lib\site-packages\perceval\backend.py", line 215, in fetch
for item in self.fetch_items(category, **kwargs):
File "C:\Users\[username]\Documents\L_Travail\Open!Next\WP2.2\pyVirtualEnv\lib\site-packages\perceval\backends\core\git.py", line 154, in fetch_items
latest_items, no_update)
File "C:\Users\[username]\Documents\L_Travail\Open!Next\WP2.2\pyVirtualEnv\lib\site-packages\perceval\backends\core\git.py", line 267, in __fetch_from_repo
repo = self.__create_git_repository()
File "C:\Users\[username]\Documents\L_Travail\Open!Next\WP2.2\pyVirtualEnv\lib\site-packages\perceval\backends\core\git.py", line 318, in __create_git_repository
repo = GitRepository.clone(self.uri, self.gitpath)
File "C:\Users\[username]\Documents\L_Travail\Open!Next\WP2.2\pyVirtualEnv\lib\site-packages\perceval\backends\core\git.py", line 826, in clone
cls._exec(cmd, env=env)
File "C:\Users\[username]\Documents\L_Travail\Open!Next\WP2.2\pyVirtualEnv\lib\site-packages\perceval\backends\core\git.py", line 1331, in _exec
raise RepositoryError(cause=cause)
perceval.errors.RepositoryError: git command - Cloning into bare repository 'testDir'...
fatal: unable to access 'https://github.com/grimoirelab/perceval.git/': Could not resolve host: github.com

We confirmed that there are no weird Internet settings on the system (e.g. no proxies, etc.). Not to mention the Github backend works fine (so far!).

I know that there is a similar issue from 2018, and we tried almost all the solutions proposed in that thread. However, the problem persists.

One of the comments there said that the git backends needs the HOME environment variable to find git configuration files. To that end, here is the output from the command git config --list --show-origin:

(pyVirtualEnv) C:\Users\[username]\Documents\L_GitRepos\wp2.2_dev\src>git config -l --show-origin
file:"C:\\ProgramData/Git/config" core.symlinks=false
file:"C:\\ProgramData/Git/config" core.autocrlf=true
file:"C:\\ProgramData/Git/config" core.fscache=true
file:"C:\\ProgramData/Git/config" color.diff=auto
file:"C:\\ProgramData/Git/config" color.status=auto
file:"C:\\ProgramData/Git/config" color.branch=auto
file:"C:\\ProgramData/Git/config" color.interactive=true
file:"C:\\ProgramData/Git/config" help.format=html
file:"C:\\ProgramData/Git/config" rebase.autosquash=true
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" http.sslcainfo=C:/Users/[username]/AppData/Local/Programs/Git/mingw64/ssl/certs/ca-bundle.crt
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" http.sslbackend=openssl
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" diff.astextplain.textconv=astextplain
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.clean=git-lfs clean -- %f
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.smudge=git-lfs smudge -- %f
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.process=git-lfs filter-process
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.required=truefile:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" credential.helper=manager
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" core.editor='C:\Program Files\Notepad++\notepad++.exe' -multiInst -notabbar -nosession -noPlugin
file:.git/config core.repositoryformatversion=0
file:.git/config core.filemode=false
file:.git/config core.bare=false
file:.git/config core.logallrefupdates=true
file:.git/config core.symlinks=false
file:.git/config core.ignorecase=true
file:.git/config submodule.active=.
file:.git/config remote.origin.url=https://github.com/OPEN-NEXT/wp2.2_dev.git
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config branch.master.remote=origin
file:.git/config branch.master.merge=refs/heads/master

And for git config --list --system --show-origin:

(pyVirtualEnv) C:\Users\[username]\Documents\L_GitRepos\wp2.2_dev\src>git config --list --system --show-origin
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" http.sslcainfo=C:/Users/[username]/AppData/Local/Programs/Git/mingw64/ssl/certs/ca-bundle.crt
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" http.sslbackend=openssl
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" diff.astextplain.textconv=astextplain
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.clean=git-lfs clean -- %f
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.smudge=git-lfs smudge -- %f
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.process=git-lfs filter-process
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" filter.lfs.required=truefile:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" credential.helper=manager
file:"C:\\Users\\[username]\\AppData\\Local\\Programs\\Git\\mingw64/etc/gitconfig" core.editor='C:\Program Files\Notepad++\notepad++.exe' -multiInst -notabbar -nosession -noPlugin

I don't know the perceval codebase well enough to figure out why the HOME variable is needed. Can anyone advise on possible workarounds and/or what git.py is looking for so that we might point it in the right direction on a Windows system?

Thank you!

[valeriocos]

Hi @penyuan, thank you for the detailed message!

The main limitation with this issue is that Perceval and the other components are developed and tests on Linux-based systems. Thus, it is pretty difficult to give help.

It would be easier to install a virtual machine using https://www.virtualbox.org/, and in there you could install and run perceval.

[penyuan]

Thanks @valeriocos for getting back to me!

The thing is, it might not be practical in our use case to set up a virtual machine just for running perceval.

Do you think the HOME environment variable mentioned two years ago has something to do with it? If so, if you can guide me in trying a few things on the Windows system, it might help pinpoint where the issue is and help us develop a solution or workaround.

[valeriocos]

Hi @penyuan , sorry for the late reply!

Do you think the HOME environment variable mentioned two years ago has something to do with it? If so, if you can guide me in trying a few things on the Windows system, it might help pinpoint where the issue is and help us develop a solution or workaround.

I'm not sure about the answer to your question. A solution with a docker container could be good for you (ref: https://github.com/chaoss/grimoirelab-perceval/tree/master/docker/images)? I don't know the status of that docker image, but I guess it will be easier to help you (since we don't have Windows systems)

[valeriocos]

Hi @penyuan ! how is it going with this issue? Were you able to advance a bit?

Thanks!

[sduenas]

I've been working on this but the error that I get is different to yours although I think it is related. What I get when Perceval does a clone is a: (Windows 10, btw)

getaddrinfo() thread failed to start

Other commands work fine.

[penyuan]

Thank you @valeriocos and @sduenas for your feedback. I don't have direct access to the Windows machine in question at the moment, but let me see if I can at least get you a minimal piece of Python code (we're using Python 3.6. BTW is GrimoireLab ready for Python 3.7+ now?) to reproduce the problem, hopefully that will help?

[penyuan]

Sorry for my late follow-up.

Unfortunately I can't even get a working Python 3.6 environment set up on my Windows test system, and in the meantime we've managed to figure out how to set up a working Ubuntu 20.04 virtual machine.

If there is interest in getting Perceval and friends to work in Windows I'm still happy to help, but for our purposes we are OK with the virtual machine solution for now...

Thanks for your help!

[sduenas]

You are welcome @penyuan . I've been trying to find the problem in Windows but without success. I'm afraid other parts of the stack will never work either. For example, Arthur uses rq, a library in Python to schedule jobs. This library uses fork() call to spawn new processes. This system call is in POSIX systems but not in Windows. It will be nice if Perceval works in Windows though.

I think we should push people to use the docker image or similar in Windows platforms.

[penyuan]

Understood, thank you.

For now, we can live with a Linux virtual machine on some of our development systems, or as you suggested, with a Docker container. However, if one day someone wants to pick this up and make Perceval work in Windows I'll be happy to help test it.

[sduenas]

We don't support Windows. Closing this ticket due to inactivity.