gitpython-developers
3.0.2: failing tests during packaging (tests should be isolated and independent)
Hi!
I'm currently packaging gitpython for Arch Linux and am about to update to version 3.0.2. However, during packaging, I tried to run the tests and many of them fail (see attachment).
We're currently on
- git 2.23.0
- python-gitdb 2.0.5
- python-smmap 2.0.5
- python-ddt 1.2.1
- python-nose 1.3.7
I'm running the tests (after build, before install) with nosetests -vd.
I'm using the pypi provided sources: https://files.pythonhosted.org/packages/51/46/993beca52f3b609d148071e129235b866626eeb6056f2faffb41d9d727a7/GitPython-3.0.2.tar.gz
I would be very grateful for any hint! python-gitpython-3.0.2-check.log
Unfortunately, the GitPython tests indeed have very specific requirements. They expect to be run in their own git repository, with certain additional requirements being met. Fortunately there is a script as run by travis which sets up these requirements.
Something I am not sure of is if it's enough to simply initialise a new git repository from the sources in the tarball.
I hope that helps, and please let me know if it worked or if there are additional issues.
I have tried to run the tests for the 3.0.3 build, but still get a lot of failing and erroring tests.
As I'm building a package, I can not rely on downloading from several different sources (am I supposed to git clone all of gitpython recursively just for testing purposes?).
That being said, there are plenty of tests, that hardcode specific commits, author names, git tags, files(!), etc. to test against. This is really bad for reproducibility and makes all of these tests basically useless for downstreams (although they would be potentially be the ones reporting integration problems, etc.)!
This is how I have to prepare the test suite, when using a tarball of gitpython (while still having 34 erroring and 9 failing tests):
local TEST_TMPDIR=$(mktemp -d -t gitpython.XXX)
local GITDB_TMPDIR=$(mktemp -d -t gitdb.XXX)
local SMMAP_TMPDIR=$(mktemp -d -t smmap.XXX)
git config --global user.name "Test User"
git config --global user.email "[email protected]"
(
cd "$SMMAP_TMPDIR"
mkdir -vp bare
(
cd bare
git init --bare
)
git clone bare clone
cd clone
for commit in {1..50}; do
touch "file${commit}"
git add "file${commit}"
git commit -m "file${commit}"
done
git push
)
(
cd "$GITDB_TMPDIR"
mkdir -vp bare
(
cd bare
git init --bare
)
git clone bare clone
cd clone
mkdir -vp gitdb/ext
git submodule add --name 'smmap' "${SMMAP_TMPDIR}/clone" gitdb/ext/smmap
git submodule init gitdb/ext/smmap
git add gitdb/ext/smmap
git commit -m "Adding smmap submodule"
for commit in {1..50}; do
touch "file${commit}"
git add "file${commit}"
git commit -m "file${commit}"
done
git push
)
(
cd "$TEST_TMPDIR"
mkdir -vp bare
(
cd bare
git init --bare
)
git clone bare clone
cd clone
mkdir -vp git/ext
git submodule add --name 'gitdb' "${GITDB_TMPDIR}/clone" git/ext/gitdb
git submodule update --init --recursive git/ext/gitdb
git add git/ext/gitdb
git commit -m "Adding gitdb submodule"
# test hardcodes this file to test against
touch AUTHORS
git add AUTHORS
git commit -m "Adding AUTHORS."
# test hardcodes this file to test against
touch MANIFEST.in
git add MANIFEST.in
git commit -m "Adding MANIFEST.in."
for commit in {1..50}; do
touch "file${commit}"
git add "file${commit}"
git commit -m "file${commit}"
done
git tag -am '0.1.4' 0.1.4
for commit in {51..100}; do
touch "file${commit}"
git add "file${commit}"
git commit -m "file${commit}"
done
git tag -am '0.1.5' 0.1.5
for commit in {101..150}; do
touch "file${commit}"
git add "file${commit}"
git commit -m "file${commit}"
done
git tag -am '0.1.6' 0.1.6
for commit in {151..160}; do
touch "file${commit}"
git add "file${commit}"
git commit -m "file${commit}"
done
git tag -am '0.3.5' 0.3.5
git tag __testing_point__
git push
git push --tags
)
export GIT_PYTHON_TEST_GIT_REPO_BASE="${TEST_TMPDIR}/clone/.git"
nosetests -vd
python-gitpython-3.0.3-build.log
I guess at this point it becomes more than evident, that this should be part of the test suite, creating assets for itself and removing all the specific hardcoding in the tests (while not relying on this repository and all of its specific components).
I absolutely agree that the way GitPython tests are done violates a lot of best practices: they are usually too long and convoluted, and downright assume they have access to a recursive clone of their own repository.
Back in the days, it seemed 'easier', not taking into consideration what this would mean further down the ~~road~~ stream. From your comment I can get a hint of the nightmare this seems to be, and I am sorry for the trouble it may cause.
Looking at the script above I would hope that eventually it manages to become a PR which helps make tests independent of their own repository. Within the test framework, everything boils down to two (?) base classes that initialise a repository for reading and/or writing. These should be one of the few places which may need adjustments on python-side.
On 2019-10-21 10:37:06 (-0700), Sebastian Thiel wrote:
Back in the days, it seemed 'easier', not taking into consideration what this would mean further down the ~~road~~ stream. From your comment I can get a hint of the nightmare this seems to be, and I am sorry for the trouble it may cause.
It is a little nightmarish indeed, but I've also seen worse (!) tbh! :)
Looking at the script above I would hope that eventually it manages to become a PR which helps make tests independent of their own repository. Within the test framework, everything boils down to two (?) base classes that initialise a repository for reading and/or writing. These should be one of the few places which may need adjustments on python-side.
I'll try to look into it, when I find the time. Got my hands pretty full with other things (mainly packages) though. Additionally, I haven't worked with nose, but mainly pytest in the past.
We used to have working tests for openSUSE/Factory with some tests skipped (mainly those which require network access), but with 3.1.31 (checkout of the tag from GitHub) it failed again (93 tests failed, 3 errors).
I don't know what changed exactly or why the tests would fail now, but judging by the log the failures seem to be due to some foundational issue.
I recommend following the instructions on how to run tests if not done - a standard clone is assumed with the HEAD not detached.