Optimize git library installation

[tobil4sk]

Closes #493. For most git installs, a (faster) shallow clone can now be performed with --depth=1, which avoids cloning the entire history and instead only clones one commit's worth of history, including for the submodules. The only exception where a shallow clone cannot be performed is when setting a library to a specific commit with a short commit id, for example:

# Reverts to full non-shallow clone:
haxelib git lime https://github.com/openfl/lime c16f278
# Performs shallow clone:
haxelib git lime https://github.com/openfl/lime c16f27818dcff17e8b5e4648624a64ea6a107119

This is due to how commit ids are resolved.

The data which was used to install the library is now stored in a .gitdata file, to avoid issues of overwriting externally created commits. With this .gitdata file, it is now possible to be smarter when reinstalling the same library, which also fixes #512.

Now when updating a git library, it will now only update if it was installed with the default branch or a specific branch. If it was installed with a specific commit or tag, it remains at that commit/tag, since there is no clear commit to update to.

~~Finally, this also fixes #593, since the submodules have to be cloned after switching to the correct branch in order to clone them with --depth=1 as well.~~ This was fixed already in #638.

Mercurial works differently, so I've just left it as is.

[Simn]

@kLabz Could you check if this makes sense git-wise?

[kLabz]

I don't think I'll have time today, so will look at this tomorrow.

It likely breaks some workflow I use from time to time (using git dependency, and doing git commands in .haxelib/somelib/git folder like changing branches etc.) but I should probably use haxelib dev for that as it's easy to lose changes like that...

[tobil4sk]

Somehow this PR causes #591 to be triggered on Windows with pretty much every haxelib git command :/ Probably requires getting rid of neko to fix it, I can't figure out what exactly is going on...

[dpomier]

The only exception where a shallow clone cannot be performed is when setting a library to a specific commit with a short commit id

It should be possible to get the full commit id from the short one once you have added a git remote:

$ git rev-parse 3cdd5d
3cdd5d19178a54d2e51b5098d43b57571241d0ab

Now when updating a git library, it will now only update if it was installed with the default branch or a specific branch. If it was installed with a specific commit or tag, it remains at that commit/tag, since there is no clear commit to update to.

It might be worth always fetching the commit id so as not to lose your optimisation when a branch or tag is specified instead of a commit id. This can be done this way:

$ git ls-remote https://github.com/HaxeFoundation/haxelib.git HEAD     
98637027327d8cf385d302acaaf104bd6107d2bf        HEAD

$ git ls-remote https://github.com/HaxeFoundation/haxelib.git 4.1.x
f17fffa97554b1bdba37750e3418051f017a5bc2        refs/heads/4.1.x

Hope this helps.

[tobil4sk]

It should be possible to get the full commit id from the short one once you have added a git remote:

This doesn't work if the commit hasn't been fetched:

$ git rev-parse 558fd07 
558fd07
fatal: ambiguous argument '558fd07': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

It might be worth always fetching the commit id so as not to lose your optimisation when a branch or tag is specified instead of a commit id.

Not sure I understand, which optimisation are you talking about that would be lost?

[MAJigsaw77]

Any progress on this?

[tobil4sk]

The main issue that's blocking this is #591, there is no point in optimising git commands if it will break them :sweat_smile:. Maybe it can work in development with the interp target and hxb, but that would mean that it doesn't work with haxe 4.3.x.

[PXshadow]

Any news on this?

[tobil4sk]

The cause for the crashes has been resolved now! I've updated the branch to resolve conflicts, and I've also split out the commits into more atomic changes and added more descriptive messages (would be good to rebase rather than squash to preserve history). This is now ready for review.

Mercurial now also is consistent with Git's behaviour.

I've also taken out the changes for #512, which will come in a separate PR.

I also had to disable the cpp client tests, because default arguments in abstract classes are broken on cpp, see: https://github.com/HaxeFoundation/haxe/issues/11666.

[PXshadow]

Tested on go2hx git library with help from @tobil4sk . It's over 200MB if not shallow cloned, and with 3 different git submodules. The time it took to install from git in total went from 14 seconds to 3 seconds on my machine, using this branch.