Add support for more git hosts

[SuperDoxin]

currently only github and bitbucket are supported. I would love to see at least gitlab.com supported too, but ideally any git url ought to work.

[profan]

Supporting this as well, seems like it would be natural to allow any given git remote.

edit: right, so it seems like the GitLab support is already in the repo, but have you given allowing arbitrary git repositories?

[s-ludwig]

The problem with arbitrary git URLs is mainly the download link. It would require downloading, extracting, zipping and storing all of such packages locally on the registry server. Currently this wouldn't be an issue, but if we ever grow to the numbers of packages that the most popular package systems have today, it will require an upgrade of the hardware infrastructure. Another potential issue is that every git repository requires its own server connection, while the HTTP hoster protocols can reuse the same connection for many consecutive requests. So this would create a lot of overhead for both, the registry and the hoster.

So, this is certainly doable, but doesn't scale well enough for the current server infrastructure.

GitLab support is basically there, but so far I've only tested it with a local GitLab instance. I'll switch that live probably in about one or two weeks.

[SuperDoxin]

so why isn't the git protocol being used here? seems to me that the dub client could just git clone whatever it needs instead of the server having to hand it a zipfile.

[s-ludwig]

Using the git protocol would mean either that the server would have to mirror all repositories, while it now only has to store the meta data, or the client would also have to use either libgit or the git command line client, both of which is problematic in one way or another. This would also mean that git gets special treatment - but we currently also support HG repositories on Bitbucket for example, because everything works using HTTP.

[sigod]

I too would love to see GitLab support.

[s-ludwig]

I've added a branch where the code I once wrote for GitLab is integrated: https://github.com/D-Programming-Language/dub-registry/tree/gitlab-support

Would be good if someone could test this and provide patches if necessary. I'm currently lacking the resources to finish this and we have internally switched to Gogs now, so when I get the time I will concentrate on that instead.

[sigod]

I'm currently lacking the resources to finish this and we have internally switched to Gogs now, so when I get the time I will concentrate on that instead.

Very interesting service. Are there any major hosting for it? I don't see how it can be supported for user hosted packages.

[s-ludwig]

Gogs would indeed just be for private/internal hosting, I don't know of any public service, although I haven't searched for it either. I want to set up a local dub-registry, so that building projects just works, without having to manually clone all dependencies.

[BlackEdder]

I'd really love to see gitlab support :) Would it be possible to merge that yet?

[BlackEdder]

Using the git protocol would mean either that the server would have to mirror all repositories, while it now only has to store the meta data, or the client would also have to use either libgit or the git command line client, both of which is problematic in one way or another. This would also mean that git gets special treatment - but we currently also support HG repositories on Bitbucket for example, because everything works using HTTP.

If you choose to support git through calling the command line, then it should be pretty trivial to do the same for the hg client? Of course this does require for those tools to be installed, but this is quite common at the moment. The slight inconvenience for requiring git to be installed, seems to be a good trade off for getting support for any git/hg repository.

[BlackEdder]

Any progress on this? I'd really prefer using gitlab over github.

[gedaiu]

I would like to use gitlab with the registry too :)

[LunaTheFoxgirl]

Is there a way you could add support of granting per-user basis a way to add their own git server to the list but only visible to them? For example I've moved my project off GitHub due to the recent Microsoft acquisition, over to my own server. (using go gogs)

[dhasenan]

I was plinking around with this a bit. The best I think you can get:

1: You don't need to check out anything if you're just checking if things are up-to-date. You can just use:

mkdir tmp
cd tmp
git init
git remote add https://some-private-repo.mysite.com/path/to/repo.git
git ls-remote --tags origin

2: you can download and work with only the affected revisions.

git fetch --depth=1 --prune origin $revisionHash
git reset --hard $revisionHash

For vibe.d, this incurs 6.17MB of network bandwidth instead of 19.88. Not awesome savings, but still savings.

3: you can save a bit of disk space with git's sparse checkout option.

git config core.sparseCheckout true
echo > .git/info/sparse-checkout <<ENDL
dub.sdl
dub.json
README.md
README.rst
README.txt
README
ENDL

Unfortunately, sparseCheckout doesn't affect the amount of storage used in .git or the amount of network bandwidth used.

Using the git API directly would let you avoid disk space, but not network bandwidth. I don't know how much bandwidth costs for the dub registry server, so I don't know whether this would be acceptable.