clib Support repositories hosted outside of GitHub

It appears clib_package.c:clib_package_install only supports repositories hosted on GitHub. Users should have the option to install packages hosted on any Git server. What's the core team's opinion on the following syntax for package.json?

{
  "name": "my-package",
  "version": "0.1.0",
  "url": "https://github.com/me/my-package.git",
  "dependencies": {
    "my-friends-package": {
        "url": "https://[email protected]/my-friend/my-friends-package.git",
        "version": "0.1.0"
    }
  },
  "install": "make install"
}

If a package declares both a url and a repo key, the url key is preferred and a warning should be displayed. If only a repo key is declared, then it's business as usual and the repo is assumed to be hosted on GitHub.

If my-friends-package uses a hash, as it does above, the url and version keys are required. If it is declared using the current syntax of "my-friends-package": "0.1.0", then it behaves as it does now. Therefore these syntax changes are backwards compatible.

If there's interest in such a feature I'd love to work on it.

Mar 18 '14 17:03 modocache

I think that would be an awesome feature. It appears we would need to make clib_package_url a bit more customizable.

See: https://github.com/stephenmathieson/clib-package/blob/master/src/clib-package.c#L473

cc @stephenmathieson

Mar 18 '14 17:03 jwerle

If the goal is BitBucket support (which I'm in support of, btw) I'd go about it a bit differently. Rather than providing "url" and "version", maybe something like:

  "dependencies": {
    "my-friend/my-friends-package": {
        "host": "bitbucket.org", // host defaults to raw.github.com
        "version": "0.1.0"
    },
  }

Then we could get file URLs with https://{host}/{owner}/{name}/{version}/{source file}.

Mar 18 '14 18:03 stephenmathieson

+1 I like that idea

Mar 18 '14 18:03 jwerle

err, actually, BitBucket raw URLs are https://{host}/{owner}/{name}/raw/{version}/{source file}.. we'll have to do this a bit differently :/

Mar 18 '14 18:03 stephenmathieson

Beyond BitBucket, parsing arbitrary URLs would allow development teams to split up large, private libraries into smaller repositories. Since enterprise solutions from GitHub and BitBucket cost a lot of money compared to self-hosting, many of these teams use internal Git servers.

Supporting arbitrary URLs would also "future-proof" clib. Using the host approach, we'd have to add more code every time a new code hosting service opens its doors.

It might be a good idea to look at how go get is implemented--it allows for arbitrary URLs as well as a shorthand when using popular hosting services like GitHub and BitBucket.

Mar 18 '14 18:03 modocache

This is true. We'd then either need libgit2 or to fetch a tar ball and extract certain files (like installing executables works currently).

I'll have to look at go get; I haven't investigated Go much.

Mar 18 '14 18:03 stephenmathieson

What about:

"dependencies": {
  "my-friend/my-friends-package": {
      "url_format": "https://{owner}@bitbucket.org/{owner}/{name}/raw/0.1.0/{file}"
  }
}

And let url_format default to https://raw.github.com/{owner}/{name}/{version}/{file}?

This will allow any arbitrary URL format to be specified, thus future-proofing and removing the need for libgit2.

Mar 18 '14 18:03 stephenmathieson

That seems fine to me. And since it's all JSON anyway we're free to change the structure in the future if need be.

For future reference, though, why not use libgit2? It seems better than rewriting git clone on our own. If we develop a way to specify a package.json for libraries outside of our control it'd also be a great opportunity to eat our own dogfood.

Mar 19 '14 07:03 modocache

Well, we're not really cloning; we're just fetching the files listed in your package.json's src. If we were cloning, we may as well just use git submodules :p

Mar 19 '14 12:03 stephenmathieson

What about:

"dependencies": { "my-friend/my-friends-package": { "url_format": "https://{owner}@bitbucket.org/{owner}/{name}/raw/0.1.0/{file}" } } And let url_format default to https://raw.github.com/{owner}/{name}/{version}/{file}?

This will allow any arbitrary URL format to be specified, thus future-proofing and >removing the need for libgit2.

Specifying a format instead of just an arbitrary URL seems like an undesirable UX imo. When would a user want to specify a format over just copy and pasting the URL from their browser?

I'll also :+1: the comment for implementing a go get variant; while it's not the best for specifying versions, it is pretty handy in most use cases.

Jan 26 '15 16:01 kellydunn