support for git dependencies

[clemos]

It doesn't seem possible to have git dependencies currently :

"dependencies": {
        "hxnodejs": "https://github.com/HaxeFoundation/hxnodejs.git"
}

The above seems to fail. Or maybe there is another syntax ? If not, then I think it would be nice to support git dependencies, pretty much like npm does.

[nadako]

note that it would be great to also be able to specify commit number/branch/tag (like npm does I think)

[clemos]

Yes, it's like npm install git+ssh://[email protected]:npm/npm.git#v1.0.27 see https://docs.npmjs.com/cli/install

[hexpunk]

+1 for this

[nadako]

As mentioned in https://github.com/HaxeFoundation/haxelib/pull/135#issuecomment-163228477, I think we should adopt NPM dependency format, see https://docs.npmjs.com/files/package.json#dependencies.

[ncannasse]

I'm fine with following anything that is quite standardized as NPM seems to be, as long as we keep backward compatibility with our previous format, so users can continue downloading and submitting libraries using the old format.

[boozook]

1. In this case haxelib.json must be exists in root of repo;
2. What about another (potential) supported systems such as Mercurial or Subversion? Should we detect what use by url? No :).

[porfirioribeiro]

Maybe something like Cargo from rust lang? http://doc.crates.io/manifest.html#the-[dependencies]-section It accepts git repo's as a depedency. The Cargo.toml is either used on Lib's or app's to specify options including dependencies.

[nadako]

That is interesting stuff, thanks!

[boozook]

I like the Cargo impl too. but this is new format and will change the our current format: From {"libname" : "ver"} to {"libname" : {"version":"", "git/hg/svn":"", "branch/tag/rev":""} }.

But we can really preserve backward compatibility. We can easy support old(current) and new formats - detect format via Reflection and use the special provider (parser) for it (Or EitherType).

[boozook]

Another one variant - namespace-like:

"dependencies": {
        "libFromHaxelibA": "",
        "libFromHaxelibB": "10.2.4",
        "libFromGit": "git::https://github.com/HaxeFoundation/libFromGit.git",
        "libFromHg": "hg::https://bitbucket.org/HaxeFoundation/libFromHg"
}

But without tags/revs/branches.

[boozook]

I love the NPMs format but it doesn't support any other control version system.

[nadako]

The npm tool itself doesn't support anything but git, but the URL format looks extensible enough (e.g. hg+http://... similar to supported-by-npm git+http://....

[nadako]

As discussed in #105, it would be also nice to support http://... urls, which is also supported by NPM. I propose we borrow npm's url format and use it for both "haxelib install" and dependency urls. We can implement it using some kind of "providers" architecture as it's done in @ibilon's sources related to #217

[nadako]

This raises more questions: should we / how to store multiple versions of a library checked out from some vcs.

For example, library A declares "somelib": "git://some/lib#rev1" dependency which is a rev1 commit from some/lib while library B wants "somelib" to be e.g. git://other/lib#rev2.

Technically, we can support that by generating some unique version identifier from a vcs url (like git__some_lib__rev1), but do we want to go that route?

[ibilon]

It depends if we can make the hypothesis that the user won't modify the content.

In that case when installing we only fetch, and when compiling we do a checkout of the required commit/branch/tag.

[grosmar]

Hello,

We have the same need so I've investigated the sources and this thread, and have some conclusion and some code now.

• As @ncannasse said we should be backward compatible, so even if we extend the syntax we should keep working with the existing one. I've considered it through all the points.
• I've found a strange concept at this moment. When we use a library, we can install it's dependencies two ways: if the library has it's own build.hxml we can install them from it with haxelib, or if it has a haxelib.json and we install it through haxelib install or haxelib git commands then it read up the dependencies. But for the haxelib.json there is no implementation to install it when is it on my local drive (like when you use npm install and it reads up the local package.json and install all the dependency from it) I think we should be able to follow the concepts of other package managers, and we should be able to express our dependencies with versions in the haxelib.json locally, even if this is our main project. And after we should just express the need in the build.hxml. But for keeping the backward-compatibility, i would implement also the git dependency support in the build.hxml So for this i've tried the following syntax: haxelib install <haxelib.json path> So the haxelib install itself looks for the haxelib.json file in the current directory. If it find it, start it's installing. If it get as a parameter the path for the haxelib.json than it will install that one.
• About multiple git dependencies: I would follow the concept that @nadako said: generate git_somebranch syntax. But only as a second step, because it raise a lot of questions. Anyway, now i would change that behavior in the code when install git dependency install it as a dev dependency. There is no point in it, it's more than enough to just set the current version to the git. It works perfectly and no need to force to that version.

To make it backward compatible, i've tried with the following syntax: build.hxml: -lib something:git:https://github.com/grosmar/something.git#commit-ish This is fully compatible with the existing haxe compiler, because it splits at ":", so it will understand to look for the "git" version folder. If we later introduce support of multiple versions, than it can be git_mybranch:https://... #commit-ish would mean anything that can be passed to the git checkout command

haxelib.json dependencies: { "something: "git:https://github.com/grosmar/something.git#commit-ish" }

Later we can introduce maybe a structured syntax for the haxelib.json, to describe all the features of git dependency (like subfolder).

So for now i would not follow fully the npm style git+..., but i would use git:... to keep the compatibility.

According to these conclusions i've made some implementation here: https://github.com/grosmar/haxelib/tree/library_git_dependency

This branch implement the following features that i've mentioned above:

• haxelib install <haxelib.json path>
• build.hxml describe git dependency
• haxelib.json describe git dependency

I still need to write the tests if we can agree on the things.

Also we could think about some extra syntax to describe subdir like git:https://github.com/grosmar/something.git#commit-ish@subDir And we should be able to express the version (tags). The #commit-ish part covers it, but maybe it would be nice to directly express the need only for tags. But i have no good idea for this. Any suggetions?

What do you think guys?

[nadako]

I'm yet to look at the code, but the proposition looks nice.

I agree about expanding haxelib.json purpose so it's not only used for "libraries", but for projects in general, similar to npm's package.json. We can even add more functionality to it, like haxelib test that reads from haxelib.json that is has to run haxe test.hxml, but that's another topic.

git:https://github.com/grosmar/something.git#commit-ish@subDir

another option would be to add another colon here, e.g. git:https://github.com/grosmar/something.git#commit-ish:subDir :)

as for tags, I think commit-ish is sufficient.

[grosmar]

Let me know if you have time to check the repository. I happily finish the devs if we can finalize the concept.

[nadako]

I'm looking at the code right now. Can't say I like it very much, but that's not about your part, but haxelib codebase in general. I think if we decide on the design here, any code will do as long as it works, then we can refactor it from there.

Also, I remembered one issue with losing dev-mode for vcs dependencies - using a subdir inside a repository for the haxelib (see https://github.com/HaxeFoundation/haxelib/issues/263#issuecomment-164477060). I wonder though if that feature used at all and worth caring about...

[nadako]

Anyway, I can't think of any issues regarding the proposed syntax.

[grosmar]

I also cannot say that i like the code, but i didn't want to change deeply the codebase because of this implementation.

I can look into the subdir devmode problem later. But for me this subdir stuff sounds a bit nasty anyway. I prefer one repository for one lib, so why should i need a subdir?

So can we agree on the followings as a final decision:

• support haxelib.json file also for local dependency descriptions, and use the following syntax to install them: haxelib install <haxelib.json path> (if no path given it tries to read haxelib.json in the current directory)
• support git dependencies in haxelib.json with the following syntax: dependencies: { "libname: "git:https://github.com/grosmar/something.git#commit-ish:subdir" }
• support git dependencies in build.hxml with the following syntax: -lib something:git:https://github.com/grosmar/something.git#commit-ish:subdir

commit-ish means anything that can be passed to git checkout command subdir is reference for the subdirectory of the git repository

And later we can improve the implementation and use versioned git folder like this: git_commit-ish_subdir

This way we can have multiple git version checked out and used. But I don't want to mix it now.

So can we agree on this syntax? If yes, i finish the development and create the tests.

[nadako]

I can look into the subdir devmode problem later. But for me this subdir stuff sounds a bit nasty anyway. I prefer one repository for one lib, so why should i need a subdir?

No idea, I think I have never needed that.

support haxelib.json file also for local dependency descriptions, and use the following syntax to install them: haxelib install <haxelib.json path> (if no path given it tries to read haxelib.json in the current directory)

Fine with me :+1:

support git dependencies in haxelib.json with the following syntax: dependencies: { "libname: "git:https://github.com/grosmar/something.git#commit-ish:subdir" } support git dependencies in build.hxml with the following syntax: -lib git:https://github.com/grosmar/something.git#commit-ish:subdir

Not sure about the subdir syntax, but if we lose that feature, we don't need it anyway :)

I don't feel like I can decide for the HF here though, so let's summon @ncannasse @Simn @andyli. Check it out guys, this feature is quite important IMO.

Also ping @fponticelli and @andywhite37, since they are using the hmm tool they wrote, so they may have an opinion on this!

[andyli]

I really want this feature too. However, I don't think we will have enough time to make sure the design and implementation is solid enough for the upcoming haxe 3.3 release. So I will suggest looking into it after that.

[nadako]

Well, the good thing is that we can release haxelib separately. In fact, there's an issue about that.

[nadako]

Hmm, but we want some release bundled with haxe though. However I don't see a problem with including the current proposed stuff in the release, the changes are minimal. IMO the only thing we have to think of is the devmode/subdir issue.

[grosmar]

I also agree the changes are minimal, and kind of new, that won't break any existing behavior.

Later i will look into devmode/subdir stuff and can talk about it. But IMO it's pretty strange to support this bad practice.

[nadako]

Later i will look into devmode/subdir stuff and can talk about it. But IMO it's pretty strange to support this bad practice.

I agree, but that's an existing feature, so removing it is a breaking change, so someone has to make a hard decision about that.

[grosmar]

Your right, so i will check it anyway :)

[grosmar]

I've looked into it. The problem is simple: There is no way to tell haxelib which subdirectory used for the current checkout, only if you set a dev path.

I know a few solutions for this:

• Implement support in the .current descriptor file the path referencing, but it has also an effect on the haxe compiler, so i would avoid it now.
• Use spare checkout feature of git, but i think it's more tricky feature, i don't know it's limitations.
• After checkout the git repository use only the given subDir, and delete everything else. This way we loose the possibility to manually update a git repository, but we can use the subfolder without dev dependecy.
• Only use library as a dev dependency when the subfolder is given. I think this is the easiest solution. Only con that I can mention is causes some inconsistent behavior with and without subDir parameter

And one more note: Because before noone had the possibility for configured git dependency in haxelib.json and build.hxml, does it really matter if we don't implement it in these? So if someone really need a subfolder he can manually install it in the oldschool way: haxelib git something https://github.com/grosmar/something.git master subDir and it will be solved with the dev dependency

What do you think?

[nadako]

When I was thinking about this while working on #263, I mostly thought about the first solution. It shouldn't affect Haxe compiler itself, since it just calls haxelib path, but still I'm very reluctant to change any file formats (especially if it's a breaking change, like this one), because if we do that, we might as well redesign the whole thing.

Solutions 2 and 3 look very ugly, let's not do that! :)

Solution 4 is a great idea IMO. Especially if we all can agree that the subdir feature is not worth it, we can deprecate it, support it with the devmode hack for some time, and then simply remove it (however I'd just remove it completely, TBH).