Support subdirectories for git deps

[lpil]

The current git dep support uses the base of the git repository as the dep rebar3 project. This generally works but some projects (such as my own repo here https://github.com/lpil/gleam) stores multiple libraries in one git repo, so the wrong directory is used.

The ability to specify the directory within the git repo would allow these mono-repo projects to be supported.

{deps,[
  {rebar, {git, "git://github.com/erlang/rebar3.git", {ref, "aef728"}, {cd, "sub/dir"}},
]}.

[tsloughter]

I've debated this. The rebar3 hex plugin supports publishing the individual apps in a multi-app project but since it is so usual to use the git version of a dep between releases if a change is needed it feels lacking that you can't do so in the case of a multi-app repo.

But hasn't been a priority. Likely will take a contribution.

[ferd]

I have discussed this with @benoitc who wanted a similar feature. He wanted to work on some monorepo plugin of which this would be an unnecessary feature, because he felt that there were better ways to do this than using a subdirectory of a git path from which to work (for example, auto-searching for the app name).

I'd be open to specifying a subpath for git deps, but not necessarily going further. We'd have to be careful not to break currently used support for rebar 2.x dep format, and {rebar, {git, "git://github.com/erlang/rebar3.git", {ref, "aef728"}, {cd, "sub/dir"}}} is a strictly non-extensible format. It would be safer for future work to allow either:

{rebar, {git, "git://github.com/erlang/rebar3.git", {ref, "aef728"}, [{subdir, "sub/dir"}]}}
{rebar, {git, "git://github.com/erlang/rebar3.git", [{ref, "aef728"}, {subdir, "sub/dir"}]}}

I feel the first one is safer since it won't allow people to specify both a ref and a branch, for example, but yeah.

[lpil]

Thanks for the feedback on the format, I wasn't very happy with my suggestion.

[benoitc]

I've debated this. The rebar3 hex plugin supports publishing the individual apps in a multi-app project

can you expand a little bit with that? do you mean running rebar3 hex publish in each folders or something else?

but since it is so usual to use the git version of a dep between releases if a change is needed it feels lacking that you can't do so in the case of a multi-app repo.

But nothing would stop to read the version of the application once the git has been fetched? What I'm thinking to do is the following:

1. fetch the git repository if it changed
2. look for the app by its name. In the case of a sub-repository each app would hav a rebar.config on its root.
3. check if the name of the application is the one given in the dependency link.
4. copy that content to the profile dir (which would mean makes this app the roo of the tmpdir used by the resource plugin

ie. when you set the following dependancy string {myapp, {git, "https://github.com/benoitc/mysubrepo.git", {tag, "SOMETAG"}}} it will fetch mysubrepo.git, look for any folder that have a rebar.config then check if the application inside that folder is myapp

One possible optimisation is to put the name of the application directly in the rebar.config so I wouldn't have to check for any app.src available in the sub folders and evaluate it. In fact having the following name, version and description set in the rebar.config would be handy in such case and I may envisage to do it. Can a provider replace such variables before any commands like compile or deps can be run ?

questions to solve:

• should I handle rebar.config.script ?
• should I handle app.config.script ?

Any feedback is welcome :)

[tsloughter]

@benoitc no, running rebar3 hex publish or rebar3 hex cut at the top level will ask which of the apps you want to publish or all.

[lpil]

Hi all! I've just taken a look at this and I was hoping to get some guidance on how to implement this feature.

Would a test belong in this suite? https://github.com/erlang/rebar3/blob/39357f1f16a573451cf4a1cfa691017b275bfcd1/test/rebar_deps_SUITE.erl#L252

Would the right approach be to move the downloaded git repo to a tmp directory and then to move the specified sub directory from the config into the desired location? If so would it be done here? https://github.com/erlang/rebar3/blob/39357f1f16a573451cf4a1cfa691017b275bfcd1/src/rebar_git_resource.erl#L123-L124

Thank you.

[ferd]

you might want to put the test suite in rebar_deps_SUITE, but the problem there is that this is all about the order of dependencies being fetched, and all mocked out because we don't want to hammer an external server to do things.

I would say that if you were to implement this in rebar_git_resource, then we'd need a test suite for rebar_git_resource itself (maybe in the systest setup? Not sure) where it makes sense to test this -- if we can assume that the ref supported yields the right app, then the behaviour should be transparent a layer above that.

Regarding the approach, there is literally no more information we have thought about than what is currently in this thread.

[benoitc]

@lpil fyi I partially implemented it in https://github.com/benoitc/rebar3_cargo . This works but the local database that keep git dependencies should be improved

[tsloughter]

Quick update on this feature. This should be done with a git sparse checkout of the directory given as an option and simply set the dir field of the app_info record appropriately.

We track all dependencies as app's in these records, we should be able to pretty easily add an additional path like apps/app-1 to the directory when constructing an app_info.

[fmv1992]

Is there an example for this? I cannot make this work. Neither this is mentioned in the docs.

[tsloughter]

@fmv1992 sorry, yea, need to update the docs. Here is an example in another repo's readme https://github.com/open-telemetry/opentelemetry-erlang#git-dependencies

[ferd]

It's already in https://rebar3.org/docs/configuration/dependencies/#declaring-dependencies

[fmv1992]

I was looking for {cd or {dir but it is {git_subdir.

Thanks for all your work on rebar3!