Better support for hexpm features

[starbelly]

Currently in rebar3_hex we hardcode hexpm as the repository an app belongs to when gathering locks to generate a dependency specification for a package. Likewise, an optional param is also hard coded (to false).

It would seem like that any hex parameters for a dependency should go in hex config along with other hex params for rebar3_hex. This however feels unnatural, perhaps it's per time spent in mix, but unnatural none the less.

Instead, it would be super neat if we supported something like the following :

{deps, [{package, "0.8.4", {hex, {organization, "repo:org"}}}, {some_package, "1.1.1", {optional, true}}]}.

This unfortunately would require a new a lockfile version I believe.

Thoughts?

[tsloughter]

I wouldn't put the hex in that tuple, and we can't put the repo either. So I'd suggest:

{package, "0.8.4", {org, "myorg"}}

Or should make that third element a map now since it also has options like pkg_name:

{package, "0.8.4", #{org => "myorg"}}

Note to Fred, we currently treat "orgs" as repos, which just doesn't work well its become clear, so we should drop that goal and treat organizations as part of the package configuration.

Repos should remain the same as they are today (which is why "repo:org" can't be the option in the deps list).

[starbelly]

I wouldn't put the hex in that tuple, and we can't put the repo either. So I'd suggest:

{package, "0.8.4", {org, "myorg"}}

Or should make that third element a map now since it also has options like pkg_name:

{package, "0.8.4", #{org => "myorg"}}

But how do we know what repo an org belongs to?

Edit:

To note, mix hex supports :

defp deps() do
  [
    # This package will be fetched from the global repository
    {:ecto, "~> 2.0"},
    # This package will be fetched from the acme organization's repository
    {:secret, "~> 1.0", organization: "acme"},
  ]
end

However, it can get away with this because it only supports (in general) hexpm for organization support AFAIK.

We can go that direction and i have no problems with that, but yeah, depends on whether we want to support orgs for private hexpm instances (or compatability services).

[tsloughter]

No other hex repo has org support. We can worry about that if it ever happens.

Issue is what happens with a proxy/mirror. I don't know how you'd use a proxy between the machine running rebar3 and hexpm that is able to cache the package. Like should we ignore the organization in the configuration if it doesn't exist...

[starbelly]

No other hex repo has org support. We can worry about that if it ever happens.

Well technically, one can run a private instance of hexpm and call it whatever, and that makes sense if you're still using official hexpm along side, but I think I'm being pedantic here :p I have no idea if anyone actually does this or not, but we support it right now. That said, yeah, moving on... :)

Issue is what happens with a proxy/mirror. I don't know how you'd use a proxy between the machine running rebar3 and hexpm that is able to cache the package. Like should we ignore the organization in the configuration if it doesn't exist...

I'm not sure I follow on this one... Could you explain more?

[tsloughter]

If I run a proxy/mirror inside my network that caches or mirrors hex packages, does that proxy have to support organizations for me to mirror my private org packages internally. That would suck since stuff like mini_repo don't support it.

[starbelly]

It looks like support for that was added in mini repo around here https://github.com/wojtekmach/mini_repo/blob/1b7a6bcbcc80f69ebdda36697c8d252a6cba9d9d/config/releases.exs#L9

Edit:

Err correction, that's not what you mean.

I guess you're angle on this, how can mini_repo auth itself to hex in order to fetch private org packages. That seems like a simple change since mini_repo uses hex_core under the hood, no?

Edit:

Worth noting mini_repo is deprecated in favor of mix hex registry. I think this is another thing we also need in rebar* but, one thing at a time. Either way, a change would need to be proposed in mix hex vs mini repo.

However, I don't think any of this should be a blocker, no?

[starbelly]

Note: The PR that was done to rebar3_hex recently will not work. The repository that is to be set in the metadata, specifically requirements must be the org name (if it came from an org and is private). Even if we pass on the repo with the org name for the dep currently published and stick in the org name in the requirements metadata, it won't be correct.

That is to say, we'll be assuming that any and all deps that requirements of a dep being published to an org belong to the org and are private.

Thus, we have to get the change alluded to here for this function properly (i.e., this information for requriements should be plucked out of the lock file).

Thus, main for rebar3_hex is currently broke. I think that's ok, as no one should be dependent on that, however if we don't want to leave it in a broken state, we need to revert the last commit for now.

Edit:

Let me try to explain that better.

I have package foo, I want it to be published to hexpm within my org, this can be deduced from command line arguments to rebar3_hex or perhaps rebar.config as sort of discussed above. No problem.

Now I have package bar, I also want it to be published to hexpm within my org, it has package foo as a dependency. In order to publish this properly and without assumption, the org must be in the lock file, so when we compute requirements we're not relying on arguments to rebar3_hex and rather the lock file to discern if a repository key/value should be put in as part of the requirements.

To build on that, maybe package bar also has package baz as a dependency, this lives on hexpm in public, yet if we assume that when -r hexpm:my_org is passed into rebar3_hex for publishing and we stuff my_org in as the repository for the baz package, when we fetch these deps, say in yet another package or application, we'll try to pull baz from our org, but it does not live there, and the op will fail.

[starbelly]

I wanted to add an example here of what the metadata rebar3_hex should be publishing to hexpm (or any repository) where the user is publishing to an org should look like, in the example we can say the user is publishing to hexpm and the org is my_org.

#{<<"app">>=><<"some_app">>,
<<"requirements">>=> #{<<"truecoat">> => #{<<"app">> => <<"truecoat">>, <<"optional">>=>false, <<"repository">>=> <<"my_org">>, <<"requirement">>=><<"0.1.0">>}}}

In other words, if the user is not publishing to an org we use hexpm or simply leave blank, if it's to an org, we have to use the org for the repository name.

Likewise, what an entry in mix.lock looks like for an application using a private dependency:

%{"some_app": {:hex, :some_app, "0.1.0", "hash",[:mix],[...deps...], "hexpm:my_org", "hash" }}

Related and relevant here is a link to cross repo deps in the hex registry v2 spec : https://github.com/hexpm/specifications/blob/main/registry-v2.md#cross-repository-dependencies

[ferd]

So should we leave some of these unspecified or always be specific? I'm wondering if we were to expand the lock file today to start supporting this, whether vague or specific would be desirable.

Do keep in mind as well that we do not expect people to manually edit the lock file, so it should represent what is derived from the rebar.config information, but shouldn't be used to add specificity to what the config means. This is important when we do something like a rebar3 upgrade, where the upgrade behavior is to unlock and re-resolve the dependency.

[starbelly]

So should we leave some of these unspecified or always be specific? I'm wondering if we were to expand the lock file today to start supporting this, whether vague or specific would be desirable.

Vague. I figured repository is vague enough. And another think to think about is whether this data even effects dependency resolution. It's more like a guarantee that "when you fetch that is dep you will always fetch from from this repository. Maybe it's even worth having a meta map in the tuple if that's true.