How to publish a package that depends on the package that exists in "hexpm:private_org"

[voluntas]

If I'm asking the wrong question in this issue, I'd appreciate it if you could point me to the right place.

I'm trying to use private_org in hexpm using rebar3 hex.

However, when I try to publish a private_org package with it specified as a deps package in rebar.confg, it tells me that this package does not exist in hexpm and I cannot publish it.

After following the source code, I found out that this is a message returned from the hexpm side.

https://github.com/hexpm/hexpm/blob/main/lib/hexpm/repository/requirement.ex#L29

How can I publish a package that depends on a package in hexpm:private_org to hexpm:private_org?

rebar.config

{deps, [
   {other_private_pkg_name, "1.0.0"}
]}.

{hex, [{repos, [#{name => <<"hexpm:private_org">>},
                #{name => <<"hexpm">>}]}]}.

publish command error

$ rebar3 hex publish -r hexpm:private_org

Publishing private_pkg_name 1.0.0 to hexpm:private_org

===> Failed to publish package: Validation error(s)
	Requirements could not be computed
other_private_pkg_name
--------------------
package does not exist in repository "hexpm"

[tsloughter]

This likely doesn't work and is a bug.

But to be clear, compiling works? So it is able to fetch that private package (and I take it, publish that package as well?) and breaks only for publishing a package depending on a private package?

I'll have to look into where we do this check it is doing that results in it saying it can't find the package.

[starbelly]

I'd agree it's a bug of sorts. Currently, there is a no way to declare a dependency in rebar.config as belonging to a private repo, thus when we send up the requirement without specifying a repo for the dep, and it ends up in the global name space and why you end up with that error.

☝️ That's an assumption without digging through hex code.

@wojtekmach is that assumption correct? Or should hexpm be searching for the dep in the users private's org regardless?

[tsloughter]

So the error is coming from hexpm?

Question I have then is if deps can depend on deps from other private orgs?

If not, then I'd agree it sounds like something hexpm could be doing since it simply has to be in the same org you are publishing to.

If they can span multiple orgs then we need to be the ones ensuring we are publishing the package with a dependency on the exact same package (meaning specifying the org) that was being used locally.

[starbelly]

Question I have then is if deps can depend on deps from other private orgs?

They can.

If they can span multiple orgs then we need to be the ones ensuring we are publishing the package with a dependency on the exact same package (meaning specifying the org) that was being used locally.

Yeah, and I think that'll have to result in a patch to rebar3 in that case. Correct me if I'm wrong though. Basically, rebar3 will have to be patched to ignore and extra field.

[starbelly]

Question I have then is if deps can depend on deps from other private orgs?

Ahh, wait, I misread... I don't believe deps can depends on deps from other private orgs, but I'm not 100% sure. They can definitely depend on other deps from within the same private org for sure though.

I suppose that part is not clear... @voluntas is other_private_pkg_name in hexpm:private_org

[tsloughter]

Not sure what you mean by ignoring an extra field.

[starbelly]

Not sure what you mean by ignoring an extra field.

It's been a minute since I've hacked on rebar3 in this area, but if we say support :

{other_private_pkg_name, "1.0.0", {repo, "hexpm:some_private_org"}}

Would we have to patch rebar3 for that? Or would rebar3 just ignore it?

[voluntas]

@tsloughter @starbelly

Thank you so much for the answers and the discussion.

Oh, I'm very sorry. I meant that other_private_pkg_name is a different package than private_pkg_name in the same private_org.

The problem was that pkg_a, which is in the same private_org, cannot be published because it depends on pkg_b.

But to be clear, compiling works? So it is able to fetch that private package (and I take it, publish that package as well?) and breaks only for publishing a package depending on a private package?

I can compile, and I can get the package. However, I am having problems only when publishing.

{other_private_pkg_name, "1.0.0", {repo, "hexpm:some_private_org"}}

This configuration is very much what I want, and I would be very happy if you could introduce this one.

[tsloughter]

We went a different way with how deps are fetched in rebar3, so

{other_private_pkg_name, "1.0.0", {repo, "hexpm:some_private_org"}}

isn't an option.

We don't want to tie packages to specific repos, it would make use of mirrors and other repos harder.

This plugin will need to check where a dep came from -- by checking the local registry -- before publishing, and use that info to tell hex what it wants to know.

[starbelly]

@tsloughter Gotcha. I'll look into this post v7.

[starbelly]

@tsloughter There's a problem with not being able to specify what repository a dep should come from specifically. You may have the package foo on hexpm but you may have a private version in your org repo hexpm:my_org. Without being able to specify this within dep config we might fetch the wrong one.

Are we sure we can not add {other_private_pkg_name, "1.0.0", {repo, "hexpm:some_private_org"}} support?

[tsloughter]

Can orgs have names of public packages in them?

And its only an issue for initial setup since the lock file will have the checksum.

Assuming hex.pm allows an org to have a package with the same name as a public package it could be confusing to a user who is initially setting up a project and for some reason wants the public package but has their private org listed first in their repos... First would want to know if that is actually allowed and then can go from there on if it is such a likely concern that we need to change anything.

[starbelly]

Yeah, it's allowed. At one company we actually had a valid case for this. There was a dependency that had a PR open for it, but we weren't sure when it would be merged, mean while everyone was fetching the dep from git, which wasn't ideal, so we published the patched version to our org and fetched it that way vs git.

[tsloughter]

But in that case it would already work. The issue is if you have a private version but want the public version.

[starbelly]

Ahh, you are right. I just tested this. There's still an edge case, but this doesn't have to be solved today either. The case would be when you want the public but not the private, yet you need to have the org repo in your config.