Use SemVer for Hex dependencies

[ericmj]

Hex requires pacakges to use semantic versioning. The packages amqp_client and rabbit_common uses strict version requirements on its dependencies which causes conflicts and ecosystem problems for users of those packages. Instead they should use the ~> version requirement operator that follows the rules of SemVer. ~> 1.0 will depend on any stable version >= 1.0.0 and < 2.0.0.

Current dependencies

amqp_client:
rabbit_common == 3.7.13

rabbit_common:
jsx == 2.9.0
lager == 3.6.5
ranch == 1.7.1
recon == 2.3.6

How dependencies should be declared

amqp_client:
rabbit_common ~> 3.7

rabbit_common:
jsx ~> 2.9
lager ~> 3.6
ranch ~> 1.7
recon ~> 2.3

[michaelklishin]

This client very likely will continue requiring an identical rabbit_common version. rabbit_common's dependencies perhaps can be relaxed. @dumbbell would there be any potential ramifications for the pipelines?

[dumbbell]

RabbitMQ does not use semantic versioning for its own components currently. As @michaelklishin said, the rabbit_common version pinning in amqp_client is strict for this reason.

About the pinning of third-party dependencies, we also want strict pinning for the following reasons:

• We don't want third-party dependencies to be updated between the beginning of our CI pipeline and the end: what was tested is what is shipped.
• We want reproducible packages at some point: if we rebuild version 1.2.3, we want it to pick the exact same dependencies.

[dumbbell]

Note that Erlang.mk, the build tool we use, does not have a "lock" mechanism like other build tools, so we can't rely on such a thing.

[ericmj]

About the pinning of third-party dependencies, we also want strict pinning for the following reasons:

This should not be be achieved by version pinning your package dependencies - what you want to do is pin your project dependencies. Rebar3 and mix support this out of the box with lock files.

If you use another build tool that does not support lock files I suggest changing the package metadata before publishing or listing project dependencies and package dependencies separately. I notice that you have built a custom tool for publishing to hex, maybe this can be added to that tool?

[michaelklishin]

What we need here is to relax rabbit_common dependencies in Hex metadata. @dumbbell can we add a Mix file to it just for dependency management at hex.pm release time? Pinning to a certain minor version of a dependency should work reasonably well in practice. Application developers can ping to a specific patch if needed.

[dumbbell]

If you use another build tool that does not support lock files I suggest changing the package metadata before publishing or listing project dependencies and package dependencies separately. I notice that you have built a custom tool for publishing to hex, maybe this can be added to that tool?

That's a good idea. Unfortunately, what we publish to Hex.pm still uses Erlang.mk as the build tool and it only accepts exact version pinning (it doesn't support the -> x.y syntax and logic).

Can we add a Mix file to it just for dependency management at hex.pm release time?

Mix is not the build time, even after rabbit_common is published to Hex.pm, so we are still limited by what Erlang.mk provides.

[ericmj]

That's a good idea. Unfortunately, what we publish to Hex.pm still uses Erlang.mk as the build tool and it only accepts exact version pinning (it doesn't support the -> x.y syntax and logic).

There's a difference what the package metadata is and what the package contents are. You can have ~> in the metadata and an exacter version in the Makefile config.

Mix is not the build time, even after rabbit_common is published to Hex.pm, so we are still limited by what Erlang.mk provides.

It seems like all the packages you publish support building with both Erlang.mk and rebar3, at least that's what is in the package metadata.

[michaelklishin]

@ericmj we understand what you are proposing. It's mostly a matter of finding a way to do that that won't be too fragile and/or require substantial changes to our build system or pipeline. I think it's doable, we'll investigate soon. Thank you for the suggestion.

[ericmj]

Thank you for looking into it!

[dumbbell]

It seems like all the packages you publish support building with both Erlang.mk and rebar3, at least that's what is in the package metadata.

Hmm, you're right, I was looking at the list of files in the output of rebar3 hex publish and didn't spot rebar.config. I assumed we didn't support rebar3 and forgot the Rebar files are not listed explicitely.

So yes, we should be able to patch the generated rebar.config to use relaxed pinning.

[tiagoefmoraes]

Found this issue while investigating why some deps in my project could not be upgraded, and attempted to patch rebar.config while keeping rabbit_common version strict.

This sed command does that(playground):

sed -i.bak -E 's/(rabbit_common,")/\1_DONT_REPLACE_/g; s/"([[:digit:]]+\.[[:digit:]]+)\.[[:digit:]]+"/"~> \1"/g; s/(rabbit_common,")_DONT_REPLACE_/\1/g' rebar.config;

example

{deps, [
-{thoas,"1.2.3"},{recon,"11.22.33"},{credentials_obfuscation,"111.222.333"},{rabbit_common,"1.2.3"}
+{thoas,"~> 1.2"},{recon,"~> 11.22"},{credentials_obfuscation,"~> 111.222"},{rabbit_common,"1.2.3"}
]}.
{erl_opts, [debug_info,warn_export_vars,warn_shadow_vars,warn_obsolete_guard,deterministic]}.

My guess is that it can be called right before $(MAKE) hex-release-publish here:

https://github.com/rabbitmq/rabbitmq-server/blob/cb4676da8880579922c92cf8ade546a86c84cf52/deps/rabbit_common/mk/rabbitmq-hexpm.mk#L54-L59

@@ -54,6 +54,7 @@ hex-publish: app rebar.config
                    rabbitmq-components.mk.not-hexpm \
                    $(RMQ_COMPONENTS_PLAIN); \
                fi' EXIT INT; \
+               sed -i.bak -E 's/(rabbit_common,")/\1_DONT_REPLACE_/g; s/"([[:digit:]]+\.[[:digit:]]+)\.[[:digit:]]+"/"~> \1"/g; s/(rabbit_common,")_DONT_REPLACE_/\1/g' rebar.config; \
                $(MAKE) hex-release-publish
 
 hex-publish-docs: app docs