Migrate to rebar3 or mix

[wohali]

We're long past due upgrading to rebar 3 (latest).

This requires using @davisp's erlang-native-compiler to continue to build our NIFs.

[tsloughter]

You might find it easier to use https://github.com/blt/port_compiler/ instead of having to call out to the external enc escript just to build NIFs.

[davisp]

The issues I had trying to use port_compiler is precisely what motivated me to write enc in the first place. I haven't paid any attention at all to changes to port_compiler in the meantime but that's precisely because I've not had any issues reported with enc in the nearly two years since I last touched it.

I'm not against changing back to port_compiler if someone wants to take a stab at it.

[nickva]

I started on this a while back:

https://github.com/cloudant/couchdb/commit/b0f3feefa8c1725b178641c024a06cf7b1defb2a

Besides the port compiler issue the other one was using raw dependencies (Fauxton and docs). rebar3 doesn't do those by default (or didn't do when I checked): https://github.com/erlang/rebar3/issues/110 There might be a plugin to handle those. The other option could be to make a "fake" Erlang OTP directory structure in those dependencies.

[tsloughter]

Yea, there is a raw resource plugin that can be used.

[iilyak]

I think we should consider trying out mix (Elixir build tool) instead of rebar3. Although we would have same compatibility issues as with rebar3 (NIFs and raw deps). Nevertheless I find mix implementation faster and easier to hack on. I am happy to go with any direction the community would decide to pursue.

[garrensmith]

+1 to using mix

---

From: iilyak [email protected] Sent: Monday, March 18, 2019 7:01 PM To: apache/couchdb Cc: Subscribed Subject: Re: [apache/couchdb] Migrate to rebar3 (#1428)

I think we should consider trying out mix (Elixir build tool) instead of rebar3. Although we would have same compatibility issues as with rebar3 (NIFs and raw deps). Nevertheless I find mix implementation faster and easier to hack on. I am happy to go with any direction the community would decide to pursue.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/apache/couchdb/issues/1428#issuecomment-474007244, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAK9AteE0ppVBV9_9dRYjSHLwZ8qOzyEks5vX8ZcgaJpZM4VH-DS.

[eiri]

I'd +1 for trying mix too. I don't have an opinion on it itself, but I personally don't find rebar3 a significant improvement over "classical" rebar.

[tsloughter]

:(

I don't work on couchdb so my opinion doesn't matter here, but I am always interested in ways to improve rebar3, so please pass along (probably not on this issue unless it somehow directly relates to the question of couchdb migration) any suggestions or complaints (particularly how it doesn't do better than rebar2 which was one of the main goals and why it was essentially a rewrite).

rebar3 is fairly different from mix, so personal preference for one or the other over particular decisions we made differently is an understandable separation that can't necessarily be bridged, but always interested in hearing about them.

[wohali]

@tsloughter Thanks so much for stopping by!

The big issues for us when rebar3 dropped were problems with NIFs (why @davisp had to build enc.), coupled with the problems Nick mentioned on raw dependencies, and some Windows compile issue (probably NIF related) I ran into that I can't remember. Because that was early and we were waiting, you have probably fixed some or all of these by now, but this hasn't been a high priority ticket. It's becoming more important as we look at architectural changes to CouchDB and newer versions of Erlang.

While mix is interesting, it matters more in my opinion if we're going to start accepting Elixir as a core language, not just for our test suite. Either way, I don't have the cycles to drive the work on either, so I hope someone picks up either and runs with it.

[iilyak]

Started discussion on ML https://lists.apache.org/thread.html/a6da2f98df4f044153ae0768c967b5bb7cf475a8dec2f2b73b907a3c@%3Cdev.couchdb.apache.org%3E

[davisp]

Looks like I never actually described my original issues with the port compiler very well. But the gist of it was that projects that had a dependency that depended on Jiffy would fail to get that NIF built correctly. There is/was also a confounding issue of making it work with rebar2 since CouchDB hadn't moved off it. Creating the standalone enc script ended up being the easiest solution at the time. Its been years so its quite possible that's all fixed now.

As to mix vs rebar3 in particular, the only worry I'd have is that Elixir is going to tighten our acceptable versions of Erlang. And in my completely limited experience it seems as though they tend to follow the newest releases quite closely. But that's super anecdata so I could be way off there.

[iilyak]

As to mix vs rebar3 in particular, the only worry I'd have is that Elixir is going to tighten our acceptable versions of Erlang.

I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.

[iilyak]

Just wanted to provide update on this one

@jiahuili430 tried to switch to rebar3 here #4120 and here #4089

I took @jiahuili430 work and tried to DRY the rebar.config.script and implement release generation here. The problem I faced is that rebar3 assumes every application either in the same repository (monorepo) or it is a dependency. In case when it is a dependency rebar3 fetches it into the _build/<profile>/lib/<app>. This means that if we create a source tarball and copy sources of dependencies into src we would need to update rebar.config. Which means two separate rebar configurations for development and building from a source archive.

In order to solve this issue I implemented a plugin which allows rebar3 to copy source files to _build/<profile>/lib/<app> from either src/<app> if it is present or fetch from git otherwise. The plugin is based on https://github.com/benoitc/rebar3_path_deps.

Another issue was the boot_script generation. Currently we use reltool here. Rebar3 don't use reltool and uses relx for release generation. To avoid maintaining two configurations I wrote a plugin to generate boot file.

dependency definition problem: However the work not yet complete. Currently it complains about cyclic dependencies.

===> Verifying dependencies...
===> Dependency cycle(s) detected:
applications: b64url bear config ets_lru folsom hyper ibrowse jiffy khash meck mochiweb recon snappy depend on each other

This is because the same dependencies defined twice. Once in applications' rebar.config files and once in the main rebar.config.script. I see two options to resolve it.

1. define dependencies in each app directly. This would mean that dependencies would be in multiple places. Also I expect problems when we would want to build from released source archive.
2. implement plugin which would take the dependencies from main repository and inject overrides into the applications which refer to them in their <app>.app.src files. Effectively, it would be the same as 1) but configuration would be maintained in one place.

I also tried experimental vendoring mechanism in recent rebar3 versions. Unfortunately it doesn't do what we need. More experimentation is needed after the dependencies definition problem is sorted out.

Unfortunately, I cannot allocate more time into it at the moment. So I decided to document my current results in a hope that it would be helpful to someone who would try to do this transition.