rebar3 bare compile: comma in path component causes error in rebar_core

[peerreynders]

Workaround: Avoid commas in file paths

Background: This error was encountered via mix phx.server of the Phoenix framework. See discussion.

Primary Manifestation:

$ rebar3 bare compile --paths "/Users/dev/sbox/phx/key,value/hello/_build/dev/lib/*/ebin"
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> When submitting a bug report, please include the output of `rebar3 report "your command"`

Context:

$ mkdir "key,value"
$ cd "key,value"
$ mix phx.new hello --no-ecto
$ cd hello
$ mix phx.server
===> Compiling ranch
===> Command 'value/hello/_build/dev/lib/*/ebin' not found in namespace bare
** (Mix) Could not compile dependency :ranch, "/Users/dev/.mix/rebar3 bare compile --paths "/Users/dev/sbox/phx/key,value/hello/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile ranch", update it with "mix deps.update ranch" or clean it with "mix deps.clean ranch"

Environment

• macOS 10.14.5
• Erlang/OTP 21.3.7 (via Homebrew)
• rebar3: 3.3.5+build.3719.ref4725d36

$ rebar3 report "bare compile --paths "/Users/dev/sbox/phx/key,value/hello/_build/dev/lib/*/ebin" report"
Rebar3 report
 version 3.3.5+build.3719.ref4725d36
 generated at 2019-06-06T13:31:14+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: bare
Entered as:
  bare compile --paths /Users/dev/sbox/phx/key,value/hello/_build/dev/lib/*/ebin report
-----------------
Operating System: x86_64-apple-darwin18.5.0
ERTS: Erlang/OTP 21 [erts-10.3.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:0] [hipe] [dtrace]
Root Directory: /usr/local/Cellar/erlang/21.3.7/lib/erlang
Library directory: /usr/local/Cellar/erlang/21.3.7/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.3.0
certifi: 0.4.0
cf: 0.2.2
common_test: 1.17.1
compiler: 7.3.2
crypto: 4.4.2
cth_readable: 1.2.3
dialyzer: 3.3.2
edoc: 0.10
erlware_commons: 1.0.0
eunit: 2.3.7
eunit_formatters: 0.3.1
getopt: 0.8.2
inets: 7.0.7
kernel: 6.3.1
providers: 1.6.0
public_key: 1.6.5
relx: 3.22.2
sasl: 3.3
snmp: 5.2.12
ssl_verify_fun: 1.1.1
stdlib: 3.8.1
syntax_tools: 2.1.7
tools: 3.1

-----------------
Escript path: /usr/local/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report shell state tar tree unlock update upgrade upgrade upgrade version xref 

• Ubuntu 19.04
• Erlang/OTP 21.1 (via asdf)
• rebar3: 3.11.1

$ export DEBUG=1
$ ./rebar3 report "bare compile --paths "/home/user/Workspace/key,value/hello/_build/dev/lib/*/ebin" report"
===> Expanded command sequence to be run: [{default,report}]
===> Provider: {default,report}
Rebar3 report
 version 3.11.1
 generated at 2019-06-06T14:37:37+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: bare
Entered as:
  bare compile --paths /home/user/Workspace/key,value/hello/_build/dev/lib/*/ebin report
-----------------
Operating System: x86_64-unknown-linux-gnu
ERTS: Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Root Directory: /home/user/.asdf/installs/erlang/21.1
Library directory: /home/user/.asdf/installs/erlang/21.1/lib
-----------------
Loaded Applications:
bbmustache: 1.6.1
certifi: 2.5.1
cf: 0.2.2
common_test: 1.16.1
compiler: 7.2.5
crypto: 4.3.3
cth_readable: 1.4.4
dialyzer: 3.3.1
edoc: 0.9.4
erlware_commons: 1.3.1
eunit: 2.3.7
eunit_formatters: 0.5.0
getopt: 1.0.1
hipe: 3.18.1
inets: 7.0.2
kernel: 6.1
providers: 1.8.1
public_key: 1.6.2
relx: 3.32.1
sasl: 3.2.1
snmp: 5.2.12
ssl_verify_fun: 1.1.5
stdlib: 3.6
syntax_tools: 2.1.6
tools: 3.0.1

-----------------
Escript path: /home/user/Workspace/key,value/hello/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

• macOS:

$ export DEBUG=1
$ rebar3 bare compile --paths "/Users/dev/sbox/phx/key,value/hello/_build/dev/lib/*/ebin"
===> Expanded command sequence to be run: []
===> Expanded command sequence to be run: [{default,app_discovery},
                                                  {bare,compile}]
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,[]}
===> Stack trace to the error location:
[{rebar_prv_bare_compile,do,1,
                         [{file,"/Users/dev/sbox/erl/rb/rebar3/_build/default/lib/rebar/src/rebar_prv_bare_compile.erl"},
                          {line,45}]},
 {rebar_core,do,2,
             [{file,"/Users/dev/sbox/erl/rb/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_prv_do,do_tasks,2,
               [{file,"/Users/dev/sbox/erl/rb/rebar3/_build/default/lib/rebar/src/rebar_prv_do.erl"},
                {line,68}]},
 {rebar_core,do,2,
             [{file,"/Users/dev/sbox/erl/rb/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar3,main,1,
         [{file,"/Users/dev/sbox/erl/rb/rebar3/_build/default/lib/rebar/src/rebar3.erl"},
          {line,66}]},
 {escript,run,2,[{file,"escript.erl"},{line,758}]},
 {escript,start,1,[{file,"escript.erl"},{line,277}]},
 {init,start_em,1,[]}]
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
$

Expected behaviour

Compilation should complete successfully even when path components in the file path include commas.

Thank You for all your efforts!

[ferd]

Thing for us to do: confirm the bug is replicable in rebar3 3.11.0, but I suspect it is. Visibly appears to be argument parsing that would ignore quotation marks around strings when breaking commas.

[tsloughter]

I reproduced it.

[ferd]

Have you tried with just --paths="string,with,commas", I believe that one would just work.

From a quick debug session, it appears that --paths "string,with,commas" is parsed differently from --paths="string,with,commas" by the getopt library, so that might represent a quick workaround.

The actual issue appears to be in getopt itself, so I'll be looking at that next.

EDIT: problem appears on our end before we submit it to getopt, that simplifies things a bit.

[ferd]

Yeah looking at things, this is a rather core bit where, in order to handle special cases that depart from the getopt format, such as rebar3 bare compile <arglist>, where neither "bare" nor "compile" are arguments, we try to do as good of a job as we can understanding the remaining argument list, but because we support rebar3 do task1 --switch arg1, task2 the argument list without the = is strictly ambiguous in a rebar3 context. If you want task2 to be part of the ordeal, you gotta do rebar3 do task1 --switch=arg1,arg2 which is understood fine.

This is a limitation because escript itself turns rebar3 command --args "quoted" into the list of arguments ["command", "--args", "quoted"] and strips the quotes in a way we can't know if they were passed or not. So unless we drop support for rebar3 do a,b, we can't assume the commas are safe.

I can't think of a way to fix this but to be stricter with the input syntax when mix calls rebar3.

This is a weird interaction between how the Erlang VM passes us arguments, ways we have decided to support command syntaxes, and Mix relying on a fuzzy syntax we added to be nice but ends up running in a corner case.

I'll have to mark this as "wontfix" because I can't think of a way to fix it without creating worse breakage. The easiest fix is for mix to just use the --name="val" form rather than --name "val" form. Note that this is also true for the short form -p="..." format

[ferd]

I guess it's more that if we fix this, it would have to go in rebar4, since we'd need a full rework of what we do with the command line argument syntax to be super strict.

[tsloughter]

Won't fix makes sense (until rebar4) since it can be "fixed" in mix.