Inconsistent behavior in "shell" & "release" when sys.config is a subdirectory and contains a reference to another config file

[yhuangsh]

Environment

Rebar3 report
 version 3.7.5
 generated at 2019-04-20T14:33:49+00:00`
=================`
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: shell
Entered as:
  shell
-----------------
Operating System: x86_64-pc-linux-musl
ERTS: Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]
Root Directory: /usr/local/lib/erlang
Library directory: /usr/local/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.6.0
certifi: 2.3.1
cf: 0.2.2
common_test: 1.16.1
compiler: 7.3
crypto: 4.4
cth_readable: 1.4.2
dialyzer: 3.3.1
edoc: 0.9.4
erlware_commons: 1.3.0
eunit: 2.3.7
eunit_formatters: 0.5.0
getopt: 1.0.1
hex_core: 0.2.0
hipe: 3.18.2
inets: 7.0.3
kernel: 6.2
providers: 1.7.0
public_key: 1.6.4
relx: 3.27.0
sasl: 3.3
snmp: 5.2.12
ssl_verify_fun: 1.1.3
stdlib: 3.7
syntax_tools: 2.1.6
tools: 3.0.2

-----------------
Escript path: /usr/local/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

It's not a crash. Command used as follows:

# DEBUG=1 rebar3 as stg_cohost shell
===> Expanded command sequence to be run: [{default,as}]
===> Provider: {default,as}
===> Expanded command sequence to be run: [{default,app_discovery},
                                                  {default,install_deps},
                                                  {default,lock},
                                                  {default,compile},
                                                  {default,shell}]
===> Provider: {default,app_discovery}
===> Evaluating config script "/project/fiftypm_api/_build/default/lib/jsx/rebar.config.script"
===> Provider: {default,install_deps}
===> Verifying dependencies...
===> Provider: {default,lock}
===> Provider: {default,compile}
===> run_hooks("/project/fiftypm_api", pre_hooks, compile) -> no hooks defined

===> Compiling fiftypm_api
===> run_hooks("/project/fiftypm_api", pre_hooks, compile) -> no hooks defined

===> run_hooks("/project/fiftypm_api", pre_hooks, erlc_compile) -> no hooks defined

===> erlopts [debug_info,{d,'STAGING'},{d,'COHOST_STAGING'}]
===> files to compile ["/project/fiftypm_api/src/fiftypm_api_badreq.erl",
                              "/project/fiftypm_api/src/fiftypm_api_session.erl",
                              "/project/fiftypm_api/src/tab_session.erl",
                              "/project/fiftypm_api/src/fiftypm_api_sup.erl",
                              "/project/fiftypm_api/src/fiftypm_api_probes.erl",
                              "/project/fiftypm_api/src/httpres.erl",
                              "/project/fiftypm_api/src/fiftypm_api_app.erl",
                              "/project/fiftypm_api/src/fiftypm_api_main.erl",
                              "/project/fiftypm_api/src/fiftypm_api_login.erl"]
===> run_hooks("/project/fiftypm_api", post_hooks, erlc_compile) -> no hooks defined

===> run_hooks("/project/fiftypm_api", pre_hooks, app_compile) -> no hooks defined

===> run_hooks("/project/fiftypm_api", post_hooks, app_compile) -> no hooks defined

===> run_hooks("/project/fiftypm_api", post_hooks, compile) -> no hooks defined

===> run_hooks("/project/fiftypm_api", post_hooks, compile) -> no hooks defined

===> Provider: {default,shell}
Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]

===> No script_file specified.
===> Found shell opts from command line option.
===> Found config from relx.
===> Loading configuration from "/project/fiftypm_api/config/sys.config"
===> Loading configuration from "/project/fiftypm_api/apikey.config"
===> The rebar3 shell is a development tool; to deploy applications in production, consider using releases (http://www.rebar3.org/docs/releases)

The issue is shown on 3rd last line:

===> Loading configuration from "/project/fiftypm_api/config/sys.config"
===> Loading configuration from "/project/fiftypm_api/apikey.config"

Expected behaviour

Expected:

===> Loading configuration from "/project/fiftypm_api/config/sys.config"
===> Loading configuration from "/project/fiftypm_api/config/apikey.config"

Because sys.config contains a line referencing another config file like this:

[
    {mnesia, [{dir, "/deploy/data/fiftypm_api"}]},
    "apikey.config"
].

In my project, I put my config files in a separate config directory like below:

project_root/
    config/
        apikey.config
        sys.config
    src/
    tools/

And my rebar.config has a relx section like the following:

{relx, 
    [
        {release,
            {fiftypm_api, "0.1.0"},
            [fiftypm_api, cowboy, jsx, inets, mnesia]
        },
        {dev_mode, false},
        {include_erts, true},
        {extended_start_script, true},
        {vm_args, "config/vm.args"},
        {sys_config, "config/sys.config"}
    ]
}.

According to Erlang document: config

File = string() Name of another .config file. Extension .config can be omitted. It is recommended to use absolute paths. If a relative path is used, File is searched, first, relative from sys.config directory, then relative to the current working directory of the emulator, for backward compatibility. This allow to use a sys.config pointing out other .config files in a release or in a node started manually using -config ... with same result whatever the current working directory.

The apikey.config should be first searched and read from the sys.config directory.

This is indeed the case and the behavior was as described when I did this:

project_root/ # rebar3 as stg_cohost release
project_root/ # _build/stg_cohost/rel/myproject/releases/bin/myproject console

The above started my app and all the variables in apikey.config were correctly imported.

However, I was not able to see the variables in apikey.config if I use shell to start my app, like this:

project_root/ # rebar3 as stg_cohost shell

Because in this case the resolved path misses the config segment in the full path, resulting in apikey.config not read. And no error message was shown. I read the source code. The referenced file name is simply prefixed with the project root. In my case apikey is simply not found and the source code shows that this error is quietly ignored.

I could fix this by adding config in front of apikey.config like this:

[
    {mnesia, [{dir, "/deploy/data/fiftypm_api"}]},
    "config/apikey.config"
].

But this breaks my relx build, because the release build searches the config/apikey.config already in the config/ directory that sys.config is located.

I've compared the related rebar3 source in both v3.7.2 (the version I'm using) and master branches. It does not seem to be an issue that was previously reported and fixed.

My workaround is to have another sys.config.dev which replicates everything from the sys.config but uses a different path.

I have my actual project hosted on https://github.com/yhuangsh/50pm.

[ferd]

Please try a newer rebar3 version, I think this is supported in newer versions.

[yhuangsh]

Tried rebar3 v3.10.0, same issue. Don't think it's fixed.