In common test sys_config is loaded prior to pre_hooks complete

[Juliusan]

Current behaviour

In a following rebar configuration

{pre_hooks, [
    {ct, "sed 's@&&f@'\"$(pwd)\"'@g' test/sys.config.template > test/sys.config"}    # Change test/sys.config file
]}.
{ct_opts, [{sys_config, ["test/sys.config"]}]}.

Running rebar3 ct doesn't take into account changes to sys.config made in the hook. That is, if you change the template file test/sys.config.template, you need two runs: first run runs the pre_hook to change the test/sys.config file, but starts common test with unchanged test/sys.config and the second run does effectively nothing to test/sys.config, however, it runs common test with new test/sys.config.

Expected behaviour

pre_hooks of ct should be completed, before ct is started and test/sys.config is read.

Environment

Here is output of my rebar3 report:

Rebar3 report
 version 3.18.0
 generated at 2022-09-23T10:30:39+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: 
Entered as:
  
-----------------
Operating System: x86_64-pc-linux-gnu
ERTS: Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]
Root Directory: /home/julius/.kerl/installs/24.3
Library directory: /home/julius/.kerl/installs/24.3/lib
-----------------
Loaded Applications:
bbmustache: 1.12.2
certifi: 2.8.0
cf: 0.3.1
common_test: 1.22.1
compiler: 8.1.1
crypto: 5.0.6.2
cth_readable: 1.5.1
dialyzer: 4.4.4
edoc: 1.1
erlware_commons: 1.5.0
eunit: 2.7
eunit_formatters: 0.5.0
getopt: 1.0.1
inets: 7.5.3
kernel: 8.3.2
providers: 1.9.0
public_key: 1.12
relx: 4.6.0
sasl: 4.1.2
snmp: 5.12
ssl_verify_fun: 1.1.6
stdlib: 3.17.2
syntax_tools: 2.6
tools: 3.5.2

-----------------
Escript path: /home/julius/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref 

[ferd]

Yeah, it's structured that way:

• https://github.com/erlang/rebar3/blob/58a6dac54ff2ad76e006f13620727f662e14e716/apps/rebar/src/rebar_prv_common_test.erl#L44
• https://github.com/erlang/rebar3/blob/58a6dac54ff2ad76e006f13620727f662e14e716/apps/rebar/src/rebar_prv_common_test.erl#L68

That can possibly be shifted around, but may break things if people expected test modules to be pre-compiled before the hooks run, in which case they won't be anymore. I'm not sure there's a safe way to change this in a backwards-compatible manner.