[bug] generated conanbuildenv appears to be missing list "for v in"

[GregDomjan]

Describe the bug

Attempting to start using conan2 building on AIX Power8 ppc64 Conan 2.12

AIX toolbox installed Python 3.7 CMake 2.16

default shell is /bin/ksh, but same happens if changing to /bin/sh

Building expat, c-ares or any other dependency so far

======== Input profiles ======== Profile host: [settings] arch=ppc64 build_type=Release compiler=gcc compiler.cppstd=gnu17 compiler.libcxx=libstdc++11 compiler.version=8 os=AIX [conf]

Profile build: [settings] arch=ppc64 build_type=Release compiler=gcc compiler.cppstd=gnu17 compiler.libcxx=libstdc++11 compiler.version=8 os=AIX [conf]

Build gives error, that looks like cmake failing, but seems to be from the generated conanbuildenv

-------- Installing package expat/2.7.1 (1 of 11) -------- expat/2.7.1: Building from source expat/2.7.1: Package expat/2.7.1:4b3487e36b23b8c80576f6b45c4385b61d49f7c8 expat/2.7.1: Copying sources to build folder expat/2.7.1: Building your package in ~/.conan2/p/b/expat06579fa46dd6b/b expat/2.7.1: Calling generate() expat/2.7.1: Generators folder: ~/.conan2/p/b/expat06579fa46dd6b/b/build/Release/generators expat/2.7.1: CMakeToolchain generated: conan_toolchain.cmake expat/2.7.1: CMakeToolchain generated: ~/.conan2/p/b/expat06579fa46dd6b/b/build/Release/generators/CMakePresets.json expat/2.7.1: CMakeToolchain generated: ~/.conan2/p/b/expat06579fa46dd6b/b/src/CMakeUserPresets.json expat/2.7.1: Generating aggregated env files expat/2.7.1: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] expat/2.7.1: Calling build() expat/2.7.1: apply_conandata_patches(): No patches defined in conandata expat/2.7.1: Running CMake.configure() expat/2.7.1: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="~/.conan2/p/b/expat06579fa46dd6b/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "~/.conan2/p/b/expat06579fa46dd6b/b/src" /bin/sh: 0403-057 Syntax error at line 4 : `newline or ;' is not expected.

expat/2.7.1: ERROR: Package '4b3487e36b23b8c80576f6b45c4385b61d49f7c8' build failed expat/2.7.1: WARN: Build folder ~/.conan2/p/b/expat06579fa46dd6b/b/build/Release ERROR: expat/2.7.1: Error in build() method, line 68 cmake.configure() ConanException: Error 2 while executing

Trying to debug I found similar error message running ~/.conan2/p/b/expat06579fa46dd6b/b/build/Release/generators/conanbuildenv-release-ppc64.sh

~/.conan2/p/b/expat06579fa46dd6b/b/build/Release/generators/conanbuildenv-release-ppc64.sh[3]: 0403-057 Syntax error at line 4 : `newline or ;' is not expected.

script_folder="/home/nds_cm/.conan2/p/b/expat06579fa46dd6b/b/build/Release/generators"
echo "echo Restoring environment" > "$script_folder/deactivate_conanbuildenv-release-ppc64.sh"
for v in
do
    is_defined="true"
    value=$(printenv $v) || is_defined="" || true
    if [ -n "$value" ] || [ -n "$is_defined" ]
    then
        echo export "$v='$value'" >> "$script_folder/deactivate_conanbuildenv-release-ppc64.sh"
    else
        echo unset $v >> "$script_folder/deactivate_conanbuildenv-release-ppc64.sh"
    fi
done

Line 4 seems to be missing a list of some sort

How to reproduce it

Cut down conan file demonstrates the issue conanfile.zip

conan install ../conanfile.py -r aauth -u -pr:b=gcc8-aix -pr:h=gcc8-aix -u --output-folder=. -s build_type=Release --build missing

[memsharded]

Hi @GregDomjan

Thanks for your report.

This is a bit unexpected, those scripts are massively used. Is it possible that you are using a different shell? Can you please try running in bash and in a basic sh shell?

[GregDomjan]

Same again launching after switching shell to /bin/bash or /bin/sh

bash.log

[GregDomjan]

Also attempted without success updated Conan 2.13 Python 3.9/3.11 CMake 3.22

[memsharded]

This is still surprising, I guess there might be something different in the AIX shells, maybe the shell version (old one?) or something like that. Do you know if there are docker images of AIX or any other way we could try to reproduce this and investigate?

[GregDomjan]

I'm not familiar, I'll see what I can find to enable this.

A first look it does seem like IBM license is going to make it difficult with getting a source image Qemu has ppc64, and this seems to be a rough instruction https://community.ibm.com/community/user/powerdeveloper/blogs/hugo-b/2024/01/17/aix-virtualization-x86-kvm-qemu

As an alternative, a bit slower perhaps, Is there something I could run up to give more detail and help dig into the problem?

[GregDomjan]

I hadn't noticed in the initial, but both the build and the run shell scripts have the same issue.

When I run conan from /bin/bash I don't get a different result, it seems to still run from /bin/sh shell Is there a way to configure and specify a specific/alternate shell?

Since changing my active shell didn't seem to change how it was running, I've tried some other ways to set the shell now I feel like I'm chasing my own tail, so confused.

the script doesn't have a shebang for the specific shell if I put in the shebang as #!/bin/bash then the error goes away if I put in the #!/bin/ksh or sh then the error was displayed if I remove the shebang, then the error remains gone, except sometimes when it doesn't.

ksh and sh have support for the syntax, but there must be some failure when no list is provided https://www.ibm.com/docs/en/aix/7.2?topic=commands-korn-shell-compound

bash-5.1$ /bin/ksh ~/.conan2/p/b/c-are5619eeba17ab1/b/build/Release/generators/conanbuildenv-release-ppc64.sh ~/.conan2/p/b/c-are5619eeba17ab1/b/build/Release/generators/conanbuildenv-release-ppc64.sh[4]: 0403-057 Syntax error at line 4 : `newline or ;' is not expected.

bash-5.1$ /bin/sh ~/.conan2/p/b/c-are5619eeba17ab1/b/build/Release/generators/conanbuildenv-release-ppc64.sh ~/.conan2/p/b/c-are5619eeba17ab1/b/build/Release/generators/conanbuildenv-release-ppc64.sh[4]: 0403-057 Syntax error at line 4 : `newline or ;' is not expected.

bash-5.1$ /bin/bash ~/.conan2/p/b/c-are5619eeba17ab1/b/build/Release/generators/conanbuildenv-release-ppc64.sh

bash-5.1$

I can't chsh to /bin/bash, but I found /usr/bin/ksh93 also seems to work. Using chsh to change the users default shell to /usr/bin/ksh93 however, I see now that /bin/sh is being used somehow as the default when conan launches it.

I'm not sure if I'm going to be able to persuade the admin that /bin/sh should be modified in any way

[memsharded]

Some quick feedback, not sure if it does make a difference regarding the error that you are seeing.

The scripts are not intended to be executed, they will not work if executed, they need to be sourced, like:

source  myfile.sh
. myfile.sh # the dot is the same as "source"

This is why they don't have shebang, because they cannot execute in a child shell, they need to execute in the current shell (sourced).

[GregDomjan]

Is my understanding correct it's actually cmake calling the script in this case via /bin/sh in some way, or is conan reporting a description of trying to perform cmake configure, which has a first step of applying the env config source conanbuildenv-release-ppc64.sh before actually calling cmake?

expat/2.7.1: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/.conan2/p/b/expat06579fa46dd6b/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/.conan2/p/b/expat06579fa46dd6b/b/src" /bin/sh: 0403-057 Syntax error at line 4 : `newline or ;' is not expected.

Just wondering if it falls into this problem https://stackoverflow.com/questions/21167014/how-to-set-shell-variable-in-makefiles-generated-by-cmake

[GregDomjan]

Wondering if always including the command end separator can get around the limitation in the shell because 'do' is on the next line often the 'in [list]' command end seperator ';' can be left out, but in this /bin/sh or ksh it seems helpful. is there an easy way to test that?

ie.

for v in ; do

[memsharded]

Is my understanding correct it's actually cmake calling the script in this case via /bin/sh in some way, or is conan reporting a description of trying to perform cmake configure, which has a first step of applying the env config source conanbuildenv-release-ppc64.sh before actually calling cmake?

Not really. If you use the conan .... -vvv to see extra verbose messages, you will see that the actual commands are more like:

$ source /path/to/conanbuild.sh && cmake ...

And it would be the sourcing of the conanbuild.sh

It seems the shell in AIX is not accepting iterating over an empty set. In linux both work:

for v in 1 2 3
do
  echo $v
done

produces

source myscript.sh
1
2
3

And

for v in
do
  echo $v
done

Produces just a blank line, nothing is output, but it doesn't fail or crash either.

[GregDomjan]

I tried earlier, just re validating, and even attempting to end the command with empty list fails

given 'test-sep.sh'

for v in ;
do
  echo $v
done

$ . test-sep.sh /bin/sh: 0403-057 Syntax error: `;' is not expected.

[GregDomjan]

I've not found a way to keep the for loop in, so asked on Stack Overflow if there is a way to do so. https://stackoverflow.com/questions/79587649/is-there-a-syntax-to-allow-empty-list-in-for-command-using-aix-7-2-bin-sh

As another alternate, if the environment list is empty, there is nothing to restore if there are no env variables to set. Could you skip the generation of the env vars code fragment of the .sh if the list is empty?

[memsharded]

/bin/sh: 0403-057 Syntax error: `;' is not expected.

But it seems that you are using sh. Can you please try with a bash shell?

[GregDomjan]

/bin/sh: 0403-057 Syntax error: `;' is not expected.

But it seems that you are using sh. Can you please try with a bash shell?

Perhaps I'm missing something. I tried changing the shell using either

• run /bin/bash (log above) to change the current terminal , or
• chsh to change the default shell for the user (I tried this with ksh98 as it should support the syntax and didn't require configuring chsh to support bash).

Neither of those option changed the log showing /bin/sh

I'll check again, back in the office next week

[GregDomjan]

Just poking around, for exec like actions, taking conan_run as example proc = subprocess.Popen(command, shell=shell, stdout=out, stderr=err, cwd=cwd)

According to python subprocess.Popen On Unix with shell=True, the shell defaults to /bin/sh. The executable argument specifies a replacement program to execute.

In the example, executable arg is not provided, so cannot configure other than /bin/sh

On linux /bin/sh is often now using bash or dash, on this AIX at least, the default is bourne or korn To change /bin/sh to be a link to any other shell has it's risks, and I need to get an admin to make the change.

[memsharded]

As an extra bit of information, I have tested the above:

for v in
do
  echo $v
done

in my Ubuntu, both with bash and the default sh (which points to dash), and it works well when empty list, so it would seem an issue of the specific (korn or bourne, which one?) shell in that AIX.

[GregDomjan]

Default /bin/sh is the korn shell (diff match with /bin/ksh)

bash-5.1$ ls -l /bin/sh -r-xr-xr-x 5 bin bin 316424 Mar 29 2019 /bin/ksh -r-xr-xr-x 2 bin bin 2384367 Mar 02 2019 /bin/ksh93 ... -r-xr-xr-x 5 bin bin 316424 Mar 29 2019 /bin/sh

bash-5.1$ diff /bin/sh /bin/ksh bash-5.1$

Running via '.' source into the shell

bash-5.1$ /bin/sh $ . test-val.sh 1 2 3 $ . test-fail.sh /bin/sh: 0403-057 Syntax error: newline or ;' is not expected. $ . test.sh /bin/sh: 0403-057 Syntax error: ;' is not expected. $

Since sh is ksh, output is basically the same

bash-5.1$ /bin/ksh $ . test-val.sh 1 2 3 $ . test-fail.sh /bin/ksh: 0403-057 Syntax error: newline or ;' is not expected. $ . test.sh /bin/ksh: 0403-057 Syntax error: ;' is not expected. $ exit

Trying with newer korn 93 shell, problem is avoided

bash-5.1$ /bin/ksh93 $ . test-val.sh 1 2 3 $ . test-fail.sh $ . test.sh $

We tried a temporary change /bin/sh to be a copy of /bin/ksh93 then I was able to successfully build the shared library, but not static archives

The admin would really prefer not to change the default /bin/sh of the system Is there any possibility to make it a configurable option ?

[GregDomjan]

other issue with linking - static build seems to be a problem

[100%] Linking C static library ../../lib/libcares ar: 0707-126 CMakeFiles/c-ares.dir/ares__addrinfo2hostent.c.o is not valid with the current object file mode. Use the -X option to specify the desired object mode. ar: 0707-126 CMakeFiles/c-ares.dir/ares__addrinfo_localhost.c.o is not valid with the current object file mode. Use the -X option to specify the desired object mode. ...

$ ar Usage: ar [-X{32|64|32_64|d64|any}] [-clsvCT] [-g|o] {-h|p|t|x} [--] Archive [File ...] ar [-X{32|64|32_64|d64|any}] [-clsvCT] [-g|o] {-m|r[u]} [{-a|b|i} {PositionName}] [--] Archive File ... ar [-X{32|64|32_64|d64|any}] [-clsvCT] [-g|o] {-d|q} [--] Archive File ... ar [-X{32|64|32_64|d64|any}] [-clvCT] {-g|o|s|w} [--] Archive

I haven't been able to identify a conan config to affect static linker flags Just going to look at see if I can apply CMAKE_STATIC_LINKER_FLAGS as a generic setting

[memsharded]

Trying with newer korn 93 shell, problem is avoided

That is good news, it sounds that the default korn shell is the older one, and the syntax support was lacking.

The admin would really prefer not to change the default /bin/sh of the system Is there any possibility to make it a configurable option ?

At the moment it is not planned, we haven't seen enough evidence of demand for this, and this still looks like a very niche use case. I can discuss this with the team and get some more opinitions, marking as "Look into"

other issue with linking - static build seems to be a problem

This sounds completely unrelated, would need a different ticket, might be a recipe (it looks cares) issue, like it is possible that some recipes in ConanCenter do not build for AIX, and it could be a thing from the Conan recipe or it might be the library itself lacking support to build in AIX.

[memsharded]

Following up on some staled tickets.

Checked with the team, this seems very niche use case to consider it for the default script generators. For generic use cases, Conan now implements the possibility of generating shell-agnostic dotenv files with the tools.env:dotenv=True configuration.

It is also solved with newer korn 93 shell, so I suggest consider this ticket solved with this recommended updated.