[BUG] scripts/activate.sh fails on pigweed

[HendrikTBB]

Reproduction steps

Follow the instructions in BUILDING.md.

The 'activate' script gets to the 'setting up python environment' and then dies with a large spew of error messages:

Setting up CIPD package manager...done (43.1s) Setting up Project actions........skipped (0.1s) Setting up Python environment.....[-] ERROR at /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_build/facade.gni:187:7: Assertion failed. assert(_dep_is_in_link_dependencies, ^----- /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_log:impl must be listed in the pw_build_LINK_DEPS build arg when the /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_log:pw_log facade is in use. Please update your toolchain configuration. See /home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_log/BUILD.gn:47:1: whence it was called. pw_facade("pw_log") { ^-------------------- See //src/lib/support/BUILD.gn:351:5: which caused the file to be included. "$dir_pw_log:impl", ^----------------- ['gn', 'gen', '/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/.environment/gn_out', '--args=chip_crypto="boringssl" dir_pigweed="/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo"']

Traceback (most recent call last): File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/virtualenv_setup/install.py", line 402, in install_packages subprocess.check_call( File "/usr/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['gn', 'gen', '/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/.environment/gn_out', '--args=chip_crypto="boringssl" dir_pigweed="/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo"']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 995, in sys.exit(main()) ^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 987, in main return EnvSetup(**vars(parse())).setup() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 577, in setup result = step(spin) ^^^^^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/env_setup.py", line 766, in virtualenv if not virtualenv_setup.install( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/virtualenv_setup/install.py", line 443, in install install_packages(gn_target) File "/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo/pw_env_setup/py/pw_env_setup/virtualenv_setup/install.py", line 410, in install_packages raise subprocess.CalledProcessError( subprocess.CalledProcessError: Command '['gn', 'gen', '/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/.environment/gn_out', '--args=chip_crypto="boringssl" dir_pigweed="/home/hendrik/Source.External/OpenThread/MatterMini/connectedhomeip/third_party/pigweed/repo"']' returned non-zero exit status 1. Installing pip requirements for all... error: externally-managed-environment

× This environment is externally managed ?-> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.

Bug prevalence

Every time

GitHub hash of the SDK that was being used

9080cd3b

Platform

other

Platform Version(s)

Debian GNU/Linux trixie/sid

Anything else?

No response

[andy31415]

@HendrikTBB is this a clean checkout of the repository or are there modifications?

There are two errors: the link-deps (which I have no idea what is going on) and a complaint about virtualenv. We can try to fix at least the virtualenv - my understanding is that newer python does not like pip install on the system python, so if you do not have virtualenv, you cannot pip install virtualenv either.

How about these steps first:

• do you have virtualenv (which virtualenv)?
• assuming no based on the output, install it (maybe apt-get install virtualenv or apt-get install python3-virtualenv) and try again

If we manage to get rid of the virtualenv error, we can figure out things further.

[HendrikTBB]

I don't have a 'virtualenv' but I do have the 'python3-venv' Debian package installed. Also, the 'python3 -m venv' mechanism worked just fine for me installing a bunch of python packages required for the nRF52840 Thread firmware. But I have no idea how to convince the automated installation script to use that mechanism instead of whatever command it is actually running. I can't even tell what python package it is trying to install if it isn't the pigweed thing. So I can't just install it myself manually.

[HendrikTBB]

Okay, I've got the venv thing working. Not sure what happened before, maybe I had a typo in the path or something. But now adding the venv bin directory to the path is causing activate.sh to pick up my local pip and install the python stuff in there.

I still get the complaints about gn and pigweed not working, though.

And I can't move on to the next step of 'Build for the host' as that also requires running 'gn' which does not appear to exist'.

[HendrikTBB]

It seems likely that the missing 'gn' is part of the Debian generate-ninja package. I installed that.

It didn't help with the gn pigweed error of the activate script but it at least lets me run the 'gn gen out/host' command.

However, the ninja step then fails complaining about a missing 'zap-cli' and tells me to install the relevant relase from github/project-chip/zap/releases. Only problem is I already did that and all it provides is 'zap'. There is no 'zap-cli' program in the zap-linux-x64.deb package.

[HendrikTBB]

I created a symlink from zap to zap-cli and lets me run the build.

However, the checks (ninja -C out/host check) failed immediately with a lot of spew about pigweed ending with "No such file or directory: PosixPath('protoc')". I'm guessing that is related to pigweed not installing in the activate.sh script.

[andy31415]

Seems the same like #31851

Could you try:

git submodule update -f --init --recursive

for the submodule update and then try the bootstrap again?

[andy31415]

also see the note in the other bug:

check to see if there is a difference between pwd and realpath . in your environment. I was able to get past this issue by making those identical:
% cd $(realpath .)

[NinjaLikesCheez]

I ran into similar issue today but on macOS. Installing protoc/protobuf resolved the build error. The BUILDING.md doesn't mention this requirement - happy to open a PR to add it (but I don't know this project and whether this is a doc bug or something else).

I raised an issue in a downstream repo in case the logs are at all useful: https://github.com/espressif/esp-matter/issues/1122

[dusikasss]

Сегодня я столкнулся с похожей проблемой, но на macOS. Установка protoc/protobuf устранила ошибку сборки. В BUILDING.md не упоминается это требование — я с радостью открою PR, чтобы добавить его (но я не знаком с этим проектом и не знаю, является ли это ошибкой в документации или чем-то другим).

Я поднял вопрос в репозитории ниже по течению на случай, если журналы окажутся полезными: espressif/esp-matter#1122

I have a similar problem to yours. Could you share the solution in more detail. In my case, installing protobuf did not solve the problem

[andy31415]

My expectation would be that bootstrap should install all required pre-requisites as a virtualenv and should not require system level package changing.

[andy31415]

Activate will generally try to re-use an existing environment so if python or other modules change that may cause issues.

Please try to clean the environment (remove .environment or $PW_ENVIRONMENT_ROOT) and re-run source ./scripts/bootstrap.sh on a clean terminal (that was not activated/bootstraped).

If you get errors, please share the logs (as an attachment please, it is hard to read them as comments inline)

[andy31415]

Once you tried a clean bootstrap, please remove the waiting for feedback tag.

I would also be interested in:

• What O/S version you are using
• What python version you are using

[andy31415]

I would also mention we are supporting ubuntu generally ... I think this should be debuggable for any O/S however it is also hard to support all possible linux distributions.

[cnyangzhenyu]

I meet same problem when building examples/lighting-app/linux: Setting up Python environment.....[|] ERROR at //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_build/facade.gni:187:7: Assertion failed. assert(_dep_is_in_link_dependencies, ^----- //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_log:impl must be listed in the pw_build_LINK_DEPS build arg when the //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_log:pw_log facade is in use. Please update your toolchain configuration. See //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_log/BUILD.gn:47:1: whence it was called. pw_facade("pw_log") { ^--------------------

OS: Ubuntu 22.04 in Docker Python Version: Python 3.10.12

[cnyangzhenyu]

change third_party/pigweed/repo/pw_build/facade.gni, print pw_build_LINK_DEPS in assert:

ERROR at //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_build/facade.gni:187:7: Assertion failed. assert(_dep_is_in_link_dependencies, ^----- //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_log:impl must be listed in the pw_build_LINK_DEPS ["//third_party/pigweed/repo/pw_assert:impl", "//third_party/pigweed/repo/pw_log:impl"] build arg when the //examples/lighting-app/linux/third_party/connectedhomeip/third_party/pigweed/repo/pw_log:pw_log facade is in use. Please update your toolchain configuration.

pw_log:impl is found in pw_build_LINK_DEPS with different path

[andy31415]

I think we have seen odd errors when the checkout directory contained symlinks in the path (this happened on mac where the home directory was a symlink). Could you verify if that is the case here?

Things to check:

• the path to the checkout root to not contain symlinks (or change to the path without symlinks)
• make sure a clean bootstrap is done

[jmeg-sfy]

@andy31415 i got the same issue.