BUG: previously working boost build fails with `error: "3.10" is not a known value of feature <python>`

[h-vetinari]

Make sure you completed the following tasks

• [x] I searched the discussions
• [x] I searched the closed and open issues
• [x] I read the contribution guidelines

Environment and version details

• Operating System+version: azure-pipeline's ubuntu-latest & macOS-12
• Compiler+version: GCC 12 & clang 16
• Shell: Bash
• B2 Version: 5.2.1 (vendored by boost 1.86)

# Place output of "b2 --debug-configuration" here.
built by boost's bootstrap.sh

Brief problem description

A previously working build in conda-forge for boost broke with version 1.86, which updates b2 from 5.1.0 (in 1.85) to 5.2.1. The build looks roughly like this (after doing bootstrap.sh):

./b2 -q \
    --prefix=./temp_prefix \
    variant=release \
    address-model="${ADDRESS_MODEL}" \
    architecture="${ARCHITECTURE}" \
    binary-format="${BINARY_FORMAT}" \
    abi="${ABI}" \
    debug-symbols=off \
    threading=multi \
    runtime-link=shared \
    link=shared \
    toolset=${TOOLSET} \
    python="3.10" \
    include="${INCLUDE_PATH}" \
    cxxflags="${CXXFLAGS}" \
    linkflags="${LINKFLAGS}" \
    cxxstd=20 \
    --layout=system \
    -j"${CPU_COUNT}" \
    install

This worked fine for boost 1.82-1.85.

Looking at the release notes for b2, python is only mentioned once:

Allow some tools to be default initialized with using multiple times. The tools are asciidoctor, fop, gettext, pkg-config, python, sass, and saxonhe.

This issue happens on all unix builds (5 different targets). Interestingly, windows is not affected. I surmise this is due to having to work-around some other issue already, which happens to avoid the problem:

:: Write python configuration, see https://github.com/boostorg/build/issues/194
@echo using python > user-config.jam
@echo : 3.10 >> user-config.jam
@echo : %PYTHON:\=\\% >> user-config.jam
@echo : %PREFIX:\=\\%\\include >> user-config.jam
@echo : %PREFIX:\=\\%\\libs >> user-config.jam
@echo ; >> user-config.jam
xcopy /Y user-config.jam %USERPROFILE%

I suspect this is the case, because when searching around for this issue, I stumbled upon this old mail, that outlines a very similar situation happening previously. To test that hypothesis, I added:

# for context, this is executed while being in $SRC_DIR
echo "using python" > user-config.jam
echo ": 3.10" >> user-config.jam
echo ": $PYTHON" >> user-config.jam
echo ": $PREFIX/include/python3.10" >> user-config.jam
echo ": $PREFIX/lib" >> user-config.jam
echo ";" >> user-config.jam
# see https://www.boost.org/build/doc/html/bbv2/overview/configuration.html
export BOOST_BUILD_PATH=$SRC_DIR

to our unix builds and then the build managed to proceed. So indeed it seems related to https://github.com/bfgroup/b2/commit/fb743b7dbea676dabfd9b679a4da588d63a93876

Steps to reproduce the issue

Build boost according to conda-forge's recipe.

Actual behavior summary

For boost 1.86, I get:

+ ./b2 -q --prefix=./temp_prefix variant=release address-model=64 architecture=x86 binary-format=elf abi=sysv debug-symbols=off threading=multi runtime-link=shared link=shared toolset=gcc python=3.10 include=$PREFIX/include 'cxxflags=-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/boost-split-1.86.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fPIC' 'linkflags= -L$PREFIX/lib' cxxstd=20 --layout=system -j2 install
$SRC_DIR/tools/build/src/build/feature.jam:491: in feature.validate-value-string from module feature
error: "3.10" is not a known value of feature <python>
error: legal values:

If a project of yours is blocked due to this bug, please, mention it explicitly.

Cannot currently release boost 1.86 in conda-forge (without the config work around I found), so pretty big blocker, especially if other people bootstrapping boost are also affected.

Expected behavior summary

boost build succeeds as before

[grisumbras]

Do you have anything in user-config.jam beyond what you've added for this workaround?

[h-vetinari]

Nothing, unless it happens to come from boost itself. It's a clean image resp. build environment otherwise - there should be no such files lurking anywhere. I double-checked that none of our used artefacts contain a user-config.jam, and the one used in the build is created from scratch as well (single >).

[grisumbras]

Having a using python : X.Y line in your user-config.jam or project-config.jam is actually the proper way to enable python=X.Y flag when running b2. Which is why I was surprised your script used to work without that line. But then I recalled that bootstrap.sh does put such a line inside project-config.jam it generates. The version (unless a specific version is requested) is guessed by analyzing certain variables in Python's sys module. This quite obviously depends on the version of the system's default Python installation. Can you please check the contents of project-config.jam generated by boostrap.sh for you?