macOS: jinja2 installed yet " ModuleNotFoundError: No module named 'jinja2' "

[jcelerier]

Summary

I have jinja2 installed with pip install jinja2 on macOS. Yet the build fails.

System information

Mbed TLS version (number or commit id): b3c3090

Operating system and version: macOS as provided on github action Configuration (if not default, please attach mbedtls_config.h): default + MBEDTLS_SSL_DTLS_SRTP

Build command

  pip install jinja2
  git clone https://github.com/Mbed-TLS/mbedtls || true
  (
    cd mbedtls
    python -m pip install -r scripts/basic.requirements.txt

    echo "#define MBEDTLS_SSL_DTLS_SRTP" >> include/mbedtls/mbedtls_config.h
    
    mkdir build
    cd build
    cmake .. \
      -DCMAKE_INSTALL_PREFIX=$BUILD_DIR \
      -DCMAKE_PREFIX_PATH=$BUILD_DIR \
      -DCMAKE_C_FLAGS="$OPT_FLAGS" \
      -DCMAKE_CXX_FLAGS="$OPT_FLAGS" \
      -DCMAKE_BUILD_TYPE=Release \
      -DENABLE_PROGRAMS=0 \
      -DBUILD_SHARED_LIBS=0 \
      -DMBEDTLS_FATAL_WARNINGS=0 \
      -DENABLE_TESTING=0

      make install
  )

Build log

# Installation of jinja2 and the other things beforehand
Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 6.1 MB/s eta 0:00:00
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_x86_64.whl (13 kB)
Installing collected packages: MarkupSafe, jinja2
Successfully installed MarkupSafe-2.1.1 jinja2-3.1.2
Cloning into 'mbedtls'...
Ignoring Jinja2: markers 'python_version < "3.10"' don't match your environment
Collecting markupsafe<2.1
  Downloading MarkupSafe-2.0.1-cp310-cp310-macosx_10_9_x86_64.whl (14 kB)
Requirement already satisfied: Jinja2>=2.10.3 in /Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages (from -r scripts/driver.requirements.txt (line 15)) (3.1.2)
Collecting types-Jinja2
  Downloading types_Jinja2-2.11.9-py3-none-any.whl (18 kB)
Collecting types-MarkupSafe
  Downloading types_MarkupSafe-1.1.10-py3-none-any.whl (4.0 kB)
Installing collected packages: types-MarkupSafe, types-Jinja2, markupsafe
  Attempting uninstall: markupsafe
    Found existing installation: MarkupSafe 2.1.1
    Uninstalling MarkupSafe-2.1.1:
      Successfully uninstalled MarkupSafe-2.1.1
Successfully installed markupsafe-2.0.1 types-Jinja2-2.11.9 types-MarkupSafe-1.1.10

# Compilation of mbedtls
-- The C compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python3: /usr/local/Frameworks/Python.framework/Versions/3.10/bin/python3.10 (found version "3.10.7") found components: Interpreter 
-- Found Perl: /usr/local/bin/perl (found version "5.36.0") 
-- Configuring done
-- Generating done
CMake Warning:
-- Build files have been written to: /Users/runner/work/witchbridge-audio/witchbridge-audio/mbedtls/build
  Manually-specified variables were not used by the project:

    CMAKE_CXX_FLAGS


[  1%] Generating version_features.c
[  1%] Generating error.c
[  2%] Generating psa_crypto_driver_wrappers.c
[  3%] Generating ssl_debug_helpers_generated.c

Traceback (most recent call last):
  File "/Users/runner/work/witchbridge-audio/witchbridge-audio/mbedtls/library/../scripts/generate_driver_wrappers.py", line 26, in <module>
    import jinja2
ModuleNotFoundError: No module named 'jinja2'

make[2]: *** [library/psa_crypto_driver_wrappers.c] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [library/CMakeFiles/mbedcrypto.dir/all] Error 2
make: *** [all] Error 2

[gilles-peskine-arm]

To me the most obvious explanation would be that jinja2 is installed for one version of python but the script is running with a different version. However I don't see any evidence of that in the logs (thanks for posting full logs by the way): both pip and cmake see python 3.10.7. Just to make sure, can you please check which python version(s) are involved here? Maybe if there's an OS native installation of Python 3.10.7 and another one under /usr/local?

type python python3
echo "$PYTHONPATH"
python3 -m pip show jinja2
python3 ./scripts/generate_driver_wrappers.py

[jcelerier]

hm yes that was the culprit, though I don't really understand why this fails which is scary :)

To anyone having the same issue, I could fix it with:

mv /usr/local/bin/python /usr/local/bin/python2
ln -s /usr/local/bin/python3 /usr/local/bin/python
python3 -m pip install -r scripts/basic.requirements.txt || true

[gilles-peskine-arm]

That's weird: our code definitely handles the case when python is Python 2. It's the case on some of our CI systems, and on my and many other team members' main work machine (Ubuntu or macOS).

CMake reports “Found Python3” and doesn't say anything about Python 2. I'm not completely sure this is the Python that it uses to invoke generate_driver_wrappers.py: that uses MBEDTLS_PYTHON_EXECUTABLE which is set in different ways depending on the CMake version. Maybe a CMake expert can chime in and say how to debug this further.

[paul-elliott-arm]

The CMake code is looking specifically for Python3, and will fail if all it can find is Python2. However, as the REQUIRED keyword is not used, the build will not fail and all that will happen is that the variable MBEDTLS_PYTHON_EXECUTABLE will not get set (I believe this was in the interest of not having to require people to have Python installed at all).

Hoever, if the generation of generated files is enabled, this will then mean that the execution of scripts/generate_ssl_debug_helpers.py will depend on its hashbang as it is now being executed directly. This hashbang is #!/usr/bin/env python3 - all good here (so long as python3 is installed), but the initial pip install jinja2 will have used python2, as that is what /usr/local/bin/python was pointing at, hence Jinja not being installed when we try and use Python3.

So, an additional problem is that this is going to fail if Python2 is installed and not Python3, but fixing this without breaking the build for those not wanting to generate files is going to be tricky. I note at this point that Python2 is being deprecated from most rolling distros as we speak, but obviously this is going to remain a problem on older distros.

[paul-elliott-arm]

I need to have a think about the best way to solve this.

[gilles-peskine-arm]

the initial pip install jinja2 will have used python2

That was my first suspicion when reading the issue description, but the logs show that jinja2 was installed for the same Python version number that CMake detects as Python3:

Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
…
Requirement already satisfied: Jinja2>=2.10.3 in /Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages (from -r scripts/driver.requirements.txt (line 15)) (3.1.2)
…
-- Found Python3: /usr/local/Frameworks/Python.framework/Versions/3.10/bin/python3.10 (found version "3.10.7") found components: Interpreter 

[tom-cosgrove-arm]

@jcelerier Does it work if you install jinja2 with pip3 install jinja2 rather than just pip install jinja2?

[paul-elliott-arm]

So, we get to my second suspicion. Multiple full installations of Python on the same machine are unfortunately not uncommon, especially on MacOS, unfortunately. If you look at the two paths above, they are not the same.

[tom-cosgrove-arm]

Yes, while it's possible that parts are installed in /Users/runner/hostedtoolcache/Python and parts in /usr/local/Frameworks/Python.framework/Versions it seems quite possible that they are two different instances

[jcelerier]

Will try asap next week

[tom-cosgrove-arm]

@jcelerier Did you get anywhere with this?

[tom-cosgrove-arm]

@jcelerier Did you get anywhere with this?

[jcelerier]

tried on gh actions as this is where my issue was, seems fixed! https://github.com/jcelerier/test-repro-mbedtls/actions/runs/8963471799/job/24613806464