[BUG]: NumPy ImportError with Embedded Interpreter in Adobe After Effects Plugin Using pybind11

[Trentonom0r3]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.11.1

Problem description

Problem Description

I'm experiencing an issue with importing NumPy when using an embedded Python interpreter in an Adobe After Effects plugin built with pybind11. The standalone Python interpreter can import NumPy without issues, but the embedded interpreter throws an ImportError related to NumPy's C-extensions.

Expected Behavior

The embedded Python interpreter should import NumPy without any ImportError, as it does in the standalone Python interpreter.

Actual Behavior

When attempting to import NumPy in the embedded interpreter, the following error occurs:

ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.

Here is the detailed error output captured:

Captured stdout:

Captured stderr:
Traceback (most recent call last):
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\__init__.py", line 24, in <module>
    from . import multiarray
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\__init__.py", line 144, in <module>
    from numpy.__config__ import show as show_config
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from "C:\Program Files\Adobe\Adobe After Effects 2023\Support Files\AfterFX.exe"
  * The NumPy version is: "1.26.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 27, in <module>
  File "C:\Users\tjerf\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\__init__.py", line 149, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.
        ```
        
#### Already Checked:
- Sys.path,
- different numpy versions
- different python versions
- trying in a shell, which works fine. 

### Reproducible example code

```text
Repo with Plugin can be found at https://github.com/Trentonom0r3/AE-Pybind-Error.


#include <pybind11/pybind11.h>
#include <pybind11/embed.h>
#include <iostream>

namespace py = pybind11;

int main() {
    try {
        py::scoped_interpreter guard{};  // Start the interpreter

        // Add the path to site-packages and capture environment information
        py::exec(R"(
        import sys
        import os
        from io import StringIO
        import sysconfig

        # Dynamically get site-packages path
        site_packages_path = sysconfig.get_paths()['purelib']

        # Add the site-packages directory to sys.path
        if site_packages_path not in sys.path:
            sys.path.insert(0, site_packages_path)

        # Collect environment information
        environment_info = {
            'sys.executable': sys.executable,
            'sys.version': sys.version,
            'sys.path': sys.path,
            'os.environ': dict(os.environ),
        }

        # Redirect stdout and stderr
        old_stdout = sys.stdout
        old_stderr = sys.stderr
        sys.stdout = StringIO()
        sys.stderr = StringIO()

        try:
            import numpy as np
            print("NumPy version:", np.__version__)
        except Exception as e:
            import traceback
            traceback.print_exc()

        # Get the captured output
        stdout_output = sys.stdout.getvalue()
        stderr_output = sys.stderr.getvalue()

        # Restore stdout and stderr
        sys.stdout = old_stdout
        sys.stderr = old_stderr
        )");

        // Retrieve the captured output from the Python execution
        py::object main_module = py::module::import("__main__");
        py::object stdout_output = main_module.attr("stdout_output");
        py::object stderr_output = main_module.attr("stderr_output");
        py::object environment_info = main_module.attr("environment_info");

        // Convert the captured output and environment information to strings
        std::string stdout_str = stdout_output.cast<std::string>();
        std::string stderr_str = stderr_output.cast<std::string>();
        auto env_info_dict = environment_info.cast<py::dict>();

        std::string env_info_str;
        for (auto item : env_info_dict) {
            env_info_str += item.first.cast<std::string>() + ": ";
            if (py::isinstance<py::str>(item.second)) {
                env_info_str += item.second.cast<std::string>() + "\n";
            } else if (py::isinstance<py::list>(item.second)) {
                auto list = item.second.cast<py::list>();
                env_info_str += "[";
                for (auto elem : list) {
                    env_info_str += elem.cast<std::string>() + ", ";
                }
                env_info_str += "]\n";
            } else if (py::isinstance<py::dict>(item.second)) {
                auto dict = item.second.cast<py::dict>();
                env_info_str += "{";
                for (auto elem : dict) {
                    env_info_str += elem.first.cast<std::string>() + ": " + elem.second.cast<std::string>() + ", ";
                }
                env_info_str += "}\n";
            }
        }

        // Output the information as a single string for debugging
        std::string debug_info = "Captured stdout:\n" + stdout_str + "\nCaptured stderr:\n" + stderr_str + "\nPython Environment Information:\n" + env_info_str;
        std::cout << debug_info << std::endl;

    } catch (const py::error_already_set& e) {
        std::string error_str = "Error in Python execution: " + std::string(e.what());
        std::cout << error_str << std::endl;
    } catch (const std::exception& e) {
        std::string error_str = "Error in Python execution: " + std::string(e.what());
        std::cout << error_str << std::endl;
    } catch (...) {
        std::string error_str = "Error in Python execution: Unknown exception";
        std::cout << error_str << std::endl;
    }

    return 0;
}

Is this a regression? Put the last known working version here if it is.

Not a regression

[Trentonom0r3]

Works with release versions, but not debug...

[valotec-cht]

@Trentonom0r3 Have you found a fix to make debug build work?