Enrico Minack

On which port did you start the ssh service on `second`? Can you ssh into `second` from `second` itself?

As above: `/usr/sbin/sshd -p 6006`, but this will start SSH on port 6006, not the default 22.

Are there any errors before this line: Could NOT find Mxnet (missing: Mxnet_LIBRARIES) (Required is at least version "1.4.0") Can you please run the following in the same environment where...

Right, you have to run that `python` command inside `horovod/cmake`. There is the `build_utils.py` that is imported by that `python` command. What is the output then?

And running `HOROVOD_WITH_MXNET=1 pip install --no-cache-dir horovod[MXNET]` in the very same environment gives you CMake Error at /tmp/horovod-cmake-tmpaicqjfxp/cmake/data/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message): Could NOT find Mxnet (missing: Mxnet_LIBRARIES) (Required is at least version...

How do you install mxnet from sources?

That looks alright, and when you execute the above `python` command right after that (without ever pip install mxnet), what is the output? I got confused with the multiple attempts...

Sorry, with "the `python` command", I meant python -c "import os; import mxnet as mx; import build_utils; print(mx.__version__); print(mx.libinfo.find_include_path()); print(' '.join(mx.libinfo.find_lib_path())); print(build_utils.is_mx_mkldnn()); print(build_utils.is_mx_onednn()); print(build_utils.is_mx_cuda())"

> i never pip install mxnet yes, but your [above comment](https://github.com/horovod/horovod/issues/3455#issuecomment-1059993696) show that `pip install horovod[mxnet]` made pip install mxnet: Successfully installed mxnet-1.9.0 So that messed up diagnostics I am...

I think the problem is that MXNet cannot find the include path: python3 -c "import mxnet as mx; print(mx.libinfo.find_include_path());" This is a problem with the MXNet installation, not Horovod. So...