[Bug]: Instructions for ROS3 driver installation might not work on Linux

[luiztauffer]

What happened?

Following the instructions here for installing h5py with ros3 driver do not seem to work on linux machines in some circumstances. I couldn't figure out what exactly is causing conda to install h5py without ros3 in that case, I cleaned all cache from both conda and pip, tried always in clean new environments and used different combinations of installation steps, but none worked. We tried in different machines, from different users, and the problem was reproduced.

I figured out a way that works in those cases though, and that is using 'h5py>3.2=mpi*', for example:

conda create -n env_ros3 -c conda-forge python=3.8 'h5py>3.2=mpi*' ipykernel 
conda activate env_ros3
pip install pynwb dandi

So maybe we should include this option in the documentation instructions?

Steps to Reproduce

Following the instructions here

Traceback

No response

Operating System

Linux

Python Executable

Python

Python Version

3.8

Package Versions

environment_for_issue.txt

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you checked the Contributing document?
• [X] Have you ensured this bug was not already reported?

[oruebel]

@luiztauffer sounds reasonable. I would make this a note box (possibly at the end of the S3 tutorials, depending on length) to describe the issue, how to check for it, and the instructions how to fix it. Feel free to suggest the changes in a PR if you have time.

[rly]

@luiztauffer Could you please post the exact steps that you used to create the environment where the h5py ROS3 driver installation failed? I can try to debug a bit.

If I test your steps as the following:

conda create -n env_ros3 -c conda-forge python=3.8 ipykernel 
conda activate env_ros3
pip install pynwb dandi
pip uninstall h5py
conda install -c conda-forge "h5py>=3.2"

OR

conda create -n env_ros3 -c conda-forge python=3.8 "h5py>=3.2" ipykernel 
conda activate env_ros3
pip install pynwb dandi

on my Ubuntu on Windows, then the streaming code works. The CircleCI check for this test also runs fine on normal Ubuntu.

[bendichter]

yes, I think it's the OSX and Windows distros that are a problem.

This line from dandi hub

conda install -c conda-forge "h5py>=3.4=mpi*"

works for me on OSX.

[CodyCBakerPhD]

yes, I think it's the OSX and Windows distros that are a problem.

The NWB Inspector CI tests the original non-mpi* version on all platforms: https://github.com/NeurodataWithoutBorders/nwbinspector/blob/dev/.github/workflows/testing.yml#L34-L35 (latest images of each)

and I've used that method with success on all personal devices over the 3 platforms.

I suspect it is something else about @luiztauffer's linux setup that causes the issue - perhaps it's an older ubuntu image, or perhaps an alternate linux dist like fedora? Or something about the configuration of the machine itself that the conda-forge source has not been compiled for.

In the end, this issue was a record to maybe add a note to the documentation instructions for this feature mentioning that if some mysterious trouble like this occurs, one can try the h5py>=3.4=mpi* alternatively and maybe that will work instead.

[luiztauffer]

yeah, I presume it is something more specific, but I couldn't figure out what that might be (I cleaned caches for both conda and pip, tried different installations orders, etc...). But not too specific, since we reproduced the problem with @h-mayorquin both are Ubuntu machines