parallel netcdf4

[apatlpo]

It'd be great to write netcdf output files. This is the standard for many users in the geophysical community. Parallel file writing is implemented in netCDF4. I failed to install it via conda however. Could it be possible to have it in the spectralDNS channel as it is the case for hdf5-parallel?

[mikaem]

Hi I have not used netcdf4, but I would not object to someone adding support for it. I see that netcdf4 is available on conda-forge, but this is only serial version of the library. And there's a libnetcdf. Are both required? Should probably only be an issue of using these recipes with hdf5-parallel instead of the serial versions used on conda-forge I think.

[mikaem]

It built quite easily on Ubuntu, so I put two versions, libnetcdf-parallel and netcdf4-parallel on Anaconda cloud. Feel free to check them out. The last one was built with python=3.6. Let me know if you need anything else.

[apatlpo]

That's great, thanks, but today I'm on osx unfortunately. Would you be able at some point to add an osx version of these libraries?

[mikaem]

I'll see what I can do.

[mikaem]

@apatlpo I have now uploaded netcdf built for osx to the spectralDNS channel.

[apatlpo]

Thanks it works like a charm for me ! Starting from the hdf5 file writer, I wrote a netcdf file writer. It works well for a simple configuration. Let me know if you want me to push a pull request to shenfun of this file.

[mikaem]

Great:-) Go ahead with the pull request. Perhaps you can even improve it? I have spent way too little time on the IO-interface.

[apatlpo]

ok, will proceed with the PR ( beware this will be my first :) ).

As far as improvements, I don't have time to do it right now but here is a to do list for the future:

• [x] Add testing
• [ ] Add slicing capability
• [ ] Optional names for coordinates
• [ ] Optional number of time steps per file

[apatlpo]

I just reinstalled shenfun via:

conda build -c conda-forge -c spectralDNS conf/conda
conda install -c conda-forge -c spectralDNS shenfun --use-local

This does not install netCDF4 which raises the following warning at imports:

>>> import shenfun
/Users/aponte/.miniconda3/envs/shenfun/lib/python3.6/site-packages/shenfun/utilities/nc_writer.py:11: UserWarning: netcdf not installed
  warnings.warn('netcdf not installed')

Is this intended?

As a side note, it may be nice to suggest the creation of a shenfun conda environment prior to running these last install lines in the doc.

[mikaem]

Not intended at all. I see that netcdf is not added to meta.yaml. Do you remember what the requirements were?

And you are correct about the last note. I'll update the readme.

[mikaem]

I think it should be netcdf4-parallel and libnetcdf-parallel, right?

[mikaem]

And there should be some tests on the netcdf io...

[mikaem]

Latest package from Anaconda cloud should now also include netcdf.

[apatlpo]

Great thanks.

Regarding tests, should I just propose additions to 'test/test_io.py' ?

[mikaem]

Yes, that would be great.

I have written a little bit about post-processing in the documentation. Have not written much about netcdf and one reason for this is that it does not have the option to store slices. Are you not interested in this feature?

[apatlpo]

just letting you know I am not dropping this.

I am fighting a ModuleNotFoundError: No module named 'mpi4py_fft at the moment.

[mikaem]

That means you haven't installed the mpi4py-fft package. Should be automatically installed if you used conda...

[apatlpo]

Yes, I am using conda. I have restarted from scratch the environment. I don't understand what is going on.

[apatlpo]

Where is the mpi4py-fft dependency explicited in the library?

[apatlpo]

Shouldn't it be added to conf/conda/meta.yaml?

[mikaem]

It's in the components-requirements file

[mikaem]

No. It is only available as source for now. There's no package available anywhere so meta.yaml won't work.

[apatlpo]

nevermind, I got confused because it is not showing up with conda list. shenfun is up and running. sorry for the noise

[mikaem]

puh:-)

[mikaem]

But you should be able to install it from the spectralDNS channel, right? I forgot it was there as well.

[apatlpo]

I followed:

conda build -c conda-forge -c spectralDNS conf/conda
conda create --name shenfun -c conda-forge -c spectralDNS shenfun --use-local
conda activate shenfun

I've added netcdf tests in test_io.py but am at a loss as to what is the best way to check if test are passing. What would you do ?

[apatlpo]

A How to contribute section on the doc could be useful ...

[mikaem]

I don't follow. You mean what is the best way to call the test itself? like py.test -v test_io.py?

[apatlpo]

yep, sorry I have never done this before ... (note for myself, this was required: pip install -U pytest)

Tests seem to pass. I'll send a PR soon.

Regarding slices, I am indeed interested in that and do not see any particular reasons for why one could not slice with netcdf.

[mikaem]

Ah, now I see. The py.test you were running before was not in the current environment and therefore no mpi4py_fft! I have made that same error a few times:-)