pynwb
pynwb copied to clipboard
Parallel io from the docs
1) Bug
If you are reporting a bug please provide the following:
Steps to Reproduce
Trying to run the code from the docs for parallel io: https://pynwb.readthedocs.io/en/latest/tutorials/general/advanced_hdf5_io.html?highlight=mpi#parallel-i-o-using-mpi
here is writenwb.py :
from mpi4py import MPI
import numpy as np
from dateutil import tz
from pynwb import NWBHDF5IO, NWBFile, TimeSeries
from datetime import datetime
from hdmf.data_utils import DataChunkIterator
start_time = datetime(2018, 4, 25, 2, 30, 3, tzinfo=tz.gettz('US/Pacific'))
fname = 'test_parallel_pynwb.nwb'
rank = MPI.COMM_WORLD.rank # The process ID (integer 0-3 for 4-process run)
# Create file on one rank. Here we only instantiate the dataset we want to
# write in parallel but we do not write any data
if rank == 0:
nwbfile = NWBFile('aa', 'aa', start_time)
data = DataChunkIterator(data=None, maxshape=(4,), dtype=np.dtype('int'))
nwbfile.add_acquisition(TimeSeries('ts_name', description='desc', data=data,
rate=100., unit='m'))
with NWBHDF5IO(fname, 'w') as io:
io.write(nwbfile)
# write to dataset in parallel
with NWBHDF5IO(fname, 'a', comm=MPI.COMM_WORLD) as io:
nwbfile = io.read()
print(rank)
nwbfile.acquisition['ts_name'].data[rank] = rank
and in read i replace the last 4 lines with:
read from dataset in parallel
with NWBHDF5IO(fname, 'r', comm=MPI.COMM_WORLD) as io:
print(io.read().acquisition['ts_name'].data[rank])
after running:
(.env) roybens@nid00227:~/tests> srun -n 3 python ./writenwb.py
0
/global/homes/r/roybens/.conda/envs/.env/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
0
/global/homes/r/roybens/.conda/envs/.env/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
0
/global/homes/r/roybens/.conda/envs/.env/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
(.env) roybens@nid00227:~/tests> vi writenwb.py
(.env) roybens@nid00227:~/tests> srun -n 3 python ./writenwb.py
0
/global/homes/r/roybens/.conda/envs/.env/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
/global/homes/r/roybens/.conda/envs/.env/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
0
/global/homes/r/roybens/.conda/envs/.env/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
0
ranks are always 0 when reading getting similiar errors
Provide a minimal code snippet here to reproduce this error.
Environment
Please describe your environment according to the following bullet points.
- Python Executable: Conda
- Python Version: Python 3.6
- Operating System: Linux
- **Pynwb Version:1.0.2 (latest from github)
2) Feature Request
If you are requesting a feature please provide the following:
Problem/Use Case
Briefly describe the needed feature as well as the reasoning behind it
Checklist
- [ ] Have you ensured the feature or change was not already reported ?
- [ ] Have you included a brief and descriptive title?
- [ ] Have you included a clear description of the problem you are trying to solve?
- [ ] Have you included a minimal code snippet that reproduces the issue you are encountering?
- [ ] Have you checked our Contributing document?