Runtime error in WepyHDF5

[SamikBose]

Hi @salotz and @alexrd,

We (I and Ceren) are having this weird issue with WepyHDF5, where in the middle of a wepy simulation (say after 100 or 200 cycles), all of a sudden we are getting this error:

Traceback (most recent call last):
  File "we_rebinding_rst4.py", line 153, in <module>
    steps_list)
  File "<boltons.funcutils.FunctionBuilder-7>", line 2, in run_simulation
  File "/home/bosesami/anaconda3/envs/wepy_new/lib/python3.7/site-packages/eliot/_action.py", line 943, in logging_wrapper
    result = wrapped_function(*args, **kwargs)
  File "/home/bosesami/software/wepy/src/wepy/sim_manager.py", line 743, in run_simulation
    self.init(num_workers=num_workers)
  File "<boltons.funcutils.FunctionBuilder-6>", line 2, in init
  File "/home/bosesami/anaconda3/envs/wepy_new/lib/python3.7/site-packages/eliot/_action.py", line 943, in logging_wrapper
    result = wrapped_function(*args, **kwargs)
  File "/home/bosesami/software/wepy/src/wepy/sim_manager.py", line 599, in init
    continue_run=continue_run)
  File "/home/bosesami/software/wepy/src/wepy/reporter/hdf5.py", line 363, in init
    alt_reps=self.alt_reps_idxs)
  File "/home/bosesami/software/wepy/src/wepy/hdf5.py", line 846, in __init__
    libver=H5PY_LIBVER, swmr=self._swmr_mode) as h5:
  File "/home/bosesami/anaconda3/envs/wepy_new/lib/python3.7/site-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/home/bosesami/anaconda3/envs/wepy_new/lib/python3.7/site-packages/h5py/_hl/files.py", line 177, in make_fid
    fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 108, in h5py.h5f.create
OSError: Unable to create file (unable to open file: name = '/dickson/s1/bosesami/comp_unbinding/lig19/vac_cavity_rebinding/sim_data/output_10000_1000_4_rst5/wepy.results.h5', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)
Exception ignored in: <function WepyHDF5.__del__ at 0x7fced6ba8b90>
Traceback (most recent call last):
  File "/home/bosesami/software/wepy/src/wepy/hdf5.py", line 913, in __del__
    self.close()
  File "/home/bosesami/software/wepy/src/wepy/hdf5.py", line 2436, in close
    if not self.closed:
AttributeError: 'WepyHDF5' object has no attribute 'closed'

Please note this is not a case of output h5 file already existing, as the error is coming up in the middle (like after 100 or 200 cycles) of the simulation (when the file is already created). I don't understand why it is trying to create the same file all of a sudden after 100 cycles. Does it seem like there is an attempt to restart the simulation (hardware issues). This is not a consistent error. Comes up once in ~10 or 15 simulations.