tftables icon indicating copy to clipboard operation
tftables copied to clipboard

ZeroDivisionError when `FileReader.get_batch` is called without `block_size`

Open yngvem opened this issue 6 years ago • 1 comments

I get a ZeroDivisionError when I try to read a batch without specifying what the block_size is. My HDF5 file is created with h5py and is a simple table.

Here is my code and the error message:

In [2]: reader = tftables.open_file('/home/yngve/table.h5', batch_size=5)

In [3]: a = reader.get_batch('/train/images', n_procs=3)
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-3-05142b89f50c> in <module>()
----> 1 a = reader.get_batch('/train/images', n_procs=3)

~/anaconda3/envs/tf/lib/python3.6/site-packages/tftables.py in get_batch(self, path, **kw_args)
    264         block_size = queue.block_size
    265         # get an example for finding data types and row sizes.
--> 266         example = self.streamer.get_remainder(path, block_size)
    267         batch_type = example.dtype
    268         inner_shape = example.shape[1:]

~/anaconda3/envs/tf/lib/python3.6/site-packages/multitables.py in get_remainder(self, path, block_size)
    459         :return: A copy of the remainder elements as a numpy array.
    460         """
--> 461         return self.__get_batch(path, length=block_size, last=True)
    462 
    463     class Queue:

~/anaconda3/envs/tf/lib/python3.6/site-packages/multitables.py in __get_batch(self, path, length, last)
    444 
    445         if last:
--> 446             example = h5_node[length*(len(h5_node)//length):].copy()
    447         else:
    448             example = h5_node[:length].copy()

ZeroDivisionError: integer division or modulo by zero

The error is resolved if I specify block_size. The code

In [2]: reader = tftables.open_file('/home/yngve/table.h5', batch_size=5)

In [3]: a = reader.get_batch('/train/images', block_size=5,  n_procs=3)

does not give any errors. I have tested this code with the FIFOQueue and it does indeed give the expected result.

System info: OS: Ubuntu 16.04 Python version: 3.6 Tensorflow version: 1.5.0 Tftables version: 1.1.2 (latest from pip)

yngvem avatar Apr 03 '18 17:04 yngvem

This definitely seems like a bug.

Thanks for the report!

ghcollin avatar Apr 04 '18 03:04 ghcollin