ffcv icon indicating copy to clipboard operation
ffcv copied to clipboard

Problem while unpacking encoded JSONField

Open anasserhussien opened this issue 3 years ago • 0 comments

Hello,

I am using FFCV for an object detection task. When writing the .beton file, I am using RGBImageField for the image and JSONField for writing the dict of bounding boxes.

writer = DatasetWriter(path, {
    'image': RGBImageField(),
    'label': JSONField()
    }
)
writer.from_indexed_dataset(ds)

Writing the .beton file works properly with no problems. However, when I am looping over the batches after loading the .beton I am getting an error while unpacking the label field to get the json back.

for i, batch in enumerate(train_data_loader):
    x, y = batch

    #this line sometimes through an error
    y = JSONField.unpack(y)

The error says the following:

sep_location = np.where(b == ord(SEPARATOR))[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

Which basically means that the SEPARATOR does not exist in the encoded array. This happens only to few number of data points, any thoughts?

anasserhussien avatar Jan 06 '23 21:01 anasserhussien