Unable to process data in gen_train_valid_test.py script

[frey123]

Hey guys,

i have problems running the gen_train_valid_test.py script. Seems to be a shape issue while loading the pickle data.

I used the flickr27_logos dataset. The crop and augmentation script worked just fine.

I get following exception:

('Unable to process data from', 'flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test.pickle', ':', ValueError('could not broadcast input array from shape (1387,32,64,3) into shape (1851,32,64,3)',)) Traceback (most recent call last): File "gen_train_valid_test.py", line 191, in <module> main() File "gen_train_valid_test.py", line 177, in main train_datasets, TRAIN_SIZE, VALID_SIZE) File "gen_train_valid_test.py", line 128, in merge_datasets train_dataset[start_t:end_t, :, :, :] = train_logo ValueError: could not broadcast input array from shape (1387,32,64,3) into shape (1851,32,64,3)

It processes all classes and failed when it is loading the single pickle files inside the class directories i think.

Has anyone an idea how to fix this issue?

best regards and thanks Frey

[satojkovic]

@frey123

Thank you for the issue report.

According to above exception message, train_datasets seem to have Ford/test.pickle incorrectly. Did you include some extra files in each class folders?

[frey123]

@satojkovic

no i ddnt include extra files. It was a clean checkout and i didnt change anything. just followed your script pipeline.

[skeeph]

Hello, I've got similar error ('Unable to process data from', 'flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test.pickle', ':', ValueError('could not broadcast input array from shape (1279,32,64,3) into shape (1851,32,64,3)',)) Traceback (most recent call last): File "gen_train_valid_test.py", line 190, in <module> main() File "gen_train_valid_test.py", line 176, in main train_datasets, TRAIN_SIZE, VALID_SIZE) File "gen_train_valid_test.py", line 127, in merge_datasets train_dataset[start_t:end_t, :, :, :] = train_logo ValueError: could not broadcast input array from shape (1279,32,64,3) into shape (1851,32,64,3)

[satojkovic]

@frey123 @skeeph

What is the output of the following command after executing the crop_and_aug.py ?

$ find flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images -name "train*" -o -name "test*"

Here is my output. Is your output same?

flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/train

[skeeph]

@satojkovic Yes, but not this order. And while running crop_and_aug.py I got output train_annotation: 4536, 7 ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') original: 1079 cropped: 217488

[satojkovic]

@skeeph

output message is fine. But the order is important because gen_train_valid_test.py assume that test and train subfolders are alternately processed. Could you show me the output of the above find command?

[mursalal]

Hey, I have same problem like skeeph. Output of the find command:

flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/test

[frey123]

Hey,

strange, i did a clean checkout and now it works. Only one thing changed. I'm on Debian Linux instead of CentOS now.

Best regards. Frey

[skeeph]

@satojkovic here is my output find flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images -name "train*" -o -name "test*" flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/test

[satojkovic]

@skeeph @mursalal

I don't know why but the order is different from my result on mac osx. What OS are you using? "test" is the first and "train" is the second in my output. "train" is the first and "test" is the second in your output. Besides my ouptut is sorted but your ouptut isn't.

Could you try the following?

gen_train_valid_test.py, at line 172,173

    #train_datasets = maybe_pickle(train_test_dirs[1::2])
    #test_datasets = maybe_pickle(train_test_dirs[0::2])
    train_datasets = maybe_pickle(train_test_dirs[0::2])  # replace 1 with 0
    test_datasets = maybe_pickle(train_test_dirs[1::2])   # replace 0 with 1

[mursalal]

@satojkovic

That helped. Thank you!

[prakritidev]

I have this problem in this script:

AttributeError: module 'common' has no attribute 'CLASS_NAME'

Can you please help me

[satojkovic]

@prakritidev

Could you try new master? 01146bd

[asbiju2000]

On what bases need to set below values I have one class TRAIN_SIZE = VALID_SIZE = TEST_SIZE =