Value Error during reshaping of data

[Jewphoria]

I'm having a persistent error in a specific section of the code that I haven't been able to resolve. Admittedly, I'm still very new to this, so I'm likely overlooking something simple. I've looked over the code to try to make sense of it and haven't made any meaningful progress. Any assistance would be greatly appreciated.

The cell in question is:

training_binary_path = os.path.join(DATA_PATH, f'training_data_{GENERATE_SQUARE}_{GENERATE_SQUARE}.npy')

print(f"Looking for file: {training_binary_path}")

if not os.path.isfile(training_binary_path): start = time.time() print("Loading training images...")

training_data = [] faces_path = os.path.join(DATA_PATH) for filename in tqdm(os.listdir(faces_path)): path = os.path.join(faces_path,filename) image = Image.open(path).resize((GENERATE_SQUARE, GENERATE_SQUARE),Image.ANTIALIAS) training_data.append(np.asarray(image)) training_data = np.reshape(training_data,(-1,GENERATE_SQUARE, GENERATE_SQUARE,IMAGE_CHANNELS)) training_data = training_data.astype(np.float32) training_data = training_data / 127.5 - 1.

print("Saving training image binary...") np.save(training_binary_path,training_data) elapsed = time.time()-start print (f'Image preprocess time: {hms_string(elapsed)}') else: print("Loading previous training pickle...") training_data = np.load(training_binary_path)

with the line in question being: image = Image.open(path).resize((GENERATE_SQUARE, GENERATE_SQUARE),Image.ANTIALIAS)

the error text reads "could not broadcast input array from shape (128,128,3) into shape (128,128) is there any fix for this? I haven't made any changes to the code.

[BakingBrains]

@Jewphoria In the dataset folder, there will be .npy files, Delete those files and re-run the cell, let me know if this resolves the issue. And also IMAGE_CHANNELS = 3 this should be set.

[Jewphoria]

@BakingBrains Thank you for your reply. I've checked through my dataset folder and can't find any .npy files. There are quite a lot of files, and colab won't let me see to the bottom of the list. I wasn't able to find any .npy files in Drive, either. IMAGE_CHANNELS = 3 is included in the cell above the one included in the original post.

[BakingBrains]

@Jewphoria just try

if os.path.isfile(os.path.join(DATA_PATH, f'training_data_{GENERATE_SQUARE}_{GENERATE_SQUARE}.npy')):
    os.remove(os.path.join(DATA_PATH, f'training_data_{GENERATE_SQUARE}_{GENERATE_SQUARE}.npy'))
    print("file deleted")

Also if it's ok share a demo image, on which you're trying to train

[Jewphoria]

Thank you again for you reply. I've attached one of the images below. The dataset is mostly large images ranging from 1080p upward beyond 4K.

I ran the code and did not detect a .npy file

[BakingBrains]

@Jewphoria try

GENERATE_RES = 6 # Generation resolution factor 
GENERATE_SQUARE = 32 * GENERATE_RES # rows/cols (should be square)
IMAGE_CHANNELS = 3

If this doesn't work switch to Progressive GAN or Style GAN. If progressive GAN I will share the code.

[Jewphoria]

Thank you. I've been running the code under the parameters you specified with the issue still persisting on the reshape. The notebook is able to successfully load the images in for training, they just can't be reshaped due to the value error.

Could you share code for progressive GAN?

[BakingBrains]

@Jewphoria can you share your mail ID, I will share the code. Thank you

[Jewphoria]

Yes, I can be reached at [email protected]

Thank you again for your assistance

[aaronbjohnson]

@Jewphoria @BakingBrains Did you all ever figure this issue out? I was able to run this successfully on one smaller collection of images. I tried to run this on a larger set of images and got a similar value error:

ValueError: could not broadcast input array from shape (64,64,3) into shape (64,64)

Any help would be greatly appreciated.

[BakingBrains]

@aaronbjohnson can you please check if there are .npy file in the dataset and if yes please delete that file, and try to run again.

[aaronbjohnson]

@BakingBrains Yes I've confirmed that there are no .npy files in the dataset. I've also tried to change the GENERATE_RES variable to both lower and higher resolutions, but am getting the same error. Could it be due to a corrupt .jpeg file in the dataset?

[BakingBrains]

@aaronbjohnson Yes that is also a possibility.

[aaronbjohnson]

@BakingBrains I've played around with this a bit more. I kept getting the error message: /usr/local/lib/python3.7/dist-packages/PIL/TiffImagePlugin.py:788: UserWarning: Corrupt EXIF data. Expecting to read 2 bytes but only got 0.

I tried making a duplicate data directory and using a tool to strip all EXIF data from the images. When running this on that set though, it couldn't even read the images. I guess this is because the tool I used removed too much data from the image to actually be usable (I'm not too familiar with EXIF data and what's needed).

So I went back to the original set and just tried to delete the images that were throwing that error. It turns out that no matter how many times I deleted the problem image and reran the program, the error always seemed to happen at the 8% mark. Any idea why this might be? It just seems so bizarre that always at the 8% mark it will throw the Corrupt EXIF data on whatever image it happens to be at.

[BakingBrains]

@aaronbjohnson can you please run the below code and check once if there any image files with 0KB.

import os

path = "D:/dataset/"

# count = 0

for dir, subdir, files in os.walk(path):
    for f in files:
        filename = dir + f
        size = os.path.getsize(dir + f)
        if size == 0:
            # count += 1
            os.remove(dir + f)
            print(f'{count}.Removed {filename} with size {size}')

[aaronbjohnson]

@BakingBrains It turns out I didn't have any files with 0KB. This little snippet of code did help me fix the EXIF error. The problem was that I did not end the path to the dataset with a "/". Once I did that I didn't receive the EXIF errors, but I still got the np.reshape error: ValueError: could not broadcast input array from shape (96,96,3) into shape (96,96)

[ShailendraLowanshi24]

Hey can you provide the solutions to the same error ?

ValueError: could not broadcast input array from shape (96,96,3) into shape (96,96)

[ShailendraLowanshi24]

I'm using the same dataset, still getting that error

[ShouryaTyagi222]

I find a easy solution to this problem , rather using pillow to load and resize the image use cv2 to open and resize the image.

[ShailendraLowanshi24]

@ShouryaTyagi222 Can you provide the code, how to solve that

[ShouryaTyagi222]

in place of line 64 image = Image.open(path).resize((GENERATE_SQUARE, GENERATE_SQUARE),Image.ANTIALIAS) training_data.append(np.asarray(image))

write,

image=cv2.imread(path)
image=cv2.resize(image,(GENERATE_SQUARE, GENERATE_SQUARE))
training_data.append(image)

and remember to import cv2