PyDataset shape error

[Sticcolet]

trafficstars

I constructed a network with an input size of (360,) and a generator with an output size of (96,360). I confirmed that the generator output size is (96,360) by calling the getitem method. But when using fit(), there will be an error message indicating inconsistent input sizes as 'expected shape=(None, 360), found shape=(1, 96, 360)'. How to solve it?

[sachinprasadhs]

Hi, Could you please provide the reproducible code to replicate the reported behavior. Thanks

[Sticcolet]

I have a simple network like this:

visible=kr.Input(shape=(360,))
x=kr.layers.Dense(256,activation='relu')(visible)
x=kr.layers.Dense(64,activation='relu')(x)
output=kr.layers.Dense(3,activation='softmax')(x)
model=kr.Model(inputs=visible, outputs=output)
model.fit(dataset,epochs=10)

and a generator.

Then I call the getitem method of the generator to get X and y, and get the shape:

X,y=dataset.__getitem__(0)
print(X.shape)
>>>(96, 360)

But there will still be errors reported expected shape=(None, 360), found shape=(1, 96, 360)

[sachinprasadhs]

Could you please try

visible=kr.Input(shape=(96,360))
x = kr.layers.Flatten()(visible)
x=kr.layers.Dense(256,activation='relu')(visible)
x=kr.layers.Dense(64,activation='relu')(x)
output=kr.layers.Dense(3,activation='softmax')(x)
model=kr.Model(inputs=visible, outputs=output)
model.fit(dataset,epochs=10)

[Sticcolet]

But doesn't this mean that my individual sample size has become 96 * 360? Actually, my single sample is a row vector of 1 * 360.

[sachinprasadhs]

I see, Thanks for the clarification are you loading the dataset using dataset = keras.utils.PyDataset(dataset)

[Sticcolet]

I do like this:

from keras.utils import PyDataset
class CustomCSVDataLoader(PyDataset):
    def __init__(self, files, labels, batch_size,shuffle=True):
        super().__init__()
        self.files = files
        self.labels = labels
        self.batch_size = batch_size
        self.num_groups = len(files)
        self.shuffle = shuffle
        
    def __len__(self):

    def __getitem__(self, index):

dataset = CustomCSVDataLoader(files, labels, batch_size)

The purpose is to read and merge data from multiple CSV files. The class implements two methods, len and getitem.

One detail is that I inputted batch_stize=32, but since I need to read data from three CSV files, the final output of the generator is 96 * 360. This batch_size was not passed to fit and should not affect the neural network.

[Sticcolet]

OK.I finally found the problem, and as expected, it was indeed a problem with the pytorch and numpy versions. I am using Python 3.10+CUDA11.8, and the corresponding numpy version should be<=1.26, while I am using 2.0. I am currently working normally after downgrading numpy. Should this question belong to Keras or PyTorch? :(

By the way, I strongly recommend providing documentation describing the versions of other modules supported by various versions of Keras to avoid similar issues. For example, I still don't know the pydot version required for keras.utils.plot_model.

[sachinprasadhs]

You need to have a Numpy version<2, since TensorFlow does not support Numpy 2.0 yet.

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No