keras icon indicating copy to clipboard operation
keras copied to clipboard

Better error message for Unexpected result of `train_function` (Empty logs). Please use `Model.compile(..., run_eagerly=True)`, or `tf.config.run_functions_eagerly(True)` for more information of where went wrong, or file a issue/bug to `tf.keras`

Open Vishu26 opened this issue 3 years ago • 6 comments

Hi!

This error message pops up when empty arrays are passed to Model.fit or Model.predict. The solution suggested in the error message does not work and in some sense is not logical. Either the error message should be updated or the mentioned functions should have additional assertions to check for empty input arrays.

Vishu26 avatar Mar 09 '22 14:03 Vishu26

Could you please provide some simple reproducible code and the error message which you are getting currently in order to expedite the troubleshooting process. Thanks!

sachinprasadhs avatar Mar 10 '22 01:03 sachinprasadhs

To be precise, this issue popped up in tensorflow==2.7.0 and keras==2.7.0 and their recent versions.

Currently, I have python==3.8.6, tensorflow==2.8.0 and keras==2.8.0.

The minimal code to reproduce the issue:

from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
Y = np.array([0, 1, 1, 0])

model = Sequential()
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam', loss='binary_crossentropy', metrics=['acc'])

idx = []

model.fit(X[idx], Y[idx])

Vishu26 avatar Mar 10 '22 12:03 Vishu26

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

google-ml-butler[bot] avatar Mar 17 '22 12:03 google-ml-butler[bot]

i have the same issue and i didnt know what is the solution

NAWRASALANI avatar Mar 20 '22 21:03 NAWRASALANI

Fix this error message to something more logical. Asap

urosjarc avatar Jul 06 '22 16:07 urosjarc

Can you check also this case: https://discuss.tensorflow.org/t/text-based-tensorflow-unexpected-result-of-train-function-empty-logs/11075

bhack avatar Jul 27 '22 21:07 bhack

Hi all,

Is there any solution to this issue?

Thank you in advance.

SJYinda avatar Mar 17 '23 17:03 SJYinda

Hi all,

Is there any solution to this issue?

Thank you in advance.

Did you find a solution to this?

mani2001 avatar Apr 03 '23 18:04 mani2001

Any update on a different (more informative) error message?

I've been working with TF for a while now and just have to say that this generic error message requires a lot more time to debug code than it seems like it should. Having a more informative message would really help code development.

pwernette avatar Apr 10 '23 13:04 pwernette

@pwernette , There is a PR created for this issue here https://github.com/keras-team/keras/pull/16216, but there is no update from the author.

sachinprasadhs avatar Apr 11 '23 21:04 sachinprasadhs

I also came across this error, but when using a tf.data.Dataset with step calculations and large batch sizes on small datasets.
I would also expect to get a better error message if I were to give steps_per_epoch=0 to fit.

Example:

import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
Y = np.array([0, 1, 1, 0])

ds = tf.data.Dataset.from_tensor_slices((X, Y))
ds = ds.repeat()
steps = 0  # from calculations such as len//batchsize

model = Sequential()
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam', loss='binary_crossentropy', metrics=['acc'])

model.fit(ds, steps_per_epoch=steps)

also results in the error:

ValueError: Unexpected result of `train_function` (Empty logs). Ple
ase use `Model.compile(..., run_eagerly=True)`, or `tf.config.run_f
unctions_eagerly(True)` for more information of where went wrong, o
r file a issue/bug to `tf.keras`.

Also if compiling the model with run_eagerly=True, as the error message suggest, you get the same error.

tomrtk avatar Apr 21 '23 14:04 tomrtk

In the hope of helping someone in a similar situation, I had the same error message, but my problem was my .fit() call was using a backup I had created using keras' BackupAndRestore callback. After deleting the backup, training could be launched.

Gumo-A avatar Dec 20 '23 12:12 Gumo-A