full version run fit_generator failed

[broadwit]

env: tensorflow-gpu 1.15.0，keras 2.3.1，cudakit 10.0.130，cudann 7.6.5

when run full version, encountered following problems, try global_variables_initializer, can't resolve: 2020-04-25 20:43:26.316669: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll Traceback (most recent call last): File "D:/work/py/Coloring-greyscale-images/Full-version/full_version.py", line 112, in model.fit_generator(image_a_b_gen(batch_size), epochs=1, steps_per_epoch=1) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\engine\training.py", line 1732, in fit_generator initial_epoch=initial_epoch) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\engine\training_generator.py", line 185, in fit_generator generator_output = next(output_generator) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\utils\data_utils.py", line 742, in get six.reraise(*sys.exc_info()) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\six.py", line 703, in reraise raise value File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\utils\data_utils.py", line 711, in get inputs = future.get(timeout=30) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\multiprocessing\pool.py", line 657, in get raise self._value File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\multiprocessing\pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\utils\data_utils.py", line 650, in next_sample return six.next(_SHARED_SEQUENCES[uid]) File "D:/work/py/Coloring-greyscale-images/Full-version/full_version.py", line 97, in image_a_b_gen embed = create_inception_embedding(grayscaled_rgb) File "D:/work/py/Coloring-greyscale-images/Full-version/full_version.py", line 81, in create_inception_embedding embed = inception.predict(grayscaled_rgb_resized) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\engine\training.py", line 1462, in predict callbacks=callbacks) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\keras\engine\training_arrays.py", line 324, in predict_loop batch_outs = f(ins_batch) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3476, in call run_metadata=self.run_metadata) File "D:\ProgramData\Anaconda3\envs\tf1_gpu\lib\site-packages\tensorflow_core\python\client\session.py", line 1472, in call run_metadata_ptr) tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found. (0) Failed precondition: Error while reading resource variable batch_normalization_177/beta from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/batch_normalization_177/beta) [[{{node batch_normalization_177/cond/ReadVariableOp}}]] [[predictions/Softmax/_7]] (1) Failed precondition: Error while reading resource variable batch_normalization_177/beta from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/batch_normalization_177/beta) [[{{node batch_normalization_177/cond/ReadVariableOp}}]] 0 successful operations. 0 derived errors ignored.

[broadwit]

Alpha and beta version can work normally in the same enviroment.

In the full version, if I replaced the following code in create_inception_embedding function ,the code could continue to run. with inception.graph.as_default(): embed = inception.predict(grayscaled_rgb_resized)

But, I tested the create_inception_embedding function with a single input tensor, it could work.

[broadwit]

I have changed to tensorflow 2.1 env, and it's ok now.

[broadwit]

But I don't know why the problems above occurred yet...

[jeanpierrefakhry]

@broadwit can you please send the code that you changed? I'm stuck on the same issue

[Maitreya-Stark]

Sorry wrong user here...

On Fri, 19 Mar 2021, 12:27 pm jeanpierrefakhry, @.***> wrote:

@broadwit https://github.com/broadwit can you please send the code that you changed? I'm stuck on the same issue

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/emilwallner/Coloring-greyscale-images/issues/12#issuecomment-802956181, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANMTVEBAHGXC2OXFMX55G2TTEN3PXANCNFSM4MQXPTRA .

[jeanpierrefakhry]

What @Maitreya-Stark ? i didn't really understand what you said

[GreihMurray]

Alpha and beta version can work normally in the same enviroment.

In the full version, if I replaced the following code in create_inception_embedding function ,the code could continue to run. with inception.graph.as_default(): embed = inception.predict(grayscaled_rgb_resized)

But, I tested the create_inception_embedding function with a single input tensor, it could work.

What did you change it too @broadwit ? Im having the same issue

[jeanpierrefakhry]

@GreihMurray I solved this issue by replacing

inception = InceptionResNetV2(weights='imagenet', include_top=True)
inception.graph = tf.get_default_graph()

with

graph = tf.Graph()
with graph.as_default():
    sess = tf.compat.v1.Session(graph=graph)
    with sess.as_default():
        inception = InceptionResNetV2(weights='imagenet', include_top=True)
        inception.graph = tf.compat.v1.get_default_graph()

Hope that helps!

[GreihMurray]

@jeanpierrefakhry hey thanks for the reply, just to be clear, is that running with the up to date versions of tenserflow and keras?

[jeanpierrefakhry]

@GreihMurray tensorflow 2.1.0

[GreihMurray]

Awesome, thank you

[Uttejreddy0496]

@jeanpierrefakhry what was the verison for keras you have used? Thanks