Grappler Errors with ConvLSTM2D

[acfrock]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution: Windows 11 Build 22000
• TensorFlow installed from: pip package
• TensorFlow version: v2.9.0-18-gd8ce9f9c301 2.9.1
• Python version: Python 3.10.4
• CUDA/cuDNN version: CUDA 11.7.0-1
• GPU model and memory: Nvidia 2080 Ti 11GiB Memory
• Exact command to reproduce: python conv_lstm.py with conv_lstm.py from the official Keras examples

Describe the problem

Using the ConvLSTM2D layer produces a grappler error: layout failed: INVALID_ARGUMENT: MutableGraphView::SortTopologically error: detected edge(s) creating cycle(s) ... when run with a GPU. Interestingly, this does not happen if TensorFlow is forced to run without a GPU.

It is not clear whether this is a problem with grappler, with the implementation of the ConvLSTM2D layer, or a bug in CUDA.

Contributing

• Do you want to contribute a PR?: no

Source code / logs

I used the example script from the Keras documentation. [Keras example page] [script]. The logs are here: output.txt.

[sushreebarsa]

@acfrock I tried to run the example script for replicating the issue and didn’t face the error reported. Could you find the gist here and let me know if I am missing something to reproduce the issue? Thank you!

[acfrock]

The error message comes from the CPP logger, so by default it doesn't show in Colab. By using wurlitzer we can see the message: gist.

[gowthamkpr]

@acfrock This is a grappler error. Please take a look at the workaround provided here. Also, please post this issue here. Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[t-kalinowski]

I've run into the same problem and am subscribed to the issue. Posting to keep the stale bot away.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[gowthamkpr]

@acfrock @t-kalinowski Looks like this issue is related to this issue. There has been a PR that has been merged. Please confirm if otherwise. Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No