keras icon indicating copy to clipboard operation
keras copied to clipboard

Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT

Open sky712345678 opened this issue 2 years ago • 12 comments

System information.

  • Have I written custom code (as opposed to using a stock example script provided in Keras): no
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.9
  • Python version: 3.7
  • Bazel version (if compiling from source): N/A
  • GPU model and memory: N/A (CPU)
  • Exact command to reproduce:
  1. Open the notebook with Google Colab
  2. Run all cells
  3. View the runtime logs image Note: We have to upgrade Tensorflow and Keras to 2.9 manually in the notebook, because the current default version is not the latest one on Colab.

Describe the problem. (Continue the issue from tensorflow_issue_57052) I got a Type inference failed error when running tf.keras.Model.fit() in Tensorflow 2.9 and Keras 2.9. I didn't see this kind of error in version 2.8 with the identical code. Although the program didn't crash, I'm afraid that there will be some error in the trained model.

Describe the current behavior. Run tf.keras.Model.fit() and the error Type inference failed shows up.

Describe the expected behavior. The error shouldn't show up.

Contributing.

  • Do you want to contribute a PR? (yes/no): no
  • If yes, please read this page for instructions
  • Briefly describe your candidate solution(if contributing):

Standalone code to reproduce the issue. Link to notebook: https://drive.google.com/file/d/1k78lpGVthB7nthEkYgUs3JNJTuR79r5E/view?usp=sharing To reproduce:

  1. Open the notebook with Google Colab
  2. Run all cells
  3. View the runtime logs

Source code / logs.

2022-08-20 17:18:05.533157: W tensorflow/core/common_runtime/forward_type_inference.cc:231] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_BOOL
    }
  }
}
 is neither a subtype nor a supertype of the combined inputs preceding it:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_LEGACY_VARIANT
    }
  }
}

        while inferring type of node 'dice_loss/cond/output/_11'

sky712345678 avatar Sep 02 '22 07:09 sky712345678

@gadagashwini I was able to replicate the issue on colab, please find the gist here. Thank you!

sushreebarsa avatar Sep 06 '22 07:09 sushreebarsa

Hi @sky712345678, W tensorflow/core/common_runtime/forward_type_inference.cc:231] Type inference failed. is just a warning, you can safely ignore it. Given code executed without any error message. Thank you!

gadagashwini avatar Sep 14 '22 08:09 gadagashwini

@gadagashwini what's the point of a warning if the response is simply you can safely ignore it.? It's clearly there for a reason

tgsmith61591 avatar Sep 20 '22 17:09 tgsmith61591

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

google-ml-butler[bot] avatar Sep 27 '22 18:09 google-ml-butler[bot]

@gadagashwini can you talk a little bit more about the reason why we can safely ignore it? Thank you!

sky712345678 avatar Sep 28 '22 02:09 sky712345678

@sky712345678 This looks like an issue from tensorflow. Can you please create this issue in tensorflow/tensorflow. Thank you!!

gowthamkpr avatar Sep 29 '22 17:09 gowthamkpr

@gowthamkpr Well, the problem was first reported in tensorflow as https://github.com/tensorflow/tensorflow/issues/57052, but the guys there told the reporter to instead post an issue here.

If you know any more details (why it is a TensorFlow issue), could you please provide more details that we can give to the TF guys?

foxik avatar Oct 02 '22 08:10 foxik

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

google-ml-butler[bot] avatar Oct 09 '22 09:10 google-ml-butler[bot]

@gowthamkpr The issue was originally a TF issue, but we were redirected to post it here. If you know any more details (why it is a TF issue and not Keras), could you please provide more details that we can give to the TF guys? Thanks!

foxik avatar Oct 09 '22 16:10 foxik

Unsubscribe

---Original--- From: "Milan @.> Date: Sun, Oct 2, 2022 16:54 PM To: @.>; Cc: @.***>; Subject: Re: [keras-team/keras] Type inference failed. This indicates aninvalid graph that escaped type checking. Error message: INVALID_ARGUMENT(Issue #16978)

@gowthamkpr Well, the problem was first reported in tensorflow as tensorflow/tensorflow#57052, but the guys there told the reporter to instead post an issue here.

If you know any more details (why it is a TensorFlow issue), could you please provide more details that we can give to the TF guys?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

zrx563010758 avatar Oct 11 '22 08:10 zrx563010758

The issue is at the level of the dice_loss. Can you try producing a reproduction script that only involves the loss function? Maybe just try to backprop through the loss function and see what happens.

I think this should be reproducible without involving any Keras logic, at which point the TF folks will definitely look at it. But anyway, as said before, this is just a warning, not something critical. You can ignore it.

fchollet avatar Oct 13 '22 17:10 fchollet

Ok, I got it. Thank you!

I wasn't sure how to reproduce it only involving the loss function, this is my try: https://colab.research.google.com/drive/1qxamrOaOqfVANzMnN-u--Sue4iPtJCtf?usp=sharing image Running this Colab notebook, I didn't see the error message in runtime logs.

sky712345678 avatar Oct 19 '22 06:10 sky712345678