TF_SIGNATURE_DEF is not used when selected on service startup

[ShamariYoti]

Description Context: Trying to load a tensorflow saved model which has multiple signature definitions.

Problem: The signature definition selected is not used. This happens randomly, when loading the container multiple times different signature definitions will be used even If TF_SIGNATURE_DEF has been set.

I see the log messages below when this happens.

2022-08-10 14:46:32.003735: W triton/tensorflow_backend_tf.cc:998] unable to find serving signature 'serving_default
2022-08-10 14:46:32.003743: W triton/tensorflow_backend_tf.cc:1000] using signature 'random_sig_def'

From looking at the code it seems like it falls into this section and chooses a signature def at random? But as I set the signature def in the config file id expect that block of code not to be executed?

Triton Information Triton version: 22.07-py3

This is the triton docker container.

To Reproduce I load in a model which has multiple signature definitions, this model has three.

I created a config.pbtxt which looks like the following, I played around with this config for a while I added all the relevant inputs and outputs to but the same outcome.

platform: "tensorflow_savedmodel"  
parameters: {
key: "TF_SIGNATURE_DEF"
value: {
  string_value: "sig_def_one"
  }
}

The model is a tensorflow saved model which has three signature definitions of which all have different inputs and outputs. (I can't provide the model here). When I run saved_model_cli show --dir ./model --tag_set serve I am able to see all sigature definition.

Expected behavior

I'd expect TF_SIGNATURE_DEF to be used when loading the models.

[Tabrizian]

@ShamariYoti Does the signature def that you are specifying exist in the model? Looks like if Triton is not able to find the correct SIGDEF it choses one randomly. I think we can fix it to explicitly return an error.

CC @jbkyang-nvi

[ShamariYoti]

Hi @Tabrizian

Yeah so when I run the following command I get the following output...

saved_model_cli show --dir model_path/ --tag serve

SignatureDef key: "onet"
SignatureDef key: "pnet"
SignatureDef key: "rnet"

^^ those are the signature defs im trying to set with TF_SIGNATURE_DEF

[Tabrizian]

@ShamariYoti Can you share your model so that we can look into this issue?

[ShamariYoti]

Hi @Tabrizian

I have shared the model. face-detection.zip

[jbkyang-nvi]

@ShamariYoti can you also share your config.pbtxt? It looks Triton is trying to find serving_default and the model doesn't contain that signature def. It might be because the config.pbtxt is not parsed properly

[ShamariYoti]

Hi @jbkyang-nvi

Yeah that's right it doesn't have a serving_default

The config.pbtxt looked something like this... config.txt

As this model has three signature definitions, I include the model three times and would create a different config.pbtxt for each one where I'd select a different signature definition and change the inputs/outputs etc, but upon startup I would see that sometimes a different signature definition was being chosen.

[jbkyang-nvi]

Hi @ShamariYoti it looks like it's a bug on our end. Thanks for reporting! In the meantime, you can try to run with --disable-auto-complete-config to fix your issue

[ShamariYoti]

@jbkyang-nvi

Thanks I will give this a try, do you have a time frame on when this bug will be fixed and released?

[jamesdhope]

Is there any update on this please? @ShamariYoti @jbkyang-nvi

[dyastremsky]

Thanks for checking on this. This was fixed in the last release. Apologies for not updating the ticket, closing now.