mesh
mesh copied to clipboard
tensorflow
Add scores for generated text in inference mode
Background:
I was using the T5 model and wanted to get the scores at inference mode along with the generated text. However, this feature is not supported by T5 at the moment and I was advised to implement this feature and raise a pull request. Please see https://github.com/google-research/text-to-text-transfer-transformer/issues/311. for more details.
This PR implemented this function to add the scores(log likelihood) along the generated text in the outputs when a model is exported in SavedModel format.
Changed file: ./mesh/mesh_tensorflow/transformer/utils.py
SignatureDef Diff Below is how a T5 MTF SavedModel SignatureDef looks like before the change:
The given SavedModel SignatureDef contains the following input(s):
inputs['input'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: inputs:0
The given SavedModel SignatureDef contains the following output(s):
outputs['inputs'] tensor_info:
dtype: DT_STRING
shape: (10)
name: SentenceTokenizer/SentenceTokenizer/SentencepieceDetokenizeOp:0
outputs['outputs'] tensor_info:
dtype: DT_STRING
shape: (10)
name: SentenceTokenizer_1/SentenceTokenizer/SentencepieceDetokenizeOp:0
Method name is: tensorflow/serving/predict
Below is how a T5 MTF SavedModel SignatureDef looks like after the change:
The given SavedModel SignatureDef contains the following input(s):
inputs['input'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: inputs:0
The given SavedModel SignatureDef contains the following output(s):
outputs['inputs'] tensor_info:
dtype: DT_STRING
shape: (10)
name: SentenceTokenizer/SentenceTokenizer/SentencepieceDetokenizeOp:0
outputs['outputs'] tensor_info:
dtype: DT_STRING
shape: (10)
name: SentenceTokenizer_1/SentenceTokenizer/SentencepieceDetokenizeOp:0
outputs['scores'] tensor_info:
dtype: DT_FLOAT
shape: (10)
name: reshape_17/parallel_0/Reshape:0
Method name is: tensorflow/serving/predict
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.
ℹ️ Googlers: Go here for more info.
Sorry for the late reply. I'm still planning to make it work but probably won't have time in the next couple of weeks. Happy for someone to take a look at it in the meantime.
On Sat, 3 Oct 2020, 12:06 am Adam Roberts, [email protected] wrote:
@allen-q https://github.com/allen-q do you plan on following up with this? thanks!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/mesh/pull/164#issuecomment-702754334, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6GK65MHIRVRNBEVT533DTSIXM7DANCNFSM4QF6TH5A .
No worries. Perhaps we can just gate this with a bool arg for now until we have the "free" version?
I am currently working on a return_logits option for sample_autoregressive that just returns the already available logits together with outputs so no extra computation is involved. If this is set to True it returns an (outputs, output_logits) tuple instead of outputs. But I think it makes no sense to return only outputs anymore so not sure another argument should be introduced.
