Incorrect predictions for Tensorflow 2.4.1 models

[scott2b]

Describe the bug Incorrect prediction values are returned for models trained in tensorflow 2.4.1 and specified with the 2.4.1 framework version.

To reproduce Train a model with tensorflow 2.4.1, deploy to a sagemaker endpoint with 2.4.1 specified as the framework version.

Expected behavior Correct predictions.

System information A description of your system. Please provide:

• sagemaker==2.29.0
• Tensorflow: 2.4.1
• Python version: 3.7.9
• CPU or GPU: CPU
• Custom Docker image (Y/N): N

Additional context Specifying 2.3 as the framework for 2.4.1 trained models yields values that are much closer to the expected prediction values.

[scott2b]

For completeness, I also tried specfying 2.4 as the framework version. This also returns incorrect values. The following outputs demonstrate a small example of classifications, all using the same inputs. The tfserving example is pulled from a docker-served tfserving model running on an ec2 instance. The remaining examples are all the same model, but deployed to sagemaker endpoints with the framework versions as specified. The model was trained and saved using tensorflow 2.4.1

tfserving: {'outputs': [[0.673829377], [0.999769], [0.0609291196]]}
2.4.1 {'outputs': [[0.854422748], [0.754358292], [0.509397566]]}
2.3 {'outputs': [[0.67383], [0.999769], [0.0609290302]]}
2.4 {'outputs': [[0.854422748], [0.754358292], [0.509397566]]}