Add check for decoded datatype precision overflow

[jhendersonHDF]

Adds a check for the case where a decoded datatype's precision could overflow SIZE_MAX due to the size of a datatype being larger than SIZE_MAX / 8

[jhendersonHDF]

Related to #4309. Adds a check to make sure that dt->shared->size * 8 doesn't overflow SIZE_MAX before we set a decoded datatype's precision or use the previous calculation later on.

[jhendersonHDF]

What about the other places that use H5T_IS_COMPLEX?

When I looked at the other occurrences previously, there are four occurrences in H5T.c, three of which are addressed here. The other one is in H5T_detect_class, which is just trying to call H5T_detect_class recursively on container datatypes and would return right away for H5T_REFERENCE.

There is one occurrence in H5Tvisit.c in H5T_visit, which only seems to be used for recursive calls of H5T__upgrade_version_cb, which returns right away for H5T_REFERENCE.

There are three occurrences in H5Tvlen.c, all in H5T__vlen_reclaim. Those ones are a bit trickier and might need an additional H5T_IS_REF check in case the datatype is something like a variable-length of array of references. I didn't see any memory leaks from this, but it's possible we aren't testing datatypes like this. I'll look closer.