Audit the Unit Test Suite for Woodwork Inference Sensitivity

[chukarsten]

In https://github.com/alteryx/evalml/pull/3626 and the much older https://github.com/alteryx/evalml/pull/2181, we've shown that EvalML is very sensitive to changes in Woodwork inference. This was a mistake in how we've written our tests and we need to determine a way to rectify this.

I think resolving this story involves coming up with a plan to 👍

1. Evaluate the changes in the linked PRs
2. Statically set woodwork typing in those tests/fixtures