Overfit/Underfit random non-ideal demo results

[astonzhang]

For both PyTorch/TensorFlow Adaptations:

In 4.4, the plots for normal fitting (4.4.4.3.) and overfitting (4.4.4.5) have random non-ideal results: they can significantly differ from the mx plots.

Here is another running output on the release branch (without my revision today).

@AnirudhDagar @terrytangyuan, can you take a look at them? Or do you have any suggestions about using a different set of hyperparameters (e.g., change max_degree) to generate more stable and consistent plots?

[terrytangyuan]

Does this occur after reducing the number of epochs?

[astonzhang]

Given the current hyperparameters, the current num_epochs is needed to demo underfitting/overfitting.

[AnirudhDagar]

@astonzhang I was able to fix the issue after setting seed. @mli @terrytangyuan do you have any other ideas to make this more robust since we won't be using a seed after all.

[murphyk]

I noticed this problem, too: I cannot get the SGD optimizer in Pytorch to overfit for the degree-20 polynomial, even when using full-batch training. (I did not try exact least squares fitting.) So this is not a compelling example of overfitting. My pytorch code is here. By contrast, my sklearn demo of overfitting using a similar polynomial regression example clearly shows the problem.