Empirical risk level sets when design matrix not full rank

[davidrosenberg]

In lectures and homework on describing the level sets of empirical risk for linear least squares models, we are not careful about the case that the design matrix is not full rank. This case is ignored here: https://davidrosenberg.github.io/mlcourse/Archive/2018/Lectures/03a.elastic-net.pdf#page=14 And explicitly mentioned here: https://davidrosenberg.github.io/mlcourse/Archive/2018/Lectures/02c.L1L2-regularization.pdf#page=25 But without any justification for the claim. Perhaps we can put caveats in the slides, and have students work it out as an optional problem in the ellipsoids homework problem: https://davidrosenberg.github.io/mlcourse/Archive/2018/Homework/hw2.pdf#page=9 Or just make a note about it and post.

[brett1479]

That homework problem seems to either be incorrect or missing something. The full rank assumption has to come at the beginning so that (X^TX)^{-1} makes sense. Without the full rank assumption you get an ellipsoid in the row space summed with the null space (so an elliptical cylinder?).

[davidrosenberg]

Yeah definitely should have the full rank condition.