The Method of Least Squares

[AleksandrHovhannisyan]

[Alexander-Dirtyhead]

Hi! My English name is alexander, too! Recently, I read your blog :The Method of Least Squares,which I have a little question about. In the second related question:2. How does this actually change the problem?, you wrote this line: we can solve for it by multiplying both sides by the inverse of ATA

And my question is that is there any possibility that the inverse of ATA does not exist, even though ATA is a symmetric matrix?

Last, lots of thanks for you blog!

[AleksandrHovhannisyan]

@Alexander-Dirtyhead

And my question is that is there any possibility that the inverse of ATA does not exist, even though ATA is a symmetric matrix?

I believe it's possible. A matrix is invertible if and only if its determinant is nonzero. And the determinant of a matrix product equals the product of the determinants of the individual matrices. So if $A$ is not invertible (determinant is zero), then neither is the product of its transpose with itself. However, we can still find a least-squares solution to $Ax = b$ using $QR$ decomposition, which doesn't require finding the inverse.

[Alexander-Dirtyhead]

@Alexander-Dirtyhead

And my question is that is there any possibility that the inverse of ATA does not exist, even though ATA is a symmetric matrix?

I believe it's possible. A matrix is invertible if and only if its determinant is nonzero. And the determinant of a matrix product equals the product of the determinants of the individual matrices. So if A is not invertible (determinant is zero), then neither is the product of its transpose with itself. However, we can still find a least-squares solution to Ax=b using QR decomposition, which doesn't require finding the inverse.

Thanks for your reply! Your reply helps me a lot. Best regards!

[EdinLiv]

Hi there, this is Mark from China. I am a novice technician in spectral flow cytometry, which utilizes least square methods extensively to perform unmixing of composite spectra from dozens of fluorescent probes labelled on each human cell in order to asnwer some real biomedical questions. Given my biological background, the concept of least square methods seemed very daunting to me until I found out your blog post. This is by far the plainest explanation of least square methods to a novice in mathematics! Although I was a bit lost with QR decomposition but to my understanding this is just a technical detail in the execution right? Anyway, highly recommended to my colleagues.

[AleksandrHovhannisyan]

@EdinLiv Glad you found it useful!

Although I was a bit lost with QR decomposition but to my understanding this is just a technical detail in the execution right?

Yes, QR decomposition is just one way of solving a least squares equation. It's mainly used in computing to minimize errors due to certain operations involved in finding the transpose and pseudo-inverse of a matrix.