Problem 1 (100 points)
Project 1
In linear regression, least square loss (L) in matrix form is ||Y − Xβ||2 = (Y − Xβ)T (Y − Xβ), where Y is the response variable and β is the coefficient. In this project, you are encouraged to find an approximation of this loss in matrix form of a||u − β||2 = a(u − β)T (u − β), where a is a constant number and u is a vector defined by yourself. Here is a toy example of my approximation:
E[(Y −Xβ)T(Y −Xβ)]=σ2
(Verify this) E[(βˆ − β)T (βˆ − β)] = σ2tr(X(XXT )−1(XT X)−1XT ) = σ2tr(A) , where βˆ is the ols estimator,
tr is the trace of that matrix. Therefore, we can approximate L by ||βˆ−β||2 tr(A)
Task 1: Follow the intuition in this toy example, establish you own approximation. Apparently you estima- tion should not be that simple and in-precision. Try to make the approximation as precise as possible.
Task 2: Use R to demonstrate your approximation is much better than mine.
There is no constraint in the way you establish you approximation, the only goal is to make the approxima- tion close to the original loss. But you should list all the papers you have referenced. If you happen to find a approximation that exactly same as the original loss, you would get a A+ in this course
Due date: 11:59 on 12/04