BASIC MULTIVARIABLE CALCULUS
CSC311 Fall 2020 (Notes by . Erdogdu)
University of Toronto
1. Basic multivariable calculus. For a given function f : Rd → R, we denote its partial deriva- tive with respect to its i-th coordinate as ∂f(x)/∂xi ∈ R. Gradient of this function is simply a vector with i-th coordinate ∂f(x)/∂xi ∈ R. That is,
Copyright By PowCoder代写 加微信 powcoder
(1.1) [∇f(x)]i = ∂f(x). ∂xi
The gradient of a function points in the direction of greatest increase, and its magnitude is the rate of increase in that direction. Therefore, when you are minimizing a function, it makes sense to move in the direction opposite to its gradient.
Similarly, we can define the second derivative of the function f, which is generally referred to as the Hessian of f. It is a matrix and its i,j-th entry is given by
(1.2) [∇2f(x)]ij = ∂2f(x). xi xj
Using the above definition, for x, y ∈ Rd and A ∈ Rd×d we obtain (a) the gradient with respect to x of xT y is y,
(b) the gradient with respect to x of xT x is 2x,
(c) the gradient with respect to x of xT Ax is 2Ax,
(d) the gradient with respect to x of Ax is A.
In some cases, you can see that the above gradients are transposed. This is a matter of definition. You should check the wikipedia page https://en.wikipedia.org/wiki/Matrix_calculus which contains a very detailed list of rules.
1.1. Least squares problem. In the least squares problem, we are given a target vector t ∈ RN , a design matrix X ∈ RN×D. We would like to find the weights w that minimizes the objective function given by the least squares problem
minimizeJ(w)=: 1∥t−Xw∥2. w2
Weknowthataminimumoccursatacriticalatwhichthepartialderivativesareequalto0.i.e.∂J(w)/wj = 0 for j = 1, .., D. This is equivalent to saying the gradient ∇J (w) = 0. We can write
J (w) =21∥t∥2 + 12w⊤X⊤Xw − t⊤Xw.
Taking derivative with respect to the vector w and setting it equal to 0, we obtain ∇J (w) = X⊤Xw − X⊤t = 0.
If X⊤X is invertible, a solution to above linear system is given by wLS = (X⊤X)−1X⊤t.
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com