Linear Algebra Review
(Adapted from ’s slides)
Introduction to Machine Learning (CSC 311) Spring 2020
University of Toronto
Copyright By PowCoder代写 加微信 powcoder
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 1 / 28
A scalar is a number.
A vector is a 1-D array of numbers. The set of vectors of length n with real elements is denoted by Rn.
Vectos can be multiplied by a scalar.
Vector can be added together if dimensions match.
A matrix is a 2-D array of numbers. The set of m ⇥ n matrices with real elements is denoted by Rm⇥n.
Matrices can be added together or multiplied by a scalar. We can multiply Matrices to a vector if dimensions match.
In the rest we denote scalars with lowercase letters like a, vectors with bold lowercase v, and matrices with bold uppercase A.
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 2 / 28
Norms measure how “large” a vector is. They can be defined for matrices too.
The `p-norm for a vector x:
kxkp = |xi|p . i
The `2-norm is known as the Euclidean norm. P The `1-norm is known as the Manhattan norm, i.e., kxk1 =
The `1 is the max (or supremum) norm, i.e., kxk1 = maxi |xi|.
i |xi|. Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 3 / 28
Dot Product
Dot product is defined as v · u = v>u = Pi uivi.
The `2 norm can be written in terms of dot product: kuk2 = pu.u.
Dot product of two vectors can be written in terms of their `2 norms and the angle ✓ between them:
a>b = kak2 kbk2 cos(✓).
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 4 / 28
Cosine Similarity
Cosine between two vectors is a measure of their similarity: cos(✓)= a·b .
Orthogonal Vectors: Two vectors a and b are orthogonal to
each other if a · b = 0.
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 5 / 28
Vector Projection
Given two vectors a and b, let bˆ = b be the unit vector in the direction of b. kbk
Then a1 = a1 · bˆ is the orthogonal projection of a onto a straight line parallel to b, where
a1=kakcos(✓)=a·bˆ=a· b kbk
Image taken from wikipedia.
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 6 / 28
Trace is the sum of all the diagonal elements of a matrix, i.e., Tr(A) = XAi,i.
Cyclic property:
Tr(ABC) = Tr(CAB) = Tr(BCA).
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 7 / 28
Multiplication
Matrix-vector multiplication is a linear transformation. In other words,
M(v1+av2)=Mv1+aMv2 =)(Mv)i=XMi,jvj. j
Matrix-matrix multiplication is the composition of linear transformations, i.e., P
(AB)v = A(Bv) =) (AB)i,j = k Ai,kBk,j.
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 8 / 28 Image taken from wikipedia.
Invertibality
I denotes the identity matrix which is a square matrix of zeros with ones along the diagonal. It has the property IA = A
(BI = B) and Iv = v
A square matrix A is invertible if A 1 exists such that A 1A = AA 1 = I.
Not all non-zero matrices are invertible, e.g., the following matrix
is not invertible:
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 9 / 28
Transposition
Transposition is an operation on matrices (and vectors) that interchange rows with columns. (A>)i,j = Aj,i.
(AB)> = B>A>.
A is called symmetric when A = A>.
A is called orthogonal when AA> = A>A = I or A 1 = A>.
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 10 / 28
Diagonal Matrix
A diagonal matrix has all entries equal to zero except the diagonal entries which might or might not be zero, e.g. identity matrix.
A square diagonal matrix with diagonal enteries given by entries of vector v is denoted by diag(v).
Multiplying vector x by a diagonal matrix is e cient: diag(v)x = v x,
where is the entrywise product.
Inverting a square diagonal matrix is e cient
diag(v) 1 = diag⇣[ 1 ,…, 1 ]>⌘. v1 vn
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 11 / 28
Determinant
Determinant of a square matrix is a mapping to scalars. det(A) or |A|
Measures how much multiplication by the matrix expands or contracts the space.
Determinant of product is the product of determinants: det(AB) = det(A)det(B)
a b = ad bc cd
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 12 / 28
List of Equivalencies
Assuming that A is a square matrix, the following statements are equivalent
Ax = b has a unique solution (for every b with correct dimension).
Ax = 0 has a unique, trivial solution: x = 0. Columns of A are linearly independent.
A is invertible, i.e. A 1 exists.
det(A) 6= 0
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 13 / 28
Zero Determinant
If det(A) = 0, then:
A is linearly dependent.
Ax = b has infinitely many solutions or no solution. These cases correspond to when b is in the span of columns of A or out of it.
Ax = 0 has a non-zero solution. (since every scalar multiple of one solution is a solution and there is a non-zero solution we get infinitely many solutions.)
Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 14 / 28
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com