Preliminaries
Probability Theory and Linear Algebra
Probability Theory Review
Rules of Probability
Probability Theory
Joint Probability
Marginal Probability
Conditional Probability
C. Bishop
Example
Probability Theory
Sum Rule
Product Rule
C. Bishop
Probability Theory see Bishop Chapter 1.2
• Pick a random box
• Pick a random fruit
• Observe the fruit type
(orange or apple)
• Put it back in the box
• Repeattrialmanytimes
What is the probability of picking an apple?
C. Bishop
The Rules of Probability
Sum Rule Product Rule
C. Bishop
Bayes’ Theorem
posterior ∝ likelihood × prior
C. Bishop
Probability Theory see Bishop Chapter 1.2
• Suppose we picked an orange
• What is the probability it came from the red box?
C. Bishop
Probability Densities for continuous variables
C. Bishop
Expectations
Conditional Expectation
(discrete)
Approximate Expectation
(discrete and continuous)
C. Bishop
Variances and Co-variances
C. Bishop
The Gaussian Distribution
C. Bishop
Gaussian Mean and Variance
C. Bishop
The Multivariate Gaussian
C. Bishop
Linear Algebra review
Matrices and vectors
Matrix Elements (entries of matrix)
“ , entry” in the row, column.
Vector: An n x 1 matrix.
element
1-indexed vs 0-indexed:
Matrix Addition
Scalar Multiplication
Combination of Operands
Linear Algebra review
Matrix-vector multiplication
Example
Details:
m x n matrix (m rows,n columns)
n x 1 matrix (n-dimensional vector)
m-dimensional vector
To get , multiply
of vector , and add them up.
’s row with elements
Example
House sizes:
How do we get predicted price as matrix-vector product?
Linear Algebra review
Matrix-matrix multiplication
Example
Details:
n x o matrix
m x n matrix (m rows, n columns)
(n rows, o columns)
m xo matrix
The column of the matrix is obtained by multiplying with the column of . (for = 1,2,…,o)
Given house sizes:
Matrix
What is the price of each house?
Have 3 competing linear functions:
1. 2. 3.
Matrix
Linear Algebra Review
Matrix multiplication properties
Let and be matrices. Then in general, (not commutative.)
E.g.
Associative
Let Compute Let Compute
Identity Matrix
Denoted (or ). Examples of identity matrices:
2 x2
3 x3
4 x4
For any matrix ,
In general, is AB = BA?
Linear Algebra review
Inverse and transpose
Not all numbers have an inverse
Matrix inverse:
If A is an m x m matrix, and if it has an inverse,
For a 2 x 2 matrix, what is a sufficient condition for it to have an inverse?
Matrices that don’t have an inverse are “singular” or “degenerate”
Matrix Transpose
Example:
Let be an m x n matrix, and let Then is an n x m matrix, and