代写 Bayesian GAUSSIAN PROCESS REGRESSION

GAUSSIAN PROCESS REGRESSION
CSE 515T

1. BACKGROUND
The kernel trick again. . .

The Kernel Trick
Consider again the linear regression model:
with prior
y(x) = φ(x)⊤w + ε, p(w) = N (w; 0, Σ).
The kernel trick is to define the function K(x, x′) = φ(x)⊤Σφ(x′),
which allows us to. . .
Introduction The Kernel Trick 3

The Kernel Trick
. . . given training data D and test inputs X∗, write the predictive distribution in this nice form:
where
p(y∗ | X∗,D,σ2) = N(y∗;μy∗|D,Ky∗|D), μy∗|D = K⊤∗ (K + σ2I)−1y
Ky∗|D = K∗∗ − K⊤∗ (K + σ2I)−1K∗, and we have defined
K = K(X,X) K∗ = K(X,X∗) K∗∗ = K(X∗,X∗).
Introduction The Kernel Trick 4

The Kernel Trick
This does more than just make the expressions pretty. In particular, it is often easier/cheaper to calculate K directly rather than explicitly compute φ(x) and take the dot product.
Example: “all subsets” kernel.
Introduction The Kernel Trick 5

The Kernel Trick
Idea: completely abandon the idea of deriving explicit feature expansions and simply derive (positive-definite) kernel functions K directly!
Introduction The Kernel Trick 6

2. GAUSSIAN PROCESSES
A reimagining of Bayesian regression

For more information
http://www.gaussianprocess.org/
(Also code!)
Gaussian Processes Introduction 9

Regression
Consider the general regression problem. Here we have:
• an input domain X (for example, Rn, but in general
anything),
• an unknown function f : X → R, and
• and (perhaps noisy) observations of the function: D=􏰫(xi,yi)􏰬,whereyi =f(xi)+εi.
Our goal is to predict the value of the function f(X∗) at some test locations X∗.
Gaussian Processes Introduction 10

Gaussian processes
• Gaussian processes take a nonparameteric approach to regression. We select a prior distribution over the function f and condition this distribution on our observations, using the posterior distribution to make predictions.
• Gaussian processes are very powerful and leverage the many convenient properties of the Gaussian distribution to enable tractable inference.
Gaussian Processes Introduction 11

From the Gaussian distribution to GPs
• How can we leverage these useful properties of the Gaussian distribution to approach the regression problem? We have a problem: the latent function f is usually infinite dimensional; however, the multivariate Gaussian distribution is only useful in finite dimensions.
• The Gaussian process is a natural generalization of the multivariate Gaussian distribution to potentially infinite settings.
Gaussian Processes GPs: Prior 12

GPs: Definition
Definition (GPs)
A Gaussian process is a (potentially infinite) collection of random variables such that the joint distribution of any finite number of them is multivariate Gaussian.
Gaussian Processes GPs: Prior 13

GPs: Notation
A Gaussian process distribution on f is written p(f) = GP(f; μ, K),
and just like the multivariate Gaussian distribution, is parameterized by its first two moments (now functions):
E[f] = μ: X → R, the mean function, and •􏰘
• E 􏰀f(x) − μ(x)􏰁􏰀f(x′) − μ(x′)􏰁􏰢 = K : X × X → R, a positive semidefinite covariance function or kernel.
Gaussian Processes GPs: Prior 14

GPs: Mean and covariance functions
• The mean function encodes the central tendency of the function, and is often assumed to be a constant (usually zero).
• The covariance function encodes information about the shape and structure we expect the function to have. A simple and very common example is the squared exponential covariance:
K(x, x′) = exp􏰀−1/2∥x − x′∥2􏰁,
which encodes the notation that “nearby points should
have similar function values.”
Gaussian Processes GPs: Prior 15

GPs: Prior on finite sets
Suppose we have selected a GP prior GP(f; μ, K) for the function f . Consider a finite set of points X ⊆ X . The GP prior on f, by definition, implies the following joint distribution on the associated function values f = f(X):
p(f | X) = N(f;μ(X),K(X,X)􏰁.
That is, we simply evaluate the mean and covariance functions at X and take the associated multivariate Gaussian distribution. Very simple!
Gaussian Processes GPs: Prior 16

Prior: Sampling examples
μ(x) ±2􏰈K(x, x) samples
0 1 2 3 4 5 6 7 8 9 10
x
K = exp􏰀−1/2∥x − x′∥2􏰁
Gaussian Processes GPs: Prior 17
2
0
−2

Prior: Sampling examples
2
0
μ(x) ±2σ samples
−2
0 1 2 3 4 5 6 7 8 9 10
x
K = λ2 exp􏰊−∥x − x′∥2 􏰋 λ = 1/2, l = 2 2l2
Gaussian Processes GPs: Prior
18

Prior: Sampling examples
4 2 0
−2 −4
μ(x) ±2σ samples
0 1 2 3 4 5 6 7 8 9 10
x
K = exp􏰀−∥x − x′∥􏰁
Gaussian Processes GPs: Prior 19

From the prior to the posterior
So far, I’ve only told you how to construct prior distributions over the function f. How do we condition our prior on some observations D = (X, f ) to make predictions about the value of f at some points X∗?
Gaussian Processes GPs: Posterior 20

From the prior to the posterior
We begin by writing the joint distribution between the training function values f(X) = f and the test function values
f(X∗) = f∗:
p(f,f∗) = N􏰎􏰌f 􏰍;􏰌μ(X)􏰍,􏰌K(X,X) K(X,X∗)􏰍􏰏… f∗ μ(X∗) K(X∗, X) K(X∗, X∗)
Gaussian Processes GPs: Posterior 21

From the prior to the posterior
. . . we then condition this multivariate Gaussian on the known training values f. We already know how to do that!
p(f∗ | X∗,D) = N􏰀f∗;μf|D(X∗),Kf|D(X∗,X∗)􏰁, where
μf |D (x) = μ(x) + K (x, X)K−1 􏰀f − μ(X)􏰁 Kf |D (x, x′ ) = K (x, x′ ) − K (x, X)K−1 K (X, x′ ).
Gaussian Processes GPs: Posterior 22

From the prior to the posterior
Notice that the functions μf|D and Kf|D are valid mean and covariance functions, respectively. That means the previous slide is telling us the posterior distribution over f is a Gaussian process!
Gaussian Processes GPs: Posterior 23

The posterior mean
One way to understand the posterior mean function μf|D is as a correction to the prior mean consisting of a weighted combination of kernel functions, one for each training data point:
μ f | D ( x ) = μ ( x ) + K􏰅( x , X ) 􏰀 K ( X , X ) 􏰁 − 1 􏰀 f − μ ( X ) 􏰁 N
= μ(x) + αiK(xi, x), i=1
whereαi =K(X,X)−1􏰀f(xi)−μ(xi)􏰁.
Gaussian Processes GPs: Posterior 24

Prior
μ(x) ±2􏰈K(x, x) samples
0 1 2 3 4 5 6 7 8 9 10
x
K = exp􏰀−1/2∥x − x′∥2􏰁
Gaussian Processes GPs: Posterior 25
2
0
−2

Posterior example
2
0
observations μ(x) ±2σ
−2
0 1 2 3 4 5 6 7 8 9 10
x
Gaussian Processes GPs: Posterior 26

Posterior: Sampling
2
0
observations μ(x) ±2σ samples
−2
0 1 2 3 4 5 6 7 8 9 10
x
Gaussian Processes GPs: Posterior 27

Dealing with noise
So far, we have assumed we can sample the function f exactly, which is uncommon in regression settings. How do we deal with observation noise?
tl;dr: the same way we did with Bayesian linear regression!
Gaussian Processes Gaussian noise 28

Dealing with noise
We must create a model for our observations given the latent function. To begin, we will choose the simple iid, zero-mean additive Gaussian noise model:
y(x) = f(x) + ε, p(ε | x) = N(ε;0,σ2);
combined we have
p(y | f) = N(y;f,σ2I).
Gaussian Processes Gaussian noise 29

Noisy posterior
To derive the posterior given noisy observations D, we again write the joint distribution between the training function values y and the test function values f∗:
p(y,f∗)=N􏰎􏰌y􏰍;􏰌μ(X)􏰍,􏰌K(X,X)+σ2I K(X,X∗)􏰍􏰏… f∗ μ(X∗) K(X∗, X) K(X∗, X∗)
(1)
Gaussian Processes Gaussian noise 30

Noisy posterior: Sampling
2
0
observations μ(x) ±2σ samples
−2
0 1 2 3 4 5 6 7 8 9 10
x
σ = 0.1
Gaussian Processes Gaussian noise 32

Noisy posterior: Sampling
2
0
observations μ(x) ±2σ samples
−2
0 1 2 3 4 5 6 7 8 9 10
x
σ = 0.5
Gaussian Processes Gaussian noise 33

3. HYPERPARAMETERS
We’re not done yet. . .

Hyperparameters
• So far, we have assumed that the Gaussian process prior distribution on f has been specified a priori.
• But this prior distribution itself has parameters, for example the length scale l, the output scale λ, and the noise variance σ2. As parameters of a prior distribution, we call these hyperparameters.
• For convenience, we will write θ to denote the vector of all hyperparameters of the model (including of μ and K).
• How do we learn θ?
Hyperparameters 35

Marginal likelihood
Assume we have chosen a parameterized prior
p(f | θ) = GP􏰀f; μ(x; θ), K(x, x′; θ)􏰁.
We will measure the quality of the fit to our training data D = (X, y) with the marginal likelihood, the probability of observing the given data under our prior:
p(y|X,θ)=􏰖 p(y|f)p(f|X,θ)df,
where we have marginalized the unknown function values f
(hence, marginal likelihood).
Hyperparameters 36

Marginal likelihood: Evaluating
Thankfully, this is an integral we can do analytically under the Gaussian noise assumption!
p(y | X,θ) = 􏰖􏰖 p(y | f)p(f | X,θ)df,
iid noise 􏰀 GP prior 􏰁
= N(y;f,σ2I)N f;μ(X;θ),K(X,X;θ) df
= N 􏰀y; μ(X; θ), K(X, X; θ) + σ2I􏰁. (Convolutions of two Gaussians are Gaussian.)
Hyperparameters 37

Marginal likelihood: Evaluating
The log-likelihood of our data under the chosen prior are then (writing V = (K(X, X; θ) + σ2I)):
logp(y | X,θ) =
data fit
Occam’s razor
−(y−μ)⊤V−1(y−μ)−logdetV −Nlog2π 222
The first term is large when the data fit the model well, and the second term is large when the volume of the prior covariance is small; that is, when the model is simpler.
Hyperparameters 38

Hyperparameters: Example
4 2 0
−2 −4
0 1 2 3 4 5 6 7 8 9 10
x
θ = (λ,l,σ) = (1,1,1/5), Hyperparameters
logp(y | X,θ) = −27.6 39

Hyperparameters: Example
4 2 0
−2 −4
0 1 2 3 4 5 6 7 8 9 10
x
θ = (λ,l,σ) = (2,1/3,1/20), Hyperparameters
logp(y | X,θ) = −46.5 40

Hyperparameters are important
Comparing the marginal likelihoods, we see that the observed data are over 100 million times more likely to have been generated by the first model rather than from the second model! Clearly hyperparameters can be quite important.
Hyperparameters 41

Can we marginalize hyperparameters?
To be fully Bayesian, we would choose a hyperprior over θ, p(θ), and marginalize the unknown hyperparameters when making predictions:
p(f∗ | x∗,D) = 􏰔 p(f∗ |􏰔x∗,D,θ)p(y | X,θ)p(θ)dθ p(y | X, θ)p(θ) dθ
Unfortunately, this integral cannot be resolved analytically. (Of course. . . )
Hyperparameters 42

Maximum likelihood-II
• Instead, if we believe the posterior distribution over θ to be well-concentrated (for example, if we have many training examples), we may approximate p(θ | D) with a delta distribution at the point with the maximum marginal likelihood:
θMLE =argmaxp(y|X,θ). θ
This is called maximum likelihood-II (ML-II) inference.
• This effectively gives the approximation
p(f∗ |x∗,D)≈p(f∗ |x∗,D,θMLE).
• How can we find θMLE?
Hyperparameters 43

Related Posts