Beacon Conference of Undergraduate Research
Generative Model
Lingqiao Liu
University of Adelaide
Outlines
University of Adelaide 2
• What is generative model?
• Auto-regressive models
– Introduction
– Recurrent Neural Network and other form of networks
– Applications
• Generative Adversarial Network
– Basic GAN
– Other development
– Application of GAN
What is generative model
University of Adelaide 3
• Generative model: Machine learning models that can model
the generative process of data
• What can generative model do?
– Sampling a data sample
– [optional] Used as the probability density function: probability
evaluation
• Generative model vs. Discriminative model
– Model P(x) or P(x|y)
– Model P(y|x)
– Can be connected through Bayes theorem
Generative model vs. Discriminative Model
• Connection: Bayes theorem
• Remark
– If P(x|y) is perfectly modelled, P(y|x) derived from the Bayes
theorem is the optimal classifier
– In real world, discriminative model usually performs better than
generative model
University of Adelaide 4
How to perform sampling?
University of Adelaide 5
• Sampling: generate a random variable from a given
distribution
– Example, generate a number from distribution [0.1,0.6,0.3]
– Modern computer allows us generate from a limited number of
distributions
• Sampling from more complex distributions?
– Decompose the generation into a sequence of simpler decision
– Build a transforming function to convert an easy-to-sample random
variable (vector) to a data sample
Generative model we have learned
University of Adelaide 6
Gaussian mixture model
Generative model we have learned
University of Adelaide 7
• Generative process:
1. Randomly select the k-th Gaussian
distribution according to
2. Randomly sample x from the
selected Gaussian distribution
Generative model we have learned
University of Adelaide 8
• 1. Sample a topic distribution
• 2. From the distribution, sample a topic
• 3. From the topic, sample a word
• 4. repeat 2-3 to sample N words in a
document
Generating a document by using LDA
Topic model
Better model? Complex data?
University of Adelaide 9
• Challenges
– More general-purpose models
– Ability to model complex internal structures
• Deep generative models
– Make use of the powerful modelling capability of deep neural network
– Active research direction in Machine learning
• Generative Adversarial Network (GAN)
• Variational Autoencoder
• Flow-based methods
• Autoregressive model
Autoregressive Model
University of Adelaide 10
• Theoretical foundation
Chain rule to factorize the joint probability
The factorization order is not unique
Autoregressive Model
University of Adelaide 11
• Key idea: model
• Especially natural for model a sequence
• But also apply to nonsequential data, e.g. images
• essentially build a predictive model to predict next
time value based on the existing historical observations
• can be implemented in various ways
Recurrent neural network
University of Adelaide 12
• Recurrent neural network
– Avoid the need of processing all the historical observations for
each time step, but keep track of the state variables.
– For f, it takes the previous output as the input to the network
– Many possible implementations for f.
– In the simplest case, f is a MLP (Vanilla RNN).
– Model design of f involves mapping functions with gates
Recurrent neural network
University of Adelaide 13
• Example: (Long short-term memory) LSTM
Recurrent neural network
University of Adelaide 14
• Example: LSTM
• Gates enable more flexible
state update in RNN
– Input gate: ability to ignore the
current input
– Forget gate: ability to reset the
state
Training auto-regressive model
• Most straightforward approach:
– The input is with different length
– The input needs to be sequentially processed by RNN
– Need a lot of storages to store all the training samples
– A lot of redundancy calculation
University of Adelaide 15
Construct pairs
A more efficient formulation: Sequence-to-
Sequence mapping
University of Adelaide 16
RNN RNN RNN RNN…
Prediction targets:
Generation by auto-regressive models
University of Adelaide 17
RNN
Generation by auto-regressive models
University of Adelaide 18
RNN RNN
Generation by auto-regressive models
University of Adelaide 19
RNN RNN RNN …
Example: generating music via RNN
• Monophonic music generation
– Use RNN to predict the next note based on previous notes
– To generate, start with a small amount of initial note, the model
will generate the next note which will be used as the input again
University of Adelaide 20
Other ways to implement Autoregressive Model
University of Adelaide 21
• Wavenet, Pixel CNN:
– Use convolutional neural network with special design
– Demo
Other ways to implement Autoregressive Model
University of Adelaide 22
• Transformer
– Use attention-based predictive function
Demo
• Input sentence:
I went to a lounge to celebrate my birthday and
• Generated sentences:
I called Donna and told her I had just adopted her. She thought my
disclosure was a donation, but I’m not sure if Donna met the criteria.
Donna was a genuinely sweet, talented woman who put her life here
as a love story. I know she thanked me because I saw her from the
photo gallery and she appreciated my outrage. It was most definitely
not a gift. I appreciate that I was letting her care about kids, and that
she saw something in me. I also didn’t have much choice but to let her
know about her new engagement, although this doesn’t mean I wasn’t
concerned, I am extremely thankful for all that she’s done to this
country. When I saw it, I said, “Why haven’t you become like Betty or
Linda?” “It’s our country’s baby and I can’t take this decision lightly.”
“But don’t tell me you’re too impatient.” Donna wept and hugged me.
She never expresses milk, otherwise I’d think sorry for her but
sometimes they immediately see how much it’s meant to her. She
apologized publicly and raised flagrant error of judgment in front of
the society of hard choices to act which is appalling and didn’t grant
my request for a birth certificate. Donna was highly emotional. …
University of Adelaide 23
Generative Adversarial Networks (GANs)
University of Adelaide 24
• Referring to GANs, Facebook’s AI research director Yann
LeCun called adversarial training “the most interesting
idea in the last 10 years in ML.”
• Do not directly modelling P(x) (but implicitly). It is
based on a simple intuition: if the generation is perfectly
done, you can not tell the difference between the real and
the generated samples
• Based on two modules: a generator and a discriminator
– Generator tries to fool the discriminator
– Discriminator tries to identify the difference
https://www.quora.com/What-are-some-recent-and-potentially-upcoming-breakthroughs-in-deep-learning
Generative Adversarial Network
University of Adelaide 25
Updating both the Generator and discriminator until reach to an equilibrium
Generative Adversarial Network
University of Adelaide 26
Generative Adversarial Network
University of Adelaide 27
Randomly choose samples from the training set
Generate a random vector from an easy-to-sample distribution
A binary classifier
Generative Adversarial Network
University of Adelaide 28
• Loss function
Ideally, D(z) gives a close-to-one value if z is a true data sample,
otherwise, D(z) gives a close-to-zero value
Generative Adversarial Network
University of Adelaide 29
• Loss function
Ideally, D(z) gives a close-to-one value if z is a true data sample,
otherwise, D(z) gives a close-to-zero value
For a given G, find the best D to tell the difference.
The quality of the discrimination is measured by max V(D,G)
Generative Adversarial Network
University of Adelaide 30
• Loss function
Ideally, D(z) gives a close-to-one value if z is a true data sample,
otherwise, D(z) gives a close-to-zero value
Find best G that can fool the best discriminator
Generative Adversarial Network
University of Adelaide 31
V(D,G): fix G, optimize D
V(D,G): fix D, optimize G
Generative Adversarial Network
University of Adelaide 32
Generative Adversarial Network
University of Adelaide 33
• Very active research field
– Many improved GANs are proposed
GAN: Application
University of Adelaide 34
• Generator can be modified as a translator
• For example, image super-resolution
High resolution
Image
Low resolution
Image
Traditional loss:
Problem: images achieves low loss values may not necessarily have authentic look
GAN: Application
University of Adelaide 35
• Using GAN
– Build a discriminator D to discriminate recovered high-
resolution images and real high-resolution images
– Train the super-resolution model to fool the discriminator
GAN: applications
University of Adelaide 36
Result