COMP9444
Neural Networks and Deep Learning
Outline
9b. Generative Adversarial Networks
Artist-Critic Co-Evolution
Co-Evolution Paradigms
Blind Watchmaker (GP Artist, Human Critic)
Evolutonary Art (GP Artist, GP or NN Critic)
Generative Adversarial Networks (CNN Artist, CNN Critic)
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444 20T3 Generative Adversarial Networks
2
COMP9444 20T3 Generative Adversarial Networks 3
Artist-Critic Co-Evolution
Co-Evolution Paradigms
Artist
Critic
Biomorph Human GP Human CPPN Human CA Human
Blind Watchmaker Interactive Evolution PicBreeder
EvoEco
(Dawkins, 1986) (Sims, 1991) (Secretan, 2011) (Kowaliw, 2012)
Generated Images
Artist Critic
Method
Reference
Real Images
GP Agents GP
Criticisrewardedfordistinguishingrealimagesfromthosegenerated by the artist.
HERCL HERCL DCNN
Co-Evolving Line Drawings HERCL Function/CNN
(Vickers, 2017) (Soderlund, 2018)
Artist is rewarded for fooling the critic into thinking that generated images are real.
DCNN DCNN DCNN DCNN
Generative Adversarial Nets Plug & Play Generative Nets
(Goodfellow, 2014) (Nguyen, 2016)
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3
Generative Adversarial Networks 1
GP
SOM NN NN NN
Artificial Creativity Computational Aesthetics Evolutionary Art Aesthetic Learning
(Saunders, 2001) (Machado, 2008) (Greenfield, 2009) (Li & Hu, 2010)
HERCL
COMP9444 20T3 Generative Adversarial Networks 4
COMP9444 20T3 Generative Adversarial Networks
5
Blind Watchmaker (Dawkins, 1986)
Blind Watchmaker Biomorphs
the Human is presented with 15 images
the chosen image(s) are used to breed the next generation
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3 Generative Adversarial Networks 6
COMP9444 20T3 Generative Adversarial Networks
7
Interactive Evolution (Sims, 1991)
PicBreeder Examples
Artist = Genetic Program (GP)
◮ used as function to compute R,G,B values for each pixel x,y
Critic = Human
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3 Generative Adversarial Networks
8
COMP9444 20T3
Generative Adversarial Networks
9
PicBreeder (Secretan, 2011)
Image Generating Paradigms
Artist = Convolutional Pattern Producing Neural Network (CPPN)
Interactive Web site (picbreeder.org) where you can choose
existing individual and use it for further breeding
Interactive Evolution is cool, but it may require a lot of work from the Human – Can the Human be replaced by an automated Critic?
HERCL(draw) HERCL(func) HERCL(func)
GAN
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20 10/23
COMP9444 20T3 Generative Adversarial Networks
10
COMP9444 20T3
Generative Adversarial Networks
11
Generative Adversarial Networks
Generative Adversarial Networks
Generator (Artist) Gθ and Discriminator (Critic) Dψ are both Deep Convolutional Neural Networks.
Alternate between:
Gradient ascent on Discriminator:
Generator Gθ : z → x, with parameters θ, generates an image x from latent variables z (sampled from a standard Normal distribution).
maxEx∼ pdata log Dψ (x) + Ez∼ pmodel log1 − Dψ (Gθ (z)) ψ
Discriminator Dψ : x → Dψ(x) ∈ (0,1), with parameters ψ, takes an image x and estimates the probability of the image being real.
Gradient descent on Generator, using:
Generator and Discriminator play a 2-player zero-sum game to compute: min maxEx∼ pdata log Dψ (x) + Ez∼ pmodel log1 − Dψ (Gθ (z))
min Ez∼ pmodel log1 − Dψ (Gθ (z)) θ
θψ
Discriminator tries to maximize the bracketed expression, Generator tries to minimize it.
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
,
Biomorph GP Picbreeder
CA
COMP9444 20T3 Generative Adversarial Networks
12
COMP9444 20T3 Generative Adversarial Networks 13
Generative Adversarial Networks
Generative Adversarial Networks
Alternate between:
Gradient ascent on Discriminator:
GAN properties:
one network aims to produces the full range of images x,
maxEx∼ pdata log Dψ (x) + Ez∼ pmodel log1 − Dψ (Gθ (z)) ψ
with different values for the latent variables z
differentials are backpropagated through the Discriminator network
Gradient descent on Generator, using:
and into the Generator network
min Ez∼ pmodel log1 − Dψ (Gθ (z)) θ
compared to previous approaches, the images produced are much more realistic!
This formula puts too much emphasis on images that are correctly classified. Better to do gradient ascent on Generator, using:
max Ez∼ pmodel logDψ (Gθ (z)) θ
This puts more emphasis on the images that are wrongly classified.
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3 Generative Adversarial Networks
14
COMP9444 20T3
Generative Adversarial Networks 15
Generative Adversarial Networks
GAN Convolutional Architectures
repeat:
for k steps do
normalize images to between −1 and +1
replace pooling layers with:
(1) sample minibatch of m latent samples {z
(m) ,…,z
} from p(z) }
(1) update Discriminator by gradient ascent on ψ:
(m)
sample minibatch of m training items {x
,…,x
◮ strided convolutions (Discriminator)
◮ fractional-strided convolutions (Generator)
1m
∇ψ m ∑log Dψ(x(i)) + log1 − Dψ(Gθ(z(i)))
use BatchNorm in both Generator and Discriminator
i=1
sample minibatch of m latent samples {z
end for
update Generator by gradient ascent on θ:
remove fully connected hidden layers for deeper architectures
end repeat COMP9444
⃝c Alan Blair, 2017-20
⃝c Alan Blair, 2017-20
1m (i) ∇θm∑logDψ(Gθ(z ))
use tanh at output layer of Generator, ReLU activation in all other layers
i=1
use LeakyReLU activation for all layers of Discriminator COMP9444
(1)
(m) ,…,z
} from p(z)
COMP9444 20T3 Generative Adversarial Networks 16
COMP9444 20T3 Generative Adversarial Networks
17
Generator Architecture
GAN Generated Images
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (Radford et al., 2016)
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3 Generative Adversarial Networks
18
COMP9444 20T3 Generative Adversarial Networks
19
GAN Image Vector Arithmetic
GAN Image Vector Arithmetic
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3 Generative Adversarial Networks 20
COMP9444 20T3 Generative Adversarial Networks
21
Oscillation and Mode Collapse
The GAN Zoo
Like any coevolution, GANs can sometimes oscillate or get stuck in a mediocre stable state.
Contex-Encoder for Image Inpainting
Texture Synthesis with Patch-based GAN Conditional GAN
Text-to-Image Synthesis
StackGAN
Patch-based Discriminator
S2-GAN
Style-GAN
Plug-and-Play Generative Networks
oscillation: GAN trains for a long time, generating a variety of images, but quality fails to improve (compare IPD)
mode collapse: Generator produces only a small subset of the desired range of images, or converges to a single image (with minor variations)
Methods for avoiding mode collapse:
Conditioning Augmentation
Minibatch Features (Fitness Sharing) Unrolled GANs
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T3
Generative Adversarial Networks 22
GAN References
http://dl.ee.cuhk.edu.hk/slides/gan.pdf cs231n.stanford.edu/slides/2017/cs231n 2017 lecture13.pdf http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf https://arxiv.org/abs/1612.00005
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20