COMP9517: Computer Vision
Applications (Part III)
Week 9 COMP9517 2021 T3 1
• Neural Architecture Search (NAS) for Cell Segmentation
• Generative Adversarial Networks (GAN) for Image Inpainting
• Style Transfer with Deep Neural Networks
Week 9 COMP9517 2021 T3 2
NAS for Cell Segmentation
Recap: several CNNs can be used for achieving cell segmentation
• Manually designing is time-consuming and labor-intensive
• Designing a network with excellent performance requires professional
knowledge and expertise
Week 9 COMP9517 2021 T3 3
NAS for Cell Segmentation
Goal: automatically design a deep neural network for a given task.
Neural Architecture Search (NAS)
• A subfield of automated machine learning
• Search space: defines which architectures can be searched
• Search strategy: details how to explore the search space
• Performance evaluation: estimates the performance of architectures
COMP9517 2021 T3 4
NAS for Cell Segmentation
Neural Architecture Search (NAS)
Search space: defines which architectures can be searched
The backbone architecture of the outer network
The selection of basic operations, e.g., convolution, depthwise-separable convolution, max-pooling, average-pooling, etc.
UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation, Zhou et al, 2019
Figure from U-Net: Convolutional Networks for Biomedical Image Segmentation, Ronneberger et al, 2015
COMP9517 2021 T3 5
NAS for Cell Segmentation
Neural Architecture Search (NAS)
• Search strategy: details how to explore the search space
• Evolutionary algorithms [1]
• Reinforcement learning based methods [2]
• Gradient-based methods [3]
[1] Lu, Zhichao, et al. “Nsga-net: neural architecture search using multi-objective genetic algorithm.” Proceedings of the Genetic and Evolutionary Computation Conference. 2019.
[2] Cai, Han, et al. “Efficient architecture search by network transformation.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.
[3] Brock, Andrew, et al. “Smash: one-shot model architecture search through hypernetworks.” arXiv preprint arXiv:1708.05344 (2017).
Week 9 COMP9517 2021 T3 6
NAS for Cell Segmentation
Neural Architecture Search (NAS)
• Performance evaluation: estimates the performance of the architectures
COMP9517 2021 T3 7
Estimate the performance of the candidate architectures to select an optimal architecture that achieves high predictive performance
Evaluate the optimal architecture for final performance
NAS for Cell Segmentation
Search Space
• Backbone architecture
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 8
NAS for Cell Segmentation
Search Space
• Define four basic operation (BO) sets
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 9
NAS for Cell Segmentation
Search Space
• Define the fundamental computing unit (CU)
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 10
NAS for Cell Segmentation
Search Space
• Basic block structure
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 11
NAS for Cell Segmentation
Search Strategy
• Two classes of parameters
• Gradient-based method
Performance estimation strategy
• Evaluate the performance of the architecture candidates without using a standard training and validation process to reduce the computation cost.
COMP9517 2021 T3 12
NAS for Cell Segmentation
Architecture of the searched network
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 13
NAS for Cell Segmentation
Architecture of the searched network
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 14
NAS for Cell Segmentation
Performance of the searched network
Figure from Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search, Zhu et al, 2021 Week 9 COMP9517 2021 T3 15
GAN for Image Inpainting
Image inpainting refers to filling in the missing parts of images such that the image is both visually and semantically consistent.
Figure from High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis, Yang et al, 2017
Week 9 COMP9517 2021 T3 16
GAN for Image Inpainting
How we the humans do the image inpainting?
Figure from Context Encoders: Feature Learning by Inpainting, Pathak et al. 2016
Week 9 COMP9517 2021 T3 17
GAN for Image Inpainting
Recap: GAN
Image from https://wiki.pathmind.com/generative-adversarial-network-gan
Week 9 COMP9517 2021 T3 18
GAN for Image Inpainting
Underlying hypothesis
If the generator is efficient in its representation, then an image that is not from the data distribution should not lie on the learned encoding manifold.
Image from Semantic Image Inpainting with Deep Generative Models, Yeh et al, 2017
COMP9517 2021 T3 19
GAN for Image Inpainting
GAN based framework for inpainting
Image from Semantic Image Inpainting with Deep Generative Models, Yeh et al, 2017
Week 9 COMP9517 2021 T3 20
GAN for Image Inpainting
Inpainting results
Image from Semantic Image Inpainting with Deep Generative Models, Yeh et al, 2017
Week 9 COMP9517 2021 T3 21
GAN for Image Inpainting
Inpainting results
Image from Semantic Image Inpainting with Deep Generative Models, Yeh et al, 2017
Week 9 COMP9517 2021 T3 22
GAN for Image Inpainting
Inpainting results
Image from Semantic Image Inpainting with Deep Generative Models, Yeh et al, 2017
Week 9 COMP9517 2021 T3 23
Style Transfer with Deep Neural Networks
Style transfer is the technique of blending style from one image into another image keeping its content intact.
Image from Image Style Transfer Using Convolutional Neural Networks, Gatys et al, 2016
Week 9 COMP9517 2021 T3 24
Style Transfer with Deep Neural Networks
How does it work?
• It deals with two sets of images: content image and style image
• Its aims is to recreate the content image in the style of the reference style image
• It employs a pre-trained CNN to differentiate between the style representations
and content representations
• It works by activating the neurons of the neural networks in a particular way, such that the output image and the content image should match particularly in the content, whereas the style image and the desired output image should match in style characteristics.
COMP9517 2021 T3 25
Style Transfer with Deep Neural Networks
How does it work?
Image from Image Style Transfer Using Convolutional Neural Networks, Gatys et al, 2016
Week 9 COMP9517 2021 T3 26
Style Transfer with Deep Neural Networks
Style transfer algorithm
Image from Image Style Transfer Using Convolutional Neural Networks, Gatys et al, 2016
Week 9 COMP9517 2021 T3 27
Style Transfer with Deep Neural Networks
Style transfer results
Image from Image Style Transfer Using Convolutional Neural Networks, Gatys et al, 2016
Week 9 COMP9517 2021 T3 28
Style Transfer with Deep Neural Networks
Style transfer results
Image from Image Style Transfer Using Convolutional Neural Networks, Gatys et al, 2016
Week 9 COMP9517 2021 T3 29
Any Questions?
Week 9 COMP9517 2021 T3 30