CS计算机代考程序代写 matlab chain algorithm [06-30213][06-30241][06-25024]

[06-30213][06-30241][06-25024]
Computer Vision and Imaging &
Robot Vision
Dr Hyung Jin Chang Dr Yixing Gao
h.j.chang@bham.ac.uk y.gao.8@bham.ac.uk
School of Computer Science

TEXTURE (SZELISKI 10.5)

Today: Texture
What defines a texture?
3

Includes: more regular patterns
Slide credit: Kristen Grauman
4

Includes: more random patterns
Slide credit: Kristen Grauman
5

Texture-related tasks • Shape from texture
– Estimate surface orientation or shape from image texture
Slide credit: Kristen Grauman
6

Shape from texture
• Use deformation of texture from point to point to estimate surface shape
Pics from A. Loh: http://www.csse.uwa.edu.au/~angie/phdpics1.html
Slide credit: Kristen Grauman
7

Texture-related tasks
• Shape from texture
– Estimate surface orientation or shape from
image texture
• Classification/segmentation from texture cues – Analyze, represent texture
– Group image regions with consistent texture
• Synthesis
– Generate new texture patches/images given some examples
Slide credit: Kristen Grauman
8

Why analyze texture? Importance to perception:
• Often indicative of a material’s properties
Slide credit: Kristen Grauman
9

10

Why analyze texture?
Importance to perception:
• Often indicative of a material’s properties
• Can be important appearance cue, especially if shape is similar across objects
Slide credit: Kristen Grauman
11

Slide credit: Kristen Grauman
12

13

Why analyze texture?
Importance to perception:
• Often indicative of a material’s properties
• Can be important appearance cue, especially if shape is similar across objects
• Aim to distinguish between boundaries and texture
Slide credit: Kristen Grauman
14

Slide credit: Kristen Grauman
http://animals.nationalgeographic.com/
15

Why analyze texture?
Importance to perception:
• Often indicative of a material’s properties
• Can be important appearance cue, especially if shape is similar across objects
• Aim to distinguish between boundaries and texture
Technically:
• Representation-wise, we want a feature one step above “building blocks” of filters, edges.
Slide credit: Kristen Grauman
16

Texture representation
• Textures are made up of repeated local
patterns, so:
– Find the patterns
• Use filters that look like patterns (spots, bars, raw patches…)
• Consider magnitude of response
– Describe their statistics within each local window
• Mean, standard deviation
• Histogram of “prototypical” feature occurrences
Slide credit: Kristen Grauman
17

Texture representation: example
mean d/dx value
mean d/dy value
Win. #1
4
10
original image
Slide credit: Kristen Grauman
derivative filter responses, squared
statistics to summarize patterns in small windows
18

Texture representation: example
mean d/dx value
mean d/dy value
Win. #1
4
10
Win.#2
18
7
original image
Slide credit: Kristen Grauman
derivative filter responses, squared
statistics to summarize patterns in small windows
19

Texture representation: example
mean d/dx value
mean d/dy value
Win. #1
4
10
Win.#2
18
7
Win.#9
20
20
original image
Slide credit: Kristen Grauman
derivative filter responses, squared
statistics to summarize patterns in small windows
20

Texture representation: example
mean d/dx value
mean d/dy value
Win. #1
4
10
Win.#2
18
7
Win.#9
20
20
Slide credit: Kristen Grauman
statistics to summarize patterns in small windows
21
Dimension 1 (mean d/dx value)
Dimension 2 (mean d/dy value)

Texture representation: example
Windows with primarily horizontal edges
Both
mean d/dx value
mean d/dy value
Win. #1
4
10
Win.#2
18
7
Win.#9
20
20
Dimension 1 (mean d/dx value)
Windows with small gradient in both directions
Slide credit: Kristen Grauman
Windows with primarily vertical edges
statistics to summarize patterns in small windows
22
Dimension 2 (mean d/dy value)

Texture representation: example
visualization of the assignment to texture “types”
original image
Slide credit: Kristen Grauman
derivative filter responses, squared
23

Texture representation: example
mean d/dx value
mean d/dy value
Win. #1
4
10
Win.#2
18
7
Win.#9
20
20
Far: dissimilar textures
Close: similar textures
Dimension 1 (mean d/dx value)
Slide credit: Kristen Grauman
statistics to summarize patterns in small windows
24
Dimension 2 (mean d/dy value)

Texture representation: example
a b
Dimension 1
Slide credit: Kristen Grauman
25
Dimension 2

Texture representation: example
a b
Dimension 1
Distance reveals how dissimilar texture from window a is from texture in window b.
a b
b
26
Slide credit: Kristen Grauman
Dimension 2

Texture representation:
window scale
• The window size (i.e., scale) for which we collect these statistics is important.
Possible to perform scale selection by looking for window scale where texture description not changing.
27

Filter banks
• Our previous example used two filters, and resulted in a 2-dimensional feature vector to describe texture in a window
– x and y derivatives revealed something about local structure
• We can generalize to apply a collection of multiple (d) filters: a “filter bank”
• Then our feature vectors will be d-dimensional – still can think of nearness, farness in feature space
Slide credit: Kristen Grauman
28

Filter banks
scales
orientations
“Edges”
“Spots”
“Bars”
• What filters to put in the bank?
– Typically we want a combination of scales and orientations, different types of patterns.
Matlab code available for these examples:
http://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html
Slide credit: Kristen Grauman
29

Slide credit: Kristen Grauman
Multivariate Gaussian
105 90 
  
=  = 16 0
= 
09
09



30
55

CNN Learning = Learning Filters Banks
31

Slide credit: Kristen Grauman
Filter bank
32

Slide credit: Kristen Grauman
33
Image from http://www.texasexplorer.com/austincap2.jpg

Slide credit: Kristen Grauman
Showing magnitude of responses
34

Slide credit: Kristen Grauman
35

Slide credit: Kristen Grauman
36

Slide credit: Kristen Grauman
37

Slide credit: Kristen Grauman
38

Slide credit: Kristen Grauman
39

Slide credit: Kristen Grauman
40

Slide credit: Kristen Grauman
41

Slide credit: Kristen Grauman
42

Slide credit: Kristen Grauman
43

Slide credit: Kristen Grauman
44

Slide credit: Kristen Grauman
45

Slide credit: Kristen Grauman
46

Slide credit: Kristen Grauman
47

Slide credit: Kristen Grauman
48

Slide credit: Kristen Grauman
49

Slide credit: Kristen Grauman
50

Slide credit: Kristen Grauman
51

You try: Can you match the texture to the response?
Filters
A
1
2
B
3
Slide credit: Derek Hoiem
Mean abs responses
52
C

Representing texture by mean abs response
Filters
Slide credit: Derek Hoiem
Mean abs responses
53

[r1, r2, …, r38]
We can form a feature vector from the list of responses at each pixel.
Slide credit: Kristen Grauman
54

d-dimensional features
Euclidean distance (L2)
D(a,b)= d (a−b)2
ii i=1
. ..
2d
Slide credit: Kristen Grauman
3d
55

Example uses of texture in vision: analysis
56

Classifying materials, “stuff”
Figure by Varma & Zisserman
57

Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121, November 2000,
58
Slide credit: Kristen Grauman
Texture features for image retrieval

Characterizing scene categories by texture
Slide credit: Kristen Grauman
59
L. W. Renninger and J. Malik. When is scene identification just texture recognition? Vision Research 44 (2004) 2301–2311

Texture-related tasks
• Shape from texture
– Estimate surface orientation or shape from
image texture
• Segmentation/classification from texture cues
– Analyze, represent texture
– Group image regions with consistent texture
• Synthesis
– Generate new texture patches/images given some examples
Slide credit: Kristen Grauman
60

Texture synthesis
• Goal: create new samples of a given texture
• Many applications: virtual environments, hole- filling, texturing surfaces
Slide credit: Kristen Grauman
61

The Challenge
• Need to model the whole spectrum: from repeated to stochastic texture
Alexei A. Efros and Thomas K. Leung, “Texture Synthesis by Non-parametric Sampling,” Proc. International Conference on Computer Vision (ICCV), 1999.
repeated
stochastic
Both?
62

Markov Chains Markov Chain
• a sequence of random variables
• is the state of the model at time t
• Markov assumption: each state is dependent only on the previous one
– dependency given by a conditional probability:
• The above is actually a first-order Markov chain
• An N’th-order Markov chain:
63
Source S. Seitz

Markov Chain Example: Text
“A dog is a man’s best friend. It’s a dog eat dog world out there.”
a
dog is
man’s best
friend it’s eat
world out
there
.
a.
p(dog|a) = ?
64
Slide credit: Adapted by Devi Parikh from Steve Seitz
there
out
world eat
it’s friend best man’s is
dog

Markov Chain Example: Text
“A dog is a man’s best friend. It’s a dog eat dog world out there.”
a
dog is
man’s best
friend it’s eat
world out
there
.
a.
2/3
1/3
1/3
1/3
1/3
1
1
1
1
1
1
1
1
1
1
65
Slide credit: Steve Seitz
there
out
world eat
it’s friend best man’s is
dog

Text synthesis
Create plausible looking poetry, love letters, term papers, etc.
Most basic algorithm
1. Build probability histogram/table
– find all blocks of N consecutive words/letters in training documents
– compute probability of occurrence
2. Givenwords
– compute by sampling from
WE NEED TO EAT CAKE
Slide credit: Steve Seitz
66

• Results:
Text synthesis
– “As I’ve commented before, really relating to someone involves standing next to impossible.”
– “One morning I shot an elephant in my arms and kissed him.”
– “I spent an interesting evening recently with a grain of salt”
Dewdney, “A potpourri of programmed prose and prosody” Scientific American, 1989.
Slide from Alyosha Efros, ICCV 1999
67

Text synthesis with LSTM • Results:
– “Albus Dumbledore, I should, do you? But he doesn’t want to adding the thing that you are at Hogwarts, so we can run and get more than one else, you see you, Harry.”
– “I know I don’t think I’ll be here in my bed!” said Ron, looking up at the owners of the Dursleys.
– “Well, you can’t be the baby way?” said Harry. “He was a great Beater, he didn’t want to ask for more time.”
– “What about this thing, you shouldn’t,” Harry said to Ron and Hermione. “I have no furious test,” said Hermione in a small voice.
https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/ 68

Markov Random Field
A Markov random field (MRF)
• generalization of Markov chains to two or more dimensions.
First-order MRF:
• probability that pixel X takes a certain value given the values
of neighbors A, B, C, and D:
A
D
X
Slide credit: Steve Seitz
C
B
69

Texture Synthesis [Efros & Leung, ICCV 99] Can apply 2D version of text synthesis
Texture corpus (sample)
Output
Slide from Alyosha Efros, ICCV 1999
70

Texture synthesis: intuition
Before, we inserted the next word based on existing nearby words…
Now we want to insert pixel intensities based on existing nearby pixel values.
Place we want to insert next
Sample of the texture (“corpus”)
Distribution of a value of a pixel is conditioned
71
on its neighbors alone. Slide credit: Kristen Grauman

• •

Find all the windows in the image that match the neighborhood
To synthesize x
– pick one matching window at random
– assign x to be the center pixel of that window
An exact neighbourhood match might not be present, so find the
best matches using SSD error and randomly choose between them,
preferring better matches with higher probability
Synthesizing One Pixel
Slide from Alyosha Efros, ICCV 1999
input image
p
synthesized image
• Whatis ?
72

Neighborhood Window
input
Slide from Alyosha Efros, ICCV 1999
73

Varying Window Size
Increasing window size Slide from Alyosha Efros, ICCV 1999
74

Synthesis results french canvas rafia weave
Slide from Alyosha Efros, ICCV 1999
75

Synthesis results white bread brick wall
Slide from Alyosha Efros, ICCV 1999
76

Synthesis results
Slide from Alyosha Efros, ICCV 1999
77

Failure Cases
Slide from Alyosha Efros, ICCV 1999
Growing garbage Verbatim copying
78

Hole Filling
Slide from Alyosha Efros, ICCV 1999
79

Extrapolation
Slide from Alyosha Efros, ICCV 1999
80

Hole Filling
https://www.nvidia.com/research/inpainting/
81

• The Efros & Leung algorithm – Simple
– Surprisingly good results
– Synthesis is easier than analysis! – …but very slow
82

Image Quilting [Efros & Freeman 2001]
non-parametric sampling
B
p
Synthesizing a block
• Observation: neighbor pixels are highly correlated
Idea: unit of synthesis = block
• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
Input image
Slide credit: Alyosha Efros
83

Input texture
block
B1
B2
B1
B2
B1
B2
Random placement of blocks
Neighboring blocks constrained by overlap
Minimal error boundary cut
Slide credit: Alyosha Efros
84

Minimal error boundary
overlapping blocks
vertical boundary
_
2
=
Slide credit: Alyosha Efros
overlap error
min. error boundary
85

Slide credit: Alyosha Efros
86

Slide credit: Alyosha Efros
87

Slide credit: Alyosha Efros
88

Slide credit: Alyosha Efros
89

Slide credit: Alyosha Efros
90

Failures
(Chernobyl Harvest)
Slide credit: Alyosha Efros
91

Texture Transfer
• Take the texture from one object and “paint” it onto another object
– This requires separating texture and shape
– That’s HARD, but we can cheat
– Assume we can capture shape by boundary and rough shading

similarity to underlying image at that spot
Then, just add another constraint when sampling:
92
Slide credit: Alyosha Efros

parmesan
+=
rice
+=
Slide credit: Alyosha Efros
93

+=
Slide credit: Alyosha Efros
94

+
=
Slide credit: Alyosha Efros
95

Gatys et al., CVPR 2016
96

Gatys et al., CVPR 2016
97

Style Transfer
https://genekogan.com/works/style-transfer/

98

Tamar et al. ICCV 2019
Slide credit: Dongmin Choi
99

Slide credit: Dongmin Choi
100
Tamar et al. ICCV 2019

Slide credit: Dongmin Choi
101
Tamar et al. ICCV 2019

Slide credit: Dongmin Choi
102
Tamar et al. ICCV 2019

Slide credit: Dongmin Choi
103
Tamar et al. ICCV 2019

Tamar et al. ICCV 2019

104

Summary
• Texture is a useful property that is often indicative of materials, appearance cues
• Texture representations attempt to summarize repeating patterns of local structure
• Filter banks useful to measure variety of structures in local neighborhood
– Feature spaces can be multi-dimensional
• Neighborhood statistics can be exploited to
“sample” or synthesize new texture regions – Example-based technique
105