CS计算机代考程序代写 ER algorithm ada Ensemble

Ensemble
Learning

Outline ensembles
Why What
choices How
Applications

Why
ensembles
Two heads
are
better
than one 个臭皮匠 胜过诸葛亮

Why
to to
Simple Easy
use
Yet Many
understand powerful
very SOTA
are
ensembles
No over filling empirically
Preferred
choice
for many

Asimpledemozr Available M individual classifiers
IG
Ensemble

ID IGu
eg.by v
x
Majority
x
x
Vote
X
o

Is
Testing results
If Icz
IS
voting
Case
X
Accuracy
x23 43

100
t Betrwnnrnn
x
ˇ

V voting
V
IS
Case I Testing results
Accuracy
x23
43 2b
2分 t
No
竟⻅太 一 到
IS Icz
x X
x
difnce

Case II Testing results
Accuracy S
xx
X V x lb
It Iz
xxt voting x X X
t Worse.nu
三个 样臭臭上加臭
Ig
S O
v

Desirable Each IC is
IC is
OK random guessing
也太剖
Better than
Diverse
Each has
造有优缺 取⻓补短
its
own
Heterogeneous
skills

What
choices

Parallel
Inputs
IC
I
IG
Bagging RF LinearRegressiohassifaim
Sequential

output
output
eg
IC

Inputs
AdaBoost
e.g
ICs
different
be all
may Similar
tree stumps
sunt Logistics 十 CART

Where to
for
friends

me
回幽
parallel friends
Ǘ
Weighting
Example
A group of
go
dinner

䞺匪毥
Boy Girl irl
sequential

How
focus
sequential
on

Boostingze
Boost Freund st Schapire
Ada
1996,1997 GBMIGBR
Extensions SGB
boost light GBM
xg

AdaBoost
Freunds.IS chapire
The
in the
1996,1997
best of the shelf classifier
world
一一
Breiman NearIPS 1996

Hierarchical structure Gnsgnymhn
G Esnì
Miwkighted
wkighted sample
G
sample G2 Esi
sample Esnìwkighted

Traiug sample
G
Esi

InputfEhTyiYiliinIyiEu.g
Initialize
Fit a classifierGm
obs M

n
weights For m 1
wi
O 二 点wi
wilt weights wi frrcrerrm I 付 㳤 Gúnnwi
Compute
to
Ìuiiyi

Gm
三ui D
g 造器 wiexpdmiyi
Compute 人 Update
Gm GgsgnxmGm版
Wi Output

Gm e.g
1 1 can be any classifier
Tree
FAN
Height71.8
四與回
weak classifier
betterthan random guessing
Remarkszn
E
Stump

More weights
For

uiwiexpdniyi W i x 全器
Wi
In next
to
weak classifier
mis classified point
l
e.mn
to log
if errm 主 I

then
so
weights less weights
iii
to to
more
round
Gm
yit Gm yi二Gm
mis classified point
classified
point

Data
Classifier IN
G
_i

X
x

Classifio 2
Gin
s
合 ㄨ
Find classier
3
hi
Class ifio 3
G
二 SggxmGm外
G
M 1
quality ofGm

Stubborn
If some points keep ding
even
mis classified more weights are assigned
of A NO
overfnngp surprising
It
IQ
danger

Extension from bing
0 yi
Same
to multi
iyD.rinn K
class
E
1,2
procedure
with
last step
Gong
max
Ěxmliy ydik

Biasvshriancel

Inputs Tree stump

output
low
is
variance
bias feature only
error use one
high
stable learner Boosting
slow
but
Gradually reducing wilt out
bias increasing variance

Statistical View
Boost
of
Ada

Nhisn
m
Let
fn

Gil fintm

三点
f
Gin
Additive

Model t

Tliliiid
AdaBoost
Stagewise AddieModelling

Forward
with
fgkeyf
exponential LG
A
loss
ˊ

Algorithm
I Initialize f
0
Er ward Stagewise Additive Modeling
2 For m 1 M_ Compute
him
egitreestunyiig fitpcj
Update ix
貳年
二 点十PmGG利

PROOF.si
For
Gnn
AdaBoost the basis functions are
Under lBm.Gmka min
G
Efl loss
argminnci
1
LlyfaDeMrf
exp
Èl.li
缈 Èacpfgexpfpyi缈
fit Èapfltp 缈
argmin
G

uiepfy off
argmin
G
wixexpfpyfg
N
where
free
G

2
stageq
stim.cn
find Gm
Fix
Given Gm find
f
Pm

First stage.ee
Now
Èuinpfpyi缈
嚪 非Glxi 可 扶Glxi Ěǐ
Ewiiftwt
巨 wig yEGGD

e

are
iteii.it
yīGlxi ēijuitei

yīGlxi
地形 獻
free of G

Hence Giargǎniniuiexpfyg
argǎninětn
argǎniny
for
fixed
B
argǎnin fee of
Enid
BJ

Secoudstagezfiargmin Èuinpfpcgpargjninli
aětbé
gipibdaé eialb
主 ln 台
Let
g

p
0

So
ytlxim fjh
win

fd
挑的
贏 喔i t
泛去
焦 win
揪划
in 主Xm


A

Update Update
fcx
W
fptpm
GG from
飛飛铡
十 Bmj
wiacpffm 生與 rrihrn
拍哪
uiapfniyiaěm
uipfniyix
w
eapfyi 追

uiapffmfz

Finally f二点
BmGm
sgnlfg
Sgf
sgnfzfmGm
sgnfxmGm

Variants of Ada Boost
Statistical be Her
BGtension
Boosting Reg
GBMIGBR
xg boost lightGBM
view understanding
Many
Varian

Boosting Regression

Data
Model Y
0
t
避到
ing continuous
LE
Lyf 火
fxt
Eii
i
Loss
fxf
Boosting reg.cn
regression residuals
sequentially

Why
幽吵
Lyfmn
ersrr
和刪二感斷同
7
min BG

ii_ 衇 明
Ǘhyim
residual in last round
甽 共遁網那衇明
Ldihii


豳州 一 㗀
11
Where

Choice
GGD tree
of Glxi D
Boostedg
basis regression
a
simple
reg Reg
Glx
Glxi
H
Boosted Linear
linear
Neural
Nets CNN Boosted NN
8

Boosted Reg Tree
regress in
tree
Gait
Glxi
8


Boostedg

Parameter M Hyperparameter
of trees cross_validation
8 d
d
7 tee depth tree stumps
no interactions interactions
learning ratj
l
d1
d
7 g
1
too big too small
might not是苡恐 too slow I
E38 10

Initialize
fdx E
0
Algorithmic
For
tree
fxef it
Output
m 17Gnn
m1
Data hi 啊
G
in
with d leaves
yifg yi M
xd
M
f
7Gm rinan

Simulation
Y
Xi Uníf 0,1 Eine NG 12
Sin 二 500
2而不 Ein
t

Boosted
M y
Reg
Tree
d2 tree
1
stumps 10,102,103
1

BoostlinearRegress
H y
a.GG
xtpgt
simplelinearreg
i.e one
feature only
P
Reg
Boosted Linear Il
Forwardsteplzvg Gradient version is be Her
Eiijdl