ILA WEAK Supervision
t Iea SourceSeparation funproblem
Weak Supervision
Bef
WE’RE past
midterm lots more material than you need More fun happyto chat
ICI INDEPENDENT Component Analysis high level story
Keyfrets likelihood morsel
cocktnlpaeyprobhe.ru IN Hw PEOPLE_ Microphones
o
qyjim.ee
SPEAKER S Sz S
DATA X
jmic2J
NB WE SEE A Mixture I At EACH m
Sz
W XL time
Intensityf
gq
IS INTENSITY AT TimEf from DEALER
WE DO Not Observe It Only XA themicrophones
AySf
microphone L SEES A mixtureof S t salt M LATENT
model
ages
a
01 OBSERVED
XA Aset LATENT
for simplicity Assume Given X X
D find S
we call A the mixing matrix AND W A unmixing matrix
WRITE W wYo So that Sj Wg d
of SPEAKERS d
of mics d ofmicrophones
d is
AND A C R d sit X’t Ask
X E R Sen ERd
speakers
SomeCaveatse
WE ASSUME A does not vary w time AND Is fullRANK
t
WO sNIVyl s There ARE INHERENT Ambiguity
WE CAN’T DETERMINE SPEAKER II cold swab 142J CAN’T DETERMINE AbsoluteIntensity
CA c SCH Asct forAny Cfo Speakers CANNIT be Gaussian
X N Nlr AAT then if UTU _I AU generates the SALE data
Nevertheless
Algorithm Just MLE SOLVED By GRAD Descent
suppose so
DETOUR C
TEMPTED
PDF of s
Density under lineartransform KegConfusion
Spring
WE CAN Recover something meaningful
Sw Uniform o D
0 25 Wurst Ps X
es PDF of 0
PDF of u
I
to
write
Pu E
PsCx7
THE Key Issue
if x cCoD OW
Pulx BLE
Is the NORMALIZATION constant
FOR INVERTIBLE MATRIX A U AS
CHANGEofWAR formula for
Integrals
detca
Pv x
PsA x IdetCAII
Psl Wx ldetcxdlf.de D
FROMHEREICAISMLE
PCs YET Ps Sj
sources are Inorseeendent AND HAVE Same distribution
RECAI
Workhorse dimensionality Reduction ICA Keyideas for Hw Introduce upto symmetry
PIX
y
USE Non Rotatonlle Invariant distribution SET PsCx 0Cgtx for gac I te TI
solve Lfw ye log g Cay x’t’s logIdeal
detail
USE GD you’re done
B W x detCw use lineartnanstonaAxe How written IN terms of X AND A o
keytechxncnlb.tn
log
Saw PCA
ExtraPuLx Ps A x IdetCA II SKIP In lecture Happy to
fixRectangle U ERd let Au y Ax y XcU RECORD SimplifiesLINEARCHANGE ofvariablesforintegrals
p e a e S o f a s t
LET P BEA Partiton AU Let Xp beAPOINT INSIDE A BsaBB
e p f K B V o Cl B
go.EE.p.int
A Le B b owns s a BoorBB fix dx I B pflingvolCB volts is thevolume
REcall AS Partlow gets finer f x VoCl B 0 then Arx getscloser to Integral
Now let’s Examine Sau f A y dy fix A Partitow al U
EVERY PARTITION al U makes A DARTINON al Au CA is fullRANK
Kegfact
IdETCA delta I
fC
BEP
volcDB
fao fatty Vol AB
dy
BEP FCA’Axy volCAB volCB
1 detail fo f x DX
SO WE Have sown Jfcay dy IdetCAl fuf x dx
Au
Recalf WE WANTED to figure out Normalization for PgCAtx IN terms of Puck i.e PsLA x c Po
the Above says C DETCA T delta as elaimers take fix Ps x
WE
Cto slides for aeroiew Simple Estimation trick
Inverse covariance GraphStructure
Xi Rd 1,130 Assassin yeE1,13
GN
X
Supeeusio Nuggets INDEP CASE
Correlations
X x cRd
Xm
Ci find Plyl X X
IDEAL Ai Is A Noisy function InaccurateIncomplete
AI
Tz
Programmarelabels No labels
Model 0 NoABsrains INDEPENDENT Errors Clear Correlates
Name In dictionary Orren case woroo
Given
Dates
UNOBSERVED
U 1
i
xIIIl l
PlyIXx
EACH larder has a Lia y Ix X AND y example
withprob P Tgx y Tj is right 1P Ax y Tisunoy
sadly
E XiY
we don’t SEE y but we x
Xi Aj’s Errors INDEPENDENT AND Symmetric inthefellowssea
PLAtx IlyDPIXx ilyi
Observe if Xn k Asgee Value is 1
Ai n Y disagree value is I
i
Pj
Pi1t lPill
2Pi I Eai definegothisway AiCCI D
Etix Iifij
E kik141 Rpg1 tCiPilaIj agree
GPiPjti tPiIt H disagree Aiaj
NOTE we didn’t use 14 1 same treefor 14 1 g kit I 4 37Ply b a.az gpcII
LEGIT
WEformAmatrixME112mm MigEkik
didn’tneed toknow bj
M CAN BE Estimated unlike Y
Agreements AND disagreements Keynoter Don’t need To SEE Y
NI
SimpleAlgorithm i
Mi
for any i g K A.ia.ae
distinct
a
Mi5Msu Mia 14
Mix
SO WE CAN Solve upto
al ai Note If we knew signCa s
then Min Sigalm SlgN a
So a a ARE solutions
A Ak signCae
Ayak
Assume WHATifMig_o
in Ai 20
CAN save for all signsfronow breaks symmetry good ow average
on Ajo
RECAP
2Pi 1 0 Pi I Ransomnoise Have Cverify Ai Bounded Away fromRANDOM Noise
CAN HANDLE w fancier tricks
Theory says let 8 min lPg 1 NEED samples Proponicoll
f1am
to clotsofword Symmetry Simple Algebraic leases gooeald useCBD
ai o this means PQi o
sign
WHatifanecorreearers
Nugget Xc
structureof INVERSE Covariance for Gaussians
X
o
KeqConcept Probaselly distribution on Graphs E kik14 Effi14 ECTly
ef Ij EfEdge
k of4 Ty o
As
Above
o 1 2 Xz
X n Nlo i
XzNX7 EzNO1XzXE
Is
X n NCEn G NcoD X x 1ExDo ECxdE tE o
E
ECT I ECxE7 E Cx 5 IELx.TT
o
2 EChB II it exits I
E XXz7
ECxf x e I
fxfEijtELEEd
IIes ldis
No aeae structure
Ef
WE SAY ftp.oba dstnbuhou p Rd o D
noeose.EE
graph G Pv xi
Now let’s look Gaussian over a graph
CU E if
for some functions
agrees on factorizes wnt A
pix CIT Pe Ivi e
Normalization Constant
Xiii
log exp
let A for hiy E E
XT x lo TPelxi.xju.TLuCx x
7CIsnoneDx
tEY
Ki
t
Paix LAI t Aji o it Ci j EE
Pu
eEeiePelogPeCxixj E for Easy notation
log
4J Ai Xing
A is symmetric so Aij o
thus if A Gaussian factors on A graph Entriesof INVERSE Area
But
More complex theory for discrete fives 104 Wainwright 014
BAEKTOOORPNOBLEM
RATNERef al
2018
fon.in
couananee matrix
a_E
I
1
Y
El 2,33
e EE
Is unobserved
1
f visibleterms
Lef O
Eo Zo
u
on those ENTRIES
some rank
1 vector
Bu 2
Inverse let D Eo
Bw’TB Bout’BT 1 UTBj
soEo Ejt2ZT Newif CijCfethenwe
Eis Bij 0
Hence
j ij
ZEE
Bij5 ZE zf
logBig logE lose this is a linearsystem in Zf Zf AND WE CAN
Solve
oyh
PAIRWISEINDEO
INNES
Higher rank VEeswws Hanble more correlations How to lead graphStructure
How to Handle Sampling Error modern me lowerbonds
RELAP
WEAKSupervision formaltheory
Nuggets About Graphs Prob distributors the graphene mocked Method ofmoments style Algorithms