Economics 430
Sampling Distributions and Limits
1
Today’s Class
• Limits
– Convergence in Probability
– Weak Law of Large Numbers
– Convergence in Probability 1
– Strong Law of Large Numbers
– Convergence in Distribution
– Central Limit Theorem
• Monte Carlo Approximations
• Normal Distribution Theory
– Chi-Squared Distribution – t-Distribution
– F-Distribution
• Sampling Distributions
Almost Sure Mean Convergence Convergence
Convergence in Probability
Convergence in Distribution
2
Sampling Distributions
Population
Sample 1
Sample 2 Sample 3
Parameter: 𝜇
Sampling Distribution of Sample Means
Statistic 1
Statistic 2 Statistic 3
3
Sampling Distributions
Anatomy of a Basic Sampling Algorithm
for i= 1:k
1.
2.
3.
end
Choose a sample size n, and a statistic (e.g., mean, proportion, etc.)
Randomly draw a sample from the population with the same size
Estimate the statistic from the sample
4
Sampling Distributions
Def: Sampling Variability
= The variability among random samples from the same population.
Def: Sampling Distribution
= A probability distribution that characterizes some aspect of sampling
variability.
= The distribution of a statistic given repeated sampling.
A sampling distribution tells us how close the resemblance between the sample and population is likely to be
Goal: We use probability theory to derive a distribution for a statistic, which allows us to make statistical inferences about population parameters.
5
Sampling Distributions
Suppose 𝑋1, 𝑋2, … , 𝑋𝑛 is a sample from some distribution, 𝑓 is a function of this sequence, and we want to estimate the distribution of the r.v., 𝑌 =
𝑓 𝑋1, 𝑋2, … , 𝑋𝑛 , i.e., 𝑌’s ‘sampling distribution’. Q: How can we obtain the distribution of 𝑌?
A: Analytically (works well when 𝑛 is small) or Approximately (through sampling)
6
Sampling Distributions
• Example: Given a sample 𝑋1, 𝑋2 of size 𝑛 = 2, with p.m.f. 𝑝𝑋, and 𝑌2 = 𝑋1𝑋2 ,/., find the distribution of 𝑌2.
Note: Y2 is known as the Geometric Average, which is widely used in finance.
• Solution: We can compute it analytically
7
Sampling Distributions
• Application: Multiple-Period Realized Return
Suppose you invest $1000 at the beginning of year 1, and get a 12% return at the end this year, and then get 8% at the end of year 2. What was your ‘average’ return over the two years?
Option 1: Arithmetic Average: (12+8)/2 = 10% Q: Is this correct? No
Q: Why is this the wrong answer? A: Does not take into account interest over interest.
– If you start with $1000, then at the end of 2 years you have
$1000 (1.12)(1.08) = $1209.6.
– The amount of money you would get if you invested $1000 at 10%/year for two
years is 1000*(1.1)2 = $1210. They are not the same!
Option 2: Geometric Average: [(1+0.12)(1+0.08)]1/2 – 1 = 0.09982=9.982%
The geometric average gives us the correct per annum return: 1000(1.09982)2=$1209.6.
8
Multiple-Period Realized Return Arithmetic vs. Geometric Average
• Example:TimeSeriesofHPRfortheS&P500 ×
Period
HPR (decimal)
Gross HPR =1+HPR
2001
-0.1189
0.8811
2002
-0.2210
0.7790
2003
0.2869
1.2869
2004
+
0.1088
1.1088
2005
1.0491
0.0491
• • •
×
Arithmetic Average:
= 1/5(-0.1189+ … +0.0491)
= 0.0210
= 2.10%
Geometric Average:
= (0.8811 × … × 1.0491)1/5 -1
= 0.0054
= 0.54%
The larger the swings in rates of return, the greater the discrepancy between the arithmetic and geometric averages.
9
Multiple-Period Realized Return Arithmetic vs. Geometric Average
• Example:S&P500annualreturnsfrom2000-2010
• Q:Giventhetwoaverages,ifyouhadinvested $10,000 in the S&P500 from 2000-2010, how much would you have in your account?
– R={-9.2, -11.9, -22.1, 28.7, 10.9, 4.9, 15.8, 5.5, -37.0, 26.5, 15.1}
• Arithmetic Avg = 2.47%, $10,247.00
• Reality?
Geometric Avg = 0.41%
correct amount
$10,041.00
After adjusting for inflation àArithmetic ($9,994), Geometric ($9,798)
10
Multiple-Period Realized Return Arithmetic vs. Geometric Average 3 of 3
Concluding Remarks
• Volatilitylowersinvestmentreturns.
• Mostreturnsarereportedasanarithmeticaverage because this is the highest average that can be reported.
– Arithmetic averages are accurate only if there is no volatility.
• Carefulwithe.g.,mutualfundandhedgefund managers! –They usually quote arithmetic averages.
11
Sampling Distributions
• Example: Given a sample 𝑋1, 𝑋2, … , 𝑋20 of size 𝑛 = 𝟐𝟎, with p.m.f. 𝑝𝑋, and 𝑌2 = (𝑋1 ⋯ 𝑋20 )1/2, find the distribution of 𝑌2.
• Note: We cannot compute it analytically! Even computationally it would be difficult because of the large number of operations.
Q: How do we find 𝑌2? A: Approximation:
12
Sampling Distributions
Density (Data) N(mu, sigma)
0.2 0.3 0.4 0.5 0.6 0.7 0.8 lnY
13
Density 0123456
Limits
Convergence in Probability
A sequence of random variables 𝑋,, 𝑋., 𝑋5, … 𝑋6 converges in probability to a random variable 𝑋, i.e., if:
Example: Let 𝑋6~ 𝐸𝑥𝑝 𝑛 , show that . That is, the sequence 𝑋,, 𝑋., 𝑋5, … 𝑋6 converges in probability to the zero random variable 𝑋.
àProof:
14
Limits
Convergence in Probability
(Textbook Example)
15
Limits
Weak Law of Large Numbers (WLLN)
Let 𝑋,, 𝑋., 𝑋5, … 𝑋6 be i.i.d random variables with a finite expectedvalue𝐸 𝑋: = 𝜇< ∞.Thenforany𝜀> 0,
Note: A common notation is 𝑀6 = 𝑋.
The WLLN is also known as Bernoulli’s Theorem. In simpler terms, it states that: “the mean of the results obtained from a large number of trials is close to the population mean”.
16
Limits
Weak Law of Large Numbers (WLLN)
Simulation of the Weak (and Strong) Law of Large Numbers
Binomial (𝜇=0.4, n=1000)
0 200 400 600 800 1000 Number of Trials
17
Sample mean
0.20 0.25 0.30 0.35 0.40
Limits
Weak Law of Large Numbers (WLLN)
Example (4.2.12): Generate i.i.d. 𝑋, , . . . , 𝑋6 distributed Exponential(5) and compute 𝑀6 when 𝑛 = 20. Repeat this N times, where N is large (if possible, take 𝑁 = 10B, otherwise as large as is feasible), and compute the proportion of values of 𝑀6 that lie between 0.19 and 0.21. Repeat this with 𝑛 = 50.
àSince 𝑋 ~ 𝐸𝑥𝑝 5 , 𝜇 = 1/5 = 0.2. Therefore, as n increases, the proportion of values that lie between 0.19 and 0.21 should increase.
0 200 400
600 800
18
n
Proportion of values of Mn that lie in [0.19, 0.21]
0.2 0.4 0.6 0.8
Limits
Convergence in Probability 1 (“almost surely”)
A sequence of random variables 𝑋,, 𝑋., … 𝑋6 converges almost surely to a random variable 𝑋, i.e., , if
Example: A fair coin is tossed once. The sample space is S={H,T}. We define a sequence of random variables 𝑋,, 𝑋., … 𝑋6 on this sample space according to:
à
(does not converge)
𝑋6 converges if S=H but not if s=T, but since the probability that 𝑋6 converges to X is equal to 1 as𝑛 →∞,then
19
Limits
Strong Law of Large Numbers
Let 𝑋,, 𝑋., … 𝑋6 be i.i.d., random variables each having a finite
m e a n 𝐸 𝑋 :
= 𝜇 < ∞ , a n d 1.00
0.75
0.50
0.25
EF G⋯GEH 6
a.s.
, t h e n M n ! μ .
Cumulative Average
0.00
P(Heads) with Fair Coin = 0.50
102 103
100 101
104
105
20
𝑀 6 =
Cumulative mean = 0.50272 mu = 0.5
Sample size = 1e+05
n (number of tosses)
Limits
Strong Law of Large Numbers
Application: Business Growth
According to the LLN as additional units are added to a sample, the average of
the sample converges to the average of the population.
àThe LLN implies that the more a company grows, the harder it is for the company to sustain that percentage of growth.
Company A: $100M (Market Cap), year 1 growth rate = 50%.
• A 50% growth rate for the next 6 years would grow the company from $100M
to $1.2B.
Company B: $20M (Market Cap) year 1 growth rate = 50%.
• A 50% growth rate for the next 6 years would grow the company from $20M
to $228M
Which company has more room for growth (stock appreciation)?
21
Limits
Convergence in Distribution
A sequence of random variables 𝑋,, 𝑋., ... 𝑋6 converges in distribution to a random variable 𝑋, i.e., d , if lim F (x) = F (x)
X n ! X n ! 1 X n X for all 𝑥 at which 𝐹E(𝑥) is continuous.
Example: Let 𝑋,, 𝑋., ... 𝑋6 be a sequence of r.v’s such that
8<:1 1 1 nx x>0 n
FXn (x) =
Show that 𝑋6 converges in distribution to Exp(1).
0 otherwise
22
Limits
Convergence in Distribution Solution:For𝑥≤0,𝐹EH 𝑥 =𝐹E(𝑥)=0,for𝑛=2,3,…
✓ ✓ 1◆nx◆= lim 1 e x =F (x), 8x
For𝑥≥0, lim FXn(x)= lim 1 1 n n!1 n!1 n!1
X
d
) X n ! X
n=2 n=4 n=8 n=10
0 20 40 60 80 100 x
23
0.0 0.2 0.4
0.6 0.8 1.0
F
Limits
Central Limit Theorem
Let𝑋,,𝑋.,…𝑋6bei.i.dr.v’swith𝐸𝑋: =𝜇<∞and 0<𝑉𝑎𝑟𝑋: =𝜎.<∞.Then,ther.v.
X μ X1 +X +2+···+Xn nμ Zn = /pn = pn
converges in distribution to the Standard Normal (𝑁 0,1 ) random variable, Φ 𝑥 as 𝑛 → ∞, i.e.,
lim P(Zn x)= (x), 8x2IR n!1
Note: Φ 𝑥 is the standard normal cdf.
24
Limits
Central Limit Theorem
Example: n= 1000 samples from a Unif[0, 1]
Distribution of One Sample
Sampling Distribution of the Sum
Distribution of z−scores
The distribution
of z-scores ~ 𝑁 0,1
0.0 0.2 0.4 0.6 0.8 1.0
my.samples[1, ]
Sampling Distribution of the Mean
4900 4950 5000 5050 5100 all.sample.sums
Sampling Distribution of the Variance
−4 −2 0 2 4
0.490 0.495 0.500 0.505 0.510 all.sample.means
0.081
0.083 0.085 all.sample.vars
z−Scores
25
Frequency 0 500 1500
Frequency
Frequency 0 500 1500
2500
0
500
Frequency
1500 2500
0
500
1000
1500
2000
2500
0
100
300 500
Frequency
Limits
26
Limits
27
Limits
• Application:InvestorsofalltypesrelyontheCLTto analyze stock returns, construct portfolios and manage risk.
• Example:Portfoliowith3ETFsallequallyweighted
DEO 33%
33% BJK
33%
MJ
28
Limits
From Jan 1st, 2016 to September 1st, 2018
Sampled Portfolio Returns
Sampled Portfolio Returns
−0.004 −0.002 0.000 Returns
Daily
0.002
0.004
−0.4
−0.2
0.0
0.2 0.4 0.6 0.8 1.0 Returns
Annual
Note: See Portfolio.R
29
Normalized Frequency
0 50 100 150 200 250 300
Normalized Frequency
0.0 0.5 1.0 1.5 2.0 2.5
Monte Carlo Approximations
• Q:If 𝑋isar.vwhere𝑋~𝑓 𝑋 then𝑔 𝑋 isalsoa r.v., but what is the probability distribution of 𝑔(𝑋)?
Example: Let 𝑋 ~ 𝑓(𝑋) = Unif[1,4] and Y=g(X) =(X-3)^2, find 𝐸[𝑌].
Step 1: 𝑝(𝑋) = 1/4 ∀ 𝑋 values in {1,2,3,4}
Step 2: 𝑔(𝑋)
X = 1àg(1) = 4 X = 2àg(2) = 1 X = 3àg(3) = 0
X=4àg(4)=1
Step3: 𝑃(𝑔(𝑋)) = 𝑃(𝑌) 𝑌 = 0,1,4
𝑃(𝑌=0) = 1/4,𝑃(𝑌=1) =1/2,𝑃(𝑌=4) = 1/4
Step4:𝐸[𝑔(𝑋)] = ∑𝑌𝑃 𝑌
= 0(1/4) + 1(1/2) + 4(1/4)
= 3/2
30
Monte Carlo Approximations
• Q:Whatifwedon’t’know𝑃(𝑌)?Doweneedit?
• Example:Let𝑋~𝑓(𝑋) =Unif[1,4]and𝑌 = 𝑔(𝑋)=(𝑋−3)2,
find 𝐸[𝑌].
E[Y]=E[g(X)]=∑YP(Y) = ∑g(X)PX(X)
= (1-3)2 P(X=1) + (2-3)2 P(X=2) + (3-3)2 P(X=3) + (4-3)2 P(X=4)
= 4 (1/4) + 1(1/4) +0(1/4) + 1(1/4)
= 3/2àsame as before without knowing P(Y)J
Conclusion: We do not need to know P(Y) in order to compute E[Y] provided we know P(X).
= Foundation of MC Approximations!
31
Monte Carlo Approximations
Application: Approximating Integrals
• Howcanweestimate: ? à
Choose the points x at random between a and b, evaluate f(x)àcompute the area A = l × w
àà
(Converges in probability) ( by the LLN)
32
Algorithmic Implementation of Sequential Bayesian Learning
Application: Approximating Integrals Generalization (arbitrary PDF)
Since for any PDF p but
then .
à(by the LLN) as
33
Monte Carlo Approximations
• Example: Approximate the integral:
Let X1, X2, ..., Xn = i.i.d r.v.s. ~ unif[0,1].àF1 = f(X1), ..., FN = f(XN) are i.i.d.
r.v’s with mean ⟨FN⟩ and E[⟨FN⟩] = F = 𝜋
Monte Carlo Integration
MC Approximation True Value
0 200 400
600 800 1000
34
N
For N =1000 àE[⟨FN⟩] = 3.15
𝜋For N =105 àE[⟨FN⟩] = 3.14
2.6 2.8 3.0
3.2 3.4 3.6 3.8
F
Monte Carlo Approximations
• Example: Value at Risk
Given a current stock price (of your choice), what is the lowest it can go (worst-
case scenario) 20 days from now with 99% probability of occurrence?
• Model: Geometric Brownian Motionà
– Standard model for stock price movements Assumptions:
MC Approximation VaR =$66.28
S0 = $100
𝜇 = 0.1 (10% drift)
𝜎 = 0.2 (20% volatility) 𝜀 ~unif[0,1]
VaR = $66.28
0 200
400 600 800 1000 Number of Simulations
35
Stock Price ($)
60 80 100 120 140 160 180 200
Normal Distribution Theory
Normal Distribution N(𝜇,𝜎)
• Suppose 𝑋: ∼ 𝑁 𝜇,𝜎:. , for 𝑖 = 1,2,...,𝑛 and that they are independent
r.v’s. Let 𝑌 = a ∑: 𝑎:𝑋:` + 𝑏!for some constants {𝑎:} and 𝑏. Then
Y⇠N Xaiμi+b,Xa2i i2 ii
Applications: finance models (e.g., mean variance analysis), business, economics, science, engineering, etc.
2
PDF: f(x)= p1 e 1(x μ)2, 1
where, μ=⌫, 2 =2⌫,M(t)=(1 2t) ⌫/2
37
χ2-Distribution Examples
1. Find the value of χ25 such that the area on the right tail is equal to 0.05.
Solution: df = 5, and α = 0.05 àχ25 = 11.070
38
χ2-Distribution Examples
2. Find the value of χ23 that represents the 97.5th percentile.
Solution: df = 3, and α = 0.025 àχ23 = 9.348
39
χ2-Distribution
Examples
3. Given X~ χ24, find P(0.484 < X < 7.779). Solution: df = 4.
àP(0.484 < X < 7.779) = 0.975-0.100 = 0.874
40
Normal Distribution Theory
t-Distribution
• The t-distribution with n degrees of freedom (or Student-t), is
the distribution of the random variable
Z=p X
(X12 +···Xn2)/n
where X1, X2, ..., Xn are i.i.d., each with the standard normal
distribution 𝑁 0,1 . Equivalently 𝑍 = E , where 𝑌 ∼ 𝜒.(𝜈). i/6
41
Normal Distribution Theory
t-Distribution
• The t-distribution with n degrees of freedom (or Student-t), is the
distribution of the random variable:
where X1, X2, ..., Xn are i.i.d., each with the standard normal distribution
Z=p X
(X12 +···Xn2)/n
𝑁 0,1 . Equivalently 𝑍 = E , where 𝑌 ∼ 𝜒.(𝜈). i/6
Applications: small sample sizes (n≾ 30), and/or limited information
(⌫+1) ✓
PDF:f(t)= p 2 2⇡ (⌫ )
1+
t2◆ (⌫+1)/2 ⌫
, 1
42
t-Distribution Examples
1. If T follows a t4 distribution, find t0 such that: P(|T|
1
F1 ↵,⌫1 ,⌫2 =
F↵,⌫2 ,⌫1
50