Probability and Random Processes: A R efres her
COMP5416
Advanced Network Technologies
Basic Concepts
• Experiment
• Outcome
• Sample space • Event
• Probability
School of Information Technologies
COMP5416 Probability – 2
Probability Axioms
• E vents A,B defined as any subsets of – 0
– Pr{
– If A and B are mutually exclusive,
Pr{ A B} Pr{ A} Pr{B}
School of Information Technologies
COMP5416 Probability – 3
Basic Probability Laws
• Laws that follow directly from axioms: – Pr{A} 1 Pr{A}
– Pr{A B} Pr{A} Pr{B} Pr{A B}
– If A and B are mutually exclusive, Pr{A B} 0
School of Information Technologies
COMP5416 Probability – 4
Conditional Probability
• Probability of event A when it is known that event B has occurred:
Pr{A|B} Pr{A B} Pr{B}
• If A,B are independent, then
Pr{A | B} Pr{A}, Pr{B | A} Pr{B}
• Events A,B are independent if Pr{A B} Pr{A}Pr{B}
School of Information Technologies
COMP5416 Probability – 5
Law of Total Probability
•𝑃𝐴=σ𝑁 𝑃𝐴𝐵𝑃(𝐵) 𝑖=1 𝑖 𝑖
• 𝐵 , 𝑖 = 1,2, … 𝑁 a set of pairwise disjoint events 𝑖
whose union is the entire sample space
Law of Total Probability
𝑃𝐴𝐵 1
A
𝑃 𝐴 𝐵2 𝑃 𝐴 𝐵3 𝑃 𝐴 𝐵𝑁
B1 B2 B3 … BN
𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 ) 123𝑁
Law of Total Probability
A
B1 B2 B3 … BN
𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 ) 123𝑁
Law of Total Probability
A
𝑃𝐴𝐵 1
B1 B2 B3 … BN
𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 ) 123𝑁
Law of Total Probability
A
𝑃𝐴𝐵 1
𝑃 𝐴 𝐵2
B1 B2 B3 … BN
𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 ) 123𝑁
Law of Total Probability
𝑃𝐴𝐵 1
A
𝑃 𝐴 𝐵2 𝑃 𝐴 𝐵3
B1 B2 B3 … BN
𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 ) 123𝑁
Law of Total Probability
𝑃𝐴𝐵 1
A
𝑃 𝐴 𝐵2 𝑃 𝐴 𝐵3 𝑃 𝐴 𝐵𝑁
B1 B2 B3 … BN 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 )
123𝑁
A outcome
Bi possible reason i
Law of Total Probability
• Suppose that two factories supply light bulbs to the market. Factory X’s bulbs work for over 5000 hours in 99% of cases, whereas factory Y’s bulbs work for over 5000 hours in 95% of cases. It is known that factory X supplies 60% of the total bulbs available. What is the chance that a purchased bulb will work for longer than 5000 hours?
• 0.99*0.6+0.95*0.4
Bayes’ theorem
𝑃 𝐴 𝐵 𝑃(𝐵 )
•𝑃𝐵𝐴=
𝑖
𝑖
𝑖 σ𝑁 𝑃𝐴𝐵𝑃(𝐵)
𝑖=1 𝑖
𝑖
Bayes’ theorem
𝑃𝐴𝐵 1
Now that A happens, what is the reason? Is it because B1 B2 or BN
𝑃 𝐴 𝐵2
A
𝑃 𝐴 𝐵3
𝑃 𝐴 𝐵𝑁
B1 B2 B3 … BN 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵) 𝑃(𝐵 )
123𝑁
𝑃 𝐵𝑖 𝐴 : Given A happens, the probability that 𝐵𝑖 is the reason
A outcome
Bi possible reason i
Example
• The entire output of a factory is produced on three machines. The three machines account for different amounts of the factory output, namely 20%, 30%, and 50%. The fraction of defective items produced is this: for the first machine, 5%; for the second machine, 3%; for the third machine, 1%. If an item is chosen at random from the total output and is found to be defective, what is the probability that it was produced by the third machine?
P(produced by 3) P(defective | produced by 3)
0.50* 0.01
0.2*0.05+0.3*0.03+0.5*0.01
P(produced by 1) P(defective | produced by 1) +P(produced by 2) P(defective | produced by 2) +P(produced by 3) P(defective | produced by 3)
Example
• The entire output of a factory is produced on three machines. The three machines account for different amounts of the factory output, namely 99%, 0.9%, and 0.1%. The fraction of defective items produced is this: for the first machine, 1%; for the second machine, 1%; for the third machine, 100%. If an item is chosen at random from the total output and is found to be defective, what is the probability that it was produced by the third machine?
0.001* 1
0.99*0.01+0.009*0.01+0.001* 1 Around 9%
Example
• The entire output of a factory is produced on three machines. The three machines account for different amounts of the factory output, namely 99%, 0.9%, and 0.1%. The fraction of defective items produced is this: for the first machine, 1%; for the second machine, 1%; for the third machine, 100%. If an item is chosen at random from the total output and is found to be defective, what is the probability that it was produced by the third machine?
Machine 3 produces 100% defective. But 9% defective products are from 3. Why?
Machine 3 produces 0.1% of the products! Both 𝑃 𝐴 𝐵𝑖 and 𝑃(𝐵𝑖) contribute!
R andom Variables
• RV: Mapping from outcome to real number • Discrete or continuous
School of Information Technologies COMP5416 Probability – 6
Dis crete random variables
• Events can take only discrete values, e.g. integer values
• P robability function pk or p(k)=Pr{X=k}
Cumulative distribution function F(x) = Pr{X
123456 value
• Probabilities sum to 1.0
1.0 pk Mean kp
k
Variance k mean 2 pk
School of Information Technologies COMP5416 Probability – 7
Example – Roll a dice
• Discrete values 1,2,3,4,5,6 • Each with probability 1/6
k 1
Variance (k 3.5)2 (
1 1 2 3 4 5 6 21
6
Mean k(6) 6 6 3.5
61
) 10.20833 Standard Deviation var iance 3.195048
k 1
6
School of Information Technologies
COMP5416 Probability – 8
E xample – B ernoulli random variable
• Discrete values
•1 with probability p,
• 0 with probability 1-p
p 1-p
Mean p
Variance (1 p) p
0
1
value
School of Information Technologies
COMP5416 Probability – 9
Example – Poisson random variable
• Discrete values 0,1,2,…,infinity • Value k has probability pk, with
pk e k
k! Mean
Variance
0 1 2 3 4 5 6 7 value
School of Information Technologies COMP5416 Probability – 10
2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
• Probabilities integrate to 1.0
Continuous random variables
• Events can take real values on arbitrary range • Continuous RV: probability density function
p(x) dF(x),whereF(x) x f(y)dy dx -
1.0 p(x)dx Mean xp(x)dx
Variance x mean p(x)dx
School of Information Technologies COMP5416 Probability – 11
E xample – exponential random
variable
Probability density function, negative exponential, =2
• Parameter
2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
p(x) e
x
Pr X x 1 e x
Mean
1
Probability distribution function, negative exponential, =2
Variance ( )2
1
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1 Standard Deviation
School of Information Technologies
COMP5416 Probability – 12
Multiple R andom Variables
• X,Y are random variables
• Joint CDF of X,Y: F(x,y)=Pr{X x,Y y}
– if X,Y discrete: joint probability function P(x,y)=Pr{X=x,Y=y}
– if X,Y continuous: joint density function
F(x,y p(x, y) )
x y
• X,Y are independent if F (x,y)=F (x)F (y)
2
School of Information Technologies
COMP5416 Probability – 13
S ome handy properties
• For any random variable X, constants a,b: – E[aX+b]=aE[X]+b
– Var[aX+b]=a2Var[X]
• For any random variables X,Y: – E[X +Y ]=E[X ]+E[Y ]
• For any independent variables X,Y: – E[XY]=E[X]E[Y]
– Var[X +Y ]=Var[X ]+Var[Y ]
School of Information Technologies
COMP5416 Probability – 14
Correlation and covariance
• Covariance: Cov[X,Y]=E[(X-E[X])(Y-E[Y])]=E[XY]- E[X ]E[Y ]
– compare with variance: Var[X]=E[(X-E[X])2]=E[X2]-E[X]2 – can be positive, negative, or zero Cov(X,Y)
•Correlationcoefficient: r(X,Y) x y -1
• X,Y are called uncorrelated if r(X,Y)=0
– if X,Y are independent, then they are uncorrelated
– if X,Y are uncorrelated, they may still not be independent
School of Information Technologies COMP5416 Probability – 15
Practice
Q1. (Review of Probability A). X and Y are two independent random variables. X is a uniformly distributed in [0, 3], and Y is uniformly distributed in [3, 6]. Find the probability density function (PDF) of X + Y.
Q2. (Review of Probability B). T is a random variable that follows exponential distribution. The probability density function of T is
𝑓(𝑡) =ቊ 0,𝑡<0 𝜆𝑒−𝜆𝑡, otherwise
Prove that P(T > a + b|T > a) = P(T > b)