sig_ratio_metrics
CTR = $\frac{Total\ Clicks}{Total\ Views}$
Copyright By PowCoder代写 加微信 powcoder
Ratio of averages so usual Central Limit theorem does not apply!
We’ve detected a 5% difference between group A and group B using the data sim_ab_test_assignment.csv
sim_ab_test_clicks.csv
sim_ab_test_views.csv
We want to test the signifiance of this difference we observe!
We talked about using the permutation test, i.e. randomize the AB assignment and re-calculate the CTR difference.
Please write down the pseudo-code for what we want to achieve.
Small pseudo-code example¶
Pseudo-code is like draft code. The pseudo-code to demonstrate the Central Limit Theorem would look like:
Simulate and store B=1000 times the following generate n=50 random numbers
calculate average from n=50 numbers
Check the distribution of the B averages
Calculate observed CTR (clicks, views, assign)
Repeat the following (B = 1000) get permute assignment of users, perm_assign(assign)
Calculate CTR (clicks, views, perm_assign)
get difference of CTR(CTR_A, CTR_B)
Check differences vs observed CTR
Task 2 – Execute your code…in pieces!¶
Write a function that takes in your data, including the assignment vector, then outputs a difference of CTR between the 2 groups. You can assume there are only 2 groups.
import numpy as np
assign = np.loadtxt(“sim_ab_test_assignment.csv”, delimiter=”,”, dtype=str)
clicks = np.loadtxt(“sim_ab_test_clicks.csv”, delimiter=”,”)
views = np.loadtxt(“sim_ab_test_views.csv”, delimiter=”,”)
tot_clicks_user = np.apply_along_axis(np.sum, 1, clicks)
tot_views_user = np.apply_along_axis(np.sum, 1, views)
len(tot_clicks_user)
groupA = assign == “True”
groupB = assign == “False”
ctr_a = sum(tot_clicks_user[groupA]) / sum(tot_views_user[groupA])
ctr_b = sum(tot_clicks_user[groupB]) / sum(tot_views_user[groupB])
obs_ctr_diff = ctr_a – ctr_b
obs_ctr_diff
0.05303067971972465
def calc_ctr_diff(tot_clicks_user, tot_views_user, assign):
groupA = assign == “True”
groupB = assign == “False”
ctr_a = sum(tot_clicks_user[groupA]) / sum(tot_views_user[groupA])
ctr_b = sum(tot_clicks_user[groupB]) / sum(tot_views_user[groupB])
return ctr_a – ctr_b
calc_ctr_diff(tot_clicks_user, tot_views_user, assign)
0.05303067971972465
Task 3 – Randomization¶
Write some code that will randomdize your assignments. Hint: numpy.random.choice, test this out first!
Do we want to do this with or without replacement?
arr = assign
perm_assign = np.random.choice(arr, len(arr), replace=False)
(assign == perm_assign).all()
np.sum(assign == “True”)
np.sum(perm_assign == “True”)
Task 4 – Repeating the process¶
Repeat the randomization & calculation B=1000 times, storing your calculation in a variable called perm_diffs
What data type should perm_diffs be?
What is the mean of perm_diffs?
What is the SD of perm_diffs?
What percentage of perm_diffs values are below k SDs from the mean? Where k is [3, 2, 1, 0, -1, -2, -3] ?
Is B=1000 sufficient? (no code! just type out what would you do)
# list or dictionary or array
perm_diffs = []
for i in range(B):
perm_assign = np.random.choice(arr, len(arr), replace=False)
perm_diff = calc_ctr_diff(tot_clicks_user, tot_views_user, perm_assign)
perm_diffs.append(perm_diff)
# Don’t know, but I think it’ll be close to ?????
perm_diffs_mean = np.mean(perm_diffs)
perm_diffs_std = np.std(perm_diffs)
ks = [3, 2, 1, 0, -1, -2, -3]
for k in ks:
cutoff = k * perm_diffs_std
perc_below = np.mean(perm_diffs < (perm_diffs_mean - cutoff)) * 100
print(perc_below)
Task 5 - Getting a p-value¶
Please calculate a p-value using your original assignment and perm_diffs, i.e. what the chance of seeing our observed statistic or something more extreme?
Would you use one-sided or a two tailed test?
obs_ctr_diff
0.05303067971972465
np.sum(perm_diffs >= obs_ctr_diff)
np.sum(perm_diffs <= -obs_ctr_diff) np.mean(np.array(perm_diffs) < 0) Task 6 - Comparing the permutation test to the Delta Method¶ The delta method says that given: $\frac{1}{\sqrt{n}} \left(\begin{bmatrix} \bar{X} \\ \bar{Y}\end{bmatrix} - \begin{bmatrix} \theta_x \\ \theta_y \end{bmatrix}\right) \stackrel{n\to\inf}{\to} N(0, \begin{bmatrix}\sigma^2_x & \sigma_{x,y} \\ \sigma_{y, x} & \sigma^2_y \end{bmatrix})$ Then a smooth transformation $g$, then $\frac{1}{\sqrt{n}} \left(g\begin{pmatrix} \bar{X} \\ \bar{Y}\end{pmatrix} - g\begin{pmatrix} \theta_x \\ \theta_y \end{pmatrix}\right) \to N(0, \nabla g\begin{pmatrix} \theta_x \\ \theta_y \end{pmatrix}^T \begin{bmatrix}\sigma^2_x & \sigma_{x,y} \\ \sigma_{y, x} & \sigma^2_y \end{bmatrix} \nabla g\begin{pmatrix} \theta_x \\ \theta_y \end{pmatrix})$ In our case, $g\begin{pmatrix} \bar{X} \\ \bar{Y}\end{pmatrix} = \frac{\bar{X}}{\bar{Y}}$ $\nabla g\begin{pmatrix} \theta_x \\ \theta_Y \end{pmatrix} = \begin{bmatrix}\frac{1}{\theta_y} \\ -\frac{\theta_x}{\theta_y^2} \end{bmatrix}$ Questions: What is the p-value coming from the Delta method? Given we haven't covered scikit-learn which can tell us the p-value directly. Try simulating numbers from a normal distribution with the mean and SD you're deriving then compare this to your observed value (hint: numpy.random.normal) What are the pro/con of the permutation test vs the delta method? array([[ 7.07154254, 47.60544945], [ 47.60544945, 929.10992593]]) np.cov(tot_clicks_user) array(7.07154254) def calc_delta(tot_clicks, tot_views): x_bar = np.mean(tot_clicks) y_bar = np.mean(tot_views) grad = np.vstack([1 / y_bar, -x_bar / y_bar**2]) cov_xy = np.cov(np.vstack([tot_clicks, tot_views])) return grad.T.dot(cov_xy).dot(grad) ctr_A_var = calc_delta(tot_clicks_user[groupA], tot_views_user[groupA]) ctr_B_var = calc_delta(tot_clicks_user[groupB], tot_views_user[groupB]) $SE(A - B) = \sqrt{Var(A - B)} = \sqrt{Var(A) + Var(B) - 2Cov(A, B)} \stackrel{ind}{=} \sqrt{Var(A) + Var(B)}$ se_ctr_diff = np.sqrt(ctr_A_var + ctr_B_var) obs_ctr_diff 0.05303067971972465 array([[0.0012168]]) array([[0.00076187]]) obs_ctr_diff / se_ctr_diff array([[1.19217642]]) p-value won't be small!¶ Delta method depends on large sample sizes Permutation test can work with arbitrary dataset size The value is larger than 1 but laess than 2. We know this p-value is larger than 0.05.¶ To get the exact p-value we'll need to load libraries like scipy. Input In [1] To get the exact p-value we'll need to load libraries like `scipy`. SyntaxError: unterminated string literal (detected at line 2) 程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com