BU.450.760 Technical Document T4.1 – Personalization experiment Prof.
R Analysis of the email personalization experiment
Here we cover the analysis of the email personalization experiment. This experiment is reported by Sahni et al. (2018).1 The dataset and codebook are in D4.1 and C4.1, respectively.
The experiment was run with the cooperation of an anonymous company. The company sells online and offline training programs for preparation of standard tests like the Chartered Financial Analyst (CFA) and the Certified Public Accountant (CPA). The company’s main target market comprises working professionals looking to improve their skills by taking certification tests. Their products are expensive; they are priced in the order of $1,000.
Copyright By PowCoder代写 加微信 powcoder
A total of 68,088 email IDs were randomized into the following two conditions:
• Control group (condition=0). Recipients in this group were sent emails in the typical format used by the company. The names of the recipients were not mentioned in the
subject line.
• Treatment group (condition=1). Recipients were sent emails with their names added to
the subject line. Specifically, the name was appended to the beginning of the subject. For example, suppose the subject line in a control email to a person named was “Learn Financial Modeling from Industry Experts.” Then the subject line for the corresponding treatment group was “Jack, Learn Financial Modeling from Industry Experts.”
The only difference between the emails received by the treatment and control group individuals was that the subject line mentioned the recipient’s name in the treatment group but not the control group. The rest of the marketing campaign –the number of emails, other email design characteristics, etc.—remained exactly the same for the two conditions.
The dataset contains 3 outcomes that we want to check: whether a customer opens the email, whether the customer generates a lead for the company, and whether the customer unsubscribes from the listserv.
1. Preliminary steps
Load data, declare factors, etc (lines 5-16). Important note: we will be estimating the models using the full dataset—training/validation data splits are no longer used.
2. Check balancing
Here we check whether there are signs that treatment/control group allocation was not random. Recall that we do this is through standardized differences, treatment versus control group. In class we covered what are the accepted thresholds to determine whether a standardized difference should be taken as evidence of problematic imbalance.
Sahni, N. S., Wheeler, S. C., & Chintagunta, P. (2018). Personalization in email marketing: The role of
noninformative advertising content. Marketing Science, 37(2), 236-258.
BU.450.760 Technical Document T4.1 – Personalization experiment Prof.
To compute standardized differences efficiently we utilize the package “CreateTableOne.” (Run “install.packages(“CreateTableOne”)” if you have not done so already.)
The table of mean comparisons that includes standardized differences is created in line 22. (The package is loaded in 21.) About this command, notice the following:
• The same line creates and prints the table. The option to print standardized difference (smd=TRUE) is part of the print command.
• The vector of variables “vars” includes both categorical (factors) and continuous variables—the command automatically recognizes which belongs to what category and runs the appropriate test (in the output below, notice that categorical variables have %’s presented in paratheses while the continuous variable has the standard deviation).
• The results output below shows (last column) that standardized differences are small, well below the thresholds discussed in class. Thus, there are no balancing concerns.
3. Treatment effect estimation
Contrary to the procedures used for prediction (where main estimations are performed on a subset of the data – the training sample), when we estimate ATEs from experimental data we use the full sample. After formatting the outcome variable and declaring factors, this is carried out below in line 31 for the outcome of opening the email.
About these results, note:
• As mentioned above, for causal analysis we use the full dataset—there is not reason to
leave anything out
• The ATE is represented by the coefficient of condition. The estimated coefficient
indicates that personalizing an email (treatment) increases the probability that a receiver will open the email by about 0.02.
BU.450.760 Technical Document T4.1 – Personalization experiment Prof.
• In predictive analyses we never cared about statistical significance—it was all about out- of-sample predictive performance—but now we do. We want to make sure that the coefficient that gives us the ATE estimate is significantly different to zero given the patterns of variation in the data. In this case, the coefficient is highly significant—the p- value of 1.24e-14 is much smaller than the standard 0.05 threshold. Thus, we can accept the ATE of about 0.02 as a real result.
• To place the ATE estimate in context: considering that the about 9% of users in the control group open the email (to obtain this statistic, run “mean(ds$open01[ds$condition==0])”), this ATE represents a lift of about 2/9 (over 20%) of the baseline.
• Notice that we are using a linear morel (family=”gaussian”) to fit a yes/no outcome instead of a logistic model. This may appear counterintuitive to you, since we have been using the logistic model for yes/no outcomes throughout the course. However, there is a good reason:
o The value of the logistic model is that it gives probability predictions within the unit interval
o But here we are not interested in these predictions. Rather, we are interested in the treatment’s ATE
o The linear model has the advantage that the parameter associated to the treatment effect (ie, the coefficient of “condition” in this case) directly gives this ATE. That is, ATE = 0.02 in this case.
o Below we will see how to compute the ATE using the logistic model. It is more complicated, which is why we don’t prefer it.
• Notice that the regression of line 31 is very simple—just a regression of the outcome on the treatment indicator. No controls (age, area, etc.) are included.
o The reason that we trust the ATE obtained from this simple regression is the balancing result. This result tells us that randomization was probably done correctly, so there are no average pre-existing differences between treatment and control groups. That is, the simple regression emulates the “two-world” comparison that we discussed in class
o The regression that adds controls (in line 32) gives a slightly more precise estimate, but essentially the same. This is another sign of consistency with properly executed randomization. (Run this regression on your own.)
In line 34 we reproduce the simple model of line 31 but using logistic regression. Recall that the logistic regression has an exponential-based functional form that is applied on top of the coefficients shown by the output below. Therefore, to compute how much the treatment increases the probability of a “yes”, we need to perform some calculations. (Roughly, these calculations average across all observations a mathematically-predicted effect of “what would happen” with the “yes” probability if the treatment was in place versus if it was not.)
BU.450.760 Technical Document T4.1 – Personalization experiment Prof.
These calculations are performed by the “margins” command, as shown by line 37. The command is available from the package “margins” and has straightforward nomenclature. The estimated marginal effect is 0.01766, which is essentially the same as the ones we have obtained before.
Lines 41 and 45 repeat the simple model but focusing on the other two outcomes of interest: lead generation and unsubscription. Estimates go in the expected direction: personalization leads to more leads and less unsubscription.
Lines 49 and 53 again focus on these outcomes, but conditional on having opened the email. The effects are still there, which suggest that at least part of the effect happens through attention.
4. Treatment effect heterogeneity
Here we investigate whether the response or personalization ATE may vary among different groups of the sample population. In other words, we perform “slicing” or treatment effect heterogeneity analysis.
We illustrate the procedures by focusing on the potentially different email opening responses of individuals of area #5 (areacode=5).
The first approach consists on separately estimating the ATE using the subsamples for each group (lines 61-62). Notice how the “!=5” symbols are used to denote all observations that are different to 5. The first regression gives an ATE = 0.002 (p>0.1) while the second gives ATE = 0.02 (p<0.01). Thus, this analysis gives that the personalization may not have worked for individuals of area 5, and that the results above are coming from individuals of other areas.
The second approach consists of creating an individual treatment effect indicator for each of the groups of interest, then estimating using the full sample. These individual treatment indicators are created in lines 64 and 65. The commands of lines 68 and 69 print the average of each of these created variables, separately for each combination of areacode and condition. When you run line 68, you will see that “cond_area5” (the treatment indicator for individuals of area 5) is only activated for individuals of area 5 that receive the personalized email. (Analogously for “cond_otherareas”.)
BU.450.760 Technical Document T4.1 – Personalization experiment Prof.
The model is estimated including both of these variables (could also include controls), using the full dataset, as in line 71. Results are qualitatively similar to the first approach, in that they indicate that the personalization treatment only worked for individuals outside area #5.
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com