Causal Inference: Problem Set
Submission deadline: 17.00, Tuesday, 26 February 2019
The problem set comprises two parts that are weighted equally. The first part includes conceptual questions, and the second part asks you to conduct experimental data analysis in Stata or R, report and interpret your results. The research article mentioned in part I, question 6, and the replication dataset for part II, are available on KEATS. Please submit your answers to the questions, along with an Appendix including your complete Stata or R code on KEATS in docx or PDF file format. Your file must include:
• Your written answers to the questions (answers need to be numbered).
• Appendix including the entire Stata or R code required to fully reproduce your answers to part II.
Part I: Conceptual Questions
Question 1 – 8 marks
Politicians and educational researchers often claim that smaller class sizes positively affect learning outcomes among high school students. A researcher decides to test this hypothesis by randomly assigning 40 6th form students to one of two small classes with 10 students (treatment), or to a large class with 20 students (control). She finds a relatively large difference in learning outcomes between students who attend the larger class and students who attend the smaller classes (the students in smaller classes appear to do better) , but the effect is not statistically significant. The researcher concludes that class size has no influence on learning outcomes.
a Is this a randomized experiment, a natural experiment, a quasi-experiment, or none of the above (an obser- vational study)? Explain your answer. (1 mark)
b How confident are you in the findings of this study? Explain your answer. (3 marks)
c Are there any issues with this research design? How could this research design be improved? (4 marks)
Question 2 – 10 marks
a What is the excludability assumption? Give an example of a published randomized field experiment in which the excludability assumption could plausibly be violated. (4 marks)
1
b What could researchers in your example have done in order to minimize the risk of violating the excludability assumption? (6 marks)
Question 3 – 2 marks
a What are potential outcomes? How do they differ from realised outcomes? (1 mark)
b Explain the following notations: Yi(1), Yi(1) | Di = 0, Yi(0) | Di = 1, and Yi(1) | di = 1. (1 mark)
Question 4 – 10 marks
a Give two examples of published randomized field experiments, one facing one-sided non-compliance, and the other facing two-sided non-compliance. How do the researchers measure compliance in these two experiments? (5 marks)
b Under two-sided non-compliance, which types of subjects will reveal which potential outcomes in the randomly assigned control group? Which types of subjects will reveal which potential outcomes in the randomly assigned treatment group? (5 marks)
Question 5 – 8 marks
Consider the schedule of outcomes in the table below. If treatment A is administered, the potential outcome is Yi(A), and if treatment B is administered, the potential outcome is Yi(B). If no treatment is administered, the potential outcome is Yi(0). The treatment effects are defined as Yi(A) − Yi(0) or Yi(B) − Yi(0).
Yi(B)
Suppose a researcher plans to assign two observations to the control group and the remaining three observations to just one of the two treatment conditions. The researcher is unsure which treatment to use.
a Applying equation 3.4 in Gerber and Green (2012, page 57) determine which treatment, A or B, will generate a sampling distribution with a smaller standard error. (4 marks)
b What does the result in part (a) imply about the feasibility of studying interventions that attempt to close an existing “achievement gap”? (4 marks)
Yi(0) Yi(A)
subject
Miriam 1 2 3 Benjamin 2 3 3 Helen 3 4 3 Eva 4 5 3 Billie 5 6 3
2
Question 6 – 12 marks
Answer the following questions based on your reading of Arceneaux and Nickerson’s (2010) article “Comparing Negative and Positive Campaign Messages: Evidence From Two Field Experiments”:
a What are the research question, and the research hypotheses? Summarize the research question, hypotheses and results of the study in no more than four sentences. (2 marks)
b How many experimental conditions are there in study 1 and study 2? Briefly describe the treatments in both experiments. (2 marks)
c What are the outcome variables in both field experiments? How are they measured and collected? (2 marks)
d Describe the random assignment procedures employed in studies 1 and 2. What is the respective unit of randomization? Why did the researchers choose a different strategy for study 2 than for study 1? (3 marks)
e “The experimental design of Study 2 offers a few complications not encountered in Study 1” (p. 64). What are these complications, and how do the researchers address them? Elaborate. (3 marks)
Part II: Experimental analysis
Researchers conducted a randomized field experiment in order to identify the effect of campaign phone calls on turnout (Foos and de Rooij 2017). The political campaign they were cooperating with managed to reach around 45% of the treatment groups by telephone, and contacted 0% of the randomly assigned control group. The campaign then collected outcome data about electoral participation from the public voter register and matched it to subjects assigned to treatment and control groups. The treatment assignment variable “treatment” is binary (0 control, 1 phone call), and the outcome variable of interest is called turnout, coded 0 if a subject did not vote in the election, and 1 if the subject voted. Using the replication data file “canvassing.dta”:
Exercise 1: ITT estimation – 8 marks
a Estimate the Intent-to-Treat Effect using the difference-in-proportions estimator. Report your ITT estimate, and interpret it. (3 marks)
b Estimate the ITT using linear regression. Compare your ITT estimate to the one you obtain in exercise 1a. Explain what you observe. (3 marks)
c Why are you asked to estimate the ITT, and not the ATE? (2 marks) 3
Exercise 2: Randomization inference – 14 marks
a What is the sharp null hypothesis in this experiment? (2 marks)
b Using randomization-inference, conduct a one-tailed and a two-tailed test of the sharp null hypothesis. Report the one-tailed and two-tailed p-values. Can you reject the sharp null hypothesis in both instances? When is it appropriate to choose a one-tailed instead of a two-tailed test? (6 marks)
c Estimate the standard error and 95% confidence-interval using linear regression with robust standard errors. Report the standard error and 95% confidence interval. What is the interpretation of the confidence interval? (6 marks)
Exercise 3: CACE estimation – 14 marks
a Estimate and report the Complier Average Causal Effect (CACE), based on an instrumental variable regression with contact as the endogenous variable (D), and treatment assignment as the exogeneous variable (Z). (4 marks)
b Estimate and report the robust standard error. (2 marks)
c Report the one-tailed and two-tailed p-values. Is your CACE estimate significantly different from 0? (3 marks)
d True or false? “The smaller the ITTD , the larger the CACE.” Explain your answer. (5 marks)
Exercise 4: Blocking – 14 marks
a The actual random assignment in this experiment was blocked on pre-treatment party-support. Re-estimate the ITT, accounting for random assignment within the three exprimental blocks. The block variable is called “pid”. Report your ITT estimate, and interpret it. (6 marks)
b Using randomization-inference, conduct one-tailed and a two-tailed tests of the sharp null hypothesis, ac- counting for random assignment within experimental blocks. Report and discuss the result. Does blocking narrow the sampling distribution of the ITT? Why/why not? (4 marks)
c Accounting for experimental blocks, report the CACE, the associated standard error, and two-tailed p-value. Interpret your results. (4 marks)
4