“`{r setup, include=FALSE} knitr::opts_chunk$set(echo = TRUE)
options(tinytex.verbose = TRUE)
options(htmltools.dir.version = FALSE)
library(knitr) opts_chunk$set( fig.align=”center”,fig.height=4, dpi=300, cache=T, echo=T)
options(“getSymbols.warning4.0″=FALSE)
“`
Front Matter
This assignment is due SUNDAY FEB 16TH AT 11:59PM ON D2L.
This Rmarkdown document is both your homework assignment and the template for you to use to complete your homework. Tasks require coding, which must be done within the code chunks as specified. Questions following each task will refer back to the results from the previous code chunk.
PART 1
We will use a small dataset of home prices and characteristics which I host online. In Part 1, you will load the data, examine the variables, clean the data, and generate some additional variables. Next, you will test a hypothesis on the mean of the variable _homeprice_. Then, you will calculate a $\hat{\beta}$ coefficient for the regression of home price on number of bedrooms, its standard error, and the t-statistic you will use to test a hypothesis.
I will tell you which functions you will need to use, but you will need to do much of the writing of the code. Remember to run each chunk sequentially (or use the “run from the top” button in a code chunk) to run as you go. In the end, your whole assignment has to run from top to bottom in order to render the file you can turn in.
Please remember to save the .html as a .pdf and upload JUST YOUR .PDF for your homework assignment. I do not need your Rmarkdown document. Remember to put your name up in the header.
Task 1.1 (8 points)
“`{r Task11, out.width=’60%’} # (a) (1 point) Load the data into an object called “df” (“df” is a common shorthand for “data.frame”) and use “head(df)” to see the top of the data.frame df = read.csv(‘https://people.duke.edu/~ajk41/HW2.csv’, stringsAsFactors = F) # this lets you load from the web directly. head(df)
(B) (2 POINTS) PRINT A LIST OF ALL OF THE VARIABLE NAMES IN THE DATA.FRAME DF USING THE “NAMES(DF)” FUNCTION.
<>
(C) (2 POINTS) THE COLUMN “X” IN DATA.FRAME DF ISN’T VERY USEFUL. WE CAN SET DF$X = NULL TO DROP IT
<>
(D) (1 POINT) LET’S LOOK AT THE RANGE OF THE HOMEPRICE. USE THE “RANGE(…)” FUNCTION ON DF$HOMEPRICE TO SEE THE MIN AND MAX
<>
(E) (2 POINTS) PLOT A SCATTERPLOT WITH THE VARIABLE HOMEPRICE ON THE Y-AXIS AND SQFT ON THE X-AXIS.
HINT: USE “PLOT” AND SPECIFY THE FORMULA IN THE FORM “Y ~ X”, THEN THE DATA NAME.
<>
“`
Question 1.1 (5 points)
(a) (1 point) How many observations are there in this dataset? <>
(b) (1 point) What are the column names?
(c) (1 point) What was the max in the range of _homeprice_
(d) (2 points) Does the plot show much of a relationship between homeprice and square footage?
Task 1.2 (7 points)
“`{r Task12, out.width=’90%’} # (a) (2 points) Using a conditional statement like: which(df$Variable > 100), create an index of which rows of df have a homeprice less than $2,000,000. Call this index largeHomeprice # <>
(D) (2 POINTS) NOW, PLOT THE RELATIONSHIP BETWEEN KHOMEPRICE AND SQFT AGAIN
<>
“`
Question 1.2 (3 points)
(a) (1 point) What do the units in _khomeprice_ refer to?
(b) (2 point) Does it look like there is a relationship between sqft (area of the home) and khomeprice? If so, what is the relationship?
Task 1.3 (10 points) – Testing hypothesis about _khomeprice_
“`{r Task13} # (a) (2 points) Generate the following: # ybar as the mean of khomeprice # N as the number of observations # A new column in df called yMinusYbar that is equal to khomeprice – ybar # Another new column in df that is equal to yMinusYbar^2. Call it df$yMinusYbar2
(B) (2 POINTS) USING DF$YMINUSYBAR2, CALCULATE THE SUM OF SQUARES AND CALL IT SSTY
(C) (2 POINTS) CALCULATE AND PRINT THE SAMPLE STANDARD DEVIATION OF KHOMEPRICE USING SSTY. CALL THIS NEW OBJECT SIGMA2Y
(D) (2 POINTS) USING THE FORMULA FOR THE STANDARD ERROR OF THE MEAN, CALCULATE AND PRINT THE STD. ERROR OF THE MEAN OF KHOMEPRICE. CALL IT SEMEAN
(E) (2 POINTS) CALCULATE AND PRINT A T-STATISTIC CALLED “TSTAT” TESTING THE HYPOTHESIS “H0: THE MEAN OF KHOMEPRICE IS 0”. SEE OUR STATS REVIEW SLIDES IF YOU FORGET HOW TO DO THIS.
(F) (2 POINTS) THE COMMAND BELOW WILL GIVE THE CRITICAL VALUES FOR A T-DISTRIBUTION’S REJECTION REGION WITH N=10 DEGREES OF FREEDOM. WE DO NOT HAVE 10 DEGREES OF FREEDOM. REPLACE “10” BELOW WITH THE CORRECT NUMBER OF DEGREES OF FREEDOM.
qt(c(.025,.975), 10)
“`
Question 1.3 (5 points)
(a) (1 point) What is the mean of _khomeprice_?
(b) (1 point) What is the t-statistic for the stated null hypothesis?
(c) (3 points) Interpret the t-statistic for the stated null hypothesis using the critical values in Task (f). Do we reject the null hypothesis?
Task 1.4 (3 points)
“`{r Task14} # (a) R has a built in function for the t-test we just ran. It is t.test(x = , mu = ). # Use t.test here to test the same null hypothesis (that mu = 0)
“`
Question 1.4 (2 points)
(A) (2 POINTS) ARE THE RESULTS THE SAME?
Task 1.5 (6 points)
“`{r Task15} # (a) (4 points) Repeat Task 1.3, testing the mean, but do it _only_ for those observations in df where df$city==’Long Beach’. The easiest # way of doing this is to set df = df[which(df$city==’LONG BEACH’),] # You will need to generate a new ybar and new columns like yMinusYbar, and you’ll have a different N # Generate a t-statistic and use qt() to find the new critical values using the new degrees of freedom.
(B) (2 POINTS) REPEAT THE T.TEST WITH R’S BUILT IN FUNCTION
“`
Question 1.5 (2 points)
(a) (1 point) What is the result of your test? Do we reject the null hypothesis? (Yes it’s a silly null hypothesis)
(b) (1 point) Does it match R’s t.test results?
Task 1.6 (10 points)
Now, we calculate our coefficient of interest for the relationship between _sqft_ and _khomeprice_. “`{r Task16} # (a) (1 point) Reload the data using the read.csv(‘https://….’) command from Task 1.1 _AND_ drop the observations in df where homeprice >2000000 as in Task 1.2. Also, re-create df$khomeprice again as in Task 1.2.
(B) (2 POINTS)
NOW WE WANT TO CALCULATE THE COEFFICIENT IN A REGRESSION OF KHOMEPRICE = BETA_0 + BETA_1 SQFT + U. CREATE:
YBAR AS THE MEAN OF DF$khomeprice # xbar as the mean of df$SQFT
N AS THE NUMBER OF ROWS (HINT: USE NROW(DF))
(C) (5 POINTS) CREATE:
A NEW COLUMN CALLED YMINUSYBAR THAT IS KHOMEPRICE – YBAR
A NEW COLUMN CALLED XMINUSXBAR THAT IS SQFT – XBAR
A NEW COLUMN CALLED YMINUSYBAR2 THAT IS YMINUSYBAR^2 (WE WILL NEED THIS LATER)
A NEW COLUMN CALLED XMINUSXBAR2
A NEW COLUMN CALLED XY
(D) (5 POINTS) NOW, CREATE:
A NEW OBJECT CALLED BETA1 THAT USES DFxyanddfXMINUSXBAR2
A NEW OBJECT CALLED BETA0 THAT USES YBAR, XBAR, AND BETA1 TO GET BETA0
beta1=0 # delete this when you create your beta0.
(E) (1 POINT) PRINT BOTH BETA1 AND BETA0
“`
Question 1.6 (13 points)
(a) (2 points) What is the intercept (beta0) and how would you interpret it in this regression of home price on square footage?
(b) (4 points) What is the beta1, and how would you interpret it?
(c) (2 points) If square footage were to increase by 300 square feet, what would be the change in expected home value?
(d) (2 points) What have we assumed about u to say that beta1 is unbiased?
(e) (3 points) What might violate this assumption? That is, think of something that might be left out of the regression that might bias our estimate?
Task 1.7 (10 points)
Now, we will calculate our standard error for beta1 “`{r Task17}
WE KNOW BETA0 AND BETA1, SO WE CAN CALCULATE Y-HAT
(A) (2 POINTS) CREATE A NEW COLUMN CALLED YHAT IN DF THAT IS EQUAL TO Y-HAT
(B) (2 POINTS) CREATE A NEW COLUMN CALLED UHAT IN DF THAT IS EQUAL TO THE RESIDUAL
(C) (2 POINTS) IN AN OBJECT CALLED SSR, SQUARE THIS COLUMN AND SUM IT UP.
(D) (2 POINTS) IN AN OBJECT CALLED SIGMA2, CALCULATE THE SIGMA^2 OF U-HAT. DON’T FORGET WE LOSE TWO DEGREES OF FREEDOM
SEE THE SLIDES FROM SINGLE REGRESSION INFERENCE IF YOU AREN’T SURE HOW TO CALCULATE THIS.
(E) (2 POINTS) IN AN OBJECT CALLED SEBETA1, CALCULATE THE STANDARD ERROR OF BETA1. DON’T FORGET, YOU’LL NEED TO TAKE THE SQUARE ROOT AT SOME POINT.
(F) (2 POINTS) IN AN OBJECT CALLED TBETA1, CALCULATE THE T-STATISTIC TESTING THE NULL HYPOTHESIS THAT BETA1 = 0.
(F) (2 POINTS) PRINT BETA1, SEBETA1, AND THE T-STATISTIC
(G) (1 POINT) GET THE CRITICAL VALUES OF THE T-DISTRIBUTION USING QT(C(.025, .975), )
“`
Question 1.7
(a) (4 points) What is the coefficient estimate for beta1? What is the interpretation of this results?
(b) (2 points) What is the t-statistic and what does it tell us about our null hypothesis (using the critical values)?
Task 1.8 (5 points)
“`{r Task18} # (a) (5 points) using lm(…), run a regression of khomeprice ~ sqft using the df data. Wrap the lm(…) call in summary() to see a complete output.
“`
Question 1.8 (3 points)
(a) (3 points) How do your estimates for beta1 (sqft) compare to the coefficients in R’s regression?
Task 1.9 (5 points)
We are going to calculate the R2 of our regression
“`{r Task 19} # (a) (5 points) # You already have SSR from Task 1.7 # And you already have SST (called SSTy) from Task 1.6 # Create an object called R2 that uses these to calculate the R2 of the regression
“`
Question 1.9 (3 points)
(a) What is the R^2 and what is the interpretation of it?
PART 2
It’s short. Don’t worry.
Task 2.1 (10 points)
We want to also include the effect of number of bedrooms in our regression since we might worry that leaving it out is creating bias. First, let’s think about how we might “sign the bias”, then let’s see if we were right. Before you get to coding, answer the following:
Question 2.1′ (7 points)
(a) (2 points) Can we sign the bias from leaving bedrooms out of our estimates in Task 1.7? What will the sign be on the relationship between bedrooms and square footage? Positive or negative? Why?
(b) (2 points) And what would the sign be on the relationship between _bedrooms_ and home price? Why?
(c) (3 points) Our slides on Multiple Regress Inference have the formula for bias (the one with the tildes). Using your answers to 2.1’a and b, what will happen to beta1 when we add in _bedrooms_ as a covariate? “`{r Task21}
(A) (3 POINTS) SINCE WE WANT TO SEE HOW BETA1 MIGHT CHANGE WHEN WE INCLUDE A COEFFICIENT, BETA2, ON BEDROOMS, WE SHOULD FIRST CHECK THE RELATIONSHIP BETWEEN SQFT AND BEDROOMS. RUN A REGRESSION USING LM(…) TO SEE. SAVE THE REGRESSION (THE LM(…) OBJECT) IN AN OBJECT CALLED LMFIRSTSTAGE, THEN PRINT SUMMARY(LMFIRSTSTAGE) TO SEE THE FULL RESULTS.
(B) (2 POINTS) WE CAN PULL THE COEFFICIENTS OUT OF THE LMFIRSTSTAGE OBJECT. WE DO THIS USING LMFIRSTSTAGE$COEFFICIENTS, MUCH LIKE WE’D USE THE $ TO SEE A COLUMN IN A DATA.FRAME. CREATE AN OBJECT CALLED DELTA THAT IS EQUAL TO LMFIRSTSTAGE$COEFFICIENTS AND PRINT IT. THEN, TRY TYPING DELTA[‘BEDROOMS’].
NOTE: DELTA, IT TURNS OUT, CAN BE INDEXED WITH NAMES, BUT IT IS NOT A DATA.FRAME, IT IS JUST A NAMED VECTOR, SO WE DON’T USE A COMMA WHEN INDEXING. JUST DELTA[‘BEDROOMS’] WILL DO.
(C) (3 POINTS) GENERATE A NEW COLUMN IN DF CALLED VHAT THAT IS THE RESIDULA OF VHAT ONCE WE REMOVE THE VHAT EXPLAINED BY BEDROOMS.
I’LL WRITE IT FOR YOU, BUT MAKE SURE YOU UNDERSTAND WHAT’S GOING ON HERE:
DFvhat = dfSQFT – DELTA[‘(INTERCEPT)’] + DELTA[‘BEDROOMS’]*DF$BEDROOMS
(C) (2 POINTS) NOW, RUN THE REGRESSION OF KHOMEPRICE ~ VHAT. THIS IS WHAT WE DID IN CLASS ON WEDNESDAY THE 5TH. PRINT THE RESULTS.
(D) (3 POINTS) RUN THE FULL (UNBIASED) REGRESSION OF KHOMEPRICE ON BEDROOMS AND SQFT. PRINT THE RESULTS.
“`
Question 2.1 (10 points)
(a) (2 points) From Task 1.7, what was beta1? Note: since you already have an R object called “beta1”, you can use an “inline code chunk” like this to print the value of beta1: r beta1. Take a look at it when you render – you’ll see the value of beta1 in there!
(b) (2 points) In Question 2.1′ above, we asked what we thought the bias would be on the “naive” single-variable estimate of beta1. How does our naive, single-variable (biased) estimate of beta1 compare to the unbiased estimate in Task 2.1 above?
(c) (3 points) What is the unbiased estimate of beta1, the coefficient on sqft, and how would you interpret it?
(d) (3 points) What is the unbiased estimate of beta2, the coefficient on bedrooms, and how would you interpret it?
Task 2.2 (5 points)
This one is easy: plot _something_ from the dataset df and explain what the plot tells us. This may involve a little research to see what sort of plots you can make. It’s your call, just plot something besides the scatterplot we made earlier!
{r Task 2.2, out.width=’90%’} # (5 points) plot something!
Postword
Don’t forget to render this to .html, then open it with your browser and print it to .pdf. Do NOT turn in your Rmarkdown file. I’ll be able to see your code in the chunk outputs.