Diagnostics part 2: error covariance assumptions
10/21/2020
Visual diagnostics
Courtesy of: Faraway, J. Linear Models with R. 2nd Edition
- To the left is a plot where:
- The horizontal (x-axis) represents the fitted value \( \hat{\mathbf{Y}} \), as in some model \( \hat{\mathbf{Y}}=\mathbf{X}\hat{\boldsymbol{\beta}} \).
- The vertical axis (y-axis) represents the corresponding residual \( \hat{\boldsymbol{\epsilon}} \) value, equal to \( \mathbf{Y} - \hat{\mathbf{Y}} \).
- Suppose that the variances of the error \( \boldsymbol{\epsilon} \) are not fixed, i.e.,
- suppose that some observations have more variation and some observations have less variation around the signal, \( \mathbf{X}\boldsymbol{\beta} \).
- In this case, there is dependence of the variation (or error \( \boldsymbol{\epsilon} \)) on the observation.
- To the left, this is actuall a well behaved situation. In particular, the residuals show variation that doesn’t seem to depend on the value of the fitted value/ observation.
- The mean of the residuals appears to be zero as well, because there isn’t a clear preference for positive or negative residuals
- The situation to the left is denoted homoscedasticity.
- Later, we will develop tests to determine if the variance is “close-enough-to-constant”, so that we are satisfied.
- On the other hand, if there are non-constant variances (as discussed above), this will often show up as a pattern in the plot to the left.
Visual diagnostics
Courtesy of: Faraway, J. Linear Models with R. 2nd Edition
- The non-constant variance of \( \hat{\boldsymbol{\epsilon}} \) to the left is known as heteroscedasticity.
- In this case, there is a clear dependence on the variation of \( \hat{\boldsymbol{\epsilon}} \) on the fitted value/ the observation.
- Breaking the constant variance assumption, the Gauss-Markov theorem no longer applies
- Heteroscedasticity does not, in itself, cause ordinary least squares coefficient estimates to be biased;
- however, the theoretical estimates of the variance of the residuals (and therefore the standard errors) will become biased.
Visual diagnostics
Courtesy of: Faraway, J. Linear Models with R. 2nd Edition
- The bias in the standard errors complicates our ability to accurate quantify the uncertainty, and therefore to accurately:
- make hypothesis tests on the parameters for significance;
- provide confidence intervals for the parameters
- provide accurate prediction intervals and confidence intervals for the mean response;
- provide explanatory power in the relationship between the response and the explanatory variables.
Visual diagnostics
Courtesy of: Faraway, J. Linear Models with R. 2nd Edition
- Using the same plot as before, we can likewise determine if there is actually nonlinearity in the residuals.
- The plot at the left exhibits a nonlinear dependence of the residual on the fitted/ observed values
- The same technique can also be used replacing the fitted values \( \hat{\mathbf{Y}} \) in the horizontal axis with any other variable in the model \( \mathbf{X}_i \), to determine dependence of the residual on the explanatory variables
- Additionally, we may consider how the residual varies across variables \( \mathbf{X}_i \) that we have data for, but have not included as explanatory variables on the response.
- If there is a dependence structure for the residuals on the variable that was left out of the model, it suggests that we should consider its impact on the response and/or if it is a variable that is tightly correlated with our existing model variables.
Savings data example
- With this model, we plot the residuals versus the fitted values:
par(mai=c(1.5,1.5,.5,.5), mgp=c(3,0,0))
plot(fitted(lmod_savings),residuals(lmod_savings),xlab="Fitted",ylab="Residuals", cex=3, cex.lab=3, cex.axis=3)
abline(h=0)
- From the initial inspection, we do not notice anything particularly structured about the residuals – indeed they are roughly symmetric around zero and in the fitted value.
Half normal distributions
Courtesy of Nagelum CC BY-SA 4.0 via Wikimedia Commons
- Therefore, to investigate the constant variance assumption more closely, we will “zoom” in on the residuals.
- In particular, we will consider the plot of the square root of the absolute residuals, i.e., \[ \begin{align} \sqrt{\vert\hat{\boldsymbol{\epsilon}}\vert} \end{align} \]
- Under the assumption \( \boldsymbol{\epsilon}\sim N(0, \mathbf{I}\sigma^2) \), the values of \( \vert\hat{\boldsymbol{\epsilon}}\vert \) are distribted according to the half normal distribution
- the relationship between the normal distribution and the corresponding half-normal distribution is illustrated on the left.
- Having excluded the nonlinearity expressed in other plots, we will focus on the absolute values to increase the resolution of the residuals as a response of the fitted values.