Further hypothesis testing, confidence intervals and regions

Outline

• The following topics will be covered in this lecture:
  • Testing a single predictor
  • The t-test
  • Testing a subspace
  • Confidence intervals
  • Confidence regions

Testing one predictor

• As a general method, we can always use the F-statistic for nested models.

• Specifically, whenever one model is given by a subspace of another:

  • \( \boldsymbol{\omega} \) consists of models over variables \( x_1, \cdots, x_{q-1} \) and corresponds to \( q \) parameters (including the intercept);
  • \( \boldsymbol{\Omega} \) consists of models over variables \( x_1, \cdots , x_{p-1} \) and corresponds to \( p \) parameters (including the itercept);
  • such that \( q \text{<} p \).

• Concretely, the null hypothesis must be \( H_0 : \boldsymbol{\beta}_i = \boldsymbol{0} \) for each \( i=q,\cdots, p-1 \).

• The alternative hypothesis is that the larger model holds,

  \[ H_1: \boldsymbol{\beta} \neq 0 \]

• We compute the F statistic as, \[ \begin{align} F &\triangleq \frac{ \left( RSS_\boldsymbol{\omega} - RSS_\boldsymbol{\Omega}\right)/ (p-1)}{RSS_\boldsymbol{\Omega}/(n-p)} . \end{align} \]

Testing one predictor – continued

• Suppose there is one particular variable that we want to determine the significance of for our model.

• Specifically, suppose we have a model,

  \[ \begin{align} \mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}, \end{align} \] with respect to some choice of variables \( \mathbf{X} \).

• Our alternative hypothesis in this case is,

  \[ \begin{align} H_1: \boldsymbol{\beta} \neq \boldsymbol{0}. \end{align} \]

• Q: if we want to determine if \( \boldsymbol{\beta}_i \) specifically gives an appreciable difference in this model, what is our null hypothesis?

• A: our null hypothesis takes the form,

  \[ \begin{align} H_0: \boldsymbol{\beta}_i = 0 \end{align} \]

Testing one predictor – continued

• We will examine this on the gala data once again.

• We define the model lmods without area as an explanatory variable for the null hypothesis. Then we compute the ANOVA table with the bigger model that contains area

require("faraway")
lmod <- lm(Species ~ Area + Elevation + Nearest + Scruz + Adjacent, gala)
lmods <- lm(Species ~ Elevation + Nearest + Scruz + Adjacent, gala)
anova(lmods, lmod)

Analysis of Variance Table

Model 1: Species ~ Elevation + Nearest + Scruz + Adjacent
Model 2: Species ~ Area + Elevation + Nearest + Scruz + Adjacent
  Res.Df   RSS Df Sum of Sq      F Pr(>F)
1     25 93469                           
2     24 89231  1    4237.7 1.1398 0.2963

• The result of the \( F \) test is to say, “With probability 29.63%, we will find a value drawn from the F distribution with this value or greater”.

• Q: do we reject or fail to reject the null hypothesis at \( 5\% \) significance here?

• A: Here we fail to reject the null hypothesis because it is reasonable that the difference between the large model and the small model could be due to random variation.

  • Note: there may be some statistical relationship, but we haven't detected one that wouldn't be surpising if it was just noise.

Student t-distribution

Courtesy of Skbkekas CC BY-SA 3.0

Student t-distribution

• Solution: we can consider the two events, \[ \begin{align} A: X\geq L & & B: X \leq -L. \end{align} \]

• Due to the symmetry of the student t-distribution about zero, we find,

  \[ \begin{align} P(A)=P(B) = \frac{\alpha}{2}. \end{align} \]

• Then, notice that,

  \[ \begin{align} P(A\cup B) &= P(A) + P(B) - P(A\cap B) \\ &= \frac{\alpha}{2} + \frac{\alpha}{2} - 0 \end{align} \] as the intersection is empty.

• Finally, the complement of the event \( A\cup B \) is the event that \( -L < X < L \), such that, \( P(-L < X < L) = 1 - \alpha \).

Testing one predictor – continued

• If we want thus to test the hypothesis that a single parameter \( \boldsymbol{\beta}_i \neq 0 \), we can follow a similar procedure under the Gaussian assumption.

• Recall, the standard error of a given parameter \( \hat{\boldsymbol{\beta}}_i \) is given \[ \begin{align} se(\hat{\boldsymbol{\beta}}_{i-1}) \triangleq \hat{\sigma}\sqrt{(\mathbf{X}^\mathrm{T}\mathbf{X})^{-1}_{ii}} \end{align} \] where \( \hat{\sigma}^2 = \frac{RSS}{n-p} \).

• The value \[ \begin{align} t_i = \frac{\hat{\boldsymbol{\beta}}_i}{se(\hat{\boldsymbol{\beta}}_i)} \end{align} \] has t-distribution in \( (n-p) \) degrees of freedom under the null hypothesis \( H_0:\boldsymbol{\beta}_i=0 \).

• Particularly, we will determine the probability of obtaining a random value \( t \) where \( \vert t \vert \geq \vert t_i \vert \) with respect to the t-distribution to determine significance.

  • If the above probability falls below our pre-chosen \( \alpha \), or equivalently, \[ \vert t_i\vert > \vert t^\alpha \vert, \] we reject the null that \( \beta_i = 0 \) with \( \alpha \) significance.

• NOTE: The null hypothesis will have a different interpretation if it is stated with respect to different alternative hypotheses.
  

summary(lm(Species ~ Area, gala))

Call:
lm(formula = Species ~ Area, data = gala)

Residuals:
    Min      1Q  Median      3Q     Max 
-99.495 -53.431 -29.045   3.423 306.137 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 63.78286   17.52442   3.640 0.001094 ** 
Area         0.08196    0.01971   4.158 0.000275 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 91.73 on 28 degrees of freedom
Multiple R-squared:  0.3817,    Adjusted R-squared:  0.3596 
F-statistic: 17.29 on 1 and 28 DF,  p-value: 0.0002748

• Q: what do you notice about the p-value for the F-statistic and the p-value for the t-statistic for area?
  • A: These correspond to the same test of the null versus alternative hypothesis in this case, i.e., leaving out area is the same null hypothesis as the data is expressed only as random variation around the mean \( \overline{\mathbf{Y}} \).
    
  • The p-value for the t-test on Area is not the same here as it was on the last slide, because this is framed with respect to a different alternative hypothesis.

Review

• We now have two methods for comparing different models: the F-test and the t-test.

• The F-test is defined to compare the likelihood of any two models, as long as one lies in a subspace of another model.

  • Particularly, this is defined as long as the small model \( \boldsymbol{\omega} \) uses some combination of the same explanatory variables of the large model \( \boldsymbol{\Omega} \).
  • This measures the ratio of the maximum likelihoods of both models, which is distributed according to an F distribution under the null hypothesis (the smaller model is accurate).

• The t-test is defined for a single parameter, rather than a combination of parameters like the F-test.

  • This computes the probability of \[ \begin{align} t_i = \frac{\hat{\boldsymbol{\beta}}_i}{se(\hat{\boldsymbol{\beta}}_i)}, \end{align} \] under the null hypothesis where \( \boldsymbol{\beta}_i = 0 \).

• In particular, the two tests are equivalent when we test the two models

1. \( \boldsymbol{\Omega} \) defined over \( X_1, \cdots, X_{p-1} \); and
2. \( \boldsymbol{\omega} \) defined over \( X_1,\cdots, X_{p-2} \)

Testing a pair of explanatory variables

• Suppose we wish to determine if the two variables area and adjacent have an effect on the response relative to the model with all other variables.

• Particularly, we obtain the null hypothesis \( H_0 : \boldsymbol{\beta}_{area}= \boldsymbol{\beta}_{adjacent}=0 \).

lmods <- lm(Species ~ Elevation + Nearest + Scruz, gala)
anova(lmods, lmod)

Analysis of Variance Table

Model 1: Species ~ Elevation + Nearest + Scruz
Model 2: Species ~ Area + Elevation + Nearest + Scruz + Adjacent
  Res.Df    RSS Df Sum of Sq      F  Pr(>F)   
1     26 158292                               
2     24  89231  2     69060 9.2874 0.00103 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

• The probability of drawing such an F value is around 1/1000, so we reject the null hypothesis.

• This tells us that it is extremely unlikely that there is no combination of the variables area and adjacent which have some effect on the response, when comparing the two models.

Testing a pair of explanatory variables

• Rejecting the null hypothesis in the last example, testing a pair of variables, cannot be inferred from their respective t-tests.

sumary(lmod)

             Estimate Std. Error t value  Pr(>|t|)
(Intercept)  7.068221  19.154198  0.3690 0.7153508
Area        -0.023938   0.022422 -1.0676 0.2963180
Elevation    0.319465   0.053663  5.9532 3.823e-06
Nearest      0.009144   1.054136  0.0087 0.9931506
Scruz       -0.240524   0.215402 -1.1166 0.2752082
Adjacent    -0.074805   0.017700 -4.2262 0.0002971

n = 30, p = 6, Residual SE = 60.97519, R-Squared = 0.77

• There is no simple way of combining the above information to test a pair of explanatory variables simultaneously.

• Testing a pair of variables needs to be performed with the F-test over the two nested models.

Testing subspaces

• Suppose we think we can make a simpler model by combining two variables as some kind of linear combination,

  • e.g., what if we want to create a new variable \[ \begin{align} \tilde{X}_i \triangleq X_i + X_j \end{align} \] corresponding to two explanatory variables \( X_i,X_j \).
  • This still makes sense with an F-test because we are again testing a null hypothesis as a nested subspace of the space of all variables.

Testing subspaces – continued

• Concretely, suppose we wish to take a new variable corresponding to the sum of the area of the island itself and the area of the adjacent island.

• Our null hypothesis will be that \( \boldsymbol{\beta}_{area} = \boldsymbol{\beta}_{adjacent} \)

lmod  <- lm(Species ~ Area + Adjacent + Elevation + Nearest + Scruz, gala)
lmods <- lm(Species ~ I(Area+Adjacent) + Elevation + Nearest + Scruz, gala)
anova(lmods,lmod)

Analysis of Variance Table

Model 1: Species ~ I(Area + Adjacent) + Elevation + Nearest + Scruz
Model 2: Species ~ Area + Adjacent + Elevation + Nearest + Scruz
  Res.Df    RSS Df Sum of Sq     F  Pr(>F)  
1     25 109591                             
2     24  89231  1     20360 5.476 0.02793 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

• In the above, notice the only difference is the I() function, which enforces that the “+” expression isn't read as a formula.

• Q: do we reject or fail to reject the null hypothesis?

• A:In this case, we can reject the null hypothesis at the \( \alpha = 5\% \) significance level.

Testing subspaces – continued

• We have learned so far how to test the significance of a given parameter \( \boldsymbol{\beta}_i=0 \), which would reduce the model space \( \boldsymbol{\Omega} \) to the subspace \( \boldsymbol{\omega} \) where \( X_i \) has no effect on the response.

• We may quite similarly test the significance of a parameter having a specific value (other than zero);

  • here \( \boldsymbol{\omega} \) will represent the subspace of the big model space \( \boldsymbol{\Omega} \) where \( X_i \) has a specified effect.

• Consider the null hypothesis \( H_0: \boldsymbol{\beta}_{Elevation} = 0.5 \), versus the alternative hypothesis where we consider the space of models over

Species ~ Area + Adjacent + Elevation + Nearest + Scruz.

• We use the function offset to fix the associated value \( \boldsymbol{\beta}_{Elevation} = 0.5 \)

lmods <- lm(Species ~ Area+ offset(0.5*Elevation) + Nearest + Scruz + Adjacent, gala)

Testing subspaces – continued

• Comparing the nested sub-model with an ANOVA table,

anova(lmods, lmod)

Analysis of Variance Table

Model 1: Species ~ Area + offset(0.5 * Elevation) + Nearest + Scruz + 
    Adjacent
Model 2: Species ~ Area + Adjacent + Elevation + Nearest + Scruz
  Res.Df    RSS Df Sum of Sq      F   Pr(>F)   
1     25 131312                                
2     24  89231  1     42081 11.318 0.002574 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

• Q: do we reject or fail to rejec the null hypothesis?

• A: we reject the null hypothesis at \( 5\% \) significance.

Testing subspaces – continued

• Note: specifying a particular value for the parameter can also be seen in terms of specifying a non-zero mean for the parameter \( \hat{\boldsymbol{\beta}}_i \) in a t-test.

• Suppose we want to test the following null hypothesis, \[ H_0: \boldsymbol{\beta}_{Elevation} = C \] for some constant \( C \) as discussed above.

• If \( \hat{\boldsymbol{\beta}}_i \) is the solution by least squares, we can compute
  \[ \begin{align} t_i = \frac{\left(\hat{\boldsymbol{\beta}}_i - C\right)}{se(\boldsymbol{\beta})} \end{align} \]

• By the same reasoning as before, this is t-distributed in \( n-p \) degrees of freedom with the assumption under the null hypothesis that

  \[ \mathbb{E}\left[\hat{\boldsymbol{\beta}}_i\right] = C \]

Confidence intervals

• We return now to the idea of confidence intervals as an alternative (and dual) idea to hypothesis testing.

• Recall that under our Gaussian assumptions,

  \[ \boldsymbol{\epsilon} \sim N\left(\boldsymbol{0}, \sigma^2 \mathbf{I} \right), \]

• we have derived,

  \[ \hat{\boldsymbol{\beta}} \sim N\left(\boldsymbol{\beta}, \left(\mathbf{X}^\mathrm{T}\mathbf{X}\right)^{-1} \sigma^2 \right). \]

• In this context, we have an unbiased, sample based estimate for the true mean \( \boldsymbol{\beta} \), with a known variance and \( n-p \) degrees of freedom.

Confidence intervals – continued

• We recall from the t-test that the value

  \[ \begin{align} t_i = \frac{\hat{\boldsymbol{\beta}}_i - \boldsymbol{\beta}_i}{se(\hat{\boldsymbol{\beta}}_i)} \end{align} \]

is t-distributed in \( n-p \) degrees of freedom.

• Suppose that \( t^\frac{\alpha}{2} \) is chosen such that \( P(X\geq t^\frac{\alpha}{2}_{n-p}) = \alpha/2 \);

  • then recall that we derived in an earlier exercise that \( P(-t^\frac{\alpha}{2}_{n-p} < X < t^\frac{\alpha}{2}_{n-p})=1- \alpha \).

• Using this dual notion to the hypothesis test, we can create an interval centered at \( \hat{\boldsymbol{\beta}} \) (using the t-distribution) with some measure of confidence that the true \( \boldsymbol{\beta} \) lives within it.

  • Again, we do not say that the “true” \( \boldsymbol{\beta} \) lives within such an interval with probability \( 100(1-\alpha)\% \), it lies there or not.
    
  • Rather, we perform a procedure that over many sets of observations, and with respect to the procedure, we have guaranteed that the true value will lie in our interval with certain confidence \( 100(1-\alpha)\% \).

Confidence intervals – continued

• Suppose that we wish to compute a confidence interval on \( \boldsymbol{\beta}_{Area} \), such that with \( 95\% \) confidence, we say the “true value” of \( \boldsymbol{\beta}_{Area} \) lives within it.

lmod <- lm(Species ~ Elevation + Nearest + Scruz + Area + Adjacent, gala)
sumary(lmod)

             Estimate Std. Error t value  Pr(>|t|)
(Intercept)  7.068221  19.154198  0.3690 0.7153508
Elevation    0.319465   0.053663  5.9532 3.823e-06
Nearest      0.009144   1.054136  0.0087 0.9931506
Scruz       -0.240524   0.215402 -1.1166 0.2752082
Area        -0.023938   0.022422 -1.0676 0.2963180
Adjacent    -0.074805   0.017700 -4.2262 0.0002971

n = 30, p = 6, Residual SE = 60.97519, R-Squared = 0.77

• We note that the standard error for area is given as \( \approx 0.022422 \),

• We compute the \( 2.5\% \) critical value for \( T_{30 - 6} \) and obtain the confidence interval as,

t_crit <- qt(0.975, 30-6)
-0.02394 + c(-1,1) * t_crit * 0.02242

[1] -0.07021261  0.02233261

• Q: does this indicate we can reject the null hypothesis \( H_0: \boldsymbol{\beta}_{Area}=0 \) at \( 5\% \) significance?

Confidence intervals – continued

• A: notice, the interval contains \( 0 \) so that we cannot reject the null at \( 5\% \) significance.

-0.02394 + c(-1,1) * t_crit * 0.02242

[1] -0.07021261  0.02233261

• This can be seen dually through the p-value of the parameter for area

sumary(lmod)

             Estimate Std. Error t value  Pr(>|t|)
(Intercept)  7.068221  19.154198  0.3690 0.7153508
Elevation    0.319465   0.053663  5.9532 3.823e-06
Nearest      0.009144   1.054136  0.0087 0.9931506
Scruz       -0.240524   0.215402 -1.1166 0.2752082
Area        -0.023938   0.022422 -1.0676 0.2963180
Adjacent    -0.074805   0.017700 -4.2262 0.0002971

n = 30, p = 6, Residual SE = 60.97519, R-Squared = 0.77

Confidence intervals – continued

• Similarly, we can compute the \( 95\% \) confidence interval for \( \boldsymbol{\beta}_{Adjacent} \) with,

  • the \( t_{30-6}^{2.5} \) critical value; and
  • the \( se(\boldsymbol{\beta}_{Adjacent}) \)

-0.07480 + c(-1,1) * t_crit * 0.01770

[1] -0.111331 -0.038269

• Q: Based on the above confidence interval, can we reject the null hypothesis \( H_0: \boldsymbol{\beta}_{Adjacent}=0 \) with \( 5\% \) significance?

• A: Yes.

  • However, while we say (with \( 95\% \) confidence) that the effect of the adjacent island area is negative on the response, but the uncertainty is large, with the width of the interval on the same order as the coefficient itself,

(-0.038269 + 0.111331)/abs(-0.07480)

[1] 0.9767647

Confidence intervals – continued

• Confidence intervals, while dual to hypothesis test, provide slightly more information than the p-values.

• Particularly, we can examine a range of equally plausible values for the parameter of interest.

  • If the confidence interval is relatively tight around the value of the parameter, we can provide more evidence of the actual effect of the predictor.
  • However, if the range of values is large, this indicates a great deal of uncertainty of the effect of the predictor and the scale of the effect.
  • When the confidence interval contains zero, then it is equally plausible that the effect of changing the predictor will be positive or negative (or nothing) on the response.

Confidence intervals – continued

• We can find the confidence intervals of all parameters simultaneously (at the \( 95\% \) confidence level) as

 confint(lmod)

                  2.5 %      97.5 %
(Intercept) -32.4641006 46.60054205
Elevation     0.2087102  0.43021935
Nearest      -2.1664857  2.18477363
Scruz        -0.6850926  0.20404416
Area         -0.0702158  0.02233912
Adjacent     -0.1113362 -0.03827344

• Setting another confidence level, e.g., \( 99\% \) can be done with a keyword argument,

 confint(lmod, level = 0.99)

                  0.5 %      99.5 %
(Intercept) -46.5049119 60.64135329
Elevation     0.1693731  0.46955638
Nearest      -2.9392105  2.95749844
Scruz        -0.8429913  0.36194283
Area         -0.0866523  0.03877562
Adjacent     -0.1243112 -0.02529848

• Q: why is the \( 99\% \) confidence interval wider than the \( 95\% \) confidence interval?

• A: to be more confident in the location of the true value, we need to widen the interval correspondingly.

Confidence regions

• The previous example only considered the univariate confidence interval, analogous to the t-test for a single parameter, i.e.,

  • \( H_0: \boldsymbol{\beta}_{p-1} = 0 \)
  • \( H_1: \boldsymbol{\beta}_{i} \neq 0 \), \( i=0,\cdots, p-1 \).

• Suppose we want (analogous to the F-test) to find a multivariate confidence region.

• For \( \alpha \) the significance level, we find the \( 100(1-\alpha)\% \) confidence region for \( \hat{\boldsymbol{\beta}} \) is given by the relationship

\[ \begin{align} \left(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta}\right)^\mathrm{T}\left(\mathbf{X}^\mathrm{T}\mathbf{X} \right) \left(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta}\right) \leq p \hat{\sigma}^2 F^{(\alpha)}_{p, n-p} \end{align} \]

• The above can be interpreted as a weighted inner product of \( \left(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta}\right) \) with itself with respect to its estimated inverse covaraince, i.e., a measure of distance squared, weighted by \( \frac{1}{\hat{\sigma}^2}\left(\mathbf{X}^\mathrm{T}\mathbf{X} \right) \).

• We say that with \( 100(1-\alpha)\% \) confidence the true parameter \( \boldsymbol{\beta} \) will lie within the domain of the weighted distance function, for which the output is less than \[ p \hat{\sigma}^2 F^{(\alpha)}_{p, n-p}. \]

Confidence regions – continued

• We can visualize such a region in 2-D, but in higher dimensions it becomes a hyper-ellipsoid.
• The package for creating confidence regions is not part of base R, and requires the installation of the package,

install.packages("ellipse"))

• We will produce a confidence ellipse for the two parameters area and adjacent simultaneously with the following code:

plot(ellipse(lmod,c(2,6)),type="l",ylim=c(-0.13,0))
points(coef(lmod)[2], coef(lmod)[6], pch=19)
abline(v=confint(lmod)[2,],lty=2)
abline(h=confint(lmod)[6,],lty=2)

• The above has the effect of creating a confidence interval for the second and sixth parameter of the model lmod.

• We plot additional points for the parameter estimates, as well as include lines in the plot for the univariate confidence intervals.

• The result is pictured next…

Confidence regions – continued

Courtesy of: Faraway, J. Linear Models with R. 2nd Edition