Analysis of variance approach to simple linear regression

Outline

• The following topics will be covered in this lecture:

  • Analysis of variance approach to regression
  • Decomposing the variation
  • Degrees of Freedom
  • Mean squares and the ANOVA table

Analysis of variance approach

• We have seen one approach now for regression analysis which will be the basic framework in which we consider these linear models.
• However, there are additional ways to approach the regression model, among which is known as Analys of Variance or ANOVA.
• This approach, which we will introduce in the following, seeks to partition the variation in the signal into different components for creating hypothesis tests.
• We will introduce the main concepts here, which will underpin a number of the techniques which we will introduce in full generality in multiple regression.

Total sum of squares

• We note, there are several forms of variation in our regression analysis.
• Among these is the variation of the response variable around its empirical, sample-based mean, \[ Y_i - \overline{Y} \]
• Analogously to how we earlier defined the RSS in terms of the squared-deviations of \( Y_i \) from the regression-estimated mean response, \[ RSS = \sum_{i=1}^n \hat{\epsilon}_i^2; \]
• we will define the Total Sum of Squares (TSS) in terms of the squared-deviations of \( Y_i \) from the sample-based mean of the response: \[ TSS\triangleq \sum_{i=1}^n \left( Y_i - \overline{Y}\right)^2. \]
• Q: if all observations of the response variable have the same value, then what value does the TSS is attain?
• A: the TSS must equal zero, as \( Y_i = \overline{Y} \) for all \( i \).
• In this regard, the greater the overall variation in the response variable across all cases, then the greater is the TSS.
• The TSS represents the variation around a null model, in which we would consider the variation present in the response to be random variation around its sample-based mean, irrespective of the explantory variable \( X \).
• In general, the RSS does not equal the TSS for the reason described above
• in particular, if there is a signal in the data, we expect there to be less variation in the RSS than in the TSS.

Residual sum of squares

• While we can consider the TSS a measure of the total variation around the null model of random variation around the mean, we can also consider how much of this variation is “explained”.
• Particularly, consider the quantity, the Explained Sum of Squares (ESS) \[ ESS = \sum_{i=1}^n\left(\hat{Y}_i - \overline{Y}\right)^2; \]
• This represents how much variation in the signal is explained by our regression model;
• if our regression model is the null model, i.e., the \( i \)-th fitted value is just the sample-based mean of the observed responses, \( \hat{Y}_i =\overline{Y} \), then \( ESS=0 \).
• Therefore, as we will show in the following, we can generally consider a larger \( ESS \) corresponding to a regression model with better performance.

Partitioning the errors

• To demonstrate the meaning of the ESS corresponding to a better performance, we consider the following partition of the variation in the response, \[ \underbrace{Y_i - \overline{Y}}_{TSS} = \underbrace{\hat{Y}_i - \overline{Y}}_{ESS} +\underbrace{ Y_i - \hat{Y}_i}_{RSS}, \] where we say each term loosely corresponds to the TSS, ESS, or RSS as above.
• This corresponds in a loose sense to decomposing the total deviation of the response around the mean into:
1. the deviation of the fitted values around the mean (ESS), plus
2. the deviation of the observed values from the fitted values (RSS)
• Q: how do we obtain the equality of the right-hand-side with the left-hand-side above?
• A: we can always add zero to any equation to acheive equality, i.e., \[ Y_i - \overline{Y} = Y_i - \overline{Y} + \left(\hat{Y}_i - \hat{Y}_i \right). \]
• Re-arranging terms recovers the decomposition as above.

Partitioning the errors – continued

• While we have motivated the decomposition of the TSS, we haven’t actually shown the decomposition.
• Specifically, we need to demonstrate that, \[ \sum_{i=1}^n \left(Y_i - \overline{Y}\right)^2 = \sum_{i=1}^n \left(\hat{Y}_i - \overline{Y}\right)^2 + \sum_{i=1}^n \left(Y_i - \hat{Y}_i \right)^2, \] which is non-trivial, and is a consequence of the choice of the estimation by least squares.
• We will begin by adding zero and expanding terms, \[ \begin{align} TSS&= \sum_{i=1}^n \left[Y_i - \overline{Y}\right]^2 \\ & = \sum_{i=1}^n \left[ \left(\hat{Y}_i - \overline{Y}\right) + \left(Y_i - \hat{Y}_i \right)\right]^2\\ &= \sum_{i=1}^n \left[ \left(\hat{Y}_i - \overline{Y}\right)^2 + \left(Y_i - \hat{Y}_i \right)^2 + 2 \left(\hat{Y}_i - \overline{Y}\right)\left(Y_i - \hat{Y}_i \right)\right]\\ &= ESS + RSS + 2\sum_{i=1}^n \left(\hat{Y}_i - \overline{Y}\right)\left(Y_i - \hat{Y}_i \right). \end{align} \]
• Therefore, we need to demonstrate that the sum of cross terms vanishes to prove the partition.

Partitioning the errors – continued

• It will be sufficient to show that \[ \sum_{i=1}^n \left(\hat{Y}_i - \overline{Y}\right)\left(Y_i - \hat{Y}_i \right) =0 \]
• We will study this property in the class activity, proving that \[ TSS = ESS + RSS. \]
• In the above form, we see the tradeoff between the two terms in the \( TSS \), particularly,
  1. When the \( RSS \) is large, this says:
     • the squared-distance between the fitted and the observed values, \( RSS= \sum_{i=1}^n\hat{\epsilon}_i^2 \), is large;
     • particularly, the \( ESS \) (explained variation) is small and the fit is close to the null model.
  2. When the \( ESS \) is large, this says:
  • the squared-distance between the fitted and the empirical, sample-based mean, \( ESS=\sum_{i=1}^n \left(\hat{Y}_i - \overline{Y}\right)^2 \), is large;
  • particularly, the \( RSS \) is small, implying a close fit between the predicted and the observed values.

Goodness of fit

• With the last discussion as a motivation, we can introduce our first metric for the “goodness of fit” of a regression model.
• A common choice to examine how well the regression model actually fits the data is called the “coefficient of determination” or “the percentage of variance explained”.
• For short, we define, \[ R^2 = 1 - \frac{\sum_{i=1}^n \left( Y_i - \hat{Y}_i\right)^2}{\sum_{i=1}^n \left(Y_i - \overline{Y}\right)^2} = 1 - \frac{RSS}{TSS} \]
• Q: recalling the realtionship \( TSS = ESS + RSS \), what is the possible range of \( R^2 \) and what the maximal and minimal value correspond to.
• A: if \( RSS=TSS \), then we have a value of \( R^2=0 \), corresponding to a null model, i.e., simply random variation about the sample-based mean.
• The smallest value \( RSS \) can attain is \( 0 \), in which case \( R^2=1 \), corresponding to the case where all fitted values equal the observed value.
• Generally, we consider a model with \( R^2 \) close to one a “good” fit, and \( R^2 \) close to zero a bad fit.
• Note: this metric has a number of flaws, which we will discuss further in the course.
• However, this metric is commonly used enough and is of great enough historical importance that we should understand it.

A visual representation

Courtesy of: Faraway, J. Linear Models with R. 2nd Edition

Computing \( R^2 \) in the R language

• Our definition for \( R^2 \) is the same one used in the language R, so we emphasize this.
• However, this definition assumes that there is an intercept term for the model.
• If there is no intercept term, i.e., \( \beta_0 = 0 \), then \( R^2 \) is equal to the correlation of the fitted values with the observed values, squared: \[ \begin{align} R^2 & = cor^2 \left(\hat{Y}, Y\right) \end{align} \]
• The value of \( R^2 \) should be computed from this definition if the model is fitted without intercept.
• If we don’t take care to do it this way, the coefficient of determination will be misleadingly high.
• Note: for this reason, we must take care about using the model summary command in the R language, when we have a model without intercept.
• Defining a model without intercept \( \beta_0=0 \) is usually an uncommon assumption, and it is taken only when there is a good “physical” meaning to taking a model without intercept.
• For example, if we are forming a model for a population size based on the food supply as the predictor, there is a clear “physical” meaning for \( \beta_0=0 \).
• It would be reasonable to require that if there is zero food supply, the population should be zero or randomly fluctuate around zero due to migrations through the study area.
• In general, however, the \( beta_0 \) is usually used simply as the intercept and may not have a real meaning for the relationship, only that it sets a base level for the response, appropriate for the scope of the model.

What are “good” values of \( R^2 \)?

• There is no universal “good” value for \( R^2 \) to attain in practice.
• For physics and engineering applications, data will be produced in tightly controlled experiments.
• Measurement noise will typically be low, and there are strong correlation and causality relationships in these settings.
• In that case, we expect \( R^2 \) to be close to one in order to say the fitted values model the observations well.
• In the social sciences, there is much more variability, typically causal relationships (if they exist) are not well understood and correlations are weaker.
• In this case, we will typically expect a “good fit” to have a much lower \( R^2 \) score.

Simulated examples of \( R^2\approx.65 \)

Courtesy of: Faraway, J. Linear Models with R. 2nd Edition

Breakdown of degrees of freedom

• We have used the notion of the “degrees of freedom” loosely up to this point, and we want to formalize this analysis.
• The degrees of freedom refer to the number of values that are free to vary (the number of free parameters or independent variables) in the computation of some statistic.
• In particular, this can be considered geometrically for a set of of \( n \) observations of the response, \( \left\{Y_i\right\}_{i=1}^n \);
• If we identify the \( n \) observations as an \( n \)-dimensional vector \[ \mathbf{Y} = \begin{pmatrix} Y_1,& \cdots, &Y_n\end{pmatrix}^\mathrm{T} , \] we say that as a random vector, it can attain a value in any subspace of the \( n \)-dimensional space \( \mathbb{R}^n \).
• Suppose we have the sample-based mean defined as before as \( \overline{Y} = \frac{1}{n}\sum_{i=1}^n Y_i \).
• Then, we can re-write the random vector \( \mathbf{Y} \) in terms of two objects, one which lives in \( 1 \)-dimensional space and one which lives in \( n-1 \)-dimensional space: \[ \mathbf{Y} = \overline{Y} \begin{pmatrix}1 & \cdots & 1\end{pmatrix}^\mathrm{T} + \begin{pmatrix} Y_1 - \overline{Y} & \cdots & Y_n - \overline{Y}\end{pmatrix}^\mathrm{T} \]
• The first quantity on the right-hand-side is constrained to live in the \( 1 \)-dimensional subspace that is spanned by the vector \( \begin{pmatrix}1 & \cdots & 1\end{pmatrix}^\mathrm{T} \); here the only “free” parameter is the value of \( \overline{Y} \).

Breakdown of degrees of freedom – continued

• Continuing our ananlysis: \[ \mathbf{Y} = \overline{Y} \begin{pmatrix}1 & \cdots & 1\end{pmatrix}^\mathrm{T} + \begin{pmatrix} Y_1 - \overline{Y} & \cdots & Y_n - \overline{Y}\end{pmatrix}^\mathrm{T} \]
• The second quantity on the right-hand-side may appear have \( n \)-dimensions of possible values, but there is a constraint implied by the used degree of freedom: \[ \sum_{i=1}^n \left(Y_i - \overline{Y}\right) =0, \] such that there are only \( n-1 \) degrees of freedom left over.
• Particularly, the second quantity (sometimes referred to as the anomalies) live in a \( n-1 \)-dimensional subspace of the full \( \mathbb{R}^n \) space.
• Therefore, when we compute the unbiased sample-based variance, the normalization by \( n-1 \) makes sense by the fact that the quantity \[ \sum_{i=1}^n \left(Y_i - \overline{Y}\right)^2, \] has only \( n-1 \) degrees of freedom, or values that are not yet determined.
• This is analogous to the earlier lecture when we discussed the over constrained/ under constrained/ unique solution to finding a line through data points in the plane.

Breakdown of degrees of freedom – continued

• In the case of estimating the regression function, we see similarly, \[ \hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X \] we estimate two parameters as linear combinations of the observed cases \( (X_i,Y_i) \).
• We consider the \( Y_i \) to be the free values here, while the two normal equations provide two constraints to the estimated regression function.
• Correspondingly, as we introduce more parameters \( p \) in the model, we will use more degrees of freedom, solving a system of equations for \( p \) parameters;
• equivalently, we will put more constraints on the system until it becomes uniquely (or eventually over-) constrained, with \( n-p \) degrees of freedom available to determine the regression function.
• Q: We said that the definition of the unbiased estimate for \( \sigma^2 \) generalizes to multiple regression, simply by increasing the number of \( p \) parameters to account for the additional estimated quantities in our regression.
• Suppose \( n=p \), what is our unbiased estimate of the variance \( \sigma^2 \)?

Breakdown of degrees of freedom – continued

• A: recall the definition, \[ \hat{\sigma}^2 \triangleq \frac{RSS}{n-p}, \] such that if \( n-p=0 \), the equation is undefined.
• Indeed, if \( n-p=0 \) this is a completely constrained system, with a unique value for the regression fuction — this is actually a serious issue of overfitting, which we will return to later.
• Particularly, one issue we can see already is that we do not have a means of uncertainty quantification for our estimates.
• If \( n-p<0 \), we have an overconstrained or “super-saturated” model for which different techniques entirely are needed for the analysis.

Degrees of freedom of TSS and RSS

• By the earlier discussion, we say that the TSS, \[ TSS = \sum_{i=1}^n \left(Y_i - \overline{Y}\right)^2 \] has \( n-1 \) degrees of freedom.
• This corresponds to the fact that there are \( n \) values that the observations can attain, with one constraint from the sample-based mean.
• Similarly, we find that the RSS, \[ \begin{align} RSS &= \sum_{i=1}^n \left( Y_i - \hat{Y}_i \right)^2\\ &= \sum_{i=1}^n \left( Y_i - \hat{\beta}_0 - \hat{\beta}_1X_i \right)^2 \end{align} \] has \( n-p \) degrees of freedom \( (p=2) \), because there are \( p \) constraints on this relationship given any \( n \) possible values that \( Y_i \) attain.

Degrees of freedom of ESS

• Let us derive the number of degrees of freedom of the explained sum of squares, \[ ESS = \sum_{i=1}^n \left(\hat{Y}_i - \overline{Y}\right)^2 \]
• Q: we will use the property that the mean of the fitted values is equal to the mean of the observed values, i.e., \[ \frac{1}{n}\sum_{i=1}^n \hat{Y}_i = \overline{Y}; \] using any of the properties we have proven already about the regression function, can you show why this is?
• A: one useful property we have shown with the normal equations is that the sum of the residuals is zero, i.e., \[ \sum_{i=1}^n \hat{\epsilon}_i = 0. \]
• Therefore, we can consider that \[ \overline{Y} =\frac{1}{n} \sum_{i=1}^n Y_i = \frac{1}{n}\sum_{i=1}^n\left( \hat{Y}_i + \hat{\epsilon}_i\right) = \frac{1}{n}\sum_{i=1}^n \hat{Y}_i \]

Degrees of freedom of ESS – continued

• Recalling the form for the explained sum of squares, \[ \begin{align} ESS &= \sum_{i=1}^n \left(\hat{Y}_i - \overline{Y}\right)^2 \\ &=\sum_{i=1}^n \left[\hat{\beta}_0 + \hat{\beta}_1X_i - \left(\frac{1}{n}\sum_{i=1}^n \hat{Y}_i \right) \right]^2 \end{align} \] where the above used the relationship we just proved.
• Q: in what way can we simplify the above expression?
• A: one way is to substitute the definition of the fitted value \( \hat{Y}_i \) once again and cancel terms, \[ \begin{align} ESS &=\sum_{i=1}^n \left\{ \hat{\beta}_0 + \hat{\beta}_1X_i - \left[\frac{1}{n}\sum_{i=1}^n \left(\hat{\beta}_0 + \hat{\beta}_1 X_i \right)\right] \right\}^2 \\ &=\sum_{i=1}^n \left\{ \hat{\beta}_0 + \hat{\beta}_1X_i - \hat{\beta}_0 - \hat{\beta}_1 \overline{X} \right\}^2\\ &= \hat{\beta}_1^2 \sum_{i=1}^n \left(X_i - \overline{X}\right)^2 \end{align} \]

Degrees of freedom decomposition

• From the last derivation, we have that \[ \begin{align} ESS &= \hat{\beta}_1^2 \sum_{i=1}^n \left(X_i - \overline{X}\right)^2 \end{align} \]
• Although the the \( ESS \) is computed from \( n \) deviations, they are all derived from the same regression line.
• If we suppose the regression line is the free value in this case, it has two degrees of freedom described by its slope \( \hat{\beta}_1 \) and its intercept \( \hat{\beta}_0 \).
• However, as we saw earlier, we cancel the terms with the intercept such that \( \hat{\beta}_1 \) is the only degree of freedom (free parameter) in the \( ESS \).
• Therefore, we say that the \( ESS \) has one degree of freedom.
• An important consequence of this for the analysis of variance approach is that the degrees of freedom, like the total variation, are additive:, \[ \underbrace{TSS}_{n-1} = \underbrace{ESS}_{p - 1} + \underbrace{RSS}_{n-p}, \] where \( p=2 \) in simple regression.
• The above concept and the geometry likewise generalize to multiple regression, which we will come to shortly.

Mean Squares

• A sum of squares, such as, the \( TSS \), \( ESS \) or \( RSS \) when divided by its associated degrees of freedom is referred to as a mean square.
• Therefore, we will identify the following quantities:
1. the regression mean square: \( \frac{ESS}{p-1} \);
2. the residual mean square error: \( \frac{RSS}{n-p} \);
• Q: we have mentioned once before that one of the above is an unbiased estimator — can you recall what is the value of, \[ \mathbb{E}\left[\frac{RSS}{n-p}\right]? \]
• A: the residual mean square error, denoted \( \hat{\sigma}^2 \) is an unbaised estimator for \( \sigma^2 \); therefore, \[ \mathbb{E}\left[\frac{RSS}{n-p}\right] = \sigma^2 \]

Mean Squares – continued

• It can be shown that similarly, the regression mean square has an expectation, \[ \mathbb{E}\left[\frac{ESS}{1} \right]= \sigma^2 + \beta_1^2 \sum_{i=1}^n \left(X_i - \overline{X}\right)^2 \]
• Note, however, while the residual mean square error takes the form for higher dimensions, the above regression mean square does not.
• Q: suppose \( \beta_1\neq 0 \), which is larger, the expected regression mean square or the expected residual mean square error?
• A: provided all cases don’t correspond to the same value \( X_i \), the sum of squares \( \sum_{i=1}^n\left(X_i - \overline{X}\right)^2 \) is positive.
• Therefore, comparing the two values of the regression mean square and the residual mean square error provides some means to determine “how likely is it that \( \beta_1=0 \)?”
• Particularly, the expected value of a mean square gives the mean around which the sample-based estimate will vary;
• if \( \beta_1 \neq 0 \), we expect the regression mean square to attain a value greater than the RSS.
• This type of comparison will underpin our hypothesis tests, which we will introduce shortly in multiple regression.

The ANOVA table

• Collecting all the information we have developed so far in the analysis of variance framework, we arrive at the ANOVA table.
• A sample ANOVA table:

  Source	Degrees of freedom	Sum of squares	Mean square	F-statistic
  Regression	\( p-1 \)	\( ESS \)	\( \frac{ESS}{p-1} \)	\( F \)
  Residual	\( n-p \)	\( RSS \)	\( \frac{ESS}{n-p} \)
  Total	\( n-1 \)	\( TSS \)	\( \frac{TSS}{1} \)

• The R language will provide an ANOVA table that arranges the information we have discussed, similarly to the above.
• The piece of data we haven’t discussed so far is the one we have been alluding to — the value of the F-statistic for hypothesis testing.
• It is not strictly necessary to compute all the elements of the table — as the originator of the table, Fisher said in 1931, it is “nothing but a convenient way of arranging the arithmetic.”
• When we introduce multiple regression, we will return to this table to interpret our results in terms of hypothesis testing versus the null model.