Instructor: Colin Grudzien

## Instructions: We will work through the following series of activities as a group and hold small group work and discussions in Zoom Breakout Rooms. Follow the instructions in each sub-section when the instructor assigns you to a breakout room. ## Activities: ### Activity 1: Deriving the normal equations The "normal" equations are derived exactly in terms of the objective function we studied earlier that measures the sum of square deviations of the proposed mean response from the observed cases: $$J\left(\overline{\beta}_0,\overline{\beta}_1\right) = \sum_{i=1}^n \left(Y_i - \overline{\beta}_0 - \overline{\beta}_1 X_i\right)^2,$$ Note that $\hat{\beta}_0,\hat{\beta}_1$ are **defined** as the unique minimizers of $J$, and are therefore critical values for the function. Use this information to derive the normal equations for $\hat{\beta}_0,\hat{\beta}_1$: $$\begin{align} \sum_{i=1}^n Y_i &= n\hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^n X_i \\ \sum_{i=1}^n X_i Y_i & = \sum_{i=1}^n \left(\hat{\beta}_0 X_i + \hat{\beta}_1 X_i^2\right) \end{align}$$ #### Solution to activity 1: The objective function is convex with a global minimum, and therefore has a single critical point corresponding to this minimum. In each of the arguments, $\overline{\beta}_0$ and $\overline{\beta}_1$, we can take the partial derivative of the objective function, $$\begin{align} \partial_{\overline{\beta}_0} J &=\sum_{i=0}^n -2\left(Y_i - \overline{\beta}_0 - \overline{\beta}_1 X_i\right) \\ \partial_{\overline{\beta}_1} J &= \sum_{i=0}^n -2 X_i \left(Y_i - \overline{\beta}_0 - \overline{\beta}_1 X_i\right) \end{align}$$ Setting these partial derivatives to zero when evaluated at $\hat{\beta}_0,\hat{\beta}_1$, we find $$\begin{align} \sum_{i=1}^n\left(Y_i\right) - n \hat{\beta}_0 - \sum_{i=1}^n \hat{\beta}_1\left( X_i\right) & = 0 \\ \sum_{i=0}^n \left( X_i Y_i\right) - \hat{\beta}_0 \sum_{i=1}^n \left(X_i\right) - \hat{\beta}_1\sum_{i=1}^n \left( X_i^2\right) & = 0 \end{align}$$ By re-arranging terms, we find the normal equations, $$\begin{align} \sum_{i=1}^n Y_i &= n\hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^n X_i \\ \sum_{i=1}^n X_i Y_i & = \sum_{i=1}^n \left(\hat{\beta}_0 X_i + \hat{\beta}_1 X_i^2\right) \end{align}$$ ### Activity 2: Use the definition of the normal equations $$\begin{align} \sum_{i=1}^n Y_i &= n\hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^n X_i ,\\ \sum_{i=1}^n X_i Y_i & = \sum_{i=1}^n \left(\hat{\beta}_0 X_i + \hat{\beta}_1 X_i^2\right), \end{align}$$ and the definition of the residuals, $$\begin{align} \hat{\epsilon}_i &= Y_i - \hat{Y}_i ,\end{align}$$ to prove the following statements:

- The sum of the residuals is zero, i.e., $$\sum_{i=1}^n \hat{\epsilon}_i = 0$$
- The sum of the observed values equals the sum of the fitted values, $$\sum_{i=1}^n Y_i = \sum_{i=1}^n \hat{Y}_i$$
- The sum of the residuals, weighted by the corresponding predictor variable, is zero, $$\sum_{i=1}^n X_i \hat{\epsilon}_i = 0$$
- The sum of the residuals, weighted by the corresponding fitted value, is zero, $$\sum_{i=1}^n \hat{Y}_i \hat{\epsilon}_i = 0$$
- The estimated regression function always passes through the point defined by the means (or the center of mass), $\left(\overline{X},\overline{Y}\right)$.