Hypothesis testing

Outline

• The following topics will be covered in this lecture:
  • A review of hypothesis testing
  • Types of errors
  • Power of statistical tests
  • P values
  • General hypothesis testing of the mean

Hypothesis testing – a review

• In hypothesis testing, the null and alternative hypotheses have special meanings philosophically and in the mathematics.

• We cannot generally “prove” a hypothesis to be true;

  • generically, we will assume that the true population parameter is unobservable.

• Instead, we can only determine if a hypothesis seems unlikely enough to reject;

  • this is similar to finding that our proposed parameter value was in far-fewer confidence intervals than predicted by the procedure.

• To begin such a test formally, we need to first make some assumption about the true parameter.

  • This always takes the form of assuming the null hypothesis \( H_0 \).

• The null hypothesis \( H_0 \) will always take the form of an equality, or an inclusive inequality.

  • That is, we take

  \[ \begin{align} H_0: & \theta \text{ is } (= / \leq / \geq) \text{ some proposed value}. \end{align} \]

  • In our example, we wrote

  \[ \begin{align} H_0∶ & \mu = 50 \text{ centimeters per second}. \end{align} \]

Hypothesis testing – a review

• The contradictory / competing hypothesis is the alternative hypothesis, written

  \[ \begin{align} H_1: & \theta \text{ is } (\neq / > / <) \text{ some proposed value} \end{align} \]

  • In our example, we wrote

  \[ \begin{align} H_1∶ & \mu \neq 50 \text{ centimeters per second}. \end{align} \]

• Once we have formed a null and alternative hypothesis:

  \[ \begin{align} H_0: & \theta \text{ is } (= / \leq / \geq) \text{ some proposed value}\\ H_1: & \theta \text{ is } (\neq / > / <) \text{ some proposed value} \end{align} \]

• we use the sample data to consider how likely or unlikely it was to observe such data with the proposed parameter.

  • If the sample doesn't seem to fit the proposed parameter value, we deem the null hypothesis unlikely.

• If the null hypothesis is sufficiently unlikely, we reject the null hypothesis in favor of the alternative hypothesis.

• However, if the evidence (the sample) doesn't contradict the null hypothesis, we tentatively keep this assumption.

  • This has not proven this assumption, it has only said that the hypothesis is not unlikely given our evidence.

• In our example, we would say either:

  1. we reject the null hypothesis of \( H_0∶ \mu = 50 \) in favor of the alternative \( H_1: \mu \neq 50 \); or
  2. we fail to reject the null hypothesis of \( H_0:\mu = 50 \).

Decision citeria for hypothesis testing – types of errors

• Let's recall the definition of \( \alpha \):

  Probability of Type I Error
  \[ \alpha = P(\text{type I error}) = P(\text{reject }H_0\text{ when }H_0\text{ is true}) \]

• Suppose that the null hypothesis is true, i.e, \( \tilde{\mu} = \mu \), yet we find that

  \[ \tilde{\mu} \notin \left( \overline{x} - \sigma_\overline{X} z_\frac{\alpha}{2} , \overline{x} + \sigma_\overline{X} z_\frac{\alpha}{2}\right). \]

• If \( H_0 \) is actually true, then concluding that \( \tilde{\mu} \) is not a reasonable value for \( \mu \) is precisely a type I error.

• If we have constructed a \( (1-\alpha)\times 100\% \) confidence interval, the rate at which

  \[ \tilde{\mu} \notin \left( \overline{X} - \sigma_\overline{X} z_\frac{\alpha}{2} , \overline{X} + \sigma_\overline{X} z_\frac{\alpha}{2}\right) \] with respect to infinite replications is precisely the rate of failure, \( \alpha \).

• Therefore, we have the equivalence:

  \[ (1-\alpha)\times 100\% \text{ confidence} \Leftrightarrow \alpha = P(\text{type I error}). \]

• The above relationship expresses the duality of confidence intervals and hypothesis tests.

• This explains, in part, why t.test() computes both a confidence interval and hypothesis test simultaneously;

  • the two procedures are formally equivalent.

Decision citeria for hypothesis testing – types of errors

• In evaluating a hypothesis-testing procedure, it is also important to examine the probability of a type II error, which we denote by \( \beta \).

Probability of Type II Error
\[ \beta = P(\text{type II error}) = P(\text{failing to reject }H_0\text{ when }H_0\text{ is false}). \] The complementary probability, \( 1- \beta \) is called the power of the hypothesis test.

• To calculate \( \beta \), we must have a specific alternative hypothesis;

  • that is, we must have a particular value of \( \mu \).

• This is because, the unknown, true alternative hypothesis for \( \mu \) will determine the sampling distribution for \( \overline{X} \).

• For example, suppose that it is important to reject the null hypothesis

  \[ H_0 : \mu = 50 \]

  whenever the mean burning rate \( \mu \) is greater than \( 52 \) centimeters per second or less than \( 48 \) centimeters per second.

• Assuming that the true sampling distribution is centered at \( \mu=52 \) or \( \mu=48 \), we can determine the probability of a type II error \( \beta \);

  • we will assume that the decision rule uses the false hypothesis \( \mu=50 \), without knowing the true parameter.

• We will estimate how the test procedure will work probabilistically if we wish to reject \( H_0 \), for a true mean value of \( \mu = 52 \) or \( \mu = 48 \).

Decision citeria for hypothesis testing – types of errors

Courtesy of Montgomery & Runger, Applied Statistics and Probability for Engineers, 7th edition

Decision citeria for hypothesis testing – types of errors

• From the last slide, we can compute the probability of a type II error, in the case that \( H_0: \mu=50 \) is false, where \( \mu=52 \) is the true value, by

  \[ \beta = P(48.5 \leq \overline{X} \leq 51.5 \text{ assuming that }\mu = 52). \]

• Recall, we said that the standard error in this example was known as,

  \[ \sigma_\overline{X} = \frac{\sigma}{\sqrt{n}} = \frac{2.5}{\sqrt{10}} \approx 0.79 \]

• Therefore, with R, this can be computed directly as follows:

se <- 2.5 / sqrt(10)
mu <- 52
beta <- pnorm(51.5, mean=mu, sd=se) - pnorm(48.5, mean=mu, sd=se)
beta

[1] 0.2635399

• (Note, once again, the values in the book differ slightly due to approximation errors throughout the manual calculations).

Decision citeria for hypothesis testing – a summary of the decision procedure

• Now that we have established the fundamental tools of hypothesis testing:

  1. type I and type II errors, and their probability;
  2. the connection between a hypothesis test decision rule and the equivalent confidence interval;

• we will discuss how one formally goes through a hypothesis test.

• Let's suppose that we have the sample of observations on speed up times, from a previous exercise:

speed_up_times <- c(3.775302,   3.350679,   4.217981,   4.030324,   4.639692, 4.139665, 4.395575,   4.824257,   4.268119,   4.584193, 4.930027, 4.315973,   4.600101)
n <- length(speed_up_times)
n

[1] 13

• We will suppose that we set a value \( \alpha=0.05 \) in advanced – this is a standard level to set the probability of type I error but can be different in practice.

  • Similarly, this corresponds to a \( (1 - \alpha)\times 100\% = 95\% \) level of confidence.

Decision citeria for hypothesis testing – a summary of the decision procedure

• We will also need to specify a null and alternative hypothesis in advanced – let these be:

  \[ \begin{align} H_0 : \mu = 4.0 & & H_1 : \mu \neq 4.00 \end{align} \]

• The procedure is then as follows:

  • We will compute the sample mean, and estimate the probability of observing such a value assuming the null hypothesis.

• We can compute the sample mean as follows:

x_bar <- mean(speed_up_times)
x_bar

[1] 4.313222

• We need to evaluate how unlikely it is to observe x_bar under the assumption that \( \mu = 4.0 \).

• The model for this probability will depend on whether the true population \( \sigma \) is known or unknown.

Decision citeria for hypothesis testing – a summary of the decision procedure

• Let's assume for simplicity at the moment that \( \sigma=0.45 \) is a known value;

  • then our model for \( \overline{X} \) is given as

  \[ \overline{X} \sim N\left(\mu, \frac{0.45^2}{13}\right). \]

• The standard error can be computed as

se <- 0.45/sqrt(13)
se

[1] 0.1248075

Decision citeria for hypothesis testing – a summary of the decision procedure

• The acceptance region for the two-sided hypothesis test is then given by,

mu <- 4.0
z_alpha_over_2 <- qnorm(0.975)
ar <- c(mu - se * z_alpha_over_2, mu + se * z_alpha_over_2)
ar

[1] 3.755382 4.244618

• while the sample mean is given by

x_bar

[1] 4.313222

• Notice that x_bar lies outside of the acceptance region – equivalently, our hypothesis value of the true mean \( 4.0 \) lies outside of the \( 95\% \) confidence interval

ci <- c(x_bar - se * z_alpha_over_2, x_bar + se * z_alpha_over_2)
ci

[1] 4.068604 4.557840

• Base on either equivalent procedure, we would say that the hypothesized value of \( \mu = 4.0 \) is unlikely given our observed sample.

Decision citeria for hypothesis testing – a summary of the decision procedure

• Specifically, with \( 95\% \) confidence, we can say that \( \mu=4.0 \) is not a plausible value for the mean based on the confidence interval.

• Alternatively, we can say that there is a probability of less than

lower_tail_probability <- pnorm(mu - se * z_alpha_over_2, mean=mu, sd=se)
upper_tail_probability <- 1 - pnorm(mu + se * z_alpha_over_2, mean=mu, sd=se)
alpha <- lower_tail_probability + upper_tail_probability
alpha

[1] 0.05

• of observing such a value for the sample mean with our model for the sampling distribution.

• Denoting \( \alpha \) the significance level we state:

  • We reject the null hypothesis \( H_0: \mu = 4.0 \) with \( \alpha=5\% \) significance.

Decision citeria for hypothesis testing – a summary of the decision procedure

• Continuing the last example, suppose we had a specific value for the alternative hypothesis in mind, \( H_1: \mu = 4.5 \).

• We can check the power of the hypothesis test versus this value of the alternative hypothesis as follows.

• Our the acceptance region for the hypothesis test is again given as

ar

[1] 3.755382 4.244618

• We fail to reject the null if the sample mean falls within this region.

• If we assume that the true model for the sample mean is given as

  \[ \begin{align} \overline{X} \sim N\left(4.5, \frac{0.45^2}{13}\right) \end{align} \]

• the probability of a type II error is given as

  \[ \begin{align} \beta = P\left( 3.755382 \leq \overline{X} \leq 4.244618\right) \end{align} \] given the above model.

Decision citeria for hypothesis testing – a summary of the decision procedure

• We can compute

  \[ \begin{align} \beta = P\left( 3.755382 \leq \overline{X} \leq 4.244618\right) \end{align} \] under the assumption

  \[ \begin{align} \overline{X} \sim N\left(4.5, \frac{0.45^2}{13}\right) \end{align} \] as

beta <- pnorm(4.244618, mean=4.5, sd=se) - pnorm(3.755382, mean=4.5, sd=se)
beta

[1] 0.02036803

• The power of the test (assuming that the true \( \mu=4.5 \) and that we have falsely hypothesized \( \mu=4.0 \)) is given as

1 - beta

[1] 0.979632

P-values in hypothesis testing

• One way to report the results of a hypothesis test is to state that the null hypothesis was or was not rejected at a specified \( \alpha \)-value or level of significance.

• This is called fixed significance level testing.

• The fixed significance level approach to hypothesis testing is very nice because it leads directly to the concepts of type II error and power;

  • these concepts are of considerable value in determining the appropriate sample sizes to use in hypothesis testing, similar to how we performed this for confidence intervals.

• However, the fixed significance level approach does have some disadvantages.

• In our last example, \( H_0 : \mu = 50 \) was rejected at the \( 0.05 \) level of significance.

• This statement of conclusions may be often inadequate because it gives the decision maker no idea about whether the computed value of the test statistic was just barely in the rejection region or whether it was very far into this region.

• Furthermore, stating the results this way imposes the predefined level of significance on other users of the information.

• This approach may be unsatisfactory because some decision makers might be uncomfortable with the risks implied by \( \alpha= 0.05 \).

• To avoid these difficulties, the P-value approach has been adopted widely in practice.

P-values in hypothesis testing

• The P-value is the probability that the test statistic will take on a value that is at least as extreme as the observed value of the statistic, when the null hypothesis \( H_0 \) is true.

• Thus, a P-value conveys much information about the weight of evidence against \( H_0 \);

  • a decision maker can draw a conclusion at any specified level of significance knowing the specific probability of the observed result.

• We now give a formal definition of a P-value.

P-Value
The P-value is the smallest level of significance that would lead to rejection of the null hypothesis \( H_0 \) with the given data.

• It is customary to consider the test statistic (and the data) significant when the null hypothesis \( H_0 \) is rejected;

  • therefore, we may think of the P-value as the smallest level \( \alpha \) at which the data are significant.

• In other words, the P-value is the observed significance level.

• Once the P-value is known, the decision maker can determine how significant the data are without the data analyst formally imposing a pre-selected level of significance.

P-values in hypothesis testing

• Recall our speedup time data:

speed_up_times <- c(3.775302,   3.350679,   4.217981,   4.030324,   4.639692, 4.139665, 4.395575,   4.824257,   4.268119,   4.584193, 4.930027, 4.315973,   4.600101)

• with sample mean

x_bar

[1] 4.313222

• Our null and alternative hypotheses were given as

  \[ \begin{align} H_0: \mu = 4.0 & & H_0: \mu \neq 4.0 \end{align} \]

• The alternative hypothesis specifies that the critical region under consideration is two-sided, like a two-sided confidence interval.

• Therefore, the P-value will measure the probability of observing a sample mean at least as far away as x_bar from \( \mu=4.0 \) in either direction, under the model

  \[ \overline{X} \sim N\left(\mu, \frac{0.45^2}{13}\right). \]

P-values in hypothesis testing

• To compute the associated probability, we can consider the z-score of the observed sample mean:

z_score <- (x_bar - 4.0)/se
z_score

[1] 2.509641

• Therefore, the observed value for the sample mean lies \( \approx 2.5 \) standard deviations to the right of the proposed mean.

• The P-value thus corresponds to the probability of observing a standard normal random variable taking a value at least as extreme as the z-score in either direction, i.e.,

  \[ \approx P(Z< -2.5 ) + P(Z> 2.5). \]

• We compute

P_value <- pnorm(-z_score) + (1 - pnorm(z_score))
P_value

[1] 0.01208539

• The observed significance level is thus \( \approx 1.2\% \).

General hypothesis testing of the mean

• When we studied confidence intervals, we differentiated how we could construct these in the case where the population variance was known or unknown.

• Particularly, we had to use the sample estimate for the standard error,

  \[ \frac{s}{\sqrt{n}} \] in place of the true standard error in our calculations.

• Under the assumption that the random sample \( X_i \) is normally distributed, we found that

  \[ T = \frac{X - \overline{x}}{\frac{s}{\sqrt{n}}} \] is t distributed in \( n-1 \) degrees of freedom.

• Using the associated \( t_\frac{\alpha}{2} \) critical value, we calculated the \( (1-\alpha)\times 100\% \) confidence interval as

  \[ \left( \overline{x} - \hat{\sigma}_\overline{X} t_\frac{\alpha}{2} , \overline{x} + \hat{\sigma}_\overline{X} t_\frac{\alpha}{2}\right). \]

• With the same development in the case \( \sigma^2 \) is known, we can replace the \( z_\frac{\alpha}{2} \) and \( z_\alpha \) critical values with t critical values to produce general hypothesis tests.

General hypothesis testing of the mean

• Specifically, if we have a formal hypothesis test

  \[ \begin{align} H_0:\mu = \tilde{\mu} & & H_1: \mu \neq \tilde{\mu}; \end{align} \] and if the variance \( \sigma^2 \) is also unknown;

• then assuming the null, we write the acceptance region as

  \[ \left( \tilde{\mu} - \hat{\sigma}_\overline{X} t_\frac{\alpha}{2} , \tilde{\mu} + \hat{\sigma}_\overline{X} t_\frac{\alpha}{2}\right). \]

• If the sample mean \( \overline{X} \) lies outside of the acceptance region, i.e., in the critical region,

  \[ \left(-\infty, \tilde{\mu} - \hat{\sigma}_\overline{X} t_\frac{\alpha}{2}\right) \cup \left( \tilde{\mu} + \hat{\sigma}_\overline{X} t_\frac{\alpha}{2}, \infty\right), \]

  • we reject the null hypothesis with \( \alpha \times 100\% \) significance.

• Alternatively, if the sample mean lies within the acceptance region, we fail to reject the null hypothesis with \( \alpha\times 100\% \) significance.

General hypothesis testing of the mean

• We similarly will compute the P value in the case where \( \sigma^2 \) is unknown.

• Our test statistic will be the value,

  \[ \begin{align} t_0 = \frac{\overline{X} - \tilde{\mu}}{\frac{s}{\sqrt{n}}} \end{align} \]

• Using the t distribution in \( n-1 \) degrees of freedom as our probability model, we calculate the probability of observing a value at least as extreme as \( T_0 \).

• If the P value falls below \( \alpha \), we can reject the null hypothesis at \( \alpha\times 100\% \) significance.

• This is how the t.test() function computes both the confidence interval and the hypothesis test simultaneously.

• In particular, it will assume a null hypothesis of \( H_0: \mu = 0 \) be default.

  • This default value is adjusted in the function keywords, along with the alternate hypothesis.

• We will now begin to consider how to use this function more generally.

General hypothesis testing of the mean – examples

• The sodium content of twenty 300-gram boxes of organic cornflakes was determined.

• The data (in milligrams) are as follows:

sodium_sample <- c(131.15, 130.69, 130.91, 129.54, 129.64, 128.77,130.72, 128.33, 128.24, 129.65, 130.14, 129.29, 128.71, 129.00, 129.39, 130.42, 129.53, 130.12, 129.78, 130.92)

• Let's suppose we want to test the hypothesis,

  \[ \begin{align} H_0: \mu = 130 & & H_1:\mu \neq 130; \end{align} \]

• If we use t.test() directly, notice the output

t.test(sodium_sample)

    One Sample t-test

data:  sodium_sample
t = 662.06, df = 19, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 129.3368 130.1572
sample estimates:
mean of x 
  129.747 

General hypothesis testing of the mean – examples

• Rather, to set the correct null and alternative hypothesis, we write,

t.test(sodium_sample, mu=130, alternative="two.sided")

    One Sample t-test

data:  sodium_sample
t = -1.291, df = 19, p-value = 0.2122
alternative hypothesis: true mean is not equal to 130
95 percent confidence interval:
 129.3368 130.1572
sample estimates:
mean of x 
  129.747 

• Notice that the above includes the test statistic \( t_0 = -1.291 \).

  • This also lists the number of degrees of freedom \( df= n-1 = 19 \) for the t distribtion.

• Most importantly, this lists the P value, \( \approx 0.2122 \).

• If we take \( \alpha=0.05 \), a common convention, we say \( P> \alpha \), such that we fail to reject the null hypothesis of \( \mu = 130 \).

General hypothesis testing of the mean – examples

• Suppose we wanted to perform a hypothesis test to make sure the mean sodium is not too high;

  • if we wanted to evaluate the one-sided hypothesis test

  \[ \begin{align} H_0: \mu \leq 130 & & H_1:\mu >130, \end{align} \]

• we would write in R

t.test(sodium_sample, mu=130, alternative="greater")

    One Sample t-test

data:  sodium_sample
t = -1.291, df = 19, p-value = 0.8939
alternative hypothesis: true mean is greater than 130
95 percent confidence interval:
 129.4081      Inf
sample estimates:
mean of x 
  129.747 

• Here, we once again fail to reject the null hypothesis at \( \alpha\times 100\% = 5\% \) significance, as \( P\approx 0.89 \).

General hypothesis testing of the mean – examples

• Computing the power of a t test is about the same as if we were computing the power with a normal distribution, replacing the appropriate z / t critical values and probabilities as necessary.

• However, computing sample size necessary for a hypothesis test to reach a certain power is very complicated, and is much more easily done with technology.

• Moreover, there is a built-in feature in R that will compute either the power of a test, or the needed sample size to attain a power, with the t test.

• The power.t.test() takes the following arguments

power.t.test(n, delta, sd, sig.level, power, alternative, type="one.sample")

• where
  • n is the sample size
  • delta is the difference between the assumed, but untrue, null hypothesis and the unknown, but assumed true, alternative hypothesis;
  • sd is the the sample standard deviation;
  • sig.level is the value of \( \alpha \);
  • power is the power of the test;
  • alternative is the alternative hypothesis; and
  • we need to specify the type="one.sample" as above.

General hypothesis testing of the mean – examples

• When we enter the power.t.test(),

power.t.test(n, delta, sd, sig.level, power, alternative, type="one.sample")

• we will actually leave out one of:

  1. power; or
  2. n

• as an argument.

• The argument that is left out, power or n, will be computed from the other arguments.

• We will continue our example with the sodium sample, now evaluating the power of our earlier tests

t.test(sodium_sample, mu=130, alternative="two.sided")

    One Sample t-test

data:  sodium_sample
t = -1.291, df = 19, p-value = 0.2122
alternative hypothesis: true mean is not equal to 130
95 percent confidence interval:
 129.3368 130.1572
sample estimates:
mean of x 
  129.747 

General hypothesis testing of the mean – examples

• To calculate the power of the hypothesis test,

  \[ \begin{align} H_0 : \mu = 130 & & H_1:\mu \neq 130 \end{align} \]

• where we assume the true alternative hypothesis is \( H_1: \mu=130.5 \),

• with a significance level of \( \alpha=0.05 \),

• we can compute this at once witht the power.t.test() as:

power.t.test(n=n, delta=delta, sd=s, sig.level=0.05, power=NULL, type="one.sample")

     One-sample t test power calculation 

              n = 20
          delta = 0.5
             sd = 0.8764288
      sig.level = 0.05
          power = 0.6775708
    alternative = two.sided

General hypothesis testing of the mean – examples

• Suppose we want to calculate power of the same type of hypothesis test, but with a different, one-sided alternative hypothesis.

  • e.g.,

  \[ \begin{align} H_0:\mu \leq 130 & & H_1 :\mu > 130. \end{align} \]

• We specify this in the function as,

power.t.test(n=n, delta=delta, sd=s, alternative="one.sided" , sig.level=0.05, power=NULL, type="one.sample")

     One-sample t test power calculation 

              n = 20
          delta = 0.5
             sd = 0.8764288
      sig.level = 0.05
          power = 0.7921742
    alternative = one.sided

General hypothesis testing of the mean – examples

• On the other hand, suppose we need to find the sample size necessary to meet a certain power with one of the earlier hypothesis tests.

• E.g., we might try to reject the null if a true mean sodium content is actually 130.1 milligrams, with a power of the test equal to 0.75.

• To do so, we now need to negelct the sample size argument n and supply the power argument power.

• The needed arguments are assigned below:

s <- sd(sodium_sample)
mu_null <- 130.0
mu_alternative <- 130.1
delta <- abs(mu_null - mu_alternative)
pow <- 0.75

General hypothesis testing of the mean – examples

• We determine the appropriate sample size via

power.t.test(n=NULL, delta=delta, sd=s, power=pow, type="one.sample")

     One-sample t test power calculation 

              n = 535.0307
          delta = 0.1
             sd = 0.8764288
      sig.level = 0.05
          power = 0.75
    alternative = two.sided

• for the two-sided test, or for the one-sided test we use

power.t.test(n=NULL, delta=delta, sd=s, alternative="one.sided", power=pow, type="one.sample")

     One-sample t test power calculation 

              n = 414.5589
          delta = 0.1
             sd = 0.8764288
      sig.level = 0.05
          power = 0.75
    alternative = one.sided