NUMBAT OER - Open Educational Resources

5. Probability, error and power

The critical significance level introduced in Section 4 implies that there is a finite possibility of error in testing the null hypothesis. If we select α = 0.05, then we are saying that we will accept our null hypothesis even if the probability that the data we are testing conform with the hypothesis is as low as 5%. This still leaves a small possibility that we will reject a null hypothesis when we really should have accepted it. In the case of α = 0.05, we are likely to make the wrong choice once in every twenty tests on average, for α = 0.01 it is once in every hundred and for α = 0.001 once in every thousand.

This error is termed a type 1 error, and is pre-defined by the critical significance level. A type 2 error occurs if you accept a false null hypothesis, and has a probability denoted by β (Greek letter beta). This is essentially an indication of some form of mismatch between the null hypothesis and the observations. The quantity (1 - β) is termed the Hypotheses and hypothesis testing page 2 of 3 'power' of the test, and is an estimate of the effectiveness of the test in avoiding a type 2 error.

Power can be increased by:

  • Increased sample size
  • A high 'signal to noise' ratio in the data, that is a strong effect of the source of variation under study and a low effect from other sources of variation