Hypothesis testing & Confidence intervals
Data Analysis for Psychology in R 1
Semester 2, Week 4
Dr Umberto Noè
Department of Psychology
The University of Edinburgh
Learning objectives
1. Interpret a confidence interval as the plausible values of a parameter that would not be rejected in a two-sided hypothesis
test.
2. Determine the decision for a two-sided hypothesis test from an appropriately constructed confidence interval.
3. Be able to explain the dierence between a significant result and an important result.
2 / 29
3 / 29
Part A
Introduction
4 / 29
Population vs Sample
We performed a study by selecting a sample from the population.
The goal is to use the sample data to answer questions about the population data. For example:
What's the population mean? Goal: estimating a parameter.
Could the population mean be zero? Goal: testing hypotheses about a parameter.
5 / 29
We said that the SE of the mean measures how far is
likely to be from the population mean .
Figure 1 is a drawing of an invisible lady walking her
dog. The dog, which is visible, is on a leash. The leash is
such that the dog is within of the lady 95% of
the time.
In one picture we can see the dog, but we would like to
know where the lady is.
Since the lady and the dog are usually within of
each other, we can take the following interval as a range
that would typically include the lady:
or, in short:
We could say that we are 95% confident that the lady is
within this interval.
Confidence interval
Figure 1. Invisible lady walking her dog.
x¯
μ
t
∗
⋅ SE
t
∗
⋅ SE
[dog − t
∗
⋅ SE,  dog + t
∗
⋅ SE]
dog ± t
∗
⋅ SE
6 / 29
Confidence interval
We have one picture, which corresponds to one sample.
We would like to know the value of the population mean , which corresponds to the invisible lady, but we cannot directly
observe it.
We can however see the sample mean , corresponding to the dog.
We use what we can see, , along with its standard error, as a way of constructing an interval that we hope will include what
we cannot see, i.e. the population mean .
The interval is a 95% confidence interval for the position of the lady.
The interval is a 95% confidence interval for the population mean .
Recall that the standard error of the mean is computed as .
μ
x¯
x¯
μ
dog ± t
∗
⋅ SE
x¯± t
∗
⋅ SE μ
SE = s/
√
n
7 / 29
Consider this example:
Sample of , with mean and
Want to test whether
The sample mean (47) is about 3 standard errors smaller
than the claimed mean (50)
Hypothesis testing
n = 30 x¯ = 47 s = 5.5
H
0
: μ = 50
H
1
: μ ≠ 50
SE = s/
√
n = 5.5/
√
30 = 1.004
t = = −2.99
47−50
1.004
8 / 29
Critical values for a distribution with :
tstar <- qt(c(0.025, 0.975), df = 29)
tstar
## [1] -2.045 2.045
As , we reject .
Hypothesis testing
t(29) α = 0.05
t ≤ −t
∗
H
0
9 / 29
10 / 29
Part B
Hypothesis testing & Confidence intervals
11 / 29
P-value method
Compute the p-value, i.e. probability of obtaining a test
statistic at least as extreme as the observed one, when
is true.
2 * pt(abs(tobs), df = n-1, lower.tail = FALSE)
Make a decision:
Reject if
Do not reject if
Critical value method
Compute the lower and upper critical values and :
qt(c(0.025, 0.975), df = n-1)
Make a decision:
Reject if or
Do not reject if
Hypothesis testing
Consider again the following example, where we wish to test the following claim at the significance level:
We start by computing the observed value of the t-statistic for our sample:
α = 0.05
H
0
: μ = μ
0
vs H
1
: μ ≠ μ
0
t =
¯
x− μ
0
s/
√
n
H
0
H
0
p ≤ α
H
0
p > α
−t
∗
t
∗
H
0
t ≤ −t
∗
t ≥ +t
∗
H
0
−t
∗
< t < +t
∗
12 / 29
Hypothesis testing
Do not reject if:H
0
: μ = μ
0
−t
∗
< t < +t
∗
−t
∗
< < +t
∗
¯
x− μ
0
s
√n
−t
∗
⋅ < x¯− μ
0
< +t
∗
⋅
s
√
n
s
√
n
−x¯− t
∗
⋅ < −μ
0
< −x¯+ t
∗
⋅
s
√
n
s
√
n
x¯+ t
∗
⋅ > μ
0
> x¯− t
∗
⋅
s
√
n
s
√
n
¯
x− t
∗
⋅ < μ
0
<
¯
x+ t
∗
⋅
s
√
n
s
√
n
13 / 29
Do not reject when: Reject when either:
Hypothesis testing
That is, we do not reject when lies within the CI:
H
0
: μ = μ
0
x¯− t
∗
⋅

lower limit of CI
< μ
0
< x¯+ t
∗
⋅

upper limit of CI
s
√
n
s
√
n
H
0
: μ = μ
0
μ
0
≥ x¯+ t
∗
⋅

upper limit of CI
s
√
n
or
μ
0
≤ x¯− t
∗
⋅

lower limit of CI
s
√
n
H
0
μ
0
Do not reject H
0
 if  [x¯− t
∗
⋅ ,   x¯+ t
∗
⋅ ]  includes μ
0
s
√
n
s
√
n
14 / 29
15 / 29
Part C
Significance vs Importance
16 / 29
head(tempsample)
## # A tibble: 6 × 2
## BodyTemp Pulse
##
## 1 36.4 69
## 2 37.4 77
## 3 37.2 75
## 4 37.1 84
## 5 36.7 71
## 6 37.2 76
dim(tempsample)
## [1] 50 2
nrow(tempsample)
## [1] 50
Body temperature example
Recall the body temperature study, where the goal was to assess whether the sample data provide evidence, at the 5%
significance level, that the population mean body temperature for healthy humans changed from the long thought 37 °C.
Data:
library(tidyverse)
tempsample <- read_csv('https://uoepsy.github.io/data/BodyTemp.csv')
17 / 29
Null and alternative hypotheses:
Chosen significance level:
Test statistic:
n <- nrow(tempsample)
xbar <- mean(tempsample$BodyTemp)
s <- sd(tempsample$BodyTemp)
se <- s / sqrt(n)
mu0 <- 37
tobs <- (xbar - mu0) / se
tobs
## [1] -3.141
Body temperature example
H
0
: μ = 37
H
1
: μ ≠ 37
α = 0.05
t =
x¯− 37
s/
√
n
18 / 29
Body temperature example
If you prefer to use tidyverse, you could equivalently do:
stats <- tempsample %>%
summarise(
n = n(),
xbar = mean(BodyTemp),
s = sd(BodyTemp),
se = s / sqrt(n),
mu0 = 37,
tobs = (xbar - mu0) / se
)
stats
## # A tibble: 1 × 6
## n xbar s se mu0 tobs
##
## 1 50 36.81 0.4252 0.06013 37 -3.141
19 / 29
Body temperature example
Recall the value of the t-statistic for the observed sample:
tobs
## [1] -3.141
To make a decision, we compare the observed t-statistic, , to the critical values from a distribution.
tstar <- qt(c(0.025, 0.975), df = n - 1)
tstar
## [1] -2.01 2.01
Two-sided alternative, so we ask: is the observed as or more extreme than the two critical values?
Is or ?
Yes, so reject .
t = −3.141 ±t
∗
t(49)
t
t ≤ −t
∗
t ≥ t
∗
H
0
20 / 29
Body temperature example
21 / 29
Body temperature example
If you were to compute the p-value, you would reach the same conclusion.
pvalue <- pt(-abs(tobs), df = n - 1) +
pt( abs(tobs), df = n - 1, lower.tail = FALSE)
pvalue
## [1] 0.002851
We could say:
At the 5% significance level, the sample data provide very strong evidence to reject the null hypothesis in favour of
the alternative one that the mean body temperature of healthy humans is dierent from the long thought 37 °C;
, two-sided.t(49) = −3.14, p = 0.003
22 / 29
tstar <- qt(c(0.025, 0.975), df = n - 1)
tstar
## [1] -2.01 2.01
xbar + tstar * se
## [1] 36.69 36.93
Body temperature example
At the significance level, the sample data provide very strong evidence to reject in favour of
.
However, this doesn't give us any idea of what the actual population mean may be. It just tells us there is very strong evidence
that it's not 37 °C.
Good practice
When you find a statistically significant result, it is good practice to follow-up the hypothesis test with a confidence
interval, in order to provide the reader with an idea of what a plausible value of the population parameter may be.
We are 95% confident that the population mean body temperature for healthy humans is between 36.69 °C and 36.93 °C.
α = 0.05 H
0
: μ = 37
H
1
: μ ≠ 37
23 / 29
Hypothesis testing and Confidence intervals
Notice that the 95% confidence interval [36.69, 36.93] does not include the hypothesised value in the null hypothesis,
.
As 37 is not within [36.69, 36.93], we reject the null hypothesis at the 5% significance level.
The 95% CI gives us a range of plausible values for the population mean.
As the claimed value 37 is not within the range, that is not a plausible value for . If you test whether the population mean
could be equal to 37, you end up rejecting that value.
Generic fact:
The 95% CI gives you a range of values for the parameter that would not be rejected at the 5% significance level in a two-sided
hypothesis test.
μ
0
= 37
μ
24 / 29
Body temperature example
The p-value is 0.003, indicating that the sample data provide very strong evidence that the population mean is significantly
dierent from 37 °C.
This is about statistical significance
That is, the result is very unlikely to come from a population where is true and the discrepancy between the sample
and population mean is only due to random sampling.
We are 95% confident that the population mean body temperature for healthy humans is between 36.69 °C and 36.93 °C.
The confidence interval tells us that perhaps the true population mean body temperature is somewhere in between
36.69 and 36.93, rather than 37.
Is this departure from 37 big enough to be of any practical impact for decision makers?
Pretty much no, it's negligible.
The result, while being significant, is not an important result.
Hence, statistical significance is dierent from practical significance (which we call importance), and a statistically
significant result may not be of practical significance at all.
H
0
25 / 29
Significance vs Importance
In scientific research, and in this course too, the word significance has a special meaning which is dierent from its everyday
language meaning.
It is a keyword reserved to mean statistical significance, i.e. the sample results being unlikely to be observed just because of
random variation due to sampling when in fact is true in the population.
This is not the same meaning as the sample results being important or practically significant.
H
0
26 / 29
Significance vs Importance
You can have a significant result, but this doesn't necessarily mean that it's important.
It's good practice to always follow-up a significant result (i.e. when you reject the null hypothesis) with a confidence
interval, to provide the reader with a measure of the "magnitude" of the departure from the hypothesised value
.
In our case, the 95% confidence interval [36.69, 36.93] °C indicates that the departure from the currently accepted
standard of 37 °C is not that big to be of much impact for decision makers.
This is an example of a statistically significant result that is not of practical importance.
H
0
: μ = μ
0
27 / 29
Why both?
If a confidence interval can also be used to perform a hypothesis test, why do we need both?
The answer is that they both provide useful and complementary information.
A p-value gives you the strength of evidence that the sample data bring against the null hypothesis. This is not conveyed
by a confidence interval.
The confidence interval gives you an idea of the "magnitude" of the population parameter, which the hypothesis test
doesn't convey.
In summary, report both if you have a significant result, i.e. if you reject the null hypothesis!
Always follow up a significant hypothesis test with a confidence interval.
Side note. If the hypothesis test is not significant, and you do not reject say, then you do have an idea about
the magnitude of the population mean. In fact, by not rejecting the claim that you are told that 0 is a plausible
value for the population mean.
H
0
: μ = 0
μ = 0
28 / 29
29 / 29