STA258H5
1
• Type I and Type II errors
• Statistical Power
• Using Power to Determine Sample Size
Type I and Type II errors
2
Decisions Errors in Tests
3
Type I Error
• Ho is true, but sampling error in the data leads you to reject Ho,
you’ve made a Type I error.
• When Ho is true, a Type I error occurs if Ho is rejected.
• The probability of a Type I error is the significance level of the
hypothesis test. The probability of a Type I error is denoted by
Type II Error
• Ho is false, but sampling error in the data does not leads you to
reject Ho, you’ve made a Type II error.
• When Ho is false, a Type II error occurs if Ho is not rejected.
• The probability of making a type II error is denoted by
Example of Decisions Errors in Tests
4
In medical disease testing, the null hypothesis is usually the assumption that a person is healthy. The alternative is
that the person has the disease we are testing for.
Ho: Healthy verses Ha: Infected
• Type I error: Reject Ho when it is true.
A Type I error is a false positive: A healthy person is diagnosed with the disease.
That is, a person must go under further test.
• Type II error: Fail to reject Ho (“Accept Ho”) when it is false.
A Type II error is a false negative in which an infected person is diagnosed as disease-free.
That is, a sick person gets untreated.
For example: If a new treatment is being tested for a disease (e.g., epilepsy), a Type I error will lead to future
patients getting a useless treatment; a Type II error means a useful treatment will remain undiscovered.
What type of error could we making in our example?
5
In 1980s, it was generally believed that congenital abnormalities affect 5% of the nation’s children.
Some people believe that the increase in the number of chemicals in the environment in recent years
has led in the incidence of abnormalities. A recent study examined 384 children and found that 46 of
them showed signs of abnormality. Is this strong evidence that the risk has increased?
Ho: = 0.05 verses Ha: > 0.05
≅ 6.28, -value < 0.0001 (which is less than = 0.05). We reject Ho and conclude Ha.
This means we could be making a Type I error. We decided that the true percentage is more than 5%
based on our data (as evidence against Ho), however, it could be that the hypothesized value of 5% is
true (e.g., 0: = 0.05 could be true).
What type of error could we making in this example?
6
According to Access and Support to Education and Training Survey (2008), of 4,756 adult Canadians, 1,581
indicated that they worked at a job or business at anytime (between July 2007 and June 2008), regardless of the
number of hours per week . Is there evidence to suggest that the true proportion is greater than 0.50?
Ho: = 0.50 Ha: > 0.50
2 = 534.24
Z = . ≅ - 23.11 (take the negative sign because the difference between ෝ = 0.3324 and = 0.5 is negative: -0.167578)
P-value = P(Z > -23.11) ≈ 1, P-value > = 0.05; We Fail to Reject Ho; We cannot conclude Ha.
This means we could be making a Type II error. We indicated that there is no evidence to conclude that the true proportion of
adult Canadians who worked at a job or business at anytime (between July 2007 and June 2008), regardless of the number of
hours per week was more than 0.50 - this conclusion implies that Ho: = 0.50 is plausible, but we could be wrong.
7Statistical Power
Statistical Power
Statistical power is the probability of correctly rejecting Ho.
• If T is close to o power will be low
• If T is far from o power will be high
8
( ) 1 ( ) (reject Ho|Ho false ) T T TP      = = − = =
What about making a correct decision? Power of Test
9
Power of test refers to probability of correctly rejecting Ho when it is false: P(reject Ho | Ho is false)
Power = 1 – Beta = 1 – P(fail to reject Ho | Ho is false)
Note: The complement of reject Ho is, fail to reject Ho. For example: P(A|B) = 1 - P(AC|B)
• When we think about power, we imagine the null hypothesis is false.
• The value of power depends on how far the truth lies from the value we hypothesize.
• We call this distance between the null hypothesis value (for example) 0 and the truth , the effect size.
What about making a correct decision? Power of Test
10
• The value of power depends on how far the truth lies from the value we hypothesize.
• We call this distance between the null hypothesis value (for example) 0 and the truth , the effect size.
• The effect size is unknown, of course, since it involves the true p.
• But we can estimate the effect size as the difference between the null value and the observed estimate.
• The effect size is central to how we think about the power of hypothesis test.
• A larger effect is easier to see and results in larger power.
• Small effects are difficult to detect. They will result in more Type II errors and therefore lower power.
• The power of the test both depends on the size of the effect and the amount of variability in the sampling model.
For proportions, we use a Normal sampling model (for Ƹ) with standard deviation inversely proportional to the
square root of the sample size, n.
Example of Power, Beta, and Sample Size
11
A newsletter reports that 90% of adults drink milk. The researchers are interested in investigating if
less than 90% of adults drink milk (at alpha = 0.05). They collect a random sample of 200 adults in
a certain region.
a. Calculate power of the test if the percentage of adults who drink milk is really 85%.
b. Calculate beta if the percentage of adults who drink milk is really 85%.
c. How many adults should you sample if you want to raise the power in part (a) to 0.80?
Example of Power
12
A newsletter reports that 90% of adults drink milk. The researchers are interested in investigating if less than 90% of
adults drink milk (at alpha = 0.05). They collect a random sample of 200 adults in a certain region.
a. Calculate power of the test if the percentage of adults who drink milk is really 85%.
Alpha = 0.05 = P(reject Ho | Ho is true)
P(Z < -1.645) = 0.05 (this is the rejection region)
Z critical value is -1.645
Power = P(reject Ho | Ho is false)
= P(
ො−0.90
0.90(1−0.90)
200
< -1.645 | P = 0.85)
= P( Ƹ < 0.8651 | P = 0.85)
= ( <
0.8651−0.85
0.85 1−0.85
200
≅ 0.60) ≅ 0.73
Example of Power
https://istats.shinyapps.io/power/
13
Example of Beta
14
A newsletter reports that 90% of adults drink milk. The researchers are interested in investigating if less than 90% of
adults drink milk (at alpha = 0.05). They collect a random sample of 200 adults in a certain region.
b. Calculate beta if the percentage of adults who drink milk is really 85%.
Beta = 1 – power = 1 – 0.73 = 0.27
Using Power to Determine Sample Size
15
Example of Sample Size
16
A newsletter reports that 90% of adults drink milk. The researchers are interested in investigating if
less than 90% of adults drink milk (at alpha = 0.05).
c. How many adults should you sample if you want to raise the power in part (a) to 0.80?
Recall part a. Calculate power of the test if the percentage of adults who drink milk is really 85%.
17
Step 1:
Write the rule for rejecting Ho in term of Ƹ∗
= P(Reject Ho | Ho is True)
= P( Ƹ < Ƹ∗ | = 0.90)
= P(
ො−0.90
0.90(1−0.90)

<
ො∗−0.90
0.90(1−0.90)

)
= P(Z <
ො∗−0.90
0.90(1−0.90)

) = 0.05
But, P(Z < -1.645) = 0.05
So, Ƹ∗ = 0.90 – 1.645
0.90(1−0.90)

18
Step 2:
The power is the probability of this event under the condition that the alternative = 0.85 is true
Standardize Ƹ using = 0.85
= P(
ො−0.85
0.85(1−0.85)

<
ො∗−0.85
0.85(1−0.85)

)
= 0.80
From R, P(Z < 0.8416212) = 0.80
So,
ො∗−0.85
0.85(1−0.85)

= 0.8416212
In terms of Ƹ∗: Ƹ∗ = 0.85 + 0.8416212
0.85(1−0.85)

19
Step 3:
Solving for n It follows from Steps 1 and 2 that
0.90 – 1.645
0.90(1−0.90)

= 0.85 + 0.8416212
0.85(1−0.85)

0.90 – 0.85 = 1.645
0.90(1−0.90)

+ 0.8416212
0.85(1−0.85)

0.05 =
1

(0.4935 + 0.300518878)
0.05 = 0.794018878
= 252.186 ≅ 253
Example of Power and Sample Size
20
A newsletter reports that 90% of adults drink milk. The researchers are interested in investigating if less than 90% of
adults drink milk (at alpha = 0.05). They collect a random sample of 100 adults in a certain region.
a. Calculate power of the test if the percentage of adults who drink milk is really 85%.
Alpha = 0.05 = P(reject Ho | Ho is true)
P(Z < -1.645) = 0.05 (this is the rejection region)
Z critical value is -1.645
Power = P(reject Ho | Ho is false)
= P(
ො−0.90
0.90(1−0.90)
100
< -1.645 | P = 0.85)
= P( Ƹ < 0.85065 | P = 0.85)
= ( <
0.85065−0.85
0.85 1−0.85
100
≅ 0.02) ≅ 0.51
Power increases as Sample Size, n increases
21
A newsletter reports that 90% of adults drink milk. The researchers are interested in investigating if less than 90%
of adults drink milk (at alpha = 0.05).
They collect a random sample of 50 adults in a
certain region. Calculate power of the test if the
percentage of adults who drink milk is really 85%.
They collect a random sample of 200 adults in a
certain region. Calculate power of the test if the
percentage of adults who drink milk is really 85%.
If we keep at the same size, larger sample sizes increase the power of test because sampling variability (sampling distributing)
are much narrower. The critical value, ∗ gets closer to 0 and farther from p.
Power Formulas
22
Need to know
• Standard deviation, s
• Significance level, a
• Effect size you want to detect,
0
0
0
( )
If : then
( )
If : then
( ) ( )
If : then 1
T
T C
T
T C
T T
T C C
n
Ha P Z z
n
Ha P Z z
n n
Ha P z Z z
 
   

 
   

   
   
 


 
 −
•  ( ) =  +  
 
 −
•  ( ) =  − +  
 
 − −
•  ( ) = − − +   +  
 
| |T  =
Sample Size Selection – Using Power
23
For : < 0
1. Select the power, P
2. Find such that ( < ) =
3. = ( < ) = < − +
(0 − )

⇒ ≥
2( + )
2
(0 − )2
For : > 0
1. Select the power, P
2. Find such that ( > ) =
3. = ( > ) = > +
(0 − )

⇒ ≥
2( − )
2
(0 − )2