Human associative learning
Lecture 2
Intro to human associative learning pt 2
Prof Mike Le Pelley
m.lepelley@unsw.edu.au
PSYC 2081
The Hats of Power
• Post thoughts, insights, jokes, memes,
observations etc to the Moodle forum
• Best entries will win one of four Hats of Power
Summary of Lecture 1
• Contingency: How strongly are events related?
• Learning is influenced by contingency (as you would hope!)
• But that’s not the whole story – for example
– Learning is gradual even though ΔP is constant; “learning curves”
– When we learn about multiple cues at the same time, we don’t learn
about each one individually based on its contingency
– Instead cues compete with each other
 Blocking as an example of “cue competition”
• Learning is not the same as calculating contingency. So what
are people doing when they learn?
• We can understand learning as the gradual formation of an
association (a link) between representations
An alternative view: Association formation
CS
cue
US
outcome
• What determines whether (and how fast) this association
strengthens?
• Proposed by Rescorla & Wagner (1972)
• Surprise is the key to learning
– We only learn when something happens that we don’t expect
 We learn when we are surprised by something
– If everything occurs as expected, we don’t learn anything new
 We don’t learn if we are not surprised
• Rescorla & Wagner captured this idea in an equation, providing a
simple but powerful model of the learning process
The Rescorla-Wagner model
Robert
Rescorla
Allan
Wagner
∆V = α β ( λ – ΣV )
Salience
of cue
Salience
of
outcome
Observed
magnitude
of outcome
Expected
magnitude
of outcome
Σ means “sum”
So ΣV means “summed
associative strength of all cues”
∆ means “change”
So ∆V means “change in
associative strength”
i.e., learning
cue outcome
V
V = predictive Value
(or associative strength)
Amount of
learning
The Rescorla-Wagner model
∆V = α β ( λ – ΣV )
Prediction error:
“Surprisingness” of
outcome
• When prediction error is zero, there is no learning
• Learning = Reduction in PE
• Learning = Things becoming more expected, and less surprising
Salience
of cue
Salience
of
outcome
Observed
magnitude
of outcome
Expected
magnitude
of outcome
Amount of
learning
The Rescorla-Wagner model
cue outcome
V
V = predictive Value
(or associative strength)
V = 0.4
∆V = α β ( λ – ΣV )
Associative
strength
Salience
of cue
Salience of
outcome
Observed
magnitude of
outcome
Expected
magnitude of
outcome
∆Vvege = 0.5 x 0.8 ( 1 – 0 ) = 0.4
Vvege = 0
Vvege = 0.4
0.5 0.8 1Let’s assume
Vege illness
Large
prediction
error
V = 0.64
∆V = α β ( λ – ΣV )
∆Vvege = 0.5 x 0.8 ( 1 – 0.4 ) = 0.24
Vvege = 0.4
Vvege = 0.64
Vege illness
Smaller
prediction
error
Associative
strength
Salience
of cue
Salience of
outcome
Observed
magnitude of
outcome
Expected
magnitude of
outcome
0.5 0.8 1Let’s assume
Trial 0: Vvege = 0
Trial 1: Vvege = 0.40
Trial 2: Vvege = 0.64
Trial 3: Vvege = 0.78
Trial 4: Vvege = 0.87
Trial 5: Vvege = 0.92
0
0.5
1
0 5 10 15
As
so
cia
tiv
e
st
re
ng
th
Trial
∆V = α β ( λ – ΣV )
SURPRISING
NOT
SURPRISING
Associative
strength
Salience
of cue
Salience of
outcome
Observed
magnitude of
outcome
Expected
magnitude of
outcome
0.5 0.8 1Let’s assume
V = 0.99
Vege illness
0
0.5
1
0 5 10 15
As
so
cia
tiv
e
st
re
ng
th
Trial
∆V = α β ( λ – ΣV )
Associative
strength
Salience
of cue
Salience of
outcome
Observed
magnitude of
outcome
Expected
magnitude of
outcome
∆Vvege = 0.5 x 0.8 ( 0 – 0.99 ) = –0.40
Vvege = 0.99
Vvege = 0.59
0.5 0.8 0Let’s assume
Negative
prediction
error
V = 0.59
Vege illness
V = 0.35
∆V = α β ( λ – ΣV )
∆Vvege = 0.5 x 0.8 ( 0 – 0.59 ) = –0.24
Vvege = 0.59
Vvege = 0.35
Vege illness
Smaller
negative
prediction
error
Associative
strength
Salience
of cue
Salience of
outcome
Observed
magnitude of
outcome
Expected
magnitude of
outcome
0.5 0.8 0Let’s assume
00.5
1
0 5 10 15 20 25
As
so
cia
tiv
e
st
re
ng
th
Trial
Acquisition
Extinction
Acquisition and extinction
SURPRISING
NOT
SURPRISING
NOT
SURPRISINGSURPRISING
∆V = α β ( λ – ΣV )
Salience
of cue
Salience of
outcome
Observed
magnitude
of outcome
Expected
magnitude
of outcome
0
0.5
1
0 5 10 15
As
so
cia
tiv
e
st
re
ng
th
Trial
Associative
strength
α = 0.5, β = 0.8
α = 0.5, β = 0.2
α = 0.2, β = 0.8
-80
-60
-40
-20
0
20
40
60
80
0 5 10 15 20 25 30 35 40
Ju
dg
m
en
t r
at
in
g
Trial
Lopez & Shanks (unpublished)
See Shanks 1995, ch2, p32
Rescorla-Wagner and contingency
People
• Learning is gradual, even though contingency (ΔP) does not change
– “Learning curves”
• The Rescorla-Wagner model also shows this pattern
∆P = 0.5
∆P = -0.5
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0 5 10 15 20 25 30 35 40
As
so
cia
tiv
e
st
re
ng
th
Trial
RW
∆P = 0.5
∆P = -0.5
01
2
3
4
Bread Dates
Ju
dg
m
en
t r
at
in
g
Aitken, Larkin & Dickinson
(2000, Expt 1)
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
Rescorla-Wagner and blocking
• Previous learning about apples blocks
subsequent learning about bread
• Cues are not learned about
independently
• Instead, cues compete with each other
00.5
1
0 5 10 15 20
As
so
cia
tiv
e
st
re
ng
th
Trial
∆Vbread : (λ – ΣV)
Vapple
Rescorla-Wagner and blocking
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
00.5
1
0 5 10 15 20
As
so
cia
tiv
e
st
re
ng
th
Trial
∆Vbread : (1 – ΣV)
Vapple
Rescorla-Wagner and blocking
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
00.5
1
0 5 10 15 20
As
so
cia
tiv
e
st
re
ng
th
Trial
∆Vbread : (1 – [Vapple + Vbread])
Vapple
Rescorla-Wagner and blocking
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
00.5
1
0 5 10 15 20
As
so
cia
tiv
e
st
re
ng
th
Trial
Vapple
Rescorla-Wagner and blocking
Outcome not
surprising
∆Vbread : (1 – [0.85 + Vbread])
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
00.5
1
0 5 10 15 20
As
so
cia
tiv
e
st
re
ng
th
Trial
Vbread
Vapple
Outcome not
surprising
Rescorla-Wagner and blocking
= 0.15
Small
prediction
error
∆Vbread : (1 – [0.85 + 0])
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
Allergy
Apples
Bread
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
Rescorla-Wagner and blocking
Allergy
Carrot
Dates
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
Rescorla-Wagner and blocking
01
2
3
4
Bread Dates
Ju
dg
m
en
t r
at
in
g
Aitken, Larkin & Dickinson
(2000, Expt 1)
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
• People learn a
weaker
association for
bread than dates
• And so does the
RW model!
0
0.2
0.4
0.6
0.8
Bread Dates
As
so
cia
tiv
e
st
re
ng
th
People RW
Rescorla-Wagner and blocking
Summary so far
• Lecture 1: Learning = Calculating contingency?
• A different view: Learning = Forming association between
representations
• The Rescorla-Wagner model
– Surprise drives association formation (learning)
– RW is one model of association formation – there are many others
 RW is probably the most influential model of associative learning
 A simple, intuitive account – but very powerful
• E.g., explains effects of contingency, learning curves, and cue
competition (blocking)
 Still often used as a “default” view of learning
An alternative view
“Many theorists maintain a belief in a learning mechanism in
which links between mental representations are formed
automatically… We conclude that learning is the consequence
of propositional reasoning processes that cooperate with the
unconscious processes involved in memory retrieval”
An alternative view
The link-based approach
(e.g., Rescorla-Wagner)
ILLV
The propositional approach
Last time I ate Vegemite I was ill.
Perhaps Vegemite makes me ill.
The propositional approach
Last 2 times I ate Vegemite I was ill.
Fairly sure Vegemite makes me ill.
0
0.5
1
0 5 10 15
St
re
ng
th
o
f b
el
ie
f
Trial
The propositional approach
• The “thinking about it” approach
• Remember what went with what in the past, try and
work out most likely explanation
Blocking
0
1
2
3
4
Bread Dates
Ju
dg
m
en
t r
at
in
g • Previous learning about effectiveness of apples
blocks subsequent learning
about bread
• Cues compete with each
other
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
Aitken, Larkin & Dickinson
(2000, Expt 1)
Propositions and blocking
• Remember what went with what in the past, try and
work out most likely explanation
“If apple and bread together cause the outcome to occur with
the same probability as apple alone, this implies that bread is
not a cause of the outcome.”
Phase 1 Phase 2 Test
Apple → allergy Apple & Bread → allergy Bread?
Carrot & Dates → allergy Dates?
Blocking and trial order
• According to the propositional account, order of the trials doesn’t matter
• Should get blocking in the standard (forward) procedure, and in a backward
procedure
Phase 1 Phase 2 Test
A → outcome A + B → outcome B?
C + D → outcome D?
“If A and B together cause the outcome to occur with the same probability and
magnitude as A alone, this implies that B is not a cause of the outcome.”
FO
RW
AR
D Proposition
Blocking
BA
CK
W
AR
D
Phase 1 Phase 2 Test
A + B → outcome A → outcome B?
C + D → outcome D?
Blocking
• In the Forward case, RW model says that blocking occurs because the outcome
is less surprising on “A+B” trials than “C+D” trials
• In the backward case, the outcome is equally surprising on “A+B” trials and
“C+D” trials
• So RW model predicts no blocking in backward condition
Blocking and trial order
Phase 1 Phase 2 Test
A → outcome A + B → outcome B?
C + D → outcome D?FO
RW
AR
D Proposition
Blocking
BA
CK
W
AR
D
Phase 1 Phase 2 Test
A + B → outcome A → outcome B?
C + D → outcome D?
Blocking
RW
Blocking
No
blocking
B D
0
2
4
6
8
Ju
dg
m
en
t r
at
in
g ✱• We’ve already seen
blocking in the
“Forward” case
– e.g., Aitken, Larkin &
Dickinson (2000), see
Lecture 1
• And it also occurs in the
“Backward” case!
– e.g., Wasserman & Berglan (1998)
• Consistent with propositional
account, but not with RW
Blocking and trial order
Phase 1 Phase 2 Test
A → outcome A + B → outcome B?
C + D → outcome D?FO
RW
AR
D Proposition
BA
CK
W
AR
D
Phase 1 Phase 2 Test
A + B → outcome A → outcome B?
C + D → outcome D?
Blocking
Blocking
RW
Blocking
No
blocking
“If A and B together cause the outcome to occur with the same
probability and magnitude as A alone, this implies that B is not
a cause of the outcome.”
• In order to draw this inference, you need to know that B has
been paired with the outcome
– Inferences based on memories of A→outcome and AB→outcome
– B is paired with the outcome, but does not cause the outcome
Blocking of memory
Phase 1 Phase 2 Test
A → outcome AB → outcome B?
Causality Memory
Propositional
account
Blocking of causal
judgement
Good memory of
outcome
Link-based account
(e.g. RW)
Blocking of causal
judgement
Blocking of memory
of outcome
Phase 1 Phase 2 Test
A → outcome AB → outcome B?
Blocking of memory
Phase 1 Phase 2 Test
A → outcome AB → outcome B?
A
B
outcome
• RW: Blocked cue does not form a
strong link with the outcome
– People don’t encode that B was ever
paired with the outcome
Blocking of memory
Phase 1 Phase 2 Test
A → outcome AB → outcome B?
Blocking of memory
Causality Memory
Propositional
account
Blocking of causal
judgement
Good memory of
outcome
Link-based account
(e.g. RW)
Blocking of causal
judgement
Blocking of memory
of outcome
Mitchell, Lovibond, Minard & Lavis (2006)
Mr X ate
Apple
He suffered
Xianethis
Mitchell, Lovibond, Minard & Lavis (2006)
Mr X ate
Peach
He suffered
Daryosis
Mitchell, Lovibond, Minard & Lavis (2006)
Mr X ate
Apple Breadand
He suffered
Xianethis
Which illness followed
Bread
during training?
Daryosis Xianethis
Plonthema Eneuritis
To what extent did the food cause
this illness?
0 1 2 3 4 5 6 7 8
definitely
not causal
possibly definitely
causal
Blocking in ratings
of causality…
… and in memory
of outcomes
A → outcome AB → outcome B = Blocked
CD → outcome D = Control
Mitchell, Lovibond, Minard & Lavis (2006)
Phase 1 Phase 2 Test
A → outcome AB → outcome B?
Blocking of memory
Causality Memory
Propositional
account
Blocking of causal
judgement
Good memory of
outcome
Link-based account
(e.g. RW)
Blocking of causal
judgement
Blocking of memory
of outcome
Summary
Propositions Links
Backward
blocking
Blocking of
memory of cue-
outcome pairings
Summary
• It’s still not clear: More research needed!
• Seems likely that both types of process operate
• Simpler experiments encourage “thinking about it”
• More complex experiments discourage thinking about it, so more
automatic link-formation mechanism dominates
Wasserman & Berglan (1998)
Backward blocking
Supports propositional
Mitchell et al (2006): Blocking of cue-outcome memory
Supports link-based
Summary
• “Associative learning” may be due to more
than one process

学霸联盟