Prediction Error Drives Updating

Repeated experiences do not necessarily produce equal amounts of updating.

Early experiences with a relationship can change behaviour substantially. As that relationship becomes better predicted, later repetitions can provide less discrepancy for some forms of updating. Then something unexpected happens — a familiar outcome fails to occur, a different outcome appears, or what happens differs from what previous experience had made likely — and learning can shift again.

This raises an important question:

Why do some experiences produce more updating than others?

One answer is that what happens matters partly in relation to what was already predicted.

An outcome that confirms an established prediction may provide relatively little new information for updating. An outcome that violates prediction can provide information that existing learning may need revision.

The discrepancy between those two things is called prediction error.

And it helps explain why:

repetition ≠ equal updating

Learning Creates Predictions

As established in D5.2 — Associative Learning, experience can change relationships among cues, actions and outcomes.

Once a relationship has been learned, one event can acquire predictive significance for another.

A cue that has repeatedly preceded an outcome can make that outcome increasingly predicted when the cue appears again.

This does not require the organism to consciously think:

I expect this to happen.

Prediction refers here to the outcome implied by prior learning or represented as expected within the relevant learning model.

prediction ≠ conscious prediction

This matters because prediction-error principles apply far beyond situations in which someone can verbally report an expectation.

Previous learning creates predictive structure.

That structure changes the informational significance of what happens next.

Prediction Error Is a Discrepancy

Suppose previous learning predicts one outcome.

Then something else occurs.

The difference between the predicted and observed outcome is the basic idea behind prediction error.

In broad terms:

Prediction error is a discrepancy between a predicted and observed outcome or quantity, defined relative to a particular learning architecture or model.

The word error can be misleading.

It does not mean that the organism made a conscious mistake.

It does not mean that the behaviour was wrong.

And it does not mean that something bad happened.

prediction error ≠ conscious mistake

The error belongs to the discrepancy between prediction and outcome, not to the person or organism.

That discrepancy can occur in different directions.

Something expected may fail to occur.

Something unexpected may appear.

An outcome may be larger or smaller than predicted.

In reward-learning models, the relevant reward-related quantity may exceed or fall below what the model predicted.

Prediction error is therefore not inherently negative.

It describes a relationship between prediction and outcome.

Unexpected Outcomes Can Create Pressure for Updating

Why should that discrepancy matter?

Within many error-driven learning models, an outcome that is already well predicted produces less discrepancy available to drive that form of updating.

But when prediction and outcome diverge, the discrepancy can provide information relevant to revising existing learning.

Prediction error can therefore create pressure for updating in the sense that the current prediction has been shown to fit the outcome imperfectly.

The larger conceptual architecture is: prediction, outcome, then discrepancy, and then existing learning becomes a candidate for revision.

This helps explain why repeated experiences need not contribute equal amounts of updating.

As an outcome becomes better predicted within an error-driven learning architecture, the discrepancy available to drive that form of updating can diminish.

But this is not a universal linear rule.

larger prediction error ≠ universally proportional learning

And a prediction error does not guarantee that learning will occur.

prediction error can drive updating ≠ prediction error guarantees updating

What actually changes can also depend on what is represented, what receives attention, the learning rate, context and other features of the learning system.

This is also where surprise needs to be handled carefully.

Unexpected outcomes are often surprising, so surprise is a useful intuition for prediction error.

But the terms are not perfectly interchangeable.

Formal theories can distinguish a signed discrepancy between prediction and outcome from other quantities describing how unexpected an observation was.

So:

prediction error ≠ surprise in every formal sense

Blocking Shows Why Co-occurrence Is Not Enough

One of the clearest demonstrations comes from D5.3 — Classical Conditioning.

Imagine that Cue A already reliably predicts an outcome.

Now a new Cue B is introduced alongside it:

A + B → outcome

Cue B repeatedly occurs together with the outcome.

If learning depended only on co-occurrence, B should acquire substantial predictive significance simply because the two repeatedly appear together.

But under appropriate conditions, learning about B can be greatly reduced.

This phenomenon is called blocking.

Why?

One influential explanation is that the outcome was already well predicted by A.

When B was introduced, relatively little new discrepancy remained to support learning about it.

The important lesson for this node is not that one particular model has therefore been proven correct. It is that the relationships established in D5.2 — Associative Learning are not explained by co-occurrence alone:

co-occurrence alone ≠ complete explanation of associative updating

What the outcome contributes depends partly on what prior learning already predicted.

Blocking therefore supports expectation-dependent updating.

It does not uniquely identify the mechanism responsible for every instance of learning.

blocking supports expectation-dependent updating ≠ blocking proves one model

Missing an Expected Outcome Can Teach Too

Prediction error does not require something unexpected to be added.

Sometimes the important discrepancy is an omission.

Suppose a cue has learned predictive significance for an outcome.

Then the cue occurs and the predicted outcome does not.

The absence itself differs from what previous learning predicted.

This is relevant to the architecture established in D5.6 — Extinction Is New Learning.

During extinction, omission of a previously expected outcome can generate discrepancy and contribute to additional learning.

That helps explain why extinction is not simply passive exposure to the cue.

The current cue–outcome contingency differs from what prior learning predicted.

But prediction error is only part of the architecture.

prediction error can contribute to extinction learning ≠ prediction error is the complete theory of extinction

Attention, context, representation and other learning processes still matter.

Reward Prediction Error Is Not Reward

Prediction-error principles become especially influential in reinforcement-learning models.

But this is also where several concepts are often collapsed together.

As established in D5.5 — Reinforcement, Reward and Pleasure, reward, reinforcement and pleasure are distinguishable.

Reward prediction error introduces another construct.

In broad terms:

Reward prediction error concerns a discrepancy between predicted and obtained reward-related quantities within a particular learning model.

A positive reward prediction error occurs when the obtained reward-related quantity exceeds the model's prediction.

A negative reward prediction error occurs when it falls below prediction.

But positive and negative do not mean emotionally positive and negative.

They describe the direction of discrepancy relative to prediction.

positive prediction error ≠ positive emotion

negative prediction error ≠ negative emotion

And, critically:

reward prediction error ≠ reward

reward prediction error ≠ reinforcement

reward prediction error ≠ pleasure

Reward prediction error belongs to an updating architecture.

The distinctions among reward, reinforcement and pleasure remain those established in D5.5 — Reinforcement, Reward and Pleasure.

Dopamine Is Not Prediction Error

Reward prediction error became especially influential because of an important body of neuroscience.

Activity in some dopamine neurons and pathways has been found to track features captured by reward-prediction-error models under particular experimental conditions.

A simplified pattern from influential experiments helps explain why this became important.

In influential recordings from particular dopamine populations, unexpected reward can evoke increased activity.

With learning, activity can shift toward cues that predict reward.

And omission of an expected reward can be accompanied by characteristic changes around the time at which that reward was predicted.

These findings provide important neural evidence compatible with reward-prediction-error accounts.

But the popular conclusion that dopamine simply is prediction error goes too far.

Dopamine is a neurotransmitter involved in complex neural systems.

Dopamine signalling varies across neural populations, projections, tasks and behavioural conditions, and dopamine systems participate in functions that cannot be reduced to one homogeneous prediction-error variable.

So:

dopamine ≠ prediction error

and:

prediction error ≠ dopamine

A more precise claim is:

Some dopamine activity can covary with variables captured by reward-prediction-error models under particular conditions.

Even that correspondence does not by itself tell us exactly what computation the nervous system implements.

That requires another distinction.

Prediction Error in a Model Is Not Automatically a Brain Mechanism

The phrase prediction error can refer to related but distinct things at different explanatory levels: a conceptual discrepancy, a formal model variable, or a neural quantity hypothesized to participate in learning.

Those levels should not be collapsed.

At the conceptual level, prediction error describes a discrepancy: prediction and outcome differed.

At the computational level, a model can formalize that discrepancy as a particular variable used in an updating rule.

For example, the Rescorla–Wagner model formalizes associative updating partly through discrepancy between expected and obtained outcomes.

Temporal-difference learning uses prediction-error ideas differently, incorporating predictions across successive moments in time.

These models are related.

They are not identical.

Rescorla–Wagner ≠ temporal-difference learning

More broadly:

shared term ≠ identical formal variable

This becomes especially important when computational models are compared with biology.

As D2.12 — Scientific Models Are Tools establishes, a model is a representation used to explain, predict or organize features of a target system.

model ≠ target system

Suppose a prediction-error model fits behaviour.

That supports the usefulness of the model.

It does not automatically prove that the organism literally implements every variable and operation contained in it.

model fit ≠ proof of literal implementation

Likewise, suppose neural activity covaries with a prediction-error variable estimated from a model.

That correspondence can be important evidence.

But:

neural correlate ≠ proof of computational implementation

and:

model variable ≠ automatically neural mechanism

Moving from behaviour fits model, to organism implements algorithm, to specific neural circuit performs that computation, requires additional evidence at each step.

This same discipline matters when this node connects to D6.2 — Predictive Processing.

Predictive-processing theories also use the term prediction error.

But shared terminology does not establish that associative learning, reinforcement learning and predictive-processing theories refer to one identical quantity, computation or biological mechanism.

learning-theory prediction error ≠ automatically predictive-processing prediction error

Prediction Error Does Not Tell Us Everything That Changes

Prediction error is powerful partly because it provides information that the current prediction may need revision.

But another question immediately follows:

What should change?

Imagine that several cues, actions and contextual features are present when an unexpected outcome occurs.

The discrepancy tells us that something about the current prediction was inadequate.

It does not, by itself, specify which cue, action, state or representation should receive the update.

This is part of the credit-assignment problem.

prediction error ≠ complete solution to credit assignment

The same limitation appears elsewhere.

What gets attended to matters.

What the system represents matters.

Context can matter.

The rate at which updating occurs can change.

Uncertainty can affect how evidence is interpreted.

Unexpected events can also become salient, but salience itself is not prediction error.

Therefore:

prediction error ≠ attention

prediction error ≠ salience

prediction error ≠ uncertainty

These processes can interact with discrepancy-driven learning without being reducible to it.

This is why even a large prediction error does not uniquely determine what will be learned.

And it is why a small error does not mean learning has permanently ended.

As an outcome becomes better predicted under current conditions, the discrepancy available to drive a particular form of updating may become small.

But circumstances can change.

New cues can appear.

Contingencies can shift.

Different outcomes can become relevant.

Representations can change.

zero prediction error ≠ permanent end of learning

The Grounded Insight

Learning depends on experience.

But experience does not arrive at a blank system.

Previous learning has already changed what is predicted.

That means the same outcome can have different implications at different times.

When an outcome is already well predicted, it may provide relatively little new information for revising that prediction.

When the outcome differs from prediction, the discrepancy can make revision more useful.

A useful conceptual map runs from prior learning, which creates predictive structure, to the predicted quantity — what the relevant learning architecture or model predicts — set against the observed outcome or quantity, which is what occurs or is represented at the relevant point in the model.

The difference between them is the prediction error within the relevant learning model: a discrepancy defined by that model.

That discrepancy provides information relevant to updating, so existing predictions become candidates for revision.

Potential updating then follows within that learning architecture, shaped additionally by representation, attention, learning rate, context, uncertainty, credit assignment and other relevant processes.

This is not one literal biological pipeline.

Nor is it one universal algorithm.

Different models define predictions, error terms and updating rules differently.

And evidence that behaviour or neural activity follows a prediction-error model does not by itself prove that the biological system literally implements that model.

The central principle is therefore powerful precisely because it is bounded:

prediction error can drive updating

but:

prediction error ≠ all learning

Prediction error can tell us that what happened differed from what was predicted within the relevant learning architecture.

It can provide information that existing learning may need revision.

But it does not, by itself, tell us everything that will change, how much it will change, or which biological mechanism will implement that change.

what happens matters partly in relation to what was expected

That is what it means to say: prediction error drives updating.