Skip to content
Mechanistic Validity

Criterion E3 — Selectivity

Section titled “Criterion E3 — Selectivity”
Validity typeExternal
Pass conditionOn-task effect exceeds off-task effect at the same intervention strength
Evidence familyBehavioral
Minimum reportingOn-task and off-task metric values at same intervention strength; selectivity ratio
Common failure modeReporting only the on-task effect; never measuring off-task metrics at the same strength

What this criterion requires

Section titled “What this criterion requires”

Selectivity is the external-validity complement to specificity (I3). Where specificity tests whether the component is specific, selectivity tests whether the intervention produces task-specific effects.

selectivity ratio = on-task effect magnitude / off-task effect magnitude

Ratio ≥ 2.0 is a reasonable pass threshold. Ratio = 1.0 means the intervention affects both tasks equally.

Choosing the control task

Section titled “Choosing the control task”

The control task should be:

  • Related but distinct: different computational structure, but close enough that trivial non-transfer explanations are ruled out.
  • Matched in difficulty: comparable full-model baseline performance, to avoid ceiling/floor effects.

For SVA: Greater-Than is a good control (different syntactic structure, similar linguistic domain). For IOI: SVA or Greater-Than. For Greater-Than: IOI.

Minimum reporting rule

Section titled “Minimum reporting rule”
  • Control task used and justification.
  • On-task and off-task values at same intervention strength.
  • Selectivity ratio.