Criterion C5 — Nomological Validity
Section titled “Criterion C5 — Nomological Validity”| Validity type | Construct |
| Pass condition | The claim fits into a broader network of established findings and theoretical expectations |
| Evidence family | Cross-cutting |
| Minimum reporting | Named theoretical framework, specific predictions it makes about this mechanism, whether they hold |
| Common failure mode | Treating a standalone finding as complete without connecting it to related work |
What this criterion requires
Section titled “What this criterion requires”Nomological validity asks whether the claimed mechanism fits into a broader theory — whether its properties are consistent with what would be expected given related findings and established principles.
Satisfied when:
- The mechanism is placed in a theoretical network. The claim states what broader framework the mechanism belongs to and what that framework predicts about it.
- Predictions from the network are tested. If the theoretical framework predicts that mechanism X should have property Y, Y is measured.
- Anomalies are acknowledged. If the mechanism violates a prediction from the broader theory, the violation is stated and explained.
MI example
Section titled “MI example”Induction heads are placed in a broader theory of in-context learning: Olsson et al. argue that induction heads are the mechanism for general in-context learning. The nomological network predicts that removing induction heads should degrade all in-context learning, not just copying. This prediction is tested and fails — induction heads are necessary for token copying but not for general ICL — which is why the general ICL claim is Disconfirmed while the token-copying claim reaches Triangulated.
Connection to the chain
Section titled “Connection to the chain”Required for Validated tier. Without nomological validity, the mechanism is characterized in isolation. Nomological validity connects it to a larger body of knowledge.