Skip to content
Mechanistic Validity

Verdict Tier 4: Triangulated

Section titled “Verdict Tier 4: Triangulated”
Tier4 of 5 (progressive)
What it meansMultiple methods with non-overlapping assumptions converge on the same mechanism
RequiresConstruct: C3 (convergent), C4 (discriminant). Internal: I5 (rival mechanism exclusion), I6 (double dissociation), I7 (confound control). External: E2 (prompt generalization), E4 (cross-model generalization)
Upgrade to ValidatedC5—C6 (nomological and complementation validity) + M1—M6 addressed + I3, I10—I12 (minimality, rescue reversibility, onset and offset coupling) + E2—E6 + V1—V5 audited
Downgrade to Mechanistically supportedIf convergent validity (C3) fails because methods disagree on core components, or a rival mechanism (I5) is not excluded

What this tier establishes

Section titled “What this tier establishes”

A Triangulated claim has been confirmed by methods whose failure modes do not overlap. The key property is robustness to methodological critique: if activation patching is shown to have distributional artifacts, the weight-space analysis still stands. If the behavioral test has confounds, the attention pattern evidence is independent.

This is a qualitative transition, not merely “more evidence.” A single methodology, no matter how well-executed, produces findings conditional on that methodology’s assumptions. Triangulation means the finding survives the failure of any single method’s assumptions. The claim’s epistemic status is fundamentally different from a well-replicated single-method result.

Convergence is formalized via the robust core: the intersection of circuits identified by each independent method. Claims about the robust core are more strongly supported than claims about the union.

Example verdict statement

Section titled “Example verdict statement”

Verdict: Triangulated — [implementational-topographic] Claim: Induction heads (L5H5, L5H1 in GPT-2 Small) implement in-context copying via QK composition with previous-token heads. Met: C3 (attention pattern analysis + QK weight decomposition + training dynamics + behavioral ablation all converge; manual identification and ACDC agree on core heads, Jaccard = 0.72), E4 (mechanism found in GPT-2 Small, Medium, and Large), I6 (double dissociation between induction and previous-token heads) Open: C6 (complementation validity — two named roles, ablated one at a time and never together), I3 (minimality, unattempted), I10 (rescue reversibility, unattempted), I12 (offset coupling, no published offset test) Scope: GPT-2 family, in-context copying of arbitrary tokens, sequences with repeated subsequences

Minimum reporting for this tier

Section titled “Minimum reporting for this tier”
  • At least two methods named, with their assumptions explicitly stated
  • Jaccard similarity (or equivalent overlap measure) between circuits identified by each method
  • The robust core (intersection) identified and distinguished from method-specific periphery
  • External robustness evidence: distributions or model sizes tested beyond discovery context
  • At least three independently testable predictions, with at least two confirmed by different methods

Upgrade and downgrade

Section titled “Upgrade and downgrade”
DirectionWhat’s required
→ ValidatedNomological and complementation validity (C5—C6). Measurement M1—M6 explicitly addressed. Minimality, rescue reversibility and onset—offset coupling (I3, I10—I12). External validity across prompts, tasks, models, dose and novel prediction (E2—E6). Interpretive validity audited (V1—V5).
→ Mechanistically supported (downgrade)Methods are shown to share a hidden assumption (their “independence” was illusory). Or external robustness fails: mechanism does not transfer to claimed distributions/sizes.

What convergence is NOT

Section titled “What convergence is NOT”

Running the same method twice (e.g., activation patching with different hyperparameters) is replication, not triangulation. The methods must have non-overlapping failure modes — if one fails due to a distributional assumption, the other must not share that assumption. Two variants of patching (mean vs. resample) are closer to replication than triangulation because both assume the same causal model of intervention.

Characteristic occupants

Section titled “Characteristic occupants”
  • Induction heads (Olsson et al., 2022) — the strongest candidate in the literature, confirmed by attention pattern analysis, QK composition analysis, training dynamics (phase transition), cross-model search, and behavioral ablation
  • IOI circuit robust core — the subset of heads where Wang et al. (2022) manual analysis, Conmy et al. (2023) ACDC, and EAP-IG all agree

Key references

Section titled “Key references”