ESM-1b attention vs. real contacts — ubiquitin

Do ESM-1b's attention heads know what a contact map is? Ubiquitin (PDB 1UBQ), 76 residues — click a cell in either panel to inspect a residue pair.

Data provenance — both panels are real. The left panel is the true contact map: Cα–Cα distances computed directly from the experimentally solved 1UBQ crystal structure (1.8 Å resolution), thresholded at 8 Å. The right panel is real ESM-1b (650M) attention — a live forward pass of esm1b_t33_650M_UR50S over this exact sequence, no fine-tuning. The precision numbers below are measured directly against the real contact map, searched over all 33×20 = 660 heads in the model: the best head found this way hits 60% precision@top-L/5 on long-range pairs; a typical early-layer head hits 0%. That's the actual range on this one protein, not an illustration of it.
Precision @ top-L/5 long-range: –
Real structure · Cα–Cα < 8 Å
Ground truth, from the actual 1UBQ coordinates. The off-diagonal streak around residues 1–8 vs 65–72 is the real β-sheet pairing that holds ubiquitin's core together.
ESM-1b attention
Real attention from a live ESM-1b forward pass. Some heads (like Layer 33 · Head 1) correlate strongly with real contacts; most early-layer heads are near-noise on this metric.
Hover or click a cell to see residue pair, sequence distance, and true Cα–Cα distance.
Reproducing this: attention was extracted from a single CPU forward pass of ESM-1b (esm1b_t33_650M_UR50S, via fair-esm) over the sequence below, no fine-tuning. Precision@top-L/5 (long-range, |i−j|≥24) was then measured for all 33×20 = 660 heads against the real 1UBQ contact map; the six heads above span that real range, from the strongest (Layer 33 · Head 1) to representative near-zero early-layer heads. Export script: precompute_esm1b_attention.py, alongside this file. Sequence: MQIFVKTLTGKTITLEVEPSDTIENVKAKIQDKEGIPPDQQRLIFAGKQLEDGRTLSDYNIQKESTLHLVLRLRGG