The Prediction That Outran the Experiment

Every two years since 1994, a small community of computational biologists has agreed to be publicly graded. Experimental groups who have solved a structure but not yet released it hand the sequence over to the organisers. The sequence goes out to everyone who wants to compete. The competitors submit their best guess at the three-dimensional shape, the organisers wait for the real coordinates to be released, and then they publish exactly how wrong everybody was.

It is a brutal format, and it was designed to be. The field had a chronic problem with methods that performed marvellously on structures their authors had already seen, and the only cure was a blind test with a fixed calendar and no way to quietly withdraw. The exercise is called the Critical Assessment of Structure Prediction, universally shortened to CASP, and for most of its life its results told a consistent and rather deflating story: prediction was improving, slowly, in the cases where a similar structure had already been solved, and remained close to useless otherwise 1.

Then came the fourteenth round, in the autumn of 2020.

Twenty-five years of modest progress

The problem CASP was set up to measure is easy to state and was, for fifty years, impossible to solve. A protein is a linear chain, and its sequence is written in the gene. In the cell that chain folds, spontaneously and usually within seconds, into one specific compact shape that determines everything it does. The information for the shape is evidently contained in the sequence, because nothing else is available to supply it. Reading it out is the hard part.

Predictions were scored on a scale of how many atoms fell within a given distance of their true positions once the two structures were optimally superimposed. High scores meant a model good enough to reason about; middling scores meant a model with the right general architecture and unreliable detail; low scores meant a plausible-looking object with no relationship to reality.

Through the 1990s and 2000s the reliable performers worked by analogy. If a target sequence resembled something already in the structural database, you could thread it onto that template and refine. This worked, but it was borrowing rather than prediction, and it failed exactly where prediction would have been most valuable — on sequences with no solved relative. Attempts to fold a chain from physics alone consumed enormous computing time and produced, in most rounds, disappointment. Groups made genuine advances; the community grew better at judging its own output; the graph of best-in-round accuracy went up. It went up gently.

The round that was not like the others

The signal that something had shifted arrived two years early. In CASP13, in 2018, an entry from a machine-learning group placed first by a clear margin using neural networks to predict contacts between residues and then assembling structures consistent with them. It was a strong result rather than a revolutionary one, and the reasonable interpretation at the time was that a well-resourced team had applied modern methods to an old problem and gained a step.

CASP14 was a different order of event. The same group's second system returned models whose median accuracy sat in the range that had previously been the province of experimental structures, and it did so on the hard targets as well as the easy ones — the free-modelling category, where no template exists, which had been the field's permanent embarrassment 3. The assessors, whose professional role is to find the failures in submitted models, found themselves in the unfamiliar position of arguing about whether discrepancies were errors in the prediction or errors in the crystal structure it was being compared against.

The method was published the following year, and the architecture turned out to lean heavily on something the field had already known was informative: the pattern of correlated mutations across the evolutionary record. If two residues touch each other in the folded structure, a change at one tends to be compensated by a change at the other, and that signal is visible in an alignment of thousands of related sequences. Earlier methods had mined it crudely. The new system learned directly from the alignment and from a representation of pairwise relationships, and refined both together until the geometry settled 2.

The reaction inside structural biology was not uncomplicated. There was genuine astonishment, and there was also unease — some of it professional, from people whose careers had been built on the difficulty of the thing that had just become easy, and some of it methodological, from people who noticed that a system trained on the accumulated output of sixty years of crystallography was being described as though it had solved the problem from first principles.

What a prediction does not replace

The triumphal telling of this story treats prediction as having retired the experiment. It has not, and the gap between the two is where most of the practical interest now lies.

A predicted model is a single static object, typically the folded state of one chain on its own. Real molecules are not usually encountered in that condition. They bind partners, and binding frequently reshapes them. They carry metal ions, cofactors, sugars and lipids, none of which the prediction places. They sit in membranes. They adopt more than one conformation, and the interesting biology is often in the switch between conformations rather than in either endpoint — a receptor that has an active and an inactive shape is not well described by a confident prediction of one of them.

Nor does a model carry an experimental error term. A crystal structure or a cryo-EM map comes with a resolution, a set of statistics, and a density map that another person can inspect to see how well the model is supported at any given point. A prediction has a self-reported confidence, which is informative and is not the same thing. Nothing was measured. There is no observation to return to.

And the most consequential limitation is one the systems themselves signal honestly: regions with low predicted confidence are frequently not failures of prediction but correct reports that there is nothing there to predict. A stretch of chain that is genuinely disordered has no single structure, and the model's uncertainty is the right answer 4.

Peptides, which are the awkward case

This is where short peptides sit, and it is worth being blunt about it, because the general enthusiasm for predicted structure tends to be applied to them without much thought.

A chain of ten or thirty residues in solution usually does not have a structure in the sense the question assumes. It samples an ensemble of conformations, interconverting rapidly, with perhaps a mild preference for a helical or extended arrangement in some segments and no preference at all elsewhere. Asking what shape it is resembles asking what shape a length of string is. Many short peptides only acquire a defined conformation on binding a partner, and which conformation depends on the partner.

The second difficulty is the input. These methods draw their power from evolutionary depth — thousands of homologous sequences whose covariation encodes contacts. A short peptide, a designed sequence, or a fragment excised from a larger protein has little or no such alignment to offer. Strip out the evolutionary signal and the prediction is running on far less than it was built for.

• Ensembles, not structures: a flexible chain is misrepresented by any single confident model of it.
• Thin alignments: short and designed sequences supply little of the covariation signal the method depends on.
• Conformation on binding: the biologically relevant shape may exist only in complex, and may differ between complexes.
• Chemical modification: non-natural residues, cyclisation and terminal capping — common in peptide chemistry — fall outside what the models were trained on.

Later systems have extended prediction to complexes, nucleic acids and small-molecule ligands, and the situation for peptides in complex is better than it was. The underlying tension remains. The method is at its most impressive on exactly the kind of molecule that crystallography was also good at — large, well-folded, evolutionarily well-sampled — and at its weakest on the kind that neither technique handles comfortably.

What it means that structure is cheap

In 2021 and 2022 the predicted structures of essentially every protein in the reference proteomes were computed and released openly, expanding structural coverage of sequence space by orders of magnitude at a stroke 5. Whatever one thinks about the epistemics, the practical effect is not in dispute: for a very large class of questions, obtaining a structure stopped being a project and became a lookup.

That is a change in economics rather than in knowledge, and changes in economics are usually the ones that reorganise a field. When a structure cost three years, it was worth solving only for targets already believed to matter, and the belief had to be justified before the work began. When a structure costs a few minutes, it can be generated for anything, including things nobody has any reason to care about — which means the scarce resource is no longer the picture but the judgement about which pictures are worth having, and the experiments that decide whether the molecule does anything at all.

It is a familiar shape of story on this subject. Synthesis became cheap and evidence did not. Weighing became routine and identity stopped being the interesting question. Now structure has followed, and the same asymmetry has widened again: the model is free, and finding out whether the model matters costs what it always did.