Critical assessment of methods of protein structure prediction (CASP)-Round XV

Abstract

Computing protein structure from amino acid sequence information has been a long-standing grand challenge. Critical assessment of structure prediction (CASP) conducts community experiments aimed at advancing solutions to this and related problems. Experiments are conducted every 2 years. The 2020 experiment (CASP14) saw major progress, with the second generation of deep learning methods delivering accuracy comparable with experiment for many single proteins. There is an expectation that these methods will have much wider application in computational structural biology. Here we summarize results from the most recent experiment, CASP15, in 2022, with an emphasis on new deep learning-driven progress. Other papers in this special issue of proteins provide more detailed analysis. For single protein structures, the AlphaFold2 deep learning method is still superior to other approaches, but there are two points of note. First, although AlphaFold2 was the core of all the most successful methods, there was a wide variety of implementation and combination with other methods. Second, using the standard AlphaFold2 protocol and default parameters only produces the highest quality result for about two thirds of the targets, and more extensive sampling is required for the others. The major advance in this CASP is the enormous increase in the accuracy of computed protein complexes, achieved by the use of deep learning methods, although overall these do not fully match the performance for single proteins. Here too, AlphaFold2 based method perform best, and again more extensive sampling than the defaults is often required. Also of note are the encouraging early results on the use of deep learning to compute ensembles of macromolecular structures. Critically for the usability of computed structures, for both single proteins and protein complexes, deep learning derived estimates of both local and global accuracy are of high quality, however the estimates in interface regions are slightly less reliable. CASP15 also included computation of RNA structures for the first time. Here, the classical approaches produced better agreement with experiment than the new deep learning ones, and accuracy is limited. Also, for the first time, CASP included the computation of protein-ligand complexes, an area of special interest for drug design. Here too, classical methods were still superior to deep learning ones. Many new approaches were discussed at the CASP conference, and it is clear methods will continue to advance.

Keywords: CASP; community wide experiment; protein structure prediction.

Figure 1:

Superposition of a large protein target, T1154, a 1040 residue archaeal S-layer protein (green) and the closest calculated structure (blue). Misalignment of the lower right-hand domain likely reflects interdomain flexibility. For the assessment, the structure was divided into two evaluation units: the lower right-hand domain, and the rest of the structure. Corresponding highest GDT_TS values are 88 and 90%. There are no detectable templates for either unit.

Figure 2:

Percentage of targets modeled to three Cα RMSD accuracy thresholds (1, 2 and 3Å) in the four most recent CASP experiments. The fraction of high accuracy structures increased dramatically from 2016 (CASP12) to 2018 (CASP13) because of the introduction of effective deep learning methods and again from 2018 to 2020 (CASP14) with the introduction of the AlphaFold2 deep learning method. Increases from 2020 to 2022 (CASP15) are more modest likely because in CASP14 many computed structures were already within experimental uncertainty, so there is not much room for further improvement. RMSD (root mean square deviation) includes all common residues in the experimental and computed structures.

Figure 3:

CASP15 performance for protein tertiary structure, compared with earlier CASPs. The Y axis shows backbone agreement with experiment in GDT_TS units (19). On this scale, a random model scores approximately 20 to 30%, a correctly folded model around 50, and a model within experimental accuracy, around 90. Open circles show CASP15 best model results for each target, trend lines show performance for each CASP. Overall best performance of CASP15 (black line) is similar to that in CASP14 (blue line). The dotted line shows best server performance in CASP15. AlphaFold2-based methods dominate best performance. However, performance with standard AlphaFold2 protocols available at the time of the experiment is lower (dashed black lines). The X-axis difficulty scale represents the extent to which homology-based could be utilized. Targets in each CASP are ordered by difficulty calculated as a cumulative rank of the sequence identity and the coverage of the target by the best homologous structure available at the time of each experiment. The templates are found by running Foldseek (20) and LGA (21) versus experimental structures deposited to the PDB.

Figure 4:

CASP15 Protein assembly subunit interface agreement with experiment. Bars show the best agreement obtained between computed and experimental interfaces, using the Interface Contact Score (ICS). Green bars indicate interfaces in targets where there are homologous structures for all component subunits and interfaces. Blue bars are those where there are homologous structures for some subunits and interfaces, and red those where there is no structural homology. More than half the interfaces reach a score of 0.8 or higher, and so may approach experimental accuracy. About a quarter reach a more stringent criterion of ICS > 90.

Figure 5:

% of CASP14 (blue) and CASP15 (red) protein assembly targets with high quality (>0.8) and low quality (<0.5) computed models as measured by the average contact precision (left) and recall (right) (11). In CASP15 the fraction of high-quality models increased from less than 10% to more than 60% by both measures. Correspondingly, the fraction of poor-quality models dropped precipitously.

Figure 6

Superposition of the experimental (green) and closest calculated structures (blue) for two protein complexes. T1113 is a phage shell homodimer with intertwined polypeptide chains at the interface. The left-hand subunit is shown in grey, to clarify the intertwined region. The interface score (ICS) is 93%. H1140 is a nanobody (right)-antigen (left) complex (ICS 81%).

Figure 7:

Superposition of the closest model to experiment (blue) onto a structure of target R1156 (green), a homolog of the SARS-CoV-2 SL5 domain. This target showed substantial experimental flexibility, and the superposition is to the second highest resolution cryo-EM map. The GDT_TS is 51%.