Incompatibility between two major innovations shaped the diversification of fish feeding mechanisms

Abstract

Innovations often shape the trajectory of macroevolution, yet their effects are usually considered independently, thus ignoring the functional and evolutionary interactions between them. Two innovations that have underpinned the ecological and evolutionary success of ray-finned fishes (Actinopterygii) are large teeth and highly protrusible jaws, which independently expanded the diversity of prey capture strategies. Here, we explore the functional relationship between these innovations across actinopterygians using high-speed videography and phylogenetic comparative methods. We find that these two innovations are functionally and evolutionarily incompatible because there is an overarching tradeoff between jaw protrusion and tooth size. Having large teeth decreases the kinematic diversity of prey capture by restricting species to overtake prey predominantly by swimming, while highly protrusible jaws are only found in species with small teeth. The space within tooth-bearing bones may impose this constraint, by limiting the maximum tooth size of species with gracile jaws adapted for high mobility and jaw protrusion. Nevertheless, some species break this constraint on tooth size through novel adaptations that accommodate exceptionally large teeth, unlocking new feeding modes which may have expanded the nature of aquatic feeding and influenced the ecosystems themselves. Although both high jaw protrusion and large teeth separately expanded prey capture strategies in fishes, they are generally not found in combination and are evolutionarily incompatible.

Fig 1. The diversity of tooth sizes and feeding kinematics across ray-finned fishes.

(A) The phylogenetic distribution of tooth size across 161 species of ray-finned fishes. Barplots at the tips correspond to the proportion of body ram (br), jaw ram (jr), and suction (s) for each species. Species (from L to R): Gymnothorax griseus, Piaractus brachypomus, Sorubim lima, Pseudobalistes fuscus, Emmelichthyops atlanticus, Rypticus maculatus, Pterois volitans, Aulostomus maculatus, Butis butis, Mastacembelus armatus, Microspathodon chrysurus, Aplocheilus lineatus, Pterophyllum scalare, Teleogramma brichardi, Boulengerochromis microlepis, Julidochromis dickfeldi, Cyprichromis leptosoma, Chilotilapia rhoadesii. (B) Ternary plot depicting the combined proportions of body ram, jaw ram, and suction; each point represents a species, and the color of the points corresponds to tooth size. The data underlying this figure can be found in S1 Data and S2 Data.

Fig 2. The relationship between prey capture kinematics and tooth size in ray-finned fishes.

(A) Phylogenetic generalized least-squares (PGLS) regressions between tooth size, quantified as a log-shape ratio, and the arcsine-transformed proportions of body ram, jaw ram, and suction. P-values and r²_pred are reported for each regression. (B) Linear regressions between kinematic variance and tooth size, for body ram, jaw ram, and suction. Ninety-five percent confidence intervals are depicted in grey. Each point corresponds to the variance quantified from 10 equal-frequency bins, based on tooth size. The data underlying this figure can be found in S1 Data and S2 Data.

Fig 3. Large teeth and highly protrusible jaws are evolutionarily incompatible.

(A) Dirichlet regression between the proportions of body ram (green), jaw ram (yellow), and suction (red) used in a strike, and tooth size, quantified as a log-shape ratio. Precision (ɸ) is plotted on the second y-axis (dotted line). (B) Distribution of optimal values for log-transformed tooth size (θ) between body ram (green) and jaw ram (yellow) feeders under a multiple optima Ornstein–Uhlenbeck model with single σ² and α parameters (OUM), based off 100 model fits. The data underlying this figure can be found in S1 Data and S3 Data.

Fig 4. Extreme morphological adaptations to accommodate large teeth in fish jaws.

Depicted are species from the upper 10% of our dataset on tooth size, which have all evolved unique modifications to accommodate large teeth in their jaws. (A) Upper jaw of Scarus iseri, in labial view. Photo credit Louis Imbeau, iNaturalist (CC BY 4.0). (B) Lower jaw of Canthigaster bennetti, in labial view, showing the fused beak. Photo credit Rickard Zerpe, Flickr (CC BY 2.0). (C) Lateral view of Zanclus cornutus upper jaw showing the procumbent implantation of teeth. The arrow indicates the location of a horizontal tooth crypt. Photo credit Laszlo Ilyes, Flickr (CC BY 2.0). (D) Lateral view of teeth from the lower jaw of Pseudobalistes flavimarginatus. The arrows indicate large attachment sites on the lingual aspect of the tooth. Photo credit Rickard Zerpe, Flickr (CC BY 2.0).