Cortical ensembles orchestrate social competition through hypothalamic outputs

Abstract

Most social species self-organize into dominance hierarchies^1,2, which decreases aggression and conserves energy^3,4, but it is not clear how individuals know their social rank. We have only begun to learn how the brain represents social rank^5-9 and guides behaviour on the basis of this representation. The medial prefrontal cortex (mPFC) is involved in social dominance in rodents^7,8 and humans^10,11. Yet, precisely how the mPFC encodes relative social rank and which circuits mediate this computation is not known. We developed a social competition assay in which mice compete for rewards, as well as a computer vision tool (AlphaTracker) to track multiple, unmarked animals. A hidden Markov model combined with generalized linear models was able to decode social competition behaviour from mPFC ensemble activity. Population dynamics in the mPFC predicted social rank and competitive success. Finally, we demonstrate that mPFC cells that project to the lateral hypothalamus promote dominance behaviour during reward competition. Thus, we reveal a cortico-hypothalamic circuit by which the mPFC exerts top-down modulation of social dominance.

Extended Data Figure 1:. Additional behavioral metrics during reward competition in unimplanted mice.

a, Mice learned the tone reward association at the same rate across social ranks. Left, latency to reward consumption following tone onset decreased over sessions. Right, the percent of trials with a reward consumption latency of less than 10 seconds increased over sessions (n=8 mice). Data are plotted as a function of social rank as measured by wins in the tube test. b, Example frames from reward competition assay showing intertrial interval time and during the tone period. c, Body weight difference between competitors does not correlate with rewards won (n=12 dyads from 8 mice, Pearson’s correlation, p=0.83). d, Relative dominant mice have higher pushing success during the tone (n=12 per group, paired t-test p=0.025). e, Latency to pick up the reward across trials for relative dominants vs subordinates (dom=101, sub=41, Two-sample Kolmogorov-Smirnov test p=0.29). f, Area occupied by dominants or subordinates in the 10 seconds prior to the tone onset for win vs lose trials (n=12 dyads). g, Distance to reward port across time by trial type and relative rank (trials n: dom win=68, dom lose=24, sub win=24, sub lose=68 from 12 dyads; early baseline −30 to −20 s prior to cue there is no effect of trial nor relative rank; 2-way ANOVA using the mean distance from −5 s to cue onset: main effect of trial type F_(1,180)=44.4, p=3×10⁻¹⁰, rank p=0.94, interaction p=0.09; 2-way ANOVA using the mean distance from 5 seconds prior to tone until 10 seconds post tone: main effect of trial type F_(1,180)=68, p=2.5×10⁻¹⁴, rank p=0.071, interaction p=0.79). Gray rectangle indicates contact range for the reward port. h, Total distance traveled immediately before the tone and during the tone period (baseline:10 seconds prior to tone; tone: 10 seconds of the tone) across trial types for relative dominant and subordinate mice (dom win=68, dom lose=24, sub win=24, sub lose=68 from 12 dyads; Wilcoxon rank-sum, baseline win p=0.79, baseline lose p=0.59, tone win p=0.028, tone lose p=0.86). Gray zone indicates contact with port. i, Percent body weight during food restriction did not differ across relative dominant and subordinate mice competing in reward competition (n=12 dyads, paired t-test, p=0.23).

Extended Data Figure 2:. AlphaTracker tracking metrics.

a, Root mean square error (RMSE) and identity error rate of AlphaTracker when tracking different body parts in videos with high resolution (1920×1080 pixels). Left plots have training and tracking done on 2 unmarked mice videos and right plots have training and tracking done on 4 unmarked mice videos. For both datasets two humans annotated the data and the RMSE between humans is indicated with the dashed line. For identity error rate 2 mouse tracking done with 9737 frame video and 4 mouse tracking done with 6020 frame video. b, Screenshot of user interface (UI) to fix errors made by AlphaTracker tracking. In addition, this UI can be used for exploring the clustering data.

Extended Data Figure 3:. AlphaTracker unsupervised clustering results.

a, Diagram depicting features used for AlphaTracker’s unsupervised clustering of the tracking datapoints. The features include head length, body length, body-head angle, displacement of the nose, distance between mice, angle between mice. b, Example frames from clips belonging to a specific cluster (cluster ID indicated with the color outline in c). c, Dendrogram and UMAP plot showing all video clips color coded by cluster ID for social behavior clustering. The mean cluster outputs are shown in (e) and features used are shown in (g). d, Dendrogram and UMAP plot showing all video clips color coded by cluster ID for individual behavior clustering. The mean cluster outputs for this clustering are shown in (f) and features used are shown in (h). e, Average normalized skeleton for nose, ears and tail base across clusters for the social behavior clustering across 500 ms of video clip time. Red arrow indicates self skeleton and green indicates the other skeleton. Each arrow represents 33.3 ms of data (1 frame). f, Average normalized skeleton for nose, ears and tail base across clusters for the individual behavior clustering across 500 ms of video clip time. Each arrow represents 33.3 ms of data (1 frame). Legend in bottom applies to panels e-f. g, Heatmap of normalized values for the self and other features used for social behavior clustering. h, Heatmap of normalized values for the self features used for individual behavior clustering.

Extended Data Figure 4:. Histological validation of electrode placements.

a, Representative images showing electrode track and lesions of mPFC electrode wires. b, Location of center for electrode lesions for all mice color coded by absolute rank across animals.

Extended Data Figure 5:. Behavior in competition with logger and HMM-GLM model controls.

a, Left, diagram of wireless electrophysiology recording device used for mPFC recordings. Image modified from SpikeGadgets’ MiniLogger product illustration. Middle, latency to collect reward over four days of training (n=16 mice). Right, latency to collect reward while performing reward task alone was not affected by wearing the logger (n=12 mice; paired t-test, p=0.83). b, Percent competitions won by absolute rank is highest for rank 1 mice in dataset used for mPFC recordings (number of competitions per rank 1 n=12; rank 2 n=12; rank 3 n=15; rank 4 n=14). c, Left, number of rewards obtained by relative dominants (dom) and subordinates (sub) during the reward competitions between animals wearing loggers (n=22 mice per group; paired t-test, p=0.86). Right, % body weight difference between competitors significantly correlates with rewards won (sub n=19 dom n=20, Pearson’s correlation, *p=0.01). For correlation only mice with same day weight measurements were used. d, Subordinates had longer latencies to pick up the reward during win trials (center line, median; box limits, upper and lower quartiles; whiskers, 1.5x interquartile range; points, outliers). Left, latency per group. Right, histogram of the distribution of latencies across all trials (dom trials n=326, sub trials n=358, Wilcoxon rank-sum, p=0.012; Two-sample Kolmogorov-Smirnov test, dom vs sub trials p=0.015; One way RM-ANOVA F_(1,24)=2.06, p=0.002). e, Percent port occupancy during tone across relative rank (n=31 sessions, paired t-test p=.90). f, Relative dominants were more successful displacing subordinates from the reward port throughout the competition (left; n=32 sessions, paired t-test, p=0.002) and during the tone time (right; n=31 sessions, paired t-test, p=0.005). g, Total time being displaced from reward port by relative rank in dataset used for mPFC recordings (n=31 sessions; paired t-test p=0.15). h, Percent time (normalized by total time per behavior) for 9 behaviors analyzed for win and lose trials separated by relative social rank. i, Percent time difference between relative dominant and subordinates for behavioral transitions during win trials (left) vs lose trials (right). j, Left, model selection for HMM-GLM state number using 10-fold cross-validation method results in a 6 state model being optimal. Error bars indicate standard error across the 10 cross-validations. Right, HMM-GLM 6 state model performance predicts behavioral label regardless of training method utilized (AUC n=9, one per each behavior label; Sign test of model performance vs chance p=0.004 for both methods). k, HMM-GLM 6 state model predicted behavioral label regardless of which dataset was used for training or testing (n=9 behavior labels using 482 trials for dom vs 478 trials for sub; Sign test performance vs 0.5 (chance) p=0.004 for all tests). l, Distribution of percent time spent in each hidden state by relative rank group (n=10 cross-validations using 482 trials for dom vs 478 trials for sub from 14 mice).

Extended Data Figure 6:. Additional data for mPFC population dynamics during social competition

a, Data arrangement across all animals (m1=mouse 1, m2=mouse 2) for the dimensionality reduction to a common subspace for the six task-relevant events. Neural trajectories were created for dominant and subordinate data using mean firing rate per event and the principal component analysis coefficients. b, Neural trajectory lengths (using principal components that captured 90% of variance) for win and lose trials are highest for relative subordinates in intermediate (ranks 2 or 3) mice (n indicated on plots; win 2-way RM-ANOVA main effects of relative rank F_(1,14)=165, p=2×10⁻⁶; lose 2-way RM-ANOVA effect of relative rank F_(1,14)=262, p=6×10⁻⁷). c, Firing rate rate of change is higher for relative dominants only in win trials (number of neurons indicated in plots, inset plot has average across groups; win trials rate of change: Kolmogorov-Smirnov (KS) test p=0.009, Wilcoxon rank sum p=0.01; lose trials rate of change: KS test p=0.40, Wilcoxon rank sum p=0.19). d, Neural trajectories for win and lose trials plotted in the first Principal Component (PC) for win and the orthogonal lose subspace show little overlap. Top right, inset of dominant neural trajectories. Bottom right, alignment of win and lose trajectories was significantly lower for dominant mice (n=13 per group; Wilcoxon rank-sum, p=1.5×10⁻⁵). e, Left, neural trajectories of mPFC population firing rate differ by relative rank for port entries that occur during the tone period in a lower dimensional common principal component (PC) sub-space (trajectories are the average across leave one out iterations leaving out one mouse at a time, total neurons recorded from dominants: n=507 and subordinates: n=490 units from 20 mice). Self entry events are aligned to port entries of the subject mouse while other entry events are aligned to the competitor’s port entries. Right, trajectory lengths (using PCs that captured 90% of variance) for self entry (top) and other entry (bottom) during the tone are higher for relative subordinates (self entry 2-way RM-ANOVA effect of relative rank F_(1,25)=452, p=5×10⁻¹⁴ and interaction of relative rank and event F_(1,25)=5950, p=1×10⁻¹⁷; other entry 2-way ANOVA effect of relative rank F_(1,25)=728, p=3×10⁻¹⁵ and interaction of relative rank and event F_(1,25)=90, p=5×10⁻⁷). f, Left, Neural trajectories of mPFC population firing rate for port entries that occur during inter-trial interval (ITI) projected into the first two principal components of the common behavioral subspace. Insets show closer look to the dominant trajectories. Right, neural trajectory lengths for self entry (top) and other entry (bottom) during the ITI (n=14 relative dom mice, n=13 relative sub mice; self entry: 2-way RM-ANOVA main effect of rank F_(1,25)=77.7, p=1×10⁻⁹; other entry: 2-way RM-ANOVA main effect of rank F_(1,25)=110, p=2×10⁻¹⁰. Self entry events are aligned to port entries of the subject mouse while other entry events are aligned to the competitor’s port entries. ITI port entries refer to port entries that occurred outside of the tone period.

Extended Data Figure 7:. mPFC population dynamics during social competition are not driven by location or mouse identity.

a, Average occupation in different parts of the chamber for win vs lose trials for the five seconds prior to tone vs first five seconds of tone. Black squares represent the reward port location. b, Distance to reward port differed by trial-type but not by rank (trials: dom win=290, dom lose=349, sub win=349, sub lose=290; 2-way ANOVA, main effect of trial-type F(1,1274)=353, p=8.8×10⁻⁷⁰, rank p=0.098 and interaction p=0.066). c, Distribution of the correlation coefficients for firing rate and distance to port for the population of mPFC single units did not differ by rank (dom=321, sub=479; KS test, p=0.48). d, To determine if distance to reward port affected the population dynamics during win and lose trials a subset of data with matched video conditions was split by distance to reward port. Neural trajectory lengths were higher for relative subordinates during win trials in which mice were close or far to the reward port during tone onset (dom n=19 sessions, sub n=18 sessions; win close to port: 2-way RM-ANOVA main effect of rank F_(1,35)=738, p=5×10⁻²¹; win far from port: 2-way RM-ANOVA main effect of rank F_(1,35)=588, p=3×10⁻²⁰). e, Neural trajectory lengths were higher for relative subordinates during lose trials in which mice were close or far from reward port during tone onset (dom n=19 sessions, sub n=18 sessions; lose close to port: 2-way RM-ANOVA main effect of rank F_(1,35)=588, p=3×10⁻²⁰; lose far from port: 2-way RM-ANOVA main effect of rank F_(1,35)=46.7, p=5×10⁻¹¹). f, To determine if reward port “place cells” contributed to neural trajectory rank differences we calculated the neural trajectory lengths without cells that were correlated to distance to port in a subset of data with equivalent video settings (video resolution and camera angle). Left, neural trajectories for self entry during the tone are highest for relative subordinates without the distance correlated cells (dom n=18 sessions, sub n=18 sessions; 2-way RM-ANOVA main effect of rank F_(1,34)=94.4, p=1×10⁻¹³). Right, neural trajectories are highest for relative subordinates without the distance correlated cells (dom n=18 sessions, sub n=18 sessions; excluding correlated cells: 2-way RM-ANOVA main effect of rank F_(1,34)=100, p=1×10⁻¹³). g, Neural trajectories of mPFC population activity for two randomly selected halves of the data for (left) win and lose trials, (middle) port entries during the tone and (right) ITI port entries (data from 49 recording sessions from 20 mice). All trajectories reflect the mean trajectories across 50 bootstrapping iterations. h, Left, trajectory lengths for win and lose trials when data is divided randomly show no effect of group indicating that the effect of rank is not due to chance (n=50; win: 2-way ANOVA, event F_(1,196)=8.41, p=0.004, group p=0.62; lose: event p=0.13, group p=0.65). Right, mean trajectory distances between groups for win and lose trials. i, Left, trajectory lengths for port entries during the tone when data is divided randomly show no effect of group (n=50; self entry: 2-way ANOVA, event F_(1,196)=14.2, p=0.0002, group p=0.97; other entry: F_(1,196)=6.76, p=0.01, group p=0.31). Right, mean trajectory distances between groups for self entry and other entry during the tone. j, Left, trajectory lengths for ITI port entries when data is divided randomly show no effect of group (n=50; self entry: 2-way ANOVA, event F_(1,196)=10.3, p=0.001, group p=0.93; other entry: event p=0.96, group p=0.87). Right, mean trajectory distances between groups for self entry and other entry during the ITI.

Extended Data Figure 8:. Decoding performance for relative and absolute social rank, and competitive success with different datasets.

a, Support Vector Machine (SVM) data pipeline to decode rank or competition outcome based on single trial population mPFC data in the common behavioral subspace. b, mPFC population encoding of win/lose in relative dominants generalizes to relative subordinates. Decoding performance (area under the receiving operating curve; AUC) when (left) training and testing on relative dominant data or (right) training on dominant and tested on relative subordinate data was higher than chance (shuffled performance indicated in gray). (Wilcoxon rank sum, dom/dom p=0.0002, dom/sub p=0.003). c, mPFC population encoding of win/lose in relative subordinates does not generalize to relative dominants. Decoding performance (area under the receiving operating curve; AUC) when (left) training and testing on relative subordinate data was higher than chance but not when (right) testing on relative dominant data (shuffled performance indicated in gray). (Wilcoxon rank sum, sub/sub p=0.0002, sub/dom p=0.14). d, Decoder performance for classifying competition outcome using training data from winner data (e.g. mouse won majority of trials) and testing data from loser data (e.g. mouse lost majority of trials) and using training data from loser data and testing data from winner data (Wilcoxon rank sum: left, baseline vs shuffle p=0.10, left, cue vs shuffle p=0.0002, right, baseline vs shuffle p=0.02, right, cue vs shuffle p=0.0002; Wilcoxon sign rank: loser base vs cue p=0.002, winner base vs cue p=0.004). All error bars indicate standard error from 10-fold cross-validation. e, SVM performance for decoding relative rank specifically for intermediate (ranks 2 or 3) mice; mean AUC vs shuffled AUC Wilcoxon rank sum: p=0.0002). f, Absolute rank can be decoded from mPFC population activity during social competition. One model was trained per absolute rank (mean performance across ranks vs shuffled data; Wilcoxon rank sum p=0.0002). g, Absolute rank can be decoded for rank 1 and 4 animals from mPFC population activity during social competition. One model was trained to discriminate rank 1 trials from rank 4 (mean performance across ranks vs shuffled data: Wilcoxon rank sum p=0.0002). h, Absolute rank can be decoded from mPFC population activity in mice performing reward task alone. One model was trained per absolute rank (mean performance across ranks vs shuffled data; Wilcoxon rank sum p=0.0002). i, Left, experimental design. In 15 mice the same neurons were recorded during alone trials and followed by competition trials. Right, mPFC population activity can decode between alone tone presentations and win trials during the competition trials (shuffle performance indicated by gray line; mean AUC vs shuffled AUC Wilcoxon rank sum p=0.0002). j, mPFC population activity is not sufficient to decode early vs late trials within task (alone mean AUC vs shuffle AUC Wilcoxon sum rank p=0.47; comp mean AUC vs shuffle AUC Wilcoxon sum rank p=0.47).

Extended Data Figure 9:. Additional data for mPFC single unit responses to task-relevant events during social competition.

a, Top, Dendrogram for functional clusters and heatmap of mean firing rate for all the neurons included in the hierarchical clustering (n=913 cells). Gray cells in the dendrogram indicate cells in functional clusters that did not meet criteria of mean z-score being higher than 2 or lower than −1 for at least one event. Bottom, distribution of mPFC cells across functional clusters in relative subordinates and relative dominants. b, Left, mPFC tone responsive cells when mice perform the reward task alone. Number of responsive cells and response magnitude to the tone does not differ between rank 1 and rank 4 mice (rank 1 exc=8 rank 1 inh=8 rank 4 exc= 8 rank 4 inh=4; Fisher’s exact test, total responsive per group p=0.16; Wilcoxon rank sum across groups: exc p=0.87, inh p=1.0). Middle, mPFC tone port entries responsive cells when mice perform the reward task alone. Number of responsive cells and response magnitude to port entries during tone does not differ across dom (rank 1) vs sub (rank 4) mice (dom exc=5 dom inh=25 sub exc=9 sub inh=16; Fisher’s exact test, total responsive per group p=0.09; Wilcoxon rank sum across groups: exc p=0.23, inh p=0.62). Right, mPFC inter trial interval (ITI) port entries responsive cells when mice perform the reward task alone. Number of responsive cells and response magnitude to port entries during ITI does not differ between rank 1 and rank 4 mice (rank 1 exc=10, rank 1 inh=23 rank 4 exc=9 rank 4 inh=49; Fisher’s exact test, total responsive per group p=0.06; Wilcoxon rank sum across groups: exc p=0.84, inh p=0.17). c, Total responsive cells and response magnitude to task-relevant event during social competition for absolute rank 1 vs rank 4 (win trials: dom exc=20, dom inh=11, sub exc=7, sub inh=14, Fisher’s exact test p=0.11, Wilcoxon rank sum exc p=0.23, inh p=0.03; lose trials: dom exc=3 dom inh=3, sub exc=0, sub inh=1, Fisher’s exact test p=0.12, Wilcoxon rank sum inh p=0.50; self entries tone: dom exc=23, dom inh=57, sub exc=24, sub inh=32, Fisher’s exact test p=0.006, Wilcoxon rank sum exc p=0.42, inh p=0.77; other entries tone: dom exc=14, dom inh 16, sub exc=27, sub inh=19, Fisher’s exact test p=0.11, Wilcoxon rank sum exc p=0.049, inh p=0.04; self entries ITI dom exc=31, dom inh=89, sub exc=21, sub inh=56, Fisher’s exact test p=2×10–5, Wilcoxon rank sum exc p=0.01, inh p=0.41; other entries ITI dom exc=13, dom inh=41, sub exc=8, sub inh=21, Fisher’s exact test p=0.001, Wilcoxon rank sum exc p=0.11, inh p=0.008). d, Total responsive cells and response magnitude to task-relevant event during social competition for intermediate rank mice (ranks 2 and 3) by relative rank (win trials: dom exc=4, dom inh=3, sub exc=5, sub inh=2, Fisher’s exact test p=0.76, Wilcoxon rank sum exc p=0.11, inh p=0.80; lose trials: dom exc=1 dom inh=3, sub exc=3, sub inh=0, Fisher’s exact test p=1, Wilcoxon rank sum exc p=1; self entries tone: dom exc=17, dom inh=30, sub exc=7, sub inh=14, Fisher’s exact test p=0.01, Wilcoxon rank sum exc p=0.89, inh p=0.57; other entries tone: dom exc=10, dom inh 23, sub exc=3, sub inh=10, Fisher’s exact test p=0.01, Wilcoxon rank sum exc p=0.46, inh p=0.79; self entries ITI dom exc=15, dom inh=42, sub exc=11, sub inh=21, Fisher’s exact test p=0.06, Wilcoxon rank sum exc p=0.11, inh p=0.44; other entries ITI dom exc=9, dom inh=26, sub exc=1, sub inh=16, Fisher’s exact test p=0.07, Wilcoxon rank sum exc p=0.20, inh p=0.90).

Extended Data Figure 10:. mPFC-LH photostimulation does not affect other motivated behaviors.

a, Representative images showing electrode lesions and mPFC-LH cells and LH axon terminals. b, Representative images showing electrode lesions and mPFC-BLA cells and BLA axon terminals. c, Responsive cells to tones and port entries while performing the reward task alone vs in social competition (alone: tone mPFC-LH n=2/13, mPFC-BLA n=3/5, non-phototagged n=54/470; entries during tone: mPFC-LH n=8/13, mPFC-BLA n=2/5, non-phototagged n=115/470; entries during ITI: mPFC-LH n=5/13, mPFC-BLA n=3/5, non-phototagged n=170/470; competition win trials: mPFC-LH n=3/43, mPFC-BLA n=1/10, non-phototagged n=62/920; self entries during tone mPFC-LH: n=11/43, mPFC-BLA n=1/10, non-phototagged n=193/920; self entries during ITI: mPFC-LH n=19/43, mPFC-BLA n=2/10, non-phototagged n=271/920, Fisher’s exact test non-photo vs LH p=0.011). d, Summary of mPFC optical fiber location (indicated with horizontal gray lines), mPFC viral expression and LH CAV2-Cre injection sites across mice for experiments shown below and in Figure 4. Distance to bregma is indicated under each brain slice. Top row shows LH injection and bottom row shows mPFC injection and fiber. e, mPFC-LH photostimulation in ChR2 mice did not change latency to pick reward while performing reward task alone (n=10; paired t-test, p=0.42). f, mPFC-LH photostimulation did not increase chow eating in the homecage (eYFP n=8, ChR2 n=7; 2-way RM ANOVA no significant effect of light, virus or interaction). g, mPFC-LH photostimulation in ChR2 mice did not change time spent in social chamber in the 3-chamber social interaction assay (n=10; paired t-test, p=0.79). h, mPFC-LH photostimulation did not change anxiety-like behavior in the open field (ChR2 n=8, eYFP n=8; 2-way repeated measures (RM) ANOVA no significant effect of light, virus or interaction). i, mPFC-LH photostimulation did not evoke conditioned placed preference or aversion (ChR2 n=5, eYFP n=5; 2-way RM ANOVA no significant effect of light, virus or interaction). j, Effort based T-maze allows mice to choose between a low reward low effort arm or a high reward high effort arm in which they must climb a wall to obtain the reward. k, mPFC-LH photostimulation did not increase high effort choice in the effort T-maze (ChR2 n=8, eYFP n=9; 2-way RM ANOVA no significant effect of light, virus or interaction for both 14 and 7 cm walls).

Figure 1:. Novel social dominance assay and deep learning tool for tracking multiple animals.

a, Reward competition behavioral paradigm schematic. b, Mice with higher relative ranks (dominant; Dom) collected more rewards than relative subordinates (Sub) when competing in dyads (n=12 dyads; Sign rank test on total rewards, p=0.008). Left, cumulative rewards across trials. Right, percent competitions won by absolute rank (n=6 competitions per rank). c, Port occupation, pushing success (pushing that resulted in displacement of competitor), and time displaced from port was higher for relative dominants (n=12 dyads; Sign rank test, occupation p=0.04; pushing success p=0.02; displaced p=0.016). d, Architecture of AlphaTracker which combines two convolutional neural networks and intersection over union for identity tracking. e, AlphaTracker precision for tracking two unmarked, near-identical mice is higher than human precision separated by body parts (left) or total (right; average precision across body parts; n=3 sub-sampled replicates). f, Left, social competition behavioral labels used for decoder models. g, Wireless device to record neural activity from mPFC (n=965 trials from 32 sessions from 13 mice). Image modified from SpikeGadgets. h, Architecture of the HMM-GLM to model the relationship between neural activity and behavioral states. i, Top, example trial with real behavioral labels and prediction for HMM-GLM 6-state model (colors from Fig. 1f). Bottom, performance across models based on area under the receiver operating curve (AUC; n=9 behaviors; Kruskal Wallis test p=6.7×10⁻⁸; model vs chance sign test p=0.004 for 6 state HMM-GLM, 9 GLMs and GLM; Wilcoxon rank sum HMM-GLM vs GLM p=4×10⁻⁵, HMM-GLM vs 9 GLMs p=4×10⁻⁵, GLM vs 9 GLMs p=0.54). HMM-GLM: hidden Markov model and generalized linear model; FT: frequency of behaviors table; SVM: support vector machine. Data are presented as mean values +/− SEM.

Figure 2:. Social rank and competitive success is decoded by mPFC population dynamics.

a, Neural trajectories of mPFC firing rate differ by relative rank during the tone presentation for win and lose trials in a lower dimensional common principal component (PC) sub-space (trajectories include a total of 20 mice, 27 sessions dominants: n=507 cells; subordinates: n=490 cells). b, Neural trajectory lengths for win (top left; 2-way RM-ANOVA effects of relative rank F_(1,25)=1090,p=2×10⁻¹⁶ and interaction,F_(1,25)=1660,p=3×10⁻¹⁴) and lose (bottom left; 2-way RM-ANOVA effect of relative rank F_(1,25)=883,p=9×10⁻¹⁶ and interaction, F_(1,25)=2995,p=9×10⁻¹⁶) trials. Firing rate variance is higher for relative subordinates (right bottom; number of neurons indicated in plots; win trials: KS test p=0.01, Wilcoxon rank-sum p=0.19; lose trials KS test p=5×10⁻⁷, Wilcoxon rank-sum p=2×10⁻⁹). c, SVM performance is higher than chance for decoding competitive success and relative rank (area under the receiving operating curve: AUC; gray: shuffled data performance; Wilcoxon rank-sum: competitive success p=2×10⁻⁴, relative rank p=2×10⁻⁴; Competitive success vs relative rank p=2×10⁻⁴). d, Neural trajectories of mPFC population firing rate by absolute rank (dominant (Dom)= rank 1; intermediates (Int)=ranks 2&3; subordinate (Sub)= rank 4) when performing the reward task alone vs in competition in a lower dimensional common PC sub-space (neurons in alone session: Dom=111, Int=259, Sub=140; competition: Dom=309, Int=359, sub=330). e, Left, trajectory during the tone is higher for subordinates during competition (2-way ANOVA main effect of rank F_(2,38)=30.4, p=1×10⁻⁸, task F_(1,38)=26.1, p=9×10⁻⁶ and interaction F_(2,38)=70.1, p=1×10⁻¹³). Right, distance between alone and competition tone trajectories increases with rank (n reflects all possible combinations of alone vs competition trajectories;1-way ANOVA main effect of rank F_(2,187)=536, p=3×10⁻⁷⁸). Post-hoc comparisons are Bonferroni-corrected t-tests, **p<0.01, ***p<0.001. 2-way ANOVAs were for rank and event (baseline vs event) or rank and trial-type and sample size indicated in plots. Data are presented as mean +/− SEM.

Figure 3:. Relative dominants have more reward seeking behavior cells while subordinates have larger responses to competitor behavior.

a, Left, heatmaps for mPFC firing rate responses to task-relevant events during the reward competition. Colors indicate clusters obtained by hierarchical clustering. Cells included in heatmap have a Z-score greater than 2 or less than −1 (Dom n=326; Sub n=306); clusters with a non-responsive population average were labeled gray. Right, difference between relative dominant (Dom) and subordinate (Sub) cells (% enrichment) across functional clusters. b, Left, response magnitude for mPFC cells during win trials differed across relative ranks (dominant D; subordinate S; Fisher’s exact test, total responsive cells per group p=0.30; Wilcoxon rank-sum across groups: excited p=0.01; inhibited p=0.06). Right, number of responsive cells and response magnitude for lose trials did not differ by relative rank (Fisher’s exact test, total responsive per group p=0.17; Wilcoxon rank-sum: excited p=0.62; inhibited p=0.28). c, Left, number of mPFC self tone port entry responsive cells was higher for relative dominants (Fisher’s exact test, total responsive cells per group p=0.0003; Wilcoxon rank-sum: excitation p=0.28 and inhibition p=0.99). Right, there was no difference in number of other tone port entry responsive cells, while the excitation magnitude was higher for relative subordinates (Fisher’s exact test, total responsive cells per group p=0.84; Wilcoxon rank-sum: excitation p=0.006, inhibition p=0.11). d, Left, relative dominants had more mPFC responsive cells to self port entries during the ITI while relative subordinates had larger excitation magnitude (Fisher’s exact test, p=9.8×10⁻⁶; Wilcoxon rank-sum: excitation p=0.006 and inhibition p=0.28). Right, relative dominants had more responsive mPFC cells to other port entries during ITI while subordinates had larger magnitudes (Fisher’s exact test, p=0.00019; Wilcoxon rank-sum: excitation p=0.015 and inhibition p=0.04). Recordings were collected from 20 mice; sample indicated in plots is cells. Data are presented as mean +/− SEM.

Figure 4:. mPFC-LH pathway encodes social competition and modulates social dominance behavior.

a, Left, viral strategy to stimulate projectors. Top, mPFC projector cells (ChrimsonR+; red) and neighbor cells (gray) responding to pulses of red light. Top scale 40 ms and 20 mV, bottom scale 40 ms and 4 mV. Bottom, average photo-latency of action potentials (AP) or excitatory postsynaptic potentials (EPSPs) for mPFC-BLA (left plot) and mPFC-LH (right plot) ChrimsonR+ cells compared to neighboring cells (mPFC-BLA ChrimsonR+ n=12 vs neighbors n=10; t-test, ****p<0.0001; mPFC-LH ChrimsonR+ n=9 vs neighbors n=8; t-test, ****p<0.0001). Dash lines indicate photoresponse latency threshold used for phototagging projector. b, Firing rate of mPFC-LH is higher than mPFC-BLA during the reward delivery (0–2 s) in win trials (mPFC-BLA n=10 neurons, mPFC-LH n=42 neurons, Wilcoxon rank-sum p=0.015). Firing rate for projector populations during tones for the reward task alone (alone data: mPFC-BLA n=5 neurons, mPFC-LH n=13 neurons, Wilcoxon rank-sum p=0.50; alone vs comp mean Z-score during tone; mPFC-LH p= 0.0189, mPFC-BLA p=0.43). Thicker lines represent the mean. c, Left, during reward competition light OFF or light ON sessions in which light was delivered in epochs (5 min light epoch of four 5 ms light pulses at 100 Hz every 200 ms). Right, cumulative rewards obtained by ChR2 mice in the light OFF vs light ON session (n=9 mice). Data are presented as mean values +/− SEM. d, mPFC-LH cell stimulation increased the number of trials won (Left; ChR2 n=9, eYFP n=7; 2-way RM ANOVA interaction of virus and light F_(1,14)=5.22, p=0.03; Bonferroni corrected t-test ChR2 p=0.01), time spent occupying the reward port (Middle; ChR2 n=9, eYFP n=7; 2-way RM ANOVA interaction of virus and light F_{(1, 14)}=6.73, p=0.02; Bonferroni corrected t-test, ChR2 p=0.02) and decreased time spent being displaced (Right; ChR2 n=9 paired t-test, p=0.005; eYFP n=7 paired t-test, p=0.28).