Extensive evaluation of the generalized relevance network approach to inferring gene regulatory networks

Abstract

Background: The generalized relevance network approach to network inference reconstructs network links based on the strength of associations between data in individual network nodes. It can reconstruct undirected networks, i.e., relevance networks, sensu stricto, as well as directed networks, referred to as causal relevance networks. The generalized approach allows the use of an arbitrary measure of pairwise association between nodes, an arbitrary scoring scheme that transforms the associations into weights of the network links, and a method for inferring the directions of the links. While this makes the approach powerful and flexible, it introduces the challenge of finding a combination of components that would perform well on a given inference task.

Results: We address this challenge by performing an extensive empirical analysis of the performance of 114 variants of the generalized relevance network approach on 47 tasks of gene network inference from time-series data and 39 tasks of gene network inference from steady-state data. We compare the different variants in a multi-objective manner, considering their ranking in terms of different performance metrics. The results suggest a set of recommendations that provide guidance for selecting an appropriate variant of the approach in different data settings.

Conclusions: The association measures based on correlation, combined with a particular scoring scheme of asymmetric weighting, lead to optimal performance of the relevance network approach in the general case. In the two special cases of inference tasks involving short time-series data and/or large networks, association measures based on identifying qualitative trends in the time series are more appropriate.

Figure 1

The first three Pareto fronts (PF) in the three-dimensional space of mean rankings of the variants of the CRN approach with respect to the three performance measures of AUROC, AUPRC, and rAUPRC. The rankings of the variants are averaged over all the network reconstruction tasks from Table 1. Each number in the legend represents the hypervolume dominated by the points in the corresponding Pareto front. The two-dimensional projection of the three-dimensional space was obtained using multidimensional scaling.

Figure 2

The association measures (left-hand side) and scoring schemes (right-hand side) used by the 11 top-ranked variants of the CRN approach from the first three Pareto fronts (PF) in the three-dimensional space of AUROC-AURPC-rAUPRC mean rankings. The rankings are averaged over the 47 tasks of GRN inference from time-series data listed in Table 1. Legend on the left-hand side: S: the class of symbolic and qualitative association measures; M: association measures based on mutual information; C: correlation-based; W: dynamic time warping; and D: distance-based measures. Legend on the right-hand side: WN: the AWE scoring scheme without time shifting; WS, MS, CS, and AS: the AWE, MRNET, CLR, and ARACNE with time shifting; and NS: the time-shifting method without a scoring scheme.

Figure 3

The association measures (left-hand side) and scoring schemes (right-hand side) used in the 7 (A) and 12 (B) top-ranked variants of the CRN approach from the first three Pareto fronts in the three-dimensional space of AUROC-AURPC-rAUPRC mean rankings. The rankings are averaged on the GRN inference tasks involving short (A, 20 tasks) and long (B, 27 tasks) time series.

Figure 4

The association measures (left-hand side) and scoring schemes (right-hand side) used in the 7 (A) and 15 (B) top-ranked variants of the CRN approach from the first three Pareto fronts in the three-dimensional space of AUROC-AURPC-rAUPRC mean rankings. The rankings are averaged over the GRN inference tasks involving small (A, 20 tasks of inference from time-series data) and large (B, 27 tasks of inference from time-series data) networks.

Figure 5

The association measures (left-hand side) and scoring schemes (right-hand side) used by the 12 top-ranked variants of the CRN approach from the first three Pareto fronts (PF) in the three-dimensional space of AUROC-AURPC-rAUPRC mean rankings. The rankings are averaged over the 39 tasks of GRN inference from steady-state data listed in Table 2. Legend on the left-hand side: M: the class of association measures based on mutual information; C: correlation-based and D: distance-based association measures. Legend on the right-hand side: WN, MN, CN, and AN: CRN-approach variants using the AWE, MRNET, CLR, and ARACNE scoring scheme; NN: variants without scoring scheme.

Figure 6

The association measures (left-hand side) and scoring schemes (right-hand side) used in the six (A) and nine (B) top-ranked variants of the CRN approach from the first three Pareto fronts in the three-dimensional space of AUROC-AURPC-rAUPRC mean rankings. The rankings are averaged over the GRN inference tasks involving small (A, 21 tasks of inference from steady-state data) and large (B, 18 tasks of inference from steady-state data) networks.