Concordance of randomized and nonrandomized studies was unrelated to translational patterns of two nutrient-disease associations

Abstract

Objective: There are several examples in nutrition of discordance between the results of observational studies and randomized controlled trials (RCTs). We hypothesized that this discordance is attributable to differences in the translational paths of nutrient-disease associations. Translational paths can be assessed using citation analysis.

Study design and setting: We compared the characteristics of citation networks using examples, where RCTs and observational studies agreed (long-chain n-3 polyunsaturated fatty acids [n-3 PUFA]) or disagreed (vitamin E). We performed systematic reviews in each example, constructed citation networks, and compared them with respect to the number of articles and citation relationships between them, as well as the distribution of articles' hub and authority scores.

Results: For n-3 PUFA, meta-analyses of 14 RCTs and 10 observational studies both suggested that higher intake was associated with lower cardiovascular mortality. For vitamin E, the meta-analysis of 14 RCTs excluded a clinically significant effect, whereas 14 observational studies reported a significant inverse association. The respective citation networks consisted of 392 (n-3 PUFA) and 351 (vitamin E) articles. No differences between the characteristics of the two networks were identified. There was no evidence that the observational studies predated RCTs in the translational process in either example.

Conclusion: In the two examples, citation network characteristics do not predict concordance in the results of observational studies and RCTs.

Fig. 1

Simplistic (a) and more complex (b) hypothetical translational paths connecting a seminal observation to eventual randomized controlled trials (RCTs) in humans t: time. Each node represents a study/publication that “informs” subsequent publications—“information” relationships are shown as arcs. (Citation relationships can be considered as a proxy of these information relationships). In the simplistic linear translation model on the left panel (a), a hypothetical first (seminal) observation starts from an in vitro study in the lab (lowest left white node), and eventually gets translated to RCT in humans through a succession of research in animal disease models and epidemiological studies in humans. Each design builds on the findings of the previous one, and there is a temporal succession: in particular, RCTs are performed after the observational studies, as they are motivated, designed, and launched based on observational data. We are not aware of any empirical evidence that the linear scenario of panel a is actually observed for nutrient–disease relationships. If anything, anecdotal observations suggest a much more circuitous path, such as that in panel b: the first observation about a nutrient–disease association is made in humans. This spurs a complex network of interrelated research activity. Related hypotheses are tested in subsequent explorations using observational data, studies in animal models, and in RCTs. For example, the observation that long-chain n-3 polyunsaturated fatty acids (n-3 PUFA) may be associated with beneficial cardiovascular disease outcomes was first made in humans, and not in the lab: Greenland Eskimos consume a diet rich in n-3 PUFA and have a low incidence of cardiovascular disease compared with genetically related Danes [109,110]. This seminal observation has spurred a large body of research of various types.

Fig. 2

Example of a citation network of six articles. The figure shows a citation network (represented as a citation graph) in a hypothetical evidence base of six articles. Each article is depicted by a vertex (A through F). The horizontal placement of the articles corresponds to their year of publication; their vertical arrangement is arbitrary–its main purpose is clarity of presentation. Arcs denote citation relationships. The direction of the arc indicates flow of information from the cited paper (beginning of the arc) to the citing paper (end of the arc). Five out of six papers (papers B through F) are connected with citation arcs. There are four observed citation arcs (B → D, C → D, C → E, and D → F) out of a total of 15 possible ones (from any earlier- to any subsequently published article). The density of the citation relationships is defined as the ratio of observed to possible citation arcs, that is, here it is 4/15 = 0.27. Citation graphs have characteristics and properties that are described in the Appendix on the journal’s Website at www.elsevier.com. Briefly, citation arcs are simple directed acyclic graphs: (1) An article does not cite itself, and there is at most one citation relationship between two articles (simple graph); (2) if one follows the direction of the citation arcs along any possible path, one cannot visit the same article twice, that is, there are no circular directed paths (directed acyclic graph). (3) In addition, there is a temporal consistency constraint, for example, an earlier-published article cannot cite a later-published article.

Fig. 3

Citation networks including all studies sending information to an index article (i.e., an article included in our systematic reviews) the long-chain n-3 polyunsaturated fatty acids (panel a) and vitamin E (panel b) examples. Red-colored vertices denote index publications of RCTs. Blue-colored vertices denote index observational studies. Empty vertices without color are articles with primary data in humans that are pertinent to the association between the nutrient and clinical cardiovascular outcomes. Small black-filled vertices denote publications that have no primary data in humans (e.g., systematic or narrative reviews, studies in animals, commentaries) or that have primary data but are not pertinent to the association of interest (e.g., report on changes in lipid profiles or blood pressure rather than clinical cardiovascular outcomes). The horizontal positioning is the year of publication. The vertical positioning is arbitrary (chosen to enhance clarity of presentation).

Fig. 4

Quantile–quantile (QQ) plots for the number of citations made to other papers (left panel) or received from other papers (right panel). The axes indicate number of citations. QQ plots are used to compare two distributions (one in the long-chain n-3 polyunsaturated fatty acids [n-3 PUFA] example and one in the vitamin E example) by plotting corresponding quantiles against each other. If the distributions were indistinguishable, the respective quantiles would be equal and all the points would line on the diagonal. The majority of points are above the diagonal in both panels, suggesting higher counts of citations made or citations received for n-3 PUFA rather than the vitamin E. The Mann–Whitney test was significant (P < 0.001 in the left panel, and P = 0.013 in the right panel). Hub score distributions were also statistically significantly different between the two topics (P < 0.001), but authority score distributions did not differ beyond chance (P = 0.16).

Fig. 5

Citation networks of the subset of studies that have primary data in humans and are pertinent to the associations examined. Panel a: long chain n-3 polyunsaturated fatty acids. Panel b: Vitamin E. Red-colored vertices denote index publications of RCT (i.e., those included in our systematic reviews). Blue-colored vertices denote index observational studies (i.e., those included in our systematic reviews). The remaining vertices are articles with primary data in humans that are pertinent to the association between the nutrient and clinical cardiovascular outcomes. Of these, RCTs are colored in pink. This figure was generated from the citation networks of Fig. 3 after excluding the small black vertices.

Fig. 6

Enrollment periods of randomized controlled trials (RCTs) and years of publication of index RCTs and observational studies (i.e., those included in our systematic reviews). Panel a: long-chain n-3 polyunsaturated fatty acids. Panel b: Vitamin E. Red-colored vertices denote index publications of RCTs. Blue-colored vertices denote index observational studies. Vertices have area proportional to the size of the each (normalized within each study type). The red horizontal bars on top denote the enrollment periods of RCTs, reported (solid) or imputed (dashed).