Measuring the evolution and output of cross-disciplinary collaborations within the NCI Physical Sciences-Oncology Centers Network

Abstract

Development of effective quantitative indicators and methodologies to assess the outcomes of cross-disciplinary collaborative initiatives has the potential to improve scientific program management and scientific output. This article highlights an example of a prospective evaluation that has been developed to monitor and improve progress of the National Cancer Institute Physical Sciences-Oncology Centers (PS-OC) program. Study data, including collaboration information, was captured through progress reports and compiled using the web-based analytic database: Interdisciplinary Team Reporting, Analysis, and Query Resource. Analysis of collaborations was further supported by data from the Thomson Reuters Web of Science database, MEDLINE database, and a web-based survey. Integration of novel and standard data sources was augmented by the development of automated methods to mine investigator pre-award publications, assign investigator disciplines, and distinguish cross-disciplinary publication content. The results highlight increases in cross-disciplinary authorship collaborations from pre- to post-award years among the primary investigators and confirm that a majority of cross-disciplinary collaborations have resulted in publications with cross-disciplinary content that rank in the top third of their field. With these evaluation data, PS-OC Program officials have provided ongoing feedback to participating investigators to improve center productivity and thereby facilitate a more successful initiative. Future analysis will continue to expand these methods and metrics to adapt to new advances in research evaluation and changes in the program.

Figure 1.

Categorization of PS-OC Program investigators and trainees into two groups, physical scientists (black) and cancer researchers (gray), using three different methods: (a) progress report data on trainees, (b) an automated classification algorithm inferring discipline based on publication history, and (c) surveys of both investigators and trainees.

Figure 2.

(A) Authorship collaboration network graphs. Nodes represent physical scientists (gray) or cancer researchers (black). Edges represent an intradisciplinary collaboration within physical scientists (gray) or cancer researchers (gray) and cross-disciplinary collaborations between physical scientists and cancer researchers (black). The edge width indicates the number of collaborations measured for a pair of scientists. (B) PS-OC investigators involved in cross-disciplinary authorship collaborations. Percentages of the 262 PS-OC (134 physical scientists, 128 cancer researchers) investigators participating exclusively in intradisciplinary collaborations or participating in cross-disciplinary collaborations pre- and post-award. (C) Pairwise collaborations. Percentages of intradisciplinary collaborations [physical scientists (PSs) only or cancer researchers (CRs) only] or cross-disciplinary collaborations out of the total number of pairwise collaborations during the pre- and post-award years.

Figure 3.

(A) Force-directed network graphs of reported collaborations generated using iTRAQR. Nodes represent a physical scientist (light gray), cancer researcher (dark gray), or unknown discipline, respectively. Edges represent all types of reported collaborations (non-publication, publication, project for within and outside the network) with the weight equal to the total number reported for that particular pair of researchers. (B) Normalized betweeness centrality value for the top 100 key nodes in the entire network diagrams for physical scientists and cancer researchers after 6 months (2010) and 3 years (2012).

Figure 4.

Characterization of the number of investigators per cross-disciplinary collaboration. (A) Survey results from respondents when asked to identify the number of investigators on a ‘successful’ cross-disciplinary collaboration. The percentages vary based on the discipline of the respondent. (B) Number of investigators per total reported collaborations in the progress reports. The number of investigators per collaboration shifts to larger numbers for cross-disciplinary collaborations versus intradisciplinary collaborations.

Figure 5.

Publications per pairwise collaboration type within the PS-OC Program. The research projects (41 projects) and cores (22 cores) produce on average less than one publication per reporting period. Trans-network projects (20 projects) have a higher publication output per collaboration.

Figure 6.

Flow diagram showing the correlation of type of authorship collaborations (cross-disciplinary (PS–CR), intradisciplinary physical scientists (PS–PS) or intradisciplinary cancer researchers (CR–CR) with the analysis of publication content (physical sciences–oncology, physical sciences, or oncology). Thickness of the lines reflects the percentage or investigators or publications contributing from one category to the next.

Figure 7.

Challenges to forming and tracking cross-disciplinary collaborations. (A) Summary of survey responses from PS-OC investigators and trainees on difficulties experienced during the cross-disciplinary collabroation. The average severity score for each difficulty is listed in parantheses (Scale: 1–5, 5 is most severe) (B) A summary of the continuity of collaborations reported by investigators in the progress reports every 6 months. Each line represents continuity of the same collaboration across two progress report periods.