Linking scientific instruments and computation: Patterns, technologies, and experiences

Abstract

Powerful detectors at modern experimental facilities routinely collect data at multiple GB/s. Online analysis methods are needed to enable the collection of only interesting subsets of such massive data streams, such as by explicitly discarding some data elements or by directing instruments to relevant areas of experimental space. Thus, methods are required for configuring and running distributed computing pipelines-what we call flows-that link instruments, computers (e.g., for analysis, simulation, artificial intelligence [AI] model training), edge computing (e.g., for analysis), data stores, metadata catalogs, and high-speed networks. We review common patterns associated with such flows and describe methods for instantiating these patterns. We present experiences with the application of these methods to the processing of data from five different scientific instruments, each of which engages powerful computers for data inversion,model training, or other purposes. We also discuss implications of such methods for operators and users of scientific facilities.

Keywords: Experiment automation; Globus; big data; computing fabric; data fabric; machine learning; scientific facility; synchrotron light source; trust fabric; workflow.

Graphical abstract

Figure 1

Two examples of instrumentation + computation applications, showing constituent flows Left: serial synchrotron crystallography; right: high-energy diffraction microscopy. In each, a variety of computing systems (including, on the right, a Cerebras AI accelerator) are used to enable rapid collection and analysis of data from synchrotron light source experiments. In each subfigure, we show, as directed acyclic graphs linking distinct actions, both the distinct flows used to automate different functions (above) and their deployment in the context of the applications (below). The callouts indicate quality of service requirements.

Figure 2

A simple example flow and its implementation From top to bottom: (1) User perspective of a simple flow that, successively (shown left to right), (A) transfers data from an instrument to an analysis computer, (B) runs an analysis, (C) asks a user to review the analysis result, and, if (D) the user review is positive, (E) publishes the data to a repository. (2) The Globus platform services engaged by the transfer, compute, query, and search action providers. (3) The resources interacted with by those platform services: instrument storage system, co-located analysis storage system and storage computer, scientist, and data repository. Not shown are the Globus Auth service that handles identities and access tokens, and the Globus Flows service that coordinates flow execution.

Figure 3

Depictions of the flows presented in the paper An x-ray photon correlation spectroscropy processing flow, XPCS; three serial synchrotron crystallography flows, SSX-Stills, SSX-Prime, and SSX-Publish; a ptychography image reconstruction flow, Ptycho; a training flow for a neural network function approximator, BraggNN; and a high-energy diffraction microscopy far-field reconstruction flow, HEDM. Text above each circle names the action; text below describes its application in the flow.

Figure 4

Resource usage over time by five experiments Total flows, data transferred, and compute time used (on 64-core ALCF Theta nodes), per quarter, for the five experiments described in application experiences.

Figure 5

The number of concurrent XPCS flows over a roughly 12 h period, March 10–11, 2022 The initial peaks are burst tests before beginning the experiment; by 00:00, a constant stream of data from the beamline is processed.

Figure 6

Distribution of runtimes for the seven flows discussed in the text Box plots show upper and lower quartiles, with whiskers to 1.5× the interquartile range.

Figure 7

For the instance of each flow with median runtime, a timeline for its constituent actions The empty spaces between steps correspond to flow orchestration overheads. Note that the Ptycho analysis times are scaled to 50% (from 2,261 to 1,130 s total) so as to better show details in the other flows.

Figure 8

Run time distributions in seconds Distributions of run time (first row) and overhead (second row), in seconds, for each of the 11 steps in the XPCS flow. MD, metadata.

Figure 9

SSX data analysis portal Facets on the left allow for selection of different proteins (nsp10nsp16 is selected here), chips, and creation dates. Search results, shown on the right, provide researchers with a quick summary of the experiment and visual representation of the analysis results.