The evolution of raw data archiving and the growth of its importance in crystallography

Abstract

The hardware for data archiving has expanded capacities for digital storage enormously in the past decade or more. The IUCr evaluated the costs and benefits of this within an official working group which advised that raw data archiving would allow ground truth reproducibility in published studies. Consultations of the IUCr's Commissions ensued via a newly constituted standing advisory committee, the Committee on Data. At all stages, the IUCr financed workshops to facilitate community discussions and possible methods of raw data archiving implementation. The recent launch of the IUCrData journal's Raw Data Letters is a milestone in the implementation of raw data archiving beyond the currently published studies: it includes diffraction patterns that have not been fully interpreted, if at all. The IUCr 75th Congress in Melbourne included a workshop on raw data reuse, discussing the successes and ongoing challenges of raw data reuse. This article charts the efforts of the IUCr to facilitate discussions and plans relating to raw data archiving and reuse within the various communities of crystallography, diffraction and scattering.

Keywords: ground truth; raw data archive hardware; raw data measuring hardware; raw data policies at photon and neutron facilities; raw data processing software.

Figure 1

Data rates (GB s⁻¹) versus calendar year of successive generations of pixel area detectors. Reproduced with the permission of Leonarski et al. (2023 ▸).

Figure 2

Example of a cascade of data storage approaches at the European X-FEL. Experimental data are captured in real time to very high-performance systems in Schenefeld, but progressively move via high-performance broadband to slower but higher-capacity media at the DESY Data Center as the focus shifts from processing and analysis to review, backup and potential reuse. Taken from Dall’Antonia (2023 ▸), originally created by Krzysztof Wrona, European XFEL.