Evaluating FAIR maturity through a scalable, automated, community-governed framework

Abstract

Transparent evaluations of FAIRness are increasingly required by a wide range of stakeholders, from scientists to publishers, funding agencies and policy makers. We propose a scalable, automatable framework to evaluate digital resources that encompasses measurable indicators, open source tools, and participation guidelines, which come together to accommodate domain relevant community-defined FAIR assessments. The components of the framework are: (1) Maturity Indicators - community-authored specifications that delimit a specific automatically-measurable FAIR behavior; (2) Compliance Tests - small Web apps that test digital resources against individual Maturity Indicators; and (3) the Evaluator, a Web application that registers, assembles, and applies community-relevant sets of Compliance Tests against a digital resource, and provides a detailed report about what a machine "sees" when it visits that resource. We discuss the technical and social considerations of FAIR assessments, and how this translates to our community-driven infrastructure. We then illustrate how the output of the Evaluator tool can serve as a roadmap to assist data stewards to incrementally and realistically improve the FAIRness of their resources.

Fig. 1

The output of assessment of the FAIR Evaluator’s interface. A summary and graph appear at the top of the report, followed by the results of individual tests as panels, with simple green (success) red (failure) button indicators on each panel. Opening one of the result panels reveals an extensive, color-coded log of every activity undertaken by the Compliance Test, including the Metadata Harvester, and whether these ended in success or failure (and why). This allows the user to quickly scan the output for details, which will also serve as a guide to understand what it takes to improve the FAIRness of their resource.

Fig. 2

The output of the “Metadata identifier persistence” test. The information in the output indicates precisely what was observed (that the GUID presented to the test is a URL) and what is tested (compliance with known permanent-url schemas).

Fig. 3

The output of the “Metadata uses FAIR vocabularies (weak)” test. Numerous lines of information indicate the activities that were undertaken by the test. The final line indicates failure of the test, and the reason for that failure. The cut-off of ‘50%’ is arbitrary, and defined in the associated MI. Other MIs and their tests might have higher (or lower) levels of rigor.

Fig. 4

Fragments of the metadata added to the Evaluator page HTML. An abbreviated metadata record, in JSON-LD format, is shown. The important metadata features required to pass additional MI Compliance Tests, are highlighted in bold. Other metadata features are provided in anticipation of future tests related to “richness” of metadata - i.e., to improve searchability and citability. The full metadata record can be seen by viewing the source code of the Evaluator homepage.

Fig. 5

The conceptual workflow for executing a FAIR Evaluation. Most steps are optional (yellow boxes), and only become necessary if the set of published MIs and/or Collections is determined to be insufficient. The only steps required for every evaluation are to select a Collection of Compliance Tests appropriate for the digital resource being evaluated, and to submit the GUID of the metadata record for that digital resource (green boxes). Governance of the evaluation of, for example, a Maturity Indicator GitHub submission, and its associated test, is currently under consideration.

Fig. 6

The Metadata Harvester workflow. Yellow boxes are the starting GUIDs, text nodes are some form of URI. Arrows show the flow of information. White boxes are resolution activities, and the Associated Accept headers used. The pink box is a suite of third-party metadata extraction tools. Tika operates on a wide range of non-textual data (e.g. PDFs) to extract embedded metadata. Extruct and Distiller extract embedded metadata within HTML in a variety of formats. Purple barrels are metadata collection steps, where all Linked Data, and hash-style metadata are cached, together with the raw output of the resolution. InChI Keys (left pathway) have a defined two-step resolution mechanism, that supports content negotiation, and are therefore treated as a special case for efficiency. DOIs and other Handles are converted into URIs, and thereafter treated in the same manner. The first step of URI harvesting is to follow all Link rel = “meta” headers after resolution, to extract any metadata from these locations following the same workflow as for other URIs. These headers are followed only one layer deep, after which the system returns to the content of the original URI resolution, using content-negotiation.