Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Oct 29;10(21):e39883.
doi: 10.1016/j.heliyon.2024.e39883. eCollection 2024 Nov 15.

A benchmark for Rey-Osterrieth complex figure test automatic scoring

Juan Guerrero-Martín  1 , María Del Carmen Díaz-Mardomingo  2 , Sara García-Herranz  3 , Rafael Martínez-Tomás  1 , Mariano Rincón  1
Affiliations

A benchmark for Rey-Osterrieth complex figure test automatic scoring

Juan Guerrero-Martín et al. Heliyon. .

Abstract

The Rey-Osterrieth complex figure (ROCF) test is a neuropsychological task that can be useful for early detection of cognitive decline in the elderly population. Several computer vision systems have been proposed to automate this complex analysis task, but the lack of public benchmarks does not allow a fair comparison of these systems. To advance in that direction, we present a benchmarking framework for the automatic scoring of the ROCF test that provides: the ROCFD528 dataset, which is the first open dataset of ROCF line drawings; and experimental results obtained by several modern deep learning models, which can be used as a baseline for comparing new proposals. We evaluate different state-of-the-art convolutional neural networks (CNNs) under traditional and transfer learning paradigms. Experimental quantitative results (MAE = 3.448) indicate that a CNN specifically designed for sketches outperforms other state of the art CNN architectures when the number of examples available is limited. This benchmark can also be a paradigmatic example within the broad field of machine learning for the development of efficient and robust models for analyzing line drawings and sketches not only in classification but also in regression tasks.

Keywords: Benchmark; Cognitive impairment detection; Deep learning; Rey-Osterrieth complex figure scoring; Transfer learning.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Figures

Figure 1
Figure 1
Socio-demographic characteristics of ROCFD528 dataset. (a) Participants' age at the time of the first evaluation; (b) Participants' academic level, measured in years of study.
Figure 2
Figure 2
Histogram of Osterrieth's QSS scores.
Figure 3
Figure 3
ROCF preprocessing. (a) Original image (472x405); (b) Resized image (384x384); (c) Binarized image (384x384).
Figure 4
Figure 4
Confusion matrices for the eleven configurations. Rows: GS score; Columns: Predicted score.
Figure 5
Figure 5
Confusion matrices comparing the scores of the two experts with the GS scores. Rows: GS score; Columns: Independent expert's score.
Figure 6
Figure 6
Confusion matrix comparing the scores of the two experts. Rows: Expert 1's score; Columns: Expert 2's score.
Figure 7
Figure 7
Moving window metric values calculated for the two human experts and the best five deep learning configurations. (a) PCC; (b) R2; (c) MAE; (d) RMSE; (e) MedAE.
See this image and copyright information in PMC

References

    1. Nichols E., Steinmetz J.D., Vollset S.E., Fukutaki K., Chalek J., Abd-Allah F., Abdoli A., Abualhasan A., Abu-Gharbieh E., Akram T.T., et al. Estimation of the global prevalence of dementia in 2019 and forecasted prevalence in 2050: an analysis for the global burden of disease study 2019. Lancet Public Health. 2022;7(2):e105–e125. - PMC - PubMed
    1. A. Rey, L'examen psychologique dans les cas d'encéphalopathie traumatique, (les problems), Archives de psychologie.
    1. P.A. Osterrieth, Le test de copie d'une figure complexe; contribution a l'etude de la perception et de la memoire, Archives de psychologie.
    1. Shin M.-S., Park S.-Y., Park S.-R., Seol S.-H., Kwon J.S. Clinical and empirical applications of the rey–osterrieth complex figure test. Nat. Protoc. 2006;1(2):892–899. - PubMed
    1. Webb S.S., Moore M.J., Yamshchikova A., Kozik V., Duta M.D., Voiculescu I., Demeyere N. Validation of an automated scoring program for a digital complex figure copy task within healthy aging and stroke. Neuropsychology. 2021;35(8):847. - PubMed

LinkOut - more resources