Validation of an integrated method for determining cognitive ability: Implications for routine assessments and clinical trials

Abstract

Introduction: Although accurate diagnosis of deficit of mild intensity is critical, various methods are used to assess, dichotomize and integrate performance, with no validated gold standard. This study described and validated a framework for the analysis of cognitive performance.

Methods: This study was performed by using the Groupe de Réflexion sur L'Evaluation des Fonctions EXécutives (GREFEX) database (724 controls and 461 patients) examined by 7 tests assessing executive functions. The first phase determined the criteria for the cutoff scores, the second phase, the effect of test number on diagnostic accuracy and the third phase, the best methods for combining test scores into an overall summary score. Four validation criteria were used: determination of impaired performance as compared to expected one, false-positive rate ≤5%, detection of both single and multiple impairments with optimal sensitivity.

Results: The procedure based on 5th percentile cutoffs determined from standardized residuals was the most appropriate procedure. Although area under the curve (AUC) increased with the number of scores (p = .0001), the false-positive rate also increased (p = .0001), resulting in suboptimal sensitivity for detecting selective impairment. Two overall summary scores, the average of the seven process scores and the Item Response Theory (IRT) score, had significantly (p = .0001) higher AUCs, even for patients with a selective impairment, and provided higher resulting prevalence of dysexecutive disorders (p = .0001).

Conclusions: The present study provides and validates a generative framework for the interpretation of cognitive data. Two overall summary score met all 4 validation criteria. A practical consequence is the need to profoundly modify the analysis and interpretation of cognitive assessments for both routine use and clinical research.

Keywords: Dementia; Diagnostic accuracy; Executive functions; Mild cognitive impairment; Stroke; sensitivity and specificity.

Figure 1

The three process phases used to interpret the raw scores obtained in battery of cognitive tests. Raw scores are obtained on a battery of tests. The first phase determines the cutoff scores used to dichotomize performance while controlling for the effects of demographic factors and the variables’ distributions. The second phase represents the common method based on numbering of impaired scores: it determines the effect of the number of tests (and thus the number of scores) on diagnostic accuracy following selection of tests. The third phase combines individual test scores into an overall summary score and determines its cutoff. Cutoff scores used for the diagnosis of deficit of an individual are represented at each phase in red color.

Figure 2

Receiver Operating Characteristic analyses as a function of the number of scores selected by the regression analysis.

Figure 3. Resulting prevalence of dysexecutive disorder using selected scores (from 1 to 8), the 19 scores of the battery (phase 2) and global summary scores (phase 3)

Results expressed as percent and 95%CI (confidence interval) *: the prevalence computed using the average of the 7 process score, IRT (item response theory) score and the 8-score set was higher (extended McNemar test, p=.0001) than that computed other overall summary scores and set-score.

Figure 4. Receiver operating characteristic analysis of the overall summary scores for executive processes in all subjects (4a, left) and in subjects with no more than 1 impaired executive process (4b, right)

SD: standard deviation