Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Sep 7;22(9):e19472.
doi: 10.2196/19472.

Validation of the Raw National Aeronautics and Space Administration Task Load Index (NASA-TLX) Questionnaire to Assess Perceived Workload in Patient Monitoring Tasks: Pooled Analysis Study Using Mixed Models

Sadiq Said #  1 , Malgorzata Gozdzik #  1 , Tadzio Raoul Roche  1 , Julia Braun  2 , Julian Rössler  1 , Alexander Kaserer  1 , Donat R Spahn  1 , Christoph B Nöthiger  1 , David Werner Tscholl  1
Affiliations

Validation of the Raw National Aeronautics and Space Administration Task Load Index (NASA-TLX) Questionnaire to Assess Perceived Workload in Patient Monitoring Tasks: Pooled Analysis Study Using Mixed Models

Sadiq Said et al. J Med Internet Res. .

Abstract

Background: Patient monitoring is indispensable in any operating room to follow the patient's current health state based on measured physiological parameters. Reducing workload helps to free cognitive resources and thus influences human performance, which ultimately improves the quality of care. Among the many methods available to assess perceived workload, the National Aeronautics and Space Administration Task Load Index (NASA-TLX) provides the most widely accepted tool. However, only few studies have investigated the validity of the NASA-TLX in the health care sector.

Objective: This study aimed to validate a modified version of the raw NASA-TLX in patient monitoring tasks by investigating its correspondence with expected lower and higher workload situations and its robustness against nonworkload-related covariates. This defines criterion validity.

Methods: In this pooled analysis, we evaluated raw NASA-TLX scores collected after performing patient monitoring tasks in four different investigator-initiated, computer-based, prospective, multicenter studies. All of them were conducted in three hospitals with a high standard of care in central Europe. In these already published studies, we compared conventional patient monitoring with two newly developed situation awareness-oriented monitoring technologies called Visual Patient and Visual Clot. The participants were resident and staff anesthesia and intensive care physicians, and nurse anesthetists with completed specialization qualification. We analyzed the raw NASA-TLX scores by fitting mixed linear regression models and univariate models with different covariates.

Results: We assessed a total of 1160 raw NASA-TLX questionnaires after performing specific patient monitoring tasks. Good test performance and higher self-rated diagnostic confidence correlated significantly with lower raw NASA-TLX scores and the subscores (all P<.001). Staff physicians rated significantly lower workload scores than residents (P=.001), whereas nurse anesthetists did not show any difference in the same comparison (P=.83). Standardized distraction resulted in higher rated total raw NASA-TLX scores (P<.001) and subscores. There was no gender difference regarding perceived workload (P=.26). The new visualization technologies Visual Patient and Visual Clot resulted in significantly lower total raw NASA-TLX scores and all subscores, including high self-rated performance, when compared with conventional monitoring (all P<.001).

Conclusions: This study validated a modified raw NASA-TLX questionnaire for patient monitoring tasks. The scores obtained correctly represented the assumed influences of the examined covariates on the perceived workload. We reported high criterion validity. The NASA-TLX questionnaire appears to be a reliable tool for measuring subjective workload. Further research should focus on its applicability in a clinical setting.

Keywords: National Aeronautics and Space Administration Task Load Index; awareness; patient monitoring; questionnaires; situation awareness; thromboelastometry; workload.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: The University of Zurich owns the intellectual property rights to the technologies described in this manuscript and registered “Visual Clot” and “Visual Patient” as trademarks. The University of Zurich and Instrumentation Laboratory Company/Werfen Corporation, Bedford, MA, USA, signed a letter of intent regarding a proposed joint development and licensing agreement to develop a product based on the concept of Visual Clot. As designated inventors DS, CN, and DT may receive royalties in the event of commercialization. The authors DT, DS, and CN are in a joint development agreement with the monitoring manufacturer Philips Healthcare (Koninklijke Philips NV) for Visual Patient. Within the framework of this cooperation, a monitoring system based on an avatar will be developed. Within the framework of licensing the technology via the University, the authors DT and CN might receive royalties as designated inventors in the event of successful product release. DS academic department is receiving grant support from the Swiss National Science Foundation, Berne, Switzerland; the Swiss Society of Anesthesiology and Reanimation (SGAR), Berne, Switzerland; the Swiss Foundation for Anesthesia Research, Zurich, Switzerland; and Vifor SA, Villars-sur-Glâne, Switzerland. DS is the co-chair of the ABC-Trauma Faculty, sponsored by unrestricted educational grants from Novo Nordisk Health Care AG, Zurich, Switzerland; CSL Behring GmbH, Marburg, Germany; LFB Biomédicaments, Courtaboeuf Cedex, France; and Octapharma AG, Lachen, Switzerland. Dr Spahn received honoraria/travel support for consulting or lecturing from Danube University of Krems, Austria; US Department of Defense, Washington, USA; European Society of Anesthesiology, Brussels, BE; Korean Society for Patient Blood Management, Seoul, Korea; Korean Society of Anesthesiologists, Seoul, Korea; Network for the Advancement of Patient Blood Management, Haemostasis and Thrombosis, Paris, France; Baxalta Switzerland AG, Volketswil, Switzerland; Bayer AG, Zürich, Switzerland; B. Braun Melsungen AG, Melsungen, Germany; Boehringer Ingelheim GmbH, Basel, Switzerland; Bristol-Myers-Squibb, Rueil-Malmaison Cedex, France and Baar, Switzerland; CSL Behring GmbH, Hattersheim am Main, Germany and Berne, Switzerland; Celgene International II Sàrl, Couvet, Switzerland; Daiichi Sankyo AG, Thalwil, Switzerland; Ethicon Sàrl, Neuchâtel, Switzerland; Haemonetics, Braintree, MA, USA; Instrumentation Laboratory (Werfen), Bedford, MA, USA; LFB Biomédicaments, Courtaboeuf Cedex, France; Merck Sharp & Dohme, Kenilworth, New Jersey, USA; PAION Deutschland GmbH, Aachen, Germany; Pharmacosmos A/S, Holbaek, Denmark; Photonics Healthcare BV, Utrecht, Netherlands; Pfizer AG, Zürich, Switzerland; Pierre Fabre Pharma, Alschwil, Switzerland; Roche Diagnostics International Ltd, Reinach, Switzerland; Sarstedt AG & Co, Sevelen, Switzerland and Nümbrecht, Germany; Shire Switzerland GmbH, Zug, Switzerland; Tem International GmbH, Munich, Germany; Vifor Pharma, Munich, Germany, Neuilly sur Seine, France, and Villars-sur-Glâne, Switzerland; Vifor (International) AG, St. Gallen, Switzerland; and Zuellig Pharma Holdings, Singapore, Singapore. CN and DT received travel support for consulting and lecturing from Instrumentation Laboratory (Werfen), Bedford, MA, USA. CN and DT received proof-of-concept funding from the University of Zurich to prototype Visual Patient. The University of Zurich and Koninklijke Philips NV, Amsterdam, Netherlands entered a joint development and licensing agreement to develop a product based on Visual Patient. As inventors, CN and DT may receive royalty payments in the event of commercialization. AK received honoraria for lecturing from Bayer AG (Switzerland). The other authors do not have any conflicts of interest.

Figures

Figure 1
Figure 1
Graphic showing the Visual Clot and Visual Patient technologies. (A) Bleeding Visual Clot illustrated with its different coagulation components. These are either present or absent, depending on the coagulation status. (B) Visual Patient with its different visualizations of vital parameters.
Figure 2
Figure 2
Correlation of different covariates with the total score of the National Aeronautics and Space Administration Task Load Index (NASA-TLX) workload assessment tool. Left and right of the dashed line indicate lower and higher perceived workload, respectively. KSW: Cantonal Hospital Winterthur; UKF: University Hospital Frankfurt.
Figure 3
Figure 3
Correlation of different covariates with subscores of the National Aeronautics and Space Administration Task Load Index (NASA-TLX) workload assessment tool. The original NASA-TLX questionnaire evaluated performance on an inverted X-axis from perfect to failure, with a low raw score corresponding to good self-rated performance. Therefore, to ensure that the X-axis in this figure is the same, the self-rated performance is displayed inverted. KSW: Cantonal Hospital Winterthur; UKF: University Hospital Frankfurt.
See this image and copyright information in PMC

References

    1. World Health Organization Safe Surgery Saves Lives. PACEsetterS. 2008;5(3):21. doi: 10.1097/01.jbi.0000393253.18990.c6. - DOI
    1. Schulz CM, Endsley MR, Kochs EF, Gelb AW, Wagner KJ. Situation awareness in anesthesia: concept and research. Anesthesiology. 2013 Mar;118(3):729–42. doi: 10.1097/ALN.0b013e318280a40f. - DOI - PubMed
    1. Reason J. Human error: models and management. West J Med. 2000 Jun;172(6):393–6. doi: 10.1136/ewjm.172.6.393. http://europepmc.org/abstract/MED/10854390 - DOI - PMC - PubMed
    1. Schulz CM, Krautheim V, Hackemann A, Kreuzer M, Kochs EF, Wagner KJ. Situation awareness errors in anesthesia and critical care in 200 cases of a critical incident reporting system. BMC Anesthesiol. 2016 Jan 16;16:4. doi: 10.1186/s12871-016-0172-7. - DOI - PMC - PubMed
    1. Nachreiner F. International standards on mental work-load--the ISO 10,075 series. Ind Health. 1999 Apr;37(2):125–33. doi: 10.2486/indhealth.37.125. https://joi.jlc.jst.go.jp/JST.Journalarchive/indhealth1963/37.125?from=P... - DOI - PubMed

Publication types