COSMOS: Interrater and Intrarater Reliability Study of a Novel Outcome Measure

Mayank Goyal^{1

2

3}, Aravind Ganesh^{1

4

3}, Salome L Bosshart^{1

5}, Alexander Stebner^{1

6}, Nishita Singh⁷, Bijoy K Menon^{1

3}, Shelagh B Coutts^{1

4

3}, Johanna M Ospel^{1

2}, Mohammed A Almekhlafi^{1

2

3}, Julie Kromm^{1

8}, Philippe Couillard^{1

3

8}, Dar Dowlatshahi⁹, Brian H Buck¹⁰, Brian van Adel^{11

12}, Karla J Ryckborst¹, Anurag Trivedi¹³, Chrysi Bogiatzi¹⁴, Alexandre Y Poppe¹⁵, Mohamad Mehdi¹, Luciana Catanese¹⁶, Ruth Whelan¹⁷, Robert Fahed¹⁸, Brindan Sivanandan¹, Mohamed A AlShamrani¹, Sachin M Mishra¹⁰, Umberto Pensato^{19

20}, Ben Agnelli¹, Arun Kathuveetil¹, Ivy A Sebastian¹, Aleksander Tkach²¹, Katrina H D Ignacio¹, Miranda Wan¹, Susan Alcock²², Rick H Swartz²³, Genoveva Maclean²⁴, Stacey Greco¹, Carol Kenney¹, Andrew M Demchuk^{1

3}, Michael D Hill^{1

3}

Affiliations

¹ Calgary Stroke Program, Department of Clinical Neurosciences, Foothills Medical Centre (M.G., A.G., S.L.B., A.S., B.K.M., S.B.C., J.M.O., M.A.Almekhlafi, J.K., P.C., K.J.R., M.M., B.S., M.A.AlShamrani, B.A., A.K., I.A.S., K.H.D.I., M.W., S.G., C.K., A.M.D., M.D.H.), University of Calgary, AB, Canada.
² Department of Diagnostic Imaging, Foothills Medical Centre (M.G., J.M.O., M.A.Almekhlafi), University of Calgary, AB, Canada.
³ Hotchkiss Brain Institute, Cumming School of Medicine, (M.G., A.G., B.K.M., M.A.Almekhlafi, S.B.C., P.C., A.M.D, M.D.H), University of Calgary, AB, Canada.
⁴ Department of Community Health Sciences (A.G., S.C.), University of Calgary, AB, Canada.
⁵ Department of Neurology, University Hospital Basel, University of Basel, Switzerland (S.L.B.).
⁶ Department of Radiology, Cantonal Hospital Munsterlingen, Switzerland (A.S.).
⁷ Rady Faculty of Health Sciences (N.S.), University of Manitoba, Winnipeg, Canada.
⁸ Department of Critical Care Medicine, Cumming School of Medicine (J.K., P.C.), University of Calgary, AB, Canada.
⁹ Department of Medicine, University of Ottawa and the Ottawa Heart Research Institute, Canada (D.D.).
¹⁰ Division of Neurology, Department of Medicine, University of Alberta, Edmonton, Canada (B.H.B., S.M.M.).
¹¹ Department of Radiology (B.v.A.), McMaster University, Hamilton, ON, Canada.
¹² Department of Neurosurgery (B.v.A.), McMaster University, Hamilton, ON, Canada.
¹³ Department of Medicine, Neurology Division (A. Trivedi), University of Manitoba, Winnipeg, Canada.
¹⁴ London Health Sciences Centre and Western University, ON, Canada (C.B.).
¹⁵ Department of Neurosciences, Université de Montréal, Canada (A.Y.P.).
¹⁶ Population Health Research Institute (L.C.), McMaster University, Hamilton, ON, Canada.
¹⁷ Royal University Hospital Stroke Program, Saskatoon, Canada (R.W.).
¹⁸ Division of Neurology, Department of Medicine, The Ottawa Hospital, Ottawa Hospital Research Institute, University of Ottawa, Ontario, Canada (R.F.).
¹⁹ Department of Neurology, IRCCS Humanitas Research Hospital, Rozzano, Milan, Italy (U.P.).
²⁰ Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy (U.P.).
²¹ Interior Health, Kelowna, British Columbia, Canada (A. Tkach).
²² Department of Radiology (S.A.), University of Manitoba, Winnipeg, Canada.
²³ Division of Neurology, Department of Medicine, Hurvitz Brain Sciences Program, Sunnybrook Health Sciences Centre, University of Toronto, ON, Canada (R.H.S.).
²⁴ Vancouver Coastal Health Research Institute, British Colombia, Canada (G.M.).

PMID: 40549842
DOI: 10.1161/STROKEAHA.125.049454

Free article

Review

COSMOS: Interrater and Intrarater Reliability Study of a Novel Outcome Measure

Mayank Goyal et al. Stroke. 2025 Jul.

Free article

. 2025 Jul;56(7):1958-1964.

doi: 10.1161/STROKEAHA.125.049454. Epub 2025 Jun 23.

Authors

Affiliations

¹ Calgary Stroke Program, Department of Clinical Neurosciences, Foothills Medical Centre (M.G., A.G., S.L.B., A.S., B.K.M., S.B.C., J.M.O., M.A.Almekhlafi, J.K., P.C., K.J.R., M.M., B.S., M.A.AlShamrani, B.A., A.K., I.A.S., K.H.D.I., M.W., S.G., C.K., A.M.D., M.D.H.), University of Calgary, AB, Canada.
² Department of Diagnostic Imaging, Foothills Medical Centre (M.G., J.M.O., M.A.Almekhlafi), University of Calgary, AB, Canada.
³ Hotchkiss Brain Institute, Cumming School of Medicine, (M.G., A.G., B.K.M., M.A.Almekhlafi, S.B.C., P.C., A.M.D, M.D.H), University of Calgary, AB, Canada.
⁴ Department of Community Health Sciences (A.G., S.C.), University of Calgary, AB, Canada.
⁵ Department of Neurology, University Hospital Basel, University of Basel, Switzerland (S.L.B.).
⁶ Department of Radiology, Cantonal Hospital Munsterlingen, Switzerland (A.S.).
⁷ Rady Faculty of Health Sciences (N.S.), University of Manitoba, Winnipeg, Canada.
⁸ Department of Critical Care Medicine, Cumming School of Medicine (J.K., P.C.), University of Calgary, AB, Canada.
⁹ Department of Medicine, University of Ottawa and the Ottawa Heart Research Institute, Canada (D.D.).
¹⁰ Division of Neurology, Department of Medicine, University of Alberta, Edmonton, Canada (B.H.B., S.M.M.).
¹¹ Department of Radiology (B.v.A.), McMaster University, Hamilton, ON, Canada.
¹² Department of Neurosurgery (B.v.A.), McMaster University, Hamilton, ON, Canada.
¹³ Department of Medicine, Neurology Division (A. Trivedi), University of Manitoba, Winnipeg, Canada.
¹⁴ London Health Sciences Centre and Western University, ON, Canada (C.B.).
¹⁵ Department of Neurosciences, Université de Montréal, Canada (A.Y.P.).
¹⁶ Population Health Research Institute (L.C.), McMaster University, Hamilton, ON, Canada.
¹⁷ Royal University Hospital Stroke Program, Saskatoon, Canada (R.W.).
¹⁸ Division of Neurology, Department of Medicine, The Ottawa Hospital, Ottawa Hospital Research Institute, University of Ottawa, Ontario, Canada (R.F.).
¹⁹ Department of Neurology, IRCCS Humanitas Research Hospital, Rozzano, Milan, Italy (U.P.).
²⁰ Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy (U.P.).
²¹ Interior Health, Kelowna, British Columbia, Canada (A. Tkach).
²² Department of Radiology (S.A.), University of Manitoba, Winnipeg, Canada.
²³ Division of Neurology, Department of Medicine, Hurvitz Brain Sciences Program, Sunnybrook Health Sciences Centre, University of Toronto, ON, Canada (R.H.S.).
²⁴ Vancouver Coastal Health Research Institute, British Colombia, Canada (G.M.).

PMID: 40549842
DOI: 10.1161/STROKEAHA.125.049454

Abstract

The vast majority of patients with minor stroke achieve what are considered good or excellent outcomes on the modified Rankin Scale (0-1/0-2), yet many are dissatisfied with their outcomes. There is a need for a functional outcome measure tailored for minor stroke that better reflects the spectrum of clinical outcomes within this population. We developed the Canadian Outcome Scale for Minor Stroke (COSMOS) and performed an interrater and intrarater reliability study. COSMOS is a 7-point scale ranging from 0 (no symptoms) to 6 (loss of independence for an instrumental or basic activity of daily living or worse), which accounts for performance limitations and losses of a person's hobbies or passions and of their employment, educational, service, or caregiving pursuits, besides just activities of daily living. One hundred test case vignettes were developed. Stroke physicians, fellows, and research nurses/staff were invited to review training materials and provide the COSMOS grade for 20 cases representing all COSMOS grades (0-6). After a minimum 2 weeks' wash-out period, participants were asked to grade the same 20 cases again. Interrater and intrarater agreement were assessed using Cohen κ, weighted κ, percentage agreement, and intraclass correlation coefficient. Among 33 participants (18 attending physicians, 9 stroke fellows, and 6 research staff/nurses; median 12.5 years of experience), COSMOS had substantial interrater reliability (80.5% agreement [95% CI, 75.7%-85.3%]; Cohen κ, 0.77 [95% CI, 0.72-0.84]) and almost-perfect intrarater reliability overall (87.1% agreement [95% CI, 84.4%-89.7%]; Cohen κ, 0.85 [95% CI, 0.82-0.88]); weighted κ showed almost perfect agreement for both interrater (0.88 [95% CI, 0.85-0.92]) and intrarater reliability (0.92 [95% CI, 0.90-0.94]). The overall chance-adjusted simultaneous intrarater/interrater agreement using intraclass correlation coefficient was 0.95 (95% CI, 0.94-0.97). Results were similar with substantial to almost-perfect agreement when considering key subgroups based on position (attendings, fellows, research nurses/staff) and years of experience. In conclusion, the newly proposed COSMOS scale demonstrated substantial interrater and intrarater reliability. The scale merits further study in cohort studies and clinical trials of minor stroke.

Keywords: activities of daily living; hemorrhagic stroke; ischemic stroke; outcome assessment, health care; physicians; reproducibility of results; stroke.

PubMed Disclaimer

Conflict of interest statement

Dr Goyal reports compensation from MicroVention, Inc, for consultant services; compensation from Mentice for consultant services; compensation from Medtronic for consultant services; grants from Medtronic to other; and grants from Johnson & Johnson Health Care Systems, Inc, to other. Dr Ganesh reports stock holdings in Collavidence, Inc, and stock options in SnapDx, Inc. Dr Stebner reports grants from Schweizerische Gesellschaft für Radiologie. Dr Almekhlafi reports compensation from Palmera Medical, Inc, for other services. Dr Poppe reports grants from Heart and Stroke Foundation of Canada; grants from Fondation Brain Canada; and grants from Canadian Institutes of Health Research. Dr Catanese reports employment by Hamilton Health Sciences; employment by McMaster University; grants from Servier Pharmaceuticals LLC; and compensation from Hoffmann-La Roche, Ltd, for consultant services. Dr Fahed reports compensation from Stryker for consultant services. Dr Ignacio reports compensation from the American Academy of Neurology for other services. Dr Swartz reports stock holdings in FollowMD, Inc, and compensation from F. Hoffmann-La Roche for consultant services. Dr Demchuk reports compensation from Medtronic for consultant services; compensation from NovaSignal for consultant services; compensation from Lumosa for data and safety monitoring services; compensation from Hoffmann-La Roche, Ltd, for consultant services; compensation from Boehringer Ingelheim for consultant services; stock holdings in Circle NVI; compensation from Philips for data and safety monitoring services; and a patent issued for stroke imaging software licensed to Circle NVI. Dr Hill reports grants from the Canadian Institutes of Health Research and employment by University of Calgary. The other authors report no conflicts.

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
Medical
- MedlinePlus Health Information

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

COSMOS: Interrater and Intrarater Reliability Study of a Novel Outcome Measure

Affiliations

COSMOS: Interrater and Intrarater Reliability Study of a Novel Outcome Measure

Authors

Affiliations

Abstract

Conflict of interest statement

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Medical