How Do Users Respond to Mass Vaccination Centers? A Cross-Sectional Study Using Natural Language Processing on Online Reviews to Explore User Experience and Satisfaction with COVID-19 Vaccination Centers

Abstract

To reach large groups of vaccine recipients, several high-income countries introduced mass vaccination centers for COVID-19. Understanding user experiences of these novel structures can help optimize their design and increase patient satisfaction and vaccine uptake. This study drew on user online reviews of vaccination centers to assess user experience and identify its key determinants over time, by sentiment, and by interaction. Machine learning methods were used to analyze Google reviews of six COVID-19 mass vaccination centers in Berlin from December 2020 to December 2021. 3647 user online reviews were included in the analysis. Of these, 89% (3261/3647) were positive according to user rating (four to five of five stars). A total of 85% (2740/3647) of all reviews contained text. Topic modeling of the reviews containing text identified five optimally latent topics, and keyword extraction identified 47 salient keywords. The most important themes were organization, friendliness/responsiveness, and patient flow/wait time. Key interactions for users of vaccination centers included waiting, scheduling, transit, and the vaccination itself. Keywords connected to scheduling and efficiency, such as "appointment" and "wait", were most prominent in negative reviews. Over time, the average rating score decreased from 4.7 to 4.1, and waiting and duration became more salient keywords. Overall, mass vaccination centers appear to be positively perceived, yet users became more critical over the one-year period of the pandemic vaccination campaign observed. The study shows that online reviews can provide real-time insights into newly set-up infrastructures, and policymakers should consider their use to monitor the population's response over time.

Keywords: health services design; keyword extraction; machine learning; mass vaccination centers; national vaccination campaign; natural language processing; online reviews; pandemic response; patient experience; patient satisfaction; text mining; topic modeling; vaccine uptake.

Figure A1

Overview of COVID-19 vaccination centers in Berlin by opening order.

Figure A2

Framework for vaccination center user experience combining key determinants of user experience and the vaccination center user journey.

Figure 1

Overview of time periods for analysis from December 2020 to December 2021.

Figure 2

Number of reviews (A), review rating (B), and keywords (C) by time period and overall.

Figure 3

Results from topic modeling. (A) Intertopic distance map of five optimally latent topics and (B) top 10 terms per topic by frequency and relevance. In the intertropic distance map, each circle represents one topic. The circle size represents the relative number of terms that belong to the topic. The distance between circles represents the relative similarity and connectedness of topics. Topic circles that are closer to each other have more terms in common. For the top 10 terms per topic, the most frequent terms within a topic are shown at λ-value 1, and the top terms combining frequency and relevance are shown at λ-value 0.6. Relevance reflects the level at which a term exclusively belongs to a single topic. The λ-values 1 and 0.6 are suggested by the prior literature to analyze topics (see Section 2.2.1).

Figure 4

Top 30 keywords identified through keyword extraction. (A) Top 30 keywords featured across all reviews (N = 2740), sorted by frequency. (B) Top 30 keywords featured in positive reviews (four- or five-star rating, N = 2426) and negative reviews (one- or two-star rating, N = 223), sorted by frequency.

Figure 5

Keywords featured in at least 1% of reviews sorted to the user experience framework. Keywords identified through the keyword extraction were grouped and then mapped against the framework to highlight key interactions and identify recurring themes across reviews. A single review could contain keywords related to several dimensions or phases, e.g., both “organization” and “staff”. A single keyword may also simultaneously be related to an enabler and the journey, e.g., “wait”, which is related to “waiting” in the journey and to “patient flow/wait time” under enabler dimensions. The frequency count of the enabler dimensions or journey phases hence do not add up to the total number of reviews.