Novel Approach to Personalized Physician Recommendations Using Semantic Features and Response Metrics: Model Evaluation Study

Abstract

Background: The rapid growth of web-based medical services has highlighted the significance of smart triage systems in helping patients find the most appropriate physicians. However, traditional triage methods often rely on department recommendations and are insufficient to accurately match patients' textual questions with physicians' specialties. Therefore, there is an urgent need to develop algorithms for recommending physicians.

Objective: This study aims to develop and validate a patient-physician hybrid recommendation (PPHR) model with response metrics for better triage performance.

Methods: A total of 646,383 web-based medical consultation records from the Internet Hospital of the First Affiliated Hospital of Xiamen University were collected. Semantic features representing patients and physicians were developed to identify the set of most similar questions and semantically expand the pool of recommended physician candidates, respectively. The physicians' response rate feature was designed to improve candidate rankings. These 3 characteristics combine to create the PPHR model. Overall, 5 physicians participated in the evaluation of the efficiency of the PPHR model through multiple metrics and questionnaires as well as the performance of Sentence Bidirectional Encoder Representations from Transformers and Doc2Vec in text embedding.

Results: The PPHR model reaches the best recommendation performance when the number of recommended physicians is 14. At this point, the model has an F₁-score of 76.25%, a proportion of high-quality services of 41.05%, and a rating of 3.90. After removing physicians' characteristics and response rates from the PPHR model, the F₁-score decreased by 12.05%, the proportion of high-quality services fell by 10.87%, the average hit ratio dropped by 1.06%, and the rating declined by 11.43%. According to whether those 5 physicians were recommended by the PPHR model, Sentence Bidirectional Encoder Representations from Transformers achieved an average hit ratio of 88.6%, while Doc2Vec achieved an average hit ratio of 53.4%.

Conclusions: The PPHR model uses semantic features and response metrics to enable patients to accurately find the physician who best suits their needs.

Keywords: PPHR; PPHR model; SBERT; Sentence Bidirectional Encoder Representations From Transformers; patient-physician hybrid recommendation; smart triage systems; text analysis; web-based medical service.

Figure 1

Sentence Bidirectional Encoder Representations from Transformers (BERT) architecture.

Figure 2

The architecture of the patient-physician hybrid recommendation model. SBERT: Sentence Bidirectional Encoder Representations from Transformers; TF-IDF: term frequency–inverse document frequency.

Figure 3

Comparison of our proposed patient-physician hybrid recommendation (PPHR) model with the patient feature–based (PFB) model and patient-physician hybrid (PPH) model in terms of various indexes, including precision (blue), recall (green), F1-score (red), and hqos_prop (orange).