Mixing web and mail methods in a survey of physicians

Abstract

Objective: To assess the effects of two different mixed-mode (mail and web survey) combinations on response rates, response times, and nonresponse bias in a sample of primary care and specialty internal medicine physicians.

Data sources/study setting: Primary data were collected from 500 physicians with an appointment in the Mayo Clinic Department of Medicine (DOM) between February and March 2005.

Study design: Physicians were randomly assigned to receive either an initial mailed survey evaluating the Electronic Medical Record (EMR) with a web survey follow-up to nonrespondents or its converse-an initial web survey followed by a mailed survey to nonrespondents. Response rates for each condition were calculated using standard formula. Response times were determined as well. Nonresponse bias was measured by comparing selected characteristics of survey respondents to similar characteristics in the full sample frame. In addition, the distributions of results on key outcome variables were compared overall and by data collection condition and phase.

Principal findings: Overall response rates were somewhat higher in the mail/web condition (70.5 percent) than in the web/mail condition (62.9 percent); differences were more pronounced before the mode switch prior to the mailing to nonrespondents. Median response time was 2 days faster in the web/mail condition than in the mail/web (median=5 and 7 days, respectively) but there was evidence of under-representation of specialist physicians and those who used the EMR a half a day or less each day in the web/mail condition before introduction of the mailed component. This did not translate into significant inconsistencies or differences in the distributions of key outcome variables, however.

Conclusions: A methodology that uses an initial mailing of a self-administered form followed by a web survey to nonrespondents provides slightly higher response rates and a more representative sample than one that starts with web and ends with a mailed survey. However, if the length of the data collection period is limited and rapid response is important, perhaps the web survey followed by a mailed questionnaire is to be preferred. Key outcome variables appear to be unaffected by the data collection method.

Figure 1

Study Sample, Data Collection Process, and Random Assignment

Figure 2

Cumulative Response by Date and Experimental Condition