Assessing response reliability of health interview surveys using reinterviews

Abstract

Data from interview surveys of households or health facilities are used to assess community parameters such as health status and factors related to the ability and willingness of individuals to pay for health services. Although the effect of sample size on confidence intervals is generally well understood by the survey designers and policy-makers who use the results, the typical survey is also subject to non-sampling errors whose magnitude may exceed that of the sampling errors. The non-sampling errors associated with surveys are only rarely assessed and reported, even though they may have a major effect on the interpretation of findings. The present study reports the non-sampling errors associated with a household survey in Sierra Leone by comparing the results of reinterviews with the responses given during the original interviews. Certain types of questions were subject to greater non-sampling errors than others. The findings should be of use to designers of similar surveys and to those who rely on such surveys for making policy decisions.

PIP: An evaluation of nonsampling errors in small scale surveys indicates that various degrees of nonsampling errors did occur in the Sierra Leone survey of health services user fees in the rural population. The focus was on response reliability when tested with reinterviews. Since there is a paucity of studies on small scale response reliability, there was little to compare results with. However, it was deemed very important that the issue of nonsampling errors be dealt with, since the growth of health interview surveys in developing countries has increased, and policies are based on research results. 5 general results that identify types of questions which are prone to errors were discussed. 1) High reliability (case error of 10%) was found for questions on sex, actions taken in response to illness, whether an injection was received as a treatment, identification of household members who were ill during the recall period, and the source of money used to pay for treatment. 2) Case errors of 10-20% were found for items on incidence of illness, whether enough money was available to pay for treatment, age of ill household members, the cost of treatment for illnesses that were self treated, and treatment costs that were low to medium. 3) High case errors of 20-30% were related to items on the result of action taken, the seriousness of illness, the reason for choosing nonmedical treatment, and high expenditures on treatment. 4) Subjective responses to questions were problematic. 5) Response errors on expenditures for treatment varied widely. The interpretation when errors is dependent on the use made of the data and the nature of the variances. A consistent treatment of "outliers" is important. A large number of small deviations will show a low correlation, which probably is insignificant when costs vary from 0-4 leones, but when a few responses show gross deviations in expenditures, the effect is noteworthy. When misclassification of categorical events is high, the interpretation of results should state this fact. Common sense says that wide confidence intervals in X2 tests indicate true differences. Before attempting new surveys in unfamiliar cultural settings, knowledge about the reliability of certain questions and the magnitude of nonsampling errors occur is dependent on the use made of the data and the nature of the variances. A consistent treatment of "outliers" is important. A large number of small deviations will show a low correlation, which probably is insignificant when costs vary from 0-4 leones, but when a few responses show gross deviations in expenditures, the effect is noteworthy. When misclassification of categorical events is high, the interpretation of results should state this fact. Common sense says that wide confidence intervals in X2 tests indicate true differences. Before attempting new surveys in unfamiliar cultural settings, knowledge about the reliability of certain questions and the magnitude of nonsampling errors should be obtained. There is discussion of study materials and methods, the reporting of errors (gross error rate, net bias, and relative net bias), and the reliability of variables.