Online Surveys: Effect of Research Design on Rates of Invalid Participation and Data Credibility

Abstract

Designing online research often involves a trade-off between procedures that maximize administration efficiency and those that encourage valid and considerate participation. This article reports on three different online research design conditions (n = 413), differing in participant screening, incentive, and anonymity, that were used in a separate study on cognition in smoking cessation. High rates of invalid participation were observed in a condition in which participants participated anonymously, received a $20 incentive, and were screened for eligibility online. In addition, this condition produced significantly different data (variance, covariance, and central tendency) than the other two conditions, which involved less incentive or personal eligibility screening without participant anonymity. Removal of apparent “invalid” participants on the basis of a data screening protocol corrected some, but not all, of these differences. Results indicate that online designs offering monetary incentives should implement procedures to enhance data integrity even at the cost of increased participation barriers.