Another Look at Norris and Ortega (2000)

Abstract

Since Norris and Ortega‟s (2000) seminal work on the effectiveness of second language instruction, there has been a proliferation of meta-analyses in the field of applied linguistics. Subsequent meta-analysts, however, have uncritically followed the methodological choices made by Norris and Ortega. This paper suggests a critical reevaluation of the methodological procedures underlying the Norris and Ortega (2000) meta-analysis. I reexamined their procedures, and reassessed the 49 unique samples they used in their meta-analysis. In doing so, I identified three key methodological limitations with the study, pertaining, respectively, to (a) the data collection procedure, (b) the coding system, and (c) the statistical analysis. I argue that the lack of data quality inherent in the primary studies, the oversimplified coding scheme, and the inappropriate use of effect size statistics combine to compromise the validity of the conclusions Norris and Ortega have drawn from their meta-analysis. I subsequently provide alternative procedures which may yield a more empirically sound research synthesis, recommending, for future meta-analysts, the „best evidence synthesis‟ approach where conclusions are drawn from combining quantitative and qualitative analyses.