Type I errors and power of tests of correlations in a matrix
Description
When testing individual correlations in a matrix for significance, researchers primarily use F or z tests repeatedly, thereby inflating the overall Type I error rate. Alternative approaches are to test whether the observed correlation matrix differs from the identity matrix or to test individual correlations for significance using procedures which control overall Type I error rates. Monte Carlo simulations were conducted to examine the power and general characteristics of: (a) the Bartlett, Brien et al., Kullback, and Steiger procedures, tests of the first type, and (b) the multistage Bonferroni, rank order method proposed by Stavig and Acock (1976), and a modified rank order method which is based on comparing the rank ordered Fisher z transform with its corresponding critical value on the ordered half-normal score distribution; these are tests of the second type. Critical value tables for ordered half-normal scores are found in Appendix A The Brien et al. procedure was the most powerful for testing the hypothesis that the observed correlation matrix differs from the identity matrix and yet showed stable empirical alpha values. The modified rank order method was the most powerful in assessing significance of the largest correlation in a matrix; however, the Type I error rate was slightly inflated for small sample sizes with larger numbers of variables. The multistage Bonferroni had a conservative Type I error rate and demonstrated moderate power. In terms of the number of significant correlations in a matrix, the modified rank order method controlled Type I error well, while possessing the greatest power. Both the multistage Bonferroni and rank order methods had conservative Type I error rates, especially as the number of variables increased. In testing all correlations in a matrix via a stepdown procedure or in assessing individual correlations after the Brien et al. procedure was first found significant, all methods were too conservative. The modified rank order method is recommended for testing all correlations in a matrix, although correction in Fisher's z or some other adjustment might be needed to reduce the nonconservative bias with a greater number of variables