The use of parallel analysis with ordinal data

Many authors recommended PA, a method which was suggested by Horn [8] in the context of PCA, to determine the number of factors or components in EFA or PCA [4,28]. In its original form, PA consists of two major steps: First, the eigenvalues of the correlation matrix of the dataset are computed. In a second step, a number of datasets which are comparable to the original dataset with respect to the sample size, the number of items and the distribution of the responses, but consist of uncorrelated items, are simulated. Based on these simulated datasets, the distribution of random eigenvalues is obtained. The number of dimensions underlying the dataset is determined by comparing the eigenvalues observed in the dataset and the distribution of random eigenvalues which are observed when there is no relationship between the items. An alternative method for conducting PA in EFA, which was based on principal axis factoring (PAFA), has been described by [11] and [29]. In the alternative approach, a modified correlation matrix is used, which contains communality estimates in its diagonal. In the simulation study presented by Timmerman and Lorenzo-Seva [2], a comparison of this modified approach to the original PA of Horn [8] indicated that the original approach led to more accurate results. This study also investigated the accuracy of PA with minimum rank factor analysis (MRFA). This approach for EFA aims at minimizing the amount of unexplained variance given a predefined number of common factors under the constraint that the reduced correlation matrix is positive-definite [2]. It was found that PA, when applied with MRFA, led to slightly more accurate results than PA applied to PCA in the datasets of their simulation study. In this study, the latent factors underlying all datasets were uncorrelated.

Timmerman and Lorenzo-Seva [2] also compared the accuracy of PA when applied to a PCA and EFA of polychoric and Pearson correlations. They found that approaches based on polychoric correlations generally led to accurate results, but also reported problems with non-convergence of the polychoric correlation matrices in the calculation of critical eigenvalues. In their study, PA was regarded non-convergent if 25 simulated matrices in a row were found to be indefinite or did not converge. In all cases where the polychoric correlation matrices converged, PCA and MRFA based on polychoric correlation matrices was found to be more accurate than PCA and MRFA based on Pearson matrices. Nevertheless, Timmerman and Lorenzo-Seva [2] concluded that the convergence problem prohibits the general application of PCA of polychoric correlations to empirical datasets. For Pearson correlations, PCA led to slightly more accurate results using the 95^th or 99^th percentile of the random eigenvalue distributions [7] than their mean [8] in datasets with uncorrelated latent factors. The simulation study of Timmerman and Lorenzo-Seva [2] calculated polychoric correlations using the two-step algorithm of Olsson [30]. This algorithm is based on maximum likelihood estimation of the correlation coefficient, while the thresholds which define the relation between the latent continuous variables and the categories of the manifest ordinal variable are calculated directly from the observed cumulative marginal proportions of each category in the data (see also the Methods section).

A recent simulation study [16] further investigated the accuracy of PA applied to PCA and MRFA in assessing the dimensionality of ordinal variables. Results indicated that PA with Pearson correlations is inaccurate with large levels of skewness and the use of PA with polychoric correlations was recommended for the dimensionality assessment of ordinal data. PCA tended to lead to more accurate solutions than MRFA in datasets with few (four) variables per factors, and also in datasets with correlated latent factors, whereas it was less accurate in datasets with a medium (eight) or large (12) number of variables per factor. Garrido et al. [16] used a smoothing algorithm described by Knol and Berger [31] (see next section) to circumvent the problem of indefinite polychoric correlation matrices.

Some further improvements on the original PA approach have been suggested as well, e.g., [7,32]. While Horn [8] originally recommended the use of the mean of the distribution of the simulated eigenvalues in PA to determine the number of components to retain, Glorfeld [7] first advocated using the 95^th percentile as a threshold to determine the number of components. This leads to more accurate results in datasets with uncorrelated factors, while Horn’s original method seems to lead to more accurate results when factors are correlated [16,33,34].

Further, several authors criticized that the procedure used in traditional parallel analysis is based on a model with 0 underlying factors [32,35]. It has been argued that, while this procedure may be appropriate to assess the presence of a single common factor in the data, it may be inappropriate for evaluating the presence of additional factors. Green et al. [32] thus have suggested incorporating all precedent k factors or components in modeling the reference distribution of the subsequent (k+1)th eigenvalue to improve the accuracy of parallel analysis. The (k+1)th critical eigenvalue is obtained by fitting a factor model with k factors on the dataset. In a second step, new datasets with the same number of variables and subjects as in the original dataset are simulated based on a model with k factors, whose loadings are identical to those observed in the original dataset. A descriptive statistic (like the mean or a specific percentile value) is calculated to summarize the distribution of the (k+1)th eigenvalue in these simulated datasets. This descriptive statistic is further used as a critical threshold for the (k+1)th eigenvalue. This method has been further investigated in a row of recent studies [36,37].

Although it has been argued that PA is no exact mathematical procedure [2,38], it has been found to provide an accurate assessment of the dimensionality of an item set under many conditions. Zwick and Velicer [5] evaluated the performance of PA in a simulation study, varying the size of the simulated item and respondent sample, the number of major components, and the magnitude of loading of the variables on each major component. They found that PA led generally to the most accurate results compared to the eigenvalue-greater-than-one criterion [6], the scree test [9], and Velicer’s minimum average partial method [10]. Humphreys and Montanelli [29] evaluated PA in EFA by applying it to simulated datasets which were created from predefined correlation matrices. They found that PA led to consistent results which corresponded to the number of dimensions defined by the correlation matrices. Peres-Neto, Jackson and Somers [39] compared the performance of PA with other stopping rules in PCA under several conditions, including different correlation matrices and different sizes of samples and variables. They found PA to be among the most accurate methods to determine the dimensionality of a test.

Some smoothing procedures for indefinite correlation matrices

The application of polychoric correlation matrices in the analysis of discrete ordinal variables assumes that a latent continuous normally distributed variable underlies each observed variable. The polychoric correlation is a maximum likelihood estimator of the Pearson correlation between the two normally distributed underlying variables [19]. However, since the estimation of the polychoric correlation matrix is based on different marginals for each correlation coefficient, the resulting matrix may be indefinite [31].

In order to apply PCA and factor analytical methods in these cases, several smoothing algorithms have been suggested, e.g., [2,16]. We will describe some selected algorithms below. All described smoothing algorithms have in common that they do not alter positive definite correlation matrices to which they are applied.

Higham [40] suggested an algorithm which searches for the symmetric positive definite matrix X which minimized the Frobenius norm to a given symmetrical matrix A. The Frobenius norm is defined by . This algorithm has been implemented in the R package Matrix [41].

An alternative approach for transforming indefinite correlation matrices into positive definite ones was described by Bentler and Yuan [42]. Given a symmetric indefinite matrix R, [42] proved that a positive definite matrix R* can be obtained by (1)

In formula (1), R₀ results from R by setting its diagonal entries to 0, D_R is the diagonal matrix containing the diagonal elements of R. Furthermore, Δ is a positive definite matrix which meets the condition that D_R − Δ²(R−D) is positive definite, where D is a diagonal matrix with nonzero elements such that (R−D) is positive semidefinite. Bentler and Yuan [42] further demonstrate a variation of their approach which is based on minimum trace factor analysis [43]. This approach can be applied using functions implemented in the R packages psych [44] and Rcsdp [45], see [26].

Knol and Berger [31] described another smoothing algorithm, which was later also suggested by [15] and applied in the simulation studies [16] and [26]. This smoothing algorithm is based on the idea to set eigenvalues which are smaller than a predefined threshold δ and to correct the associated eigenvectors accordingly to obtain a correlation matrix whose eigenvalues are thus greater than or equal to δ. Let R = KDK^t be the eigendecomposition of the symmetric matrix R, with D being a diagonal matrix of its eigenvalues λ_i, K the matrix of the corresponding normalized eigenvectors, and D⁺ a diagonal matrix in which all eigenvalues lower than δ have been replaced with δ, see [31]. Then, if Diag(X) denotes the matrix containing the entries of the main diagonal of X, a correlation matrix with eigenvalues greater than δ can be obtained by calculating. R(δ) = [Diag(KD⁺K^t)]^−1/2KD⁺K^t[Diag(KD⁺K^t)]^−1/2. Debelak and Tran [26] reported that this smoothing algorithm leads to results similar to those obtained with the algorithm of Higham [40] when used in PCA of binary items.

Another approach for smoothing indefinite correlation matrices has been proposed by Knol and ten Berge [23]. In contrast to the smoothing algorithms by Bentler and Yuan [42], Higham [40], and Knol and Berger [31], this smoothing algorithm allows defining rows and columns of the original correlation matrix which should not be altered by the smoothing algorithm. For the remaining parts of the correlation matrix R, an approximation matrix R* is determined which is positive definite, symmetrical with Diag(R*) = I and additionally meets the criterion that the trace of (R − R*)² is minimized. For finding this minimum, an iterative algorithm based on [43] is proposed. Although the Knol and ten Berge algorithm [23] is interesting for many practical applications, it is, to our knowledge, currently not implemented in any widely available software program.

Additional smoothing algorithms have been made available in specific software programs and packages. For the users of the R programing language, appropriate functions are contained in several software packages, like sfsmisc [46] and psych [44]. However, these algorithms are based on similar ideas as the ones described above, and are therefore not described in further detail.

It should be noted that the approaches described by Bentler and Yuan [42] and Knol and Berger [31] generally alter only few cells in an indefinite correlation matrix to obtain a positive definite correlation matrix, which constitutes an important difference to the algorithm of Higham [40] that makes small modifications to the complete correlation matrix. In contrast to this algorithm, the method of Bentler and Yuan [42] is based on the idea of rescaling variables of the indefinite correlation matrix which correspond to Heywood cases. In their simulation study on binary variables, Debelak and Tran [26] found that the algorithm of Bentler and Yuan [42] led to more accurate results than the algorithm of Higham [40] under most examined conditions.

Goals of the current study

A number of studies compared the accuracy of PA in assessing the dimensionality of an item set in the context of PCA and EFA when applied to polychoric correlation matrices with its accuracy when applied to Pearson correlation matrices [2,15,16,26]. These studies generally found that PA based on polychoric matrices led to an accurate assessment of the dimensionality of an item set, but the problem of indefinite polychoric correlation matrices led to the recommendation to use smoothing algorithms [15], and to the evaluation of various smoothing algorithms [2,16, 26]. Although the application of smoothing algorithms has been shown to enhance the accuracy of PA of polychoric correlations, little is known on the effects of different smoothing algorithms on the accuracy of PA. Although this issue has been examined in [26], this study was constrained to the investigation of PCA applied to tetrachoric correlations, so the generalization of these results to more general conditions (i.e., items with three or more response categories) or to the application of EFA remains unclear. The main goal of the current study was therefore to evaluate the application of three different widely available smoothing algorithms, and their effect on the accuracy of Horn’s [8] and Glorfeld’s [7] variations of PA in the context of PCA and principal axes factor analysis (PAFA) ([11,29]) as a method of EFA. It should be noted that several recent simulation studies [2,16] included MRFA as a promising method of EFA. This method was not included in this simulation study, since it is, to the best of our knowledge, currently not available in published R packages.

In our study, we restricted the application of smoothing algorithms to the three algorithms described by Bentler and Yuan [42], Higham [40] and Knol and Berger [31]. This choice was motivated by a number of considerations: First, these smoothing algorithms are available in the open source software R [47] and can therefore be easily applied in practical research. Second, these smoothing algorithms are based on specific, but different criteria for the smoothing method, and lead to different positive definite matrices when applied to an indefinite matrix; they can therefore not be regarded as interchangeable. Third, these smoothing algorithms tend to be computationally fast, which makes them attractive in practical application. The algorithm of Knol and ten Berge [23] was not included in our study, since it is, to our best of knowledge, currently not implemented in any available software package and seems to be hardly used in practical research. Since it is also possible to apply PCA and PAFA to indefinite correlation matrices, we further evaluated the accuracy of PA if no smoothing algorithms are used in the application of PA on polychoric correlation matrices.

Size of the respondent sample

Three different sizes of samples were used, which consisted of 200, 500 and 1000 respondents. This choice aimed to reflect the sample sizes found in practical applications of PCA and EFA. Numerous studies have already demonstrated that the size of the respondent sample has a significant impact on the accuracy of PA [2,16,26,33]. We therefore expected to obtain more accurate results in datasets with a greater respondent sample.

Underlying factor model

The response matrices analyzed in our study were based on a factor model which contained major, minor, and unique factors. Following earlier studies [2,48], the inclusion of many minor factors in our simulations aimed at simulating the effect of random factors on the observed variables which go beyond the effects of unique factors. These minor factors can be thought of as representing random influences on the observed variables and noise which cannot be controlled in applied research. Similar to the simulation study of Timmerman and Lorenzo-Seva [2], the population correlation matrix ∑_pop of the latent variables underlying the discrete responses to the items was given as (2)

In this formula, Λ_ma and Λ_mi stand for the factor loading matrices of the major and minor factors, respectively, with and being the transposed loading matrices. In our simulations, each major factor loaded on a constant number of variables, which was 5 or 15, depending on the simulated condition. Since we simulated conditions with one or three major factors, cases with 5, 15 or 45 variables were simulated. If a major factor loaded on a specific variable, the specific loading Λ_ma was set to 1, otherwise it was set to 0. The simulated factor models can be thus interpreted as having a simple structure with uncorrelated factors.Λ_mi contained the loadings of minor factors, which were included in some of our simulations. If minor factors were present, their number was set to be 40. The loadings of the minor factors on the variables were drawn from an uniform distribution in the interval [-1;1]. I_J is an identity matrix of size J, with J being the number of the variables. w_ma, w_mi and w_un are weights which determined the loadings of major and minor factors in the simulated dataset. The analyzed datasets differed with regard to the size of the loadings of the underlying factors on the observed variables. In conditions with high and medium factor loadings, w_ma was set to be 0.5 and 0.3 respectively. w_mi was set to 0 in simulated datasets where minor factors were absent, and to where minor factors were present in the data. w_un was set to (1 − w_mi − w_ma). It follows thus that in datasets without minor factors about 50% and 30% of the variance of the observed variables could be explained by the major factors in conditions with high and medium factor loadings, respectively. The resulting relationship between all major factors and the observed variables allowed us to use the number of major factors as criterion for the dimensionality of each dataset. Based on the population correlation matrix ∑_pop a set of 5, 15 or 45 continuous variables was defined which followed a multivariate standard normal distribution. This set was further transformed into a set of discrete ordinal variables by applying thresholds, which further defined the number of the response categories and the skewness of the response distribution.

Number of categories

In our simulation study, items with two, three or four response categories were simulated. While we expected the differences between methods based on Pearson and polychoric correlations to be most pronounced in conditions with two response categories, items with three or more response categories are regularly used in psychological and clinical assessments.

Skewness of the response distribution

For each number of response categories, a symmetrical and a skewed distribution of the observed variables were simulated. In case of a symmetrical distribution, the expected frequencies of observations were set to be equal for all response categories. In the skewed response distributions, the thresholds for defining the ordinal variables were set so that the response categories were observed with expected relative frequencies of [0.2; 0.8], [0.1; 0.2; 0.7], and [0.1; 0.1; 0.1; 0.7] under the conditions of simulated variables with two, three or four response categories, respectively. This led to response distributions with an estimated skewness of about 1.4, which may be interpreted as a meaningful departure from normality, see [16]. We expected the difference between methods based on Pearson and polychoric correlation matrices to be more pronounced under skewed response distributions.

In summary, 2 (number of major factors) x 2 (number of variables per major factor) x 3 (number of simulated respondents) x 3 (number of the response categories) x 2 (skewness of the response distribution) x 2 (presence of minor factors) x 2 (factor loadings) = 288 different conditions were simulated. Under each condition, the datasets were analyzed using four different variations of PA, which were based on PCA or EFA and used the criterion of Horn [8] or Glorfeld [7], respectively. 100 response matrices were simulated under each condition. In each simulated dataset, it was assessed how many factors or components were retained, and whether this number was identical to the number of major factors. The code used in this simulation study can be provided by the first author.

Calculation of the polychoric correlation matrices was based on algorithms implemented in the R package psych [44]. In all cases we used a two-step procedure which is based on the algorithm of Olsson [30], using item pair-specific estimators of the threshold coefficients for estimating the polychoric correlation matrix. Based on the results of a preliminary study, this approach appeared to be numerically more stable than an alternative procedure which estimated the threshold coefficients simultaneously for the whole matrix. For implementing the smoothing algorithms of Higham [40] and Bentler and Yuan [42], methods from the R packages Matrix [41], psych [44] and Rcsdp [45] were used. In order to apply the smoothing algorithm of Knol and Berger [31], code was written in R, which can be obtained from the authors. In our simulations, the constant δ was set to 0.1. We further investigated the accuracy of each parallel analysis method if no smoothing algorithm was applied.

Overall results

We first present some tables which summarize the accuracy of the dimensionality assessment of each method under each condition. Each dimensionality assessment was considered as accurate if the number of major factors was correctly measured. In general, all smoothing algorithms led to comparable results across all simulated conditions. Therefore, only the results obtained with the algorithms of Bentler and Yuan [42] and Knol and Berger [31] will be presented in detail. Overall, these two algorithms led to the most accurate assessments of dimensionality, with the algorithm of Bentler and Yuan [42] correctly identifying the number of greater factors in about 91.5% of the simulated datasets and the algorithm of Knol and Berger [31] in about 90.7% of the simulated datasets. The algorithm of Higham [40] (90.3% correct assessments) and the analysis of the indefinite correlation matrices (89.14% correct assessments) led to only slightly less accurate results. The methods based on the analysis of Pearson correlation matrices led to a correct assessment of dimensionality in 91.3% of the simulated datasets. We now present several tables which present the percentage of simulated datasets in which the number of major factors was reported correctly by the application of parallel analysis. The entries in the individual cells of these tables are based on 3600 datasets (in columns which present results for different numbers of major factors or different numbers of variables per factor) or 2400 datasets (in columns which present results for different numbers of respondents or different response categories). The confidence intervals for the individual entries in these tables are smaller than ± 1.6 for entries based on 3600 datasets and smaller than ± 2 for entries based on 2400 datasets, based on a significance level of 0.05. However, it should be noted that the size of the confidence interval is smaller for entries closer to 0 and 100. For entries above 90, the respective confidence intervals are smaller than ± 1.0 for entries based on 3600 datasets and smaller than ± 1.2 for entries based on 2400 datasets, again based on a significance level of 0.05. Additional detailed results for the accuracy of parallel analysis in datasets with high and medium factor loadings are provided in S1 Appendix.

Table 1 summarizes the results of the simulations where the response distribution was symmetrical, with no minor factors present.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Rate (in percent) of correctly detected number of major factors under different variations of PA when minor factors were not present under a symmetrical response distribution.

https://doi.org/10.1371/journal.pone.0148143.t001

Table 2 presents the analogous results of the simulations with symmetrical response distribution, with minor factors present.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Rate (in percent) of correctly detected number of major factors under different variations of PA when major and minor factors were present under a symmetrical response distribution.

https://doi.org/10.1371/journal.pone.0148143.t002

The Tables 3 and 4 present the results with skewed response distribution without (Table 3) and with minor factors (Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Rate (in percent) of correctly detected number of major factors under different variations of PA when only major factors were present under a skewed response distribution.

https://doi.org/10.1371/journal.pone.0148143.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Rate (in percent) of correctly detected number of major factors under different variations of PA when major and minor factors were present under a skewed response distribution.

https://doi.org/10.1371/journal.pone.0148143.t004

Indefinite correlation matrices

Overall, indefinite correlation matrices were observed in 8.1% of all simulated datasets. As can be seen in Tables 1 to 4, indefinite polychoric correlation matrices were generally observed more often in datasets with a large number of variables, a large number of factors and a low number of respondents. Polychoric correlation matrices also tended to be indefinite more often when the response distribution was skewed, compared to conditions when the response distribution was symmetrical, and when the number of response categories was low, compared with conditions with a higher number of response categories. In general, fewer indefinite correlation matrices were observed when factor loadings were lower.

Overall effects of study characteristics on the accuracy of dimensionality assessment

Independent of the approach used in dimensionality assessment, some general statements can be made with regard to which conditions allowed a higher accuracy in the detection of the major factors in a dataset. In general, the number of major factors could be detected more accurately when the number of respondents was large, the response distribution was symmetrical, the number of response categories was large, when the factor loadings were high, and when no minor factors were present in the dataset.

Accuracy of approaches based on polychoric correlation matrices and of approaches based on Pearson correlations

In datasets where indefinite polychoric correlation matrices were observed, the accuracy of methods based on Pearson correlations could be compared against methods based on the three smoothing algorithms. Compared to methods based on polychoric correlation matrices which applied the smoothing algorithm of Bentler and Yuan [42], which led to the most accurate dimensionality assessments in our simulation study, methods based on Pearson correlations were slightly less accurate. For both methods, 1152 dimensionality assessments were obtained, resulting from the four used variations of PA which were applied under 288 simulated conditions described in the method section. In these dimensionality assessments, the approach based on polychoric correlation matrices led to more accurate results in 213 cases, to less accurate results in 190 cases, and to equally accurate results in 749 cases. In general, methods based on Pearson correlations led to more accurate results only in datasets with medium factor loadings, no minor factors and a skewed response distribution, when the number of respondents was small.

Of all methods investigated in this study, the application of the modified PA of Glorfeld to a PCA of Pearson correlation and to a PCA of polychoric correlation matrices which were smoothed used the smoothing algorithm of Bentler and Yuan were the most accurate over all conditions simulated in this study. While the application of parallel analysis to a PCA of Pearson correlations led to an accurate assessment in 99.76% of all datasets, the application of PCA to polychoric correlations which were smoothed using the algorithm of Bentler and Yuan led to an accurate assessment in 99.34% of all datasets.

Comparison of the accuracy of parallel analysis based on PCA and parallel analysis based on PAFA

Over all conditions, the combination of parallel analysis and PCA was more accurate in the determination of the number of major factors than the modified parallel analysis which was used with PAFA. The specific results depended on the type of correlation matrix and, in the case of polychoric correlation matrices, the smoothing algorithm used in the analysis. For PCA of polychoric correlation matrices with smoothing algorithms of Knol and Berger [31] or Bentler and Yuan [42], the accuracy of the dimensionality assessment tended to be near 100% under all conditions. The parallel analysis with PAFA, when applied to polychoric correlation matrices with or without smoothing algorithms, was less accurate. Under almost all conditions, variations of parallel analysis which used the criterion of Glorfeld [7] were more accurate then approaches which used the original criterion of Horn [8]. Parallel analysis with PCA of Pearson correlation matrices led in general to an accurate assessment of the dimensionality of a dataset, while parallel analysis with PAFA of Pearson correlation matrices was less accurate than when applied to polychoric correlation matrices when the response distribution was symmetrical, or skewed with high factor loadings, but more accurate when the response distribution was skewed with medium factor loadings.