Exploratory and Confirmatory Factor Analysis: Survey Scale Validation Workflow

Validating a new survey scale requires exploratory and confirmatory factor analysis across split samples. Here is the step-by-step workflow from item screening to lavaan CFA reporting.

HumanPen Team

· 10 min read

Why exploratory and confirmatory factor analysis require independent or split samples

Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) address fundamentally different psychometric objectives. EFA identifies latent dimensional structure without prior constraints, whereas CFA tests whether a hypothesized measurement model replicates. Performing both procedures on the same sample introduces severe capitalization on chance, yielding artificially inflated fit indices and almost certain desk rejection. Researchers must collect independent cohorts or randomly split their dataset into two distinct calibration and validation subsamples.

Methodological pitfall: Never use EFA to trim cross-loading items and then immediately fit a CFA to the surviving items using the exact same respondents. That circular procedure merely tests whether sampling idiosyncrasies match the data that generated them.

Whether you are translating an established scale across cultures or developing a new multi-item instrument, psychometric validation demands an orderly, transparent progression. When preparing revisions for peer review, consulting a structured methods and results revision checklist helps ensure all sampling splits, software packages, and model parameters are fully disclosed.

  • Randomization and partition ratio: Use simple or stratified random assignment with a fixed pseudo-random seed (such as `set.seed(42)`) to allocate 40% to 50% of cases to the EFA calibration subset and the remainder to the CFA validation subset.
  • Sample size thresholds: Plan for at least 5 to 10 respondents per manifest item in the EFA calibration set (minimum `N = 150` to `200`) and at least `N = 200` to `300` cases for stable maximum likelihood estimation in CFA.
  • Complete participant independence: Ensure that individual observation identifiers never overlap between calibration and validation subsets to prevent error covariance leakage.

Sampling suitability and factor retention: why parallel analysis outperforms the Kaiser rule

Before extracting factors from the calibration dataset, researchers must verify that the correlation matrix contains adequate common variance. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy quantifies the ratio of shared variance to total variance across items, where overall values above 0.80 indicate good factorability and values below 0.60 require eliminating poorly behaving items. Simultaneously, Bartlett's test of sphericity tests the null hypothesis that the item correlation matrix equals an identity matrix; a statistically significant result (`p < 0.001`) confirms that item intercorrelations are non-zero.

Psychometric standard: Kaiser's classic eigenvalue greater than one rule (`eigenvalue > 1.0`) was derived for theoretical populations rather than sample estimates. In empirical survey data, random sampling variation inflates early eigenvalues, causing the Kaiser rule to consistently over-extract trivial factors.

To prevent over-extraction, psychometricians rely on Horn's parallel analysis (1965), available in R through `psych::fa.parallel`. Parallel analysis generates dozens of synthetic datasets containing uncorrelated random variables matching the exact sample size and variable count of the empirical dataset. By plotting the empirical eigenvalues against the 95th percentile of the simulated random distributions on a scree plot, researchers retain only those factors whose real variance surpasses pure sampling noise.

  • Step 1 — Audit sampling adequacy: Confirm that the overall `KMO > 0.80`, individual item Measures of Sampling Adequacy (MSA) exceed 0.70, and Bartlett's test achieves `p < 0.001`.
  • Step 2 — Execute Horn's parallel analysis: Generate 100 random simulated datasets in `psych` and retain only factors whose sample eigenvalues exceed the 95th percentile of simulated eigenvalues.
  • Step 3 — Inspect scree plot inflection: Verify that the retention cutoff aligns with the scree plot elbow, where the curve's descent transitions from steep drops to a flat plateau.
  • Step 4 — Verify theoretical coherence: Require that each retained factor contains a minimum of three items with high primary loadings and clear conceptual meaning.

Rotation methods and handling cross-loadings in exploratory factor analysis

Once the number of factors is established, factor rotation aligns the coordinate axes to produce simple structure, making factor loadings interpretable. In social science, psychological, and behavioral research, latent constructs virtually always correlate to some degree. Forcing orthogonality through `varimax` rotation artificially restricts factor correlations to zero, which distorts loading patterns and misrepresents underlying construct relationships. Instead, researchers should default to oblique rotation techniques such as `oblimin` (direct quartimin) or `promax`.

Rotation guideline: Always begin with an oblique rotation. If the resulting factor correlation matrix reveals that inter-factor correlations satisfy |r| < 0.20 across all factor pairs, an orthogonal solution may be reported for parsimony; otherwise, oblique rotation is methodologically mandatory.

During EFA, researchers frequently encounter cross-loading items that threaten simple structure. A problematic item displays a primary loading below 0.40, a secondary loading exceeding 0.30, or a difference between primary and secondary loadings under 0.20. When trimming such items, researchers must follow an iterative deletion protocol: delete only one item at a time and re-run both factor extraction and rotation after each step, because removing any variable alters the entire residual correlation matrix.

  • Examine rotated pattern matrix: Inspect loadings in the pattern matrix rather than the structure matrix to evaluate the unique regression relationship between each item and factor.
  • Apply cross-loading thresholds: Flag items exhibiting secondary loadings above 0.30 or primary-to-secondary loading differentials below 0.20 for potential removal.
  • Delete sequentially: In each iteration, remove only the single most problematic item—prioritizing items with severe cross-loadings or inadequate communalities (`h² < 0.30`).
  • Re-estimate the model: Recalculate KMO, re-run parallel analysis, and re-estimate rotated loadings after every deletion until all retained items load cleanly on their intended dimension.

Evaluating CFA measurement models in lavaan: fit indices and construct validity

After establishing a clean factor structure in the calibration split, the researcher specifies the measurement model in R using `lavaan::cfa` and estimates parameters on the holdout validation subsample. Reporting global model fit requires evaluating multiple fit indices simultaneously, as established by Hu & Bentler (1999). However, modern psychometric consensus emphasizes that cutoffs must not be applied blindly: fit indices vary with sample size, item reliability, and factor complexity, requiring researchers to pair global fit evaluation with inspection of localized residuals and parameter estimates.

Fit IndexFull NameReference Cutoff (Hu & Bentler, 1999)Diagnostic Role & Sensitivity Caveats
`χ²/df`Relative Chi-Square`< 3.0` (acceptable `< 5.0` in large samples)Highly sensitive to sample size when `N > 400`; statistical significance `p < 0.05` does not indicate poor fit
`CFI`Comparative Fit Index`≥ 0.95` (acceptable `≥ 0.90`)Compares target model to a null baseline model; sensitive to small average correlations among indicators
`TLI`Tucker-Lewis Index (NNFI)`≥ 0.95` (acceptable `≥ 0.90`)Rewards model parsimony; values can fall outside the `0–1` range under small samples or poor specification
`RMSEA`Root Mean Square Error of Approximation`≤ 0.06` (acceptable `≤ 0.08`; 90% CI upper bound `< 0.08`)Parsimony-adjusted absolute index; always report point estimate alongside the full 90% confidence interval
`SRMR`Standardized Root Mean Square Residual`≤ 0.08` (strict `≤ 0.05`)Measures average discrepancy between observed and model-implied covariances; sensitive to mis-specified loadings
Construct validity checkpoint: Acceptable global fit indices alone do not establish construct validity. Researchers must verify convergent validity (standardized loadings λ > 0.50, average variance extracted AVE > 0.50, composite reliability CR > 0.70) and discriminant validity via the Fornell-Larcker criterion or the heterotrait-monotrait (HTMT) ratio.

When examining model misfit via modification indices (`modindices(fit, sort = TRUE)`), researchers must avoid post-hoc correlation of residual errors (`item1 ~~ item2`) solely to inflate fit metrics. Correlating item error terms capitalizes on sample-specific noise and violates the local independence assumption of latent variable measurement. Error covariances are permissible only when justified by documented methodological artifacts, such as identical item phrasing or reverse-coding orientation.

  • Report model specification and estimator: Document the estimation method (`ML` for normal continuous indicators, `MLR` or `WLSMV` for ordinal Likert scales) and latent metric scaling constraints.
  • Present multi-index fit summary: Tabulate Chi-square, degrees of freedom, p-value, `CFI`, `TLI`, `RMSEA` with its 90% confidence interval, and `SRMR`.
  • Report standardized parameter estimates: Display standardized factor loadings (`λ`), standard errors, and z-values, ensuring all loadings are statistically significant.
  • Present convergent and discriminant validity: Provide a summary table reporting Cronbach's alpha, McDonald's omega, CR, AVE, and inter-construct correlations.

Automating split-sample EFA and CFA reporting with HumanPen Start Project

Assembling a fully reproducible psychometric workflow—partitioning datasets with fixed seeds, executing iterative EFA item screening, fitting measurement equations in `lavaan`, and generating publication-ready tables—demands hours of custom scripting. Researchers preparing scale validation chapters or journal manuscripts can orchestrate this entire analytical cycle through HumanPen Start Project. For integrating these statistical findings into your broader manuscript narrative, consult our guide on revising literature reviews without breaking citations.

Attach your data and papers, describe what you need, approve the plan before work starts, and get the reviewed report, charts, and code ready to preview online or download.

Inside HumanPen Start Project (`/projects`), researchers can upload raw survey datasets (`CSV`, `Excel`, up to 2 GB) alongside scale items and study protocols. The analysis runs inside an isolated computing sandbox executing clean R scripts (`psych`, `lavaan`) and Python data pipelines. Within your approved project plan, you specify your preferred split ratio (such as 50/50), parallel analysis extraction rules, rotation criteria, and target fit thresholds. The platform delivers formatted statistical tables, 300 dpi scree plots, and reproducible R scripts. Public example projects on the platform demonstrate how full analytical pipelines, reports, and codebases are structured and reviewed.

  • Exploratory calibration dossier (`data/efa_summary.docx`): Tabulates item descriptive statistics, KMO metrics, Bartlett's test results, parallel analysis comparison tables, and rotated pattern matrices.
  • Confirmatory validation dossier (`data/cfa_summary.docx`): Contains the `lavaan` model syntax, global fit index summary (`CFI`, `TLI`, `RMSEA`, `SRMR`), standardized factor loadings, and composite reliability calculations.
  • High-resolution scree figures (`figures/scree_plot.png`): Generates 300 dpi vector and raster scree plots contrasting empirical eigenvalues against 95th percentile simulated eigenvalues.
  • Reproducible R package (`scripts/scale_validation_pipeline.R`): Delivers fully documented, self-contained R scripts with seeded data partitions, package imports, and parameter estimation calls.

Academic integrity and reproducibility note: Deliverables provided through HumanPen Start Project serve as a transparent methodological and analytical reference scaffold. Graduate students and principal investigators remain responsible for examining raw response distributions, verifying item screening decisions, and ensuring that all data handling aligns with institutional review boards (IRB) and target journal data policies.

Frequently asked questions

What should I do if my total sample size is too small to split for EFA and CFA? When your total sample size is under 300 cases (making it impossible to partition two robust subsamples of `N ≥ 150`), do not attempt to run both EFA and CFA on the same dataset. If your scale is an established instrument adapted from prior literature, skip EFA entirely and conduct a pre-registered CFA on the full sample. If you are developing an entirely new measure, conduct only EFA on the complete sample, explicitly describe the study as an exploratory preliminary validation in your paper, and call for future independent studies to perform confirmatory testing.

Why is Horn's parallel analysis preferred over Kaiser's eigenvalue greater than one rule? Kaiser's `eigenvalue > 1` rule systematically over-extracts factors in empirical research because random sampling error generates eigenvalues exceeding 1.0 even in completely uncorrelated synthetic data. Horn's parallel analysis (1965) resolves this distortion by simulating dozens of random correlation matrices matched to the exact sample size and number of variables of your survey. By comparing empirical eigenvalues to the 95th percentile of random eigenvalues, it ensures that only factors accounting for more variance than pure sampling noise are retained.

Can I correlate error terms in lavaan to improve poor CFA model fit? Adding post-hoc error covariances (such as `item1 ~~ item2`) simply to achieve `CFI > 0.95` or `RMSEA < 0.06` represents capitalization on chance that peer reviewers routinely challenge. Error correlations violate the assumption of local independence in latent measurement models. You should only allow correlated residuals when supported by compelling methodological justification—such as identical wording, consecutive positioning, or method effects from reverse-scored items—and you must transparently explain the rationale in your methods section.

Which estimator should I select in lavaan for Likert-scale survey data? Standard maximum likelihood (`estimator = "ML"`) assumes continuous multivariate normal data, whereas survey questionnaires use ordinal Likert scales. If items display mild non-normality (skewness between `-2` and `+2`), robust maximum likelihood (`estimator = "MLR"`) with Satorra-Bentler scaled corrections is widely recommended. For scales with 4 or fewer response options, or when indicators exhibit severe skewness, diagonally weighted least squares (`estimator = "WLSMV"`) should be used to estimate parameters from polychoric correlation matrices.

Sources

KEEP READING