Omissions by Design in a Survey: Is This a Good Choice when using Structural Equation Models?

Paula  C. R. Vicente

doi:10.2478/ngoe-2024-0018

Paula C. R. Vicente

DOI: https://doi.org/10.2478/ngoe-2024-0018

Abstract

Missing observations can arise due to the effort required to answer many questions in long surveys and the cost required to obtain some responses. Implementing a planned missing design in surveys helps reduce the number of questions each respondent needs to answer, thereby lowering survey fatigue and cutting down on implementation costs. The three-form and the two-method design are two different types of planned missing designs. An important consideration when designing a study with omissions by design is to know how it will affect statistical results. In this work, a simulation study is conducted to analyze how the usual fit measures, root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker-Lewis index (TLI) perform in the adjustment of a Structural Equation Model. The results revealed that the CFI, TLI, and SRMR indices exhibit sensitivity to omissions with small samples, low factor loadings and large models. Overall, this study contributes to our understanding of the importance of considering omissions by design in market research.

Downloads

Download data is not yet available.

References

Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. In Marcoulides, G. A., & Schumacker, R. E. (eds.), Advanced structural equation modelling, 243–277. Erlbaum.

Azar, B. (2002). Finding a solution for missing data. Monitor on Psychology, 33,70.

Bandalous, D. L., & Finney, S. J. (2010). Factor analysis: Exploratory and confirmatory. In Hancock, G. R. & Mueller, R. O. (eds.), The reviewer’s guide to quantitative methods in the social sciences. New York, NY: Routledge. DOI: https://doi.org/10.4324/9780203861554-15

Bentler, P. M. (1990). Comparative Fit Indexes in Structural Equation Models. Psychological Bulletin, 107, 238–246. DOI: https://doi.org/10.1037/0033-2909.107.2.238

Bollen, K. (1989). Structural Equations with Latent Variables. New York, USA: Wile

Cangur, S., & Ercan, I. (2015). Comparison of model Fit Indices Used in Structural Equation Modeling Under Multivariate Normality, 14 (1), 152–167. DOI: https://doi.org/10.22237/jmasm/1430453580

Davey, A. (2005). Issues in evaluating model fit with missing data. Struct. Equ. Model. Multidiscip. J., 12, 578–597. DOI: https://doi.org/10.1207/s15328007sem1204_4

Enders, C. K. (2010). Applied Missing Data. New York, USA: The Guilford Press.

Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods and model specification on structural equation modelling fit indexes. Struct. Equ. Model. Multidiscip. J., 6, 56–83. DOI: https://doi.org/10.1080/10705519909540119

Fürst, G. (2020). Measuring creativity with planned missing data. The Journal of Creative Behavior, 54(1), 150–164. DOI: https://doi.org/10.1002/jocb.352

Garnier-Vilarreal, M., Rhemtulla, M., & Little, T.D. (2014). Two-method planned missing designs for longitudinal research. International Journal of Behavioral Development, 38(5), 411–422. DOI: https://doi.org/10.1177/0165025414542711

Graham, J., Taylor, B., Olchowski, A., & Cumsville, P. (2006). Planned missing data designs in psychological research. Psychological Methods, 11, 323–343. DOI: https://doi.org/10.1037/1082-989x.11.4.323

Graham,J., Hofer,S. & Mackinnon,D. (1996). Maximizing the usefulness of data obtained with planned missing value patterns: An application of maximum likelihood procedures. Multivariate Behavioral Research, 31, 197–218.

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis (6th ed). Columbus, Oh: Pearson.

Hancock, G. R. & Mueller, R. O. (2011). The reliability paradox in assessing structural relations within covariance structure models. Educ. Psychol. Meas., 71, 306–324. DOI: https://doi.org/10.1177/0013164410384856

Hoyle, R. H. (2012). Handbook of Structural equation Modelling. New York, USA: Guilford Press.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. DOI: https://doi.org/10.1080/10705519909540118

Jia, F., Moore, E. W. G., Kinai, R. Crowe, K. S., Schoemann, A. M., & Little, T. D. (2014). Planned missing data designs with small sample sizes: How small is too small? Int. J. Behav. Dev., 38, 435–452. DOI: https://doi.org/10.1177/0165025414531095

Kaplan, D. (2000). Structural Equation Modeling – Foundations and Extensions. Sage Publications. DOI: https://doi.org/10.4135/9781452226576

Kenny, D. A., Kanishan, B. & McCoach, D. B. (2015). The performance of RMSEA in models with small degrees of freedom. Sociological Methods & Research, 44, 486–507. DOI: https://doi.org/10.1177/0049124114543236

Kenny, D. A. & McCoach, D. B. (2003). Effect of the number of variables on measures of fit in structural equation modelling. Structural Equation Modeling, 10, 333–351. DOI: https://doi.org/10.1177/0049124114543236

Kline, R. B. (2010). Principles and Practice of Structural Equation Modeling (3rd ed.). Guilford Press, New York.

Lang, K. M., Moore, E. W. G., & Grandfield, E. M. (2020). A novel item-allocation procedure for the three-form planned missing design. MethodsX, 7, 100941. DOI: https://doi.org/10.1016/j.mex.2020.100941

Lawes, M., Schultze, M., & Eid, M. J. A. (2020). Making the most of your research budget: Efficiency of a three-method measurement design with planned missing data. Assessment, 27(5), 903–920. DOI: https://doi.org/10.1177/1073191118798050

Little, T. D., Jorgensen, T. D., Lang, K. M., & Moore, W. G. (2013). On the Joys of Missing Data. Journal of Pediatric Psychology, 39(2), 151–162. DOI: https://doi.org/10.1093/jpepsy/jst048

McNeish, D., & Wolf, M. G. (2021). Dynamic Fit Index Cutoffs for Confirmatory Factor Analysis Models. Pshycological Methods. DOI: https://doi.org/10.1080/00223891.2017.1281286

Moore, E. W. G., Lang, K. M., & Grandfield, E. M. (2020). Maximizing data quality and shortening survey time: Three-form planned missing data survey design. Psychology of sport & Exercise, 51, 1–12. DOI: https://doi.org/10.1016/j.psychsport.2020.101701

Moshagen, M. (2012). The model size effect in SEM: Inflated goodness of fit statistics are due to the size of the covariance matrix. Struct. Equ. Model. Multidiscip. J., 19, 86–98. DOI: https://doi.org/10.1080/10705511.2012.634724

Nye, C. D. (2022). Reviewer Resources: Confirmatory Factor Analysis. Organ. Res. Methods. Online publishing. DOI: https://doi.org/10.1177/10944281221120541

Pornprasertmanit, S., Miller, P., Schoemann, A., & Jorgensen, T. D. (2021). simsem: SIMulated Structural Equation Modeling (version 0.5-16) [R package]. Retrieved from http://simsem.org/

Rhemtulla, M., & Hancock, G. R. (2016). Planned Missing Data Designs in Educational Psychology Research. Educational Psychologist, 51(3-4), 305–316.

Rioux, C., Lewin, A., Odejimi, O. A., & Little, T. D. (2020). International Journal of Epidemiology, 1702–1711. DOI: https://doi.org/10.1093/ije/dyaa042

Rubin, D. (1976). Inference and missing data. Biometrika, 63(3), 581–592. DOI: https://doi.org/10.1093/biomet/63.3.581

Schoemann, A. M., Miller, P., Pornprasertmanit, S, & Wu, W. (2014). Using Monte Carlo simulations to determine power and sample size for planned missing designs. International Journal of Behavioural Development, 38(5), 471–479. DOI: https://doi.org/10.1177/0165025413515169

Shi, D., Lee, T., & Maydeu-Olivares, A. (2019). Understanding the Model Size Effect on SEM Fit Indices. Educational and Psychological Measurement, 79(2), 310–334. DOI: https://doi.org/10.1177/0013164418783530

Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173–180. DOI: https://doi.org/10.1207/s15327906mbr2502_4

Tucker, L. R., & Lewis, C. (1973). The reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. DOI: https://doi.org/10.1007/bf02291170

Vicente, P. C. R. (2023). Evaluating the Effect of Planned Missing Designs in Structural Equation Model Fit Measures. Psych, 5, 983–995. DOI: https://doi.org/10.3390/psych5030064

Wu, W., & Jia, F. (2021). Applying planned missingness designs to longitudinal panel studies in developmental science: An overview. Child & Adolescent Development, 2021 (175), 35–63. DOI: https://doi.org/10.1002/cad.20391

Yuan, K.-H. (2005). Fit indices versus test statistics. Multivariate Behavioural Research, 40(1), 115–148. DOI: https://doi.org/10.1207/s15327906mbr4001_5

Zhang, X., & Savalei, V. (2023). New computations for RMSEA and CFI following FIML and TS estimation with missing data. Psychol. Methods, 28, 263–283. DOI: https://doi.org/10.1080/10705511.2019.1642111