Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data

Bhandari, P. M.; Levis, B.; Neupane, D.; Patten, S. B.; Shrier, I.; Thombs, B. D.; Benedetti, A.; Sun, Y.; He, C.; Rice, D. B.; Krishnan, A.; Wu, Y.; Azar, M.; Sanchez, T. A.; Chiovitti, M. J.; Saadat, N.; Riehm, K. E.; Imran, M.; Negeri, Z.; Boruff, J. T.; Cuijpers, P.; Gilbody, S.; Ioannidis, J. P. A.; Kloda, L. A.; Ziegelstein, R. C.; Comeau, L.; Mitchell, N. D.; Tonelli, M.; Vigod, S. N.; Aceti, F.; Alvarado, R.; Alvarado-Esquivel, C.; Bakare, M. O.; Barnes, J.; Bavle, A. D.; Beck, C. T.; Bindt, C.; Boyce, P. M.; Bunevicius, A.; Castro e Couto, T.; Chaudron, L. H.; Correa, H.; de Figueiredo, F. P.; Eapen, V.; Favez, N.; Felice, E.; Fernandes, M.; Figueiredo, B.; Fisher, J. R. W.; Garcia-Esteve, L.; Giardinelli, L.; Helle, N.; Howard, L. M.; Khalifa, D. S.; Kohlhoff, J.; Kozinszky, Z.; Kusminskas, L.; Lelli, L.; Leonardou, A. A.; Maes, M.; Meuti, V.; Rados, S. N.; Garcia, P. N.; Nishi, D.; Luwa E-Andjafono, D. O.; Pawlby, S. J.; Quispel, C.; Robertson-Blackmore, E.; Rochat, T. J.; Rowe, H. J.; Sharp, D. J.; Siu, B. W. M.; Skalkidou, A.; Stein, A.; Stewart, R. C.; K. -P., Su; Sundstrom-Poromaa, I.; Tadinac, M.; Tandon, S. D.; Tendais, I.; Thiagayson, P.; Toreki, A.; Torres-Gimenez, A.; Tran, T. D.; Trevillion, K.; Turner, K.; Vega-Dienstmaier, J. M.; Wynter, K.; Yonkers, K. A.

doi:10.1016/j.jclinepi.2021.03.031

Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates.Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity-1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values.Results: Optimal cutoffs in simulated samples ranged from >= 5 to >= 17 for n = 100, >= 6 to >= 16 for n = 200, >= 6 to >= 14 for n = 500, and >= 8 to >= 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (>= 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000.Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods. (C) 2021 Elsevier Inc. All rights reserved.

Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data / Bhandari, P. M.; Levis, B.; Neupane, D.; Patten, S. B.; Shrier, I.; Thombs, B. D.; Benedetti, A.; Sun, Y.; He, C.; Rice, D. B.; Krishnan, A.; Wu, Y.; Azar, M.; Sanchez, T. A.; Chiovitti, M. J.; Saadat, N.; Riehm, K. E.; Imran, M.; Negeri, Z.; Boruff, J. T.; Cuijpers, P.; Gilbody, S.; Ioannidis, J. P. A.; Kloda, L. A.; Ziegelstein, R. C.; Comeau, L.; Mitchell, N. D.; Tonelli, M.; Vigod, S. N.; Aceti, F.; Alvarado, R.; Alvarado-Esquivel, C.; Bakare, M. O.; Barnes, J.; Bavle, A. D.; Beck, C. T.; Bindt, C.; Boyce, P. M.; Bunevicius, A.; Castro e Couto, T.; Chaudron, L. H.; Correa, H.; de Figueiredo, F. P.; Eapen, V.; Favez, N.; Felice, E.; Fernandes, M.; Figueiredo, B.; Fisher, J. R. W.; Garcia-Esteve, L.; Giardinelli, L.; Helle, N.; Howard, L. M.; Khalifa, D. S.; Kohlhoff, J.; Kozinszky, Z.; Kusminskas, L.; Lelli, L.; Leonardou, A. A.; Maes, M.; Meuti, V.; Rados, S. N.; Garcia, P. N.; Nishi, D.; Luwa E-Andjafono, D. O.; Pawlby, S. J.; Quispel, C.; Robertson-Blackmore, E.; Rochat, T. J.; Rowe, H. J.; Sharp, D. J.; Siu, B. W. M.; Skalkidou, A.; Stein, A.; Stewart, R. C.; Su, K. -P.; Sundstrom-Poromaa, I.; Tadinac, M.; Tandon, S. D.; Tendais, I.; Thiagayson, P.; Toreki, A.; Torres-Gimenez, A.; Tran, T. D.; Trevillion, K.; Turner, K.; Vega-Dienstmaier, J. M.; Wynter, K.; Yonkers, K. A.. - In: JOURNAL OF CLINICAL EPIDEMIOLOGY. - ISSN 0895-4356. - 137:(2021), pp. 137-147. [10.1016/j.jclinepi.2021.03.031]