Correcting the predictive validity of a selection test for the effect of indirect range restriction.

Authors:
Stefan Zimmermann
Stefan Zimmermann
University of Heidelberg
Germany
Dietrich Klusmann
Dietrich Klusmann
Universit├Ątsklinikum Hamburg-Eppendorf
Germany
Wolfgang Hampe
Wolfgang Hampe
University Medical Center Hamburg-Eppendorf
Germany

BMC Med Educ 2017 Dec 11;17(1):246. Epub 2017 Dec 11.

Department of Biochemistry and Molecular Cell Biology, University Medical Center Hamburg-Eppendorf, Martinistra├če 52, D-20246, Hamburg, Germany.

Background: The validity of selection tests is underestimated if it is determined by simply calculating the predictor-outcome correlation found in the admitted group. This correlation is usually attenuated by two factors: (1) the combination of selection variables which can compensate for each other and (2) range restriction in predictor and outcome due to the absence of outcome measures for rejected applicants.

Methods: Here we demonstrate the logic of these artifacts in a situation typical for student selection tests and compare four different methods for their correction: two formulas for the correction of direct and indirect range restriction, expectation maximization algorithm (EM) and multiple imputation by chained equations (MICE). First we show with simulated data how a realistic estimation of predictive validity could be achieved; second we apply the same methods to empirical data from one medical school.

Results: The results of the four methods are very similar except for the direct range restriction formula which underestimated validity.

Conclusion: For practical purposes Thorndike's case C formula is a relatively straightforward solution to the range restriction problem, provided distributional assumptions are met. With EM and MICE more precision is obtained when distributional requirements are not met, but access to a sophisticated statistical package such as R is needed. The use of true score correlation has its own problems and does not seem to provide a better correction than other methods.

Download full-text PDF

Source
http://dx.doi.org/10.1186/s12909-017-1070-5DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5725878PMC
December 2017
3 Reads

Publication Analysis

Top Keywords

range restriction
20
selection tests
8
validity selection
8
indirect range
8
predictive validity
8
restriction
5
range
5
restriction formula
4
second apply
4
achieved second
4
validity achieved
4
data realistic
4
realistic estimation
4
estimation predictive
4
apply methods
4
direct range
4
medical schoolresults
4
schoolresults methods
4
data medical
4
empirical data
4

Similar Publications

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies.

BMC Med 2013 Nov 14;11:243. Epub 2013 Nov 14.

UCL Medical School, University College London, Gower Street, London WC1E 6BT, UK.

Background: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities. Read More

View Article
November 2013

A Missing Data Approach to Correct for Direct and Indirect Range Restrictions with a Dichotomous Criterion: A Simulation Study.

PLoS One 2016 28;11(3):e0152330. Epub 2016 Mar 28.

Department of Applied Psychology: Work, Education, Economy, Faculty of Psychology, University of Vienna, Vienna, Austria.

A recurring methodological problem in the evaluation of the predictive validity of selection methods is that the values of the criterion variable are available for selected applicants only. This so-called range restriction problem causes biased population estimates. Correction methods for direct and indirect range restriction scenarios have widely studied for continuous criterion variables but not for dichotomous ones. Read More

View Article
August 2016

The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools.

BMC Med 2013 Nov 14;11:244. Epub 2013 Nov 14.

UCL Medical School, University College London, Gower Street, London WC1E 6BT, UK.

Background: Most UK medical schools use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the United Kingdom Clinical Aptitude Test (UKCAT), and its four sub-scales, along with measures of educational attainment, individual and contextual socio-economic background factors, as predictors of performance in the first year of medical school training.

Methods: A prospective study of 4,811 students in 12 UK medical schools taking the UKCAT from 2006 to 2008 as a part of the medical school application, for whom first year medical school examination results were available in 2008 to 2010. Read More

View Article
November 2013

Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study.

BMC Med Educ 2012 Aug 8;12:69. Epub 2012 Aug 8.

Hull York Medical School, University of York, York YO10 5DD, UK.

Background: Over two-thirds of UK medical schools are augmenting their selection procedures for medical students by using the United Kingdom Clinical Aptitude Test (UKCAT), which employs tests of cognitive and non-cognitive personal qualities, but clear evidence of the tests' predictive validity is lacking. This study explores whether academic performance and professional behaviours that are important in a health professional context can be predicted by these measures, when taken before or very early in the medical course.

Methods: This prospective cohort study follows the progress of the entire student cohort who entered Hull York Medical School in September 2007, having taken the UKCAT cognitive tests in 2006 and the non-cognitive tests a year later. Read More

View Article
August 2012