Assessing the practical differences between model selection methods in inferences about choice response time tasks.

Authors:
Nathan J Evans
Nathan J Evans
University of Wisconsin

Psychon Bull Rev 2019 Feb 19. Epub 2019 Feb 19.

Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.

Evidence accumulations models (EAMs) have become the dominant modeling framework within rapid decision-making, using choice response time distributions to make inferences about the underlying decision process. These models are often applied to empirical data as "measurement tools", with different theoretical accounts being contrasted within the framework of the model. Some method is then needed to decide between these competing theoretical accounts, as only assessing the models on their ability to fit trends in the empirical data ignores model flexibility, and therefore, creates a bias towards more flexible models. However, there is no objectively optimal method to select between models, with methods varying in both their computational tractability and theoretical basis. I provide a systematic comparison between nine different model selection methods using a popular EAM-the linear ballistic accumulator (LBA; Brown & Heathcote, Cognitive Psychology 57(3), 153-178 2008)-in a large-scale simulation study and the empirical data of Dutilh et al. (Psychonomic Bulletin and Review, 1-19 2018). I find that the "predictive accuracy" class of methods (i.e., the Akaike Information Criterion [AIC], the Deviance Information Criterion [DIC], and the Widely Applicable Information Criterion [WAIC]) make different inferences to the "Bayes factor" class of methods (i.e., the Bayesian Information Criterion [BIC], and Bayes factors) in many, but not all, instances, and that the simpler methods (i.e., AIC and BIC) make inferences that are highly consistent with their more complex counterparts. These findings suggest that researchers should be able to use simpler "parameter counting" methods when applying the LBA and be confident in their inferences, but that researchers need to carefully consider and justify the general class of model selection method that they use, as different classes of methods often result in different inferences.

Download full-text PDF

Source
http://dx.doi.org/10.3758/s13423-018-01563-9DOI Listing
February 2019

Publication Analysis

Top Keywords

model selection
12
empirical data
12
selection methods
8
methods
8
theoretical accounts
8
response time
8
class methods
8
choice response
8
inferences
6
model
5
models
5
dutilh psychonomic
4
psychonomic bulletin
4
bulletin review
4
data dutilh
4
simulation study
4
2008-in large-scale
4
large-scale simulation
4
review 1-19
4
study empirical
4

References

(Supplied by CrossRef)
Article in IEEE Transactions on Automatic Control
H Akaike et al.
IEEE Transactions on Automatic Control 1974

GE Box et al.
1987
Article in Monthey Weather Review
GW Brier et al.
Monthey Weather Review 1950
Article in Cognitive Psychology
SD Brown et al.
Cognitive Psychology 2008
Article in Psychological Review
SD Brown et al.
Psychological Review 2008
Article in Behavior Research Methods
C Donkin et al.
Behavior Research Methods 2009
Article in Psychonomic Bulletin and Review
C Donkin et al.
Psychonomic Bulletin and Review 2009
Article in Psychonomic Bulletin and Review
G Dutilh et al.
Psychonomic Bulletin and Review 2009

Similar Publications