Statistical Uncertainty Analysis for Small-Sample, High Log-Variance Data: Cautions for Bootstrapping and Bayesian Bootstrapping.

Authors:
Barmak Mostofian
Barmak Mostofian
UT/ORNL Center for Molecular Biophysics
Daniel M Zuckerman
Daniel M Zuckerman
University of Pittsburgh
Pittsburgh | United States

J Chem Theory Comput 2019 May 7. Epub 2019 May 7.

Department of Biomedical Engineering, School of Medicine , Oregon Health & Science University , Portland , Oregon 97239-3098 , United States.

Recent advances in molecular simulations allow the evaluation of previously unattainable observables, such as rate constants for protein folding. However, these calculations are usually computationally expensive, and even significant computing resources may result in a small number of independent estimates spread over many orders of magnitude. Such small-sample, high "log-variance" data are not readily amenable to analysis using the standard uncertainty (i.e., "standard error of the mean") because unphysical negative limits of confidence intervals result. Bootstrapping, a natural alternative guaranteed to yield a confidence interval within the minimum and maximum values, also exhibits a striking systematic bias of the lower confidence limit in log space. As we show, bootstrapping artifactually assigns high probability to improbably low mean values. A second alternative, the Bayesian bootstrap strategy, does not suffer from the same deficit and is more logically consistent with the type of confidence interval desired. The Bayesian bootstrap provides uncertainty intervals that are more reliable than those from the standard bootstrap method but must be used with caution nevertheless. Neither standard nor Bayesian bootstrapping can overcome the intrinsic challenge of underestimating the mean from small-size, high log-variance samples. Our conclusions are based on extensive analysis of model distributions and reanalysis of multiple independent atomistic simulations. Although we only analyze rate constants, similar considerations will apply to related calculations, potentially including highly nonlinear averages like the Jarzynski relation.

Download full-text PDF

Source
http://dx.doi.org/10.1021/acs.jctc.9b00015DOI Listing
May 2019
1 Read

Publication Analysis

Top Keywords

high log-variance
8
bayesian bootstrapping
8
small-sample high
8
confidence interval
8
bayesian bootstrap
8
rate constants
8
bootstrapping
5
analysis model
4
values exhibits
4
maximum values
4
model distributions
4
interval minimum
4
minimum maximum
4
striking systematic
4
conclusions based
4
confidence limit
4
limit log
4
lower confidence
4
bias lower
4
extensive analysis
4

Similar Publications