Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models.

Med Decis Making 2017 10 11;37(7):790-801. Epub 2017 Apr 11.

Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).

Background: Risk prediction models allow for the incorporation of individualized risk and clinical effectiveness information to identify patients for whom therapy is most appropriate and cost-effective. This approach has the potential to identify inefficient (or harmful) care in subgroups at different risks, even when the overall results appear favorable. Here, we explore the value of personalized risk information and the factors that influence it.

Methods: Using an expected value of individualized care (EVIC) framework, which monetizes the value of customizing care, we developed a general approach to calculate individualized incremental cost effectiveness ratios (ICERs) as a function of individual outcome risk. For a case study (tPA v. streptokinase to treat possible myocardial infarction), we used a simulation to explore how an EVIC is influenced by population outcome prevalence, model discrimination (c-statistic) and calibration, and willingness-to-pay (WTP) thresholds.

Results: In our simulations, for well-calibrated models, which do not over- or underestimate predicted v. observed event risk, the EVIC ranged from $0 to $700 per person, with better discrimination (higher c-statistic values) yielding progressively higher EVIC values. For miscalibrated models, the EVIC ranged from -$600 to $600 in different simulated scenarios. The EVIC values decreased as discrimination improved from a c-statistic of 0.5 to 0.6, before becoming positive as the c-statistic reached values of ~0.8.

Conclusions: Individualizing treatment decisions using risk may produce substantial value but also has the potential for net harm. Good model calibration ensures a non-negative EVIC. Improvements in discrimination generally increase the EVIC; however, when models are miscalibrated, greater discriminating power can paradoxically reduce the EVIC under some circumstances.

Download full-text PDF

Source
http://dx.doi.org/10.1177/0272989X17704855DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5577366PMC
October 2017
21 Reads

Publication Analysis

Top Keywords

evic
9
evic values
8
prediction models
8
evic ranged
8
risk
6
models
5
discrimination
5
explore evic
4
decisions risk
4
simulation explore
4
streptokinase treat
4
treat myocardial
4
risk produce
4
myocardial infarction
4
infarction simulation
4
evic influenced
4
c-statistic
4
discrimination c-statistic
4
c-statistic calibration
4
individualizing treatment
4

Similar Publications