Publications by authors named "Richard D Riley"

163 Publications

Prognostic models for predicting relapse or recurrence of major depressive disorder in adults.

Cochrane Database Syst Rev 2021 May 6;5:CD013491. Epub 2021 May 6.

Mental Health and Addiction Research Group, Department of Health Sciences, University of York, York, UK.

Background: Relapse (the re-emergence of depressive symptoms after some level of improvement but preceding recovery) and recurrence (onset of a new depressive episode after recovery) are common in depression, lead to worse outcomes and quality of life for patients and exert a high economic cost on society. Outcomes can be predicted by using multivariable prognostic models, which use information about several predictors to produce an individualised risk estimate. The ability to accurately predict relapse or recurrence while patients are well (in remission) would allow the identification of high-risk individuals and may improve overall treatment outcomes for patients by enabling more efficient allocation of interventions to prevent relapse and recurrence.

Objectives: To summarise the predictive performance of prognostic models developed to predict the risk of relapse, recurrence, sustained remission or recovery in adults with major depressive disorder who meet criteria for remission or recovery.

Search Methods: We searched the Cochrane Library (current issue); Ovid MEDLINE (1946 onwards); Ovid Embase (1980 onwards); Ovid PsycINFO (1806 onwards); and Web of Science (1900 onwards) up to May 2020. We also searched sources of grey literature, screened the reference lists of included studies and performed a forward citation search. There were no restrictions applied to the searches by date, language or publication status .

Selection Criteria: We included development and external validation (testing model performance in data separate from the development data) studies of any multivariable prognostic models (including two or more predictors) to predict relapse, recurrence, sustained remission, or recovery in adults (aged 18 years and over) with remitted depression, in any clinical setting. We included all study designs and accepted all definitions of relapse, recurrence and other related outcomes. We did not specify a comparator prognostic model.

Data Collection And Analysis: Two review authors independently screened references; extracted data (using a template based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS)); and assessed risks of bias of included studies (using the Prediction model Risk Of Bias ASsessment Tool (PROBAST)). We referred any disagreements to a third independent review author. Where we found sufficient (10 or more) external validation studies of an individual model, we planned to perform a meta-analysis of its predictive performance, specifically with respect to its calibration (how well the predicted probabilities match the observed proportions of individuals that experience the outcome) and discrimination (the ability of the model to differentiate between those with and without the outcome). Recommendations could not be qualified using the GRADE system, as guidance is not yet available for prognostic model reviews.

Main Results: We identified 11 eligible prognostic model studies (10 unique prognostic models). Seven were model development studies; three were model development and external validation studies; and one was an external validation-only study. Multiple estimates of performance measures were not available for any of the models and, meta-analysis was therefore not possible. Ten out of the 11 included studies were assessed as being at high overall risk of bias. Common weaknesses included insufficient sample size, inappropriate handling of missing data and lack of information about discrimination and calibration. One paper (Klein 2018) was at low overall risk of bias and presented a prognostic model including the following predictors: number of previous depressive episodes, residual depressive symptoms and severity of the last depressive episode. The external predictive performance of this model was poor (C-statistic 0.59; calibration slope 0.56; confidence intervals not reported). None of the identified studies examined the clinical utility (net benefit) of the developed model.

Authors' Conclusions: Of the 10 prognostic models identified (across 11 studies), only four underwent external validation. Most of the studies (n = 10) were assessed as being at high overall risk of bias, and the one study that was at low risk of bias presented a model with poor predictive performance. There is a need for improved prognostic research in this clinical area, with future studies conforming to current best practice recommendations for prognostic model development/validation and reporting findings in line with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/14651858.CD013491.pub2DOI Listing
May 2021

Developing more generalizable prediction models from pooled studies and large clustered data sets.

Stat Med 2021 May 5. Epub 2021 May 5.

Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor-outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal-external cross-validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold-out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta-analysis of calibration and discrimination performance in each hold-out cluster shows that trade-offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8981DOI Listing
May 2021

Clinical prediction models: diagnosis versus prognosis.

J Clin Epidemiol 2021 Apr;132:142-145

Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands.

Clinical prediction models play an increasingly important role in contemporary clinical care, by informing healthcare professionals, patients and their relatives about outcome risks, with the aim to facilitate (shared) medical decision making and improve health outcomes. Diagnostic prediction models aim to calculate an individual's risk that a disease is already present, whilst prognostic prediction models aim to calculate the risk of particular heath states occurring in the future. This article serves as a primer for diagnostic and prognostic clinical prediction models, by discussing the basic terminology, some of the inherent challenges, and the need for validation of predictive performance and the evaluation of impact of these models in clinical care.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.jclinepi.2021.01.009DOI Listing
April 2021

Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model.

Stat Med 2021 Mar 26. Epub 2021 Mar 26.

Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK.

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8959DOI Listing
March 2021

Authors' reply to Sabour and Ghajari "Clinical prediction models to predict the risk of multiple binary outcomes: Methodological issues".

Stat Med 2021 Mar;40(7):1861-1862

Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK.

View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8872DOI Listing
March 2021

External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb.

J Clin Epidemiol 2021 Feb 14;135:79-89. Epub 2021 Feb 14.

Centre for Prognosis Research, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom.

Introduction: Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.

Methods: Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.

Results: Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.

Conclusion: Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.jclinepi.2021.02.011DOI Listing
February 2021

Community-based complex interventions to sustain independence in older people, stratified by frailty: a protocol for a systematic review and network meta-analysis.

BMJ Open 2021 02 15;11(2):e045637. Epub 2021 Feb 15.

School of Medicine, University of Nottingham, Nottingham, UK.

Introduction: Maintaining independence is a primary goal of community health and care services for older people, but there is currently insufficient guidance about which services to implement. Therefore, we aim to synthesise evidence on the effectiveness of community-based complex interventions to sustain independence for older people, including the effect of frailty, and group interventions to identify the best configurations.

Methods And Analysis: Systematic review and network meta-analysis (NMA). We will include randomised controlled trials (RCTs) and cluster RCTs of community-based complex interventions to sustain independence for older people living at home (mean age ≥65 years), compared with usual care or another complex intervention. We will search MEDLINE (1946 to September 2020), Embase (1947 to September 2020), CINAHL (1981 to September 2020), PsycINFO (1806 to September 2020), CENTRAL and clinical trial registries from inception to September 2020, without date/language restrictions, and scan included papers' reference lists. Main outcomes were: living at home, activities of daily living (basic/instrumental), home-care services usage, hospitalisation, care home admission, costs and cost effectiveness. Additional outcomes were: health status, depression, loneliness, falls and mortality. Interventions will be coded, summarised and grouped. An NMA using a multivariate random-effects model for each outcome separately will determine the relative effects of different complex interventions. For each outcome, we will produce summary effect estimates for each pair of treatments in the network, with 95% CI, ranking plots and measures, and the borrowing of strength statistic. Inconsistency will be examined using a 'design-by-treatment interaction' model. We will assess risk of bias (Cochrane tool V.2) and certainty of evidence using the Grading of Recommendations Assessment, Development and Evaluation for NMA approach.

Ethics And Dissemination: This research will use aggregated, anonymised, published data. Findings will be reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidance. They will be disseminated to policy-makers, commissioners and providers, and via conferences and scientific journals.

Prospero Registration Number: CRD42019162195.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1136/bmjopen-2020-045637DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7887376PMC
February 2021

Association between antihypertensive treatment and adverse events: systematic review and meta-analysis.

BMJ 2021 02 10;372:n189. Epub 2021 Feb 10.

Nuffield Department of Primary Care Health Sciences, Radcliffe Primary Care Building, University of Oxford, Oxford, OX2 6GG, UK

Objective: To examine the association between antihypertensive treatment and specific adverse events.

Design: Systematic review and meta-analysis.

Eligibility Criteria: Randomised controlled trials of adults receiving antihypertensives compared with placebo or no treatment, more antihypertensive drugs compared with fewer antihypertensive drugs, or higher blood pressure targets compared with lower targets. To avoid small early phase trials, studies were required to have at least 650 patient years of follow-up.

Information Sources: Searches were conducted in Embase, Medline, CENTRAL, and the Science Citation Index databases from inception until 14 April 2020.

Main Outcome Measures: The primary outcome was falls during trial follow-up. Secondary outcomes were acute kidney injury, fractures, gout, hyperkalaemia, hypokalaemia, hypotension, and syncope. Additional outcomes related to death and major cardiovascular events were extracted. Risk of bias was assessed using the Cochrane risk of bias tool, and random effects meta-analysis was used to pool rate ratios, odds ratios, and hazard ratios across studies, allowing for between study heterogeneity (τ).

Results: Of 15 023 articles screened for inclusion, 58 randomised controlled trials were identified, including 280 638 participants followed up for a median of 3 (interquartile range 2-4) years. Most of the trials (n=40, 69%) had a low risk of bias. Among seven trials reporting data for falls, no evidence was found of an association with antihypertensive treatment (summary risk ratio 1.05, 95% confidence interval 0.89 to 1.24, τ=0.009). Antihypertensives were associated with an increased risk of acute kidney injury (1.18, 95% confidence interval 1.01 to 1.39, τ=0.037, n=15), hyperkalaemia (1.89, 1.56 to 2.30, τ=0.122, n=26), hypotension (1.97, 1.67 to 2.32, τ=0.132, n=35), and syncope (1.28, 1.03 to 1.59, τ=0.050, n=16). The heterogeneity between studies assessing acute kidney injury and hyperkalaemia events was reduced when focusing on drugs that affect the renin angiotensin-aldosterone system. Results were robust to sensitivity analyses focusing on adverse events leading to withdrawal from each trial. Antihypertensive treatment was associated with a reduced risk of all cause mortality, cardiovascular death, and stroke, but not of myocardial infarction.

Conclusions: This meta-analysis found no evidence to suggest that antihypertensive treatment is associated with falls but found evidence of an association with mild (hyperkalaemia, hypotension) and severe adverse events (acute kidney injury, syncope). These data could be used to inform shared decision making between doctors and patients about initiation and continuation of antihypertensive treatment, especially in patients at high risk of harm because of previous adverse events or poor renal function.

Registration: PROSPERO CRD42018116860.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1136/bmj.n189DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7873715PMC
February 2021

Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Diagn Progn Res 2021 Jan 11;5(1). Epub 2021 Jan 11.

Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.

Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, "living" (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1186/s41512-020-00090-3DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7797885PMC
January 2021

Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient.

J Clin Epidemiol 2020 Dec 28;133:53-60. Epub 2020 Dec 28.

Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK. Electronic address:

Objective: Developing clinical prediction models (CPMs) on data of sufficient sample size is critical to help minimize overfitting. Using prostate cancer as a clinical exemplar, we aimed to investigate to what extent existing CPMs adhere to recent formal sample size criteria, or historic rules of thumb of events per predictor parameter (EPP)≥10.

Study Design And Setting: A systematic review to identify CPMs related to prostate cancer, which provided enough information to calculate minimum sample size. We compared the reported sample size of each CPM against the traditional 10 EPP rule of thumb and formal sample size criteria.

Results: About 211 CPMs were included. Three of the studies justified the sample size used, mostly using EPP rules of thumb. Overall, 69% of the CPMs were derived on sample sizes that surpassed the traditional EPP≥10 rule of thumb, but only 48% surpassed recent formal sample size criteria. For most CPMs, the required sample size based on formal criteria was higher than the sample sizes to surpass 10 EPP.

Conclusion: Few of the CPMs included in this study justified their sample size, with most justifications being based on EPP. This study shows that, in real-world data sets, adhering to the classic EPP rules of thumb is insufficient to adhere to recent formal sample size criteria.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.jclinepi.2020.12.011DOI Listing
December 2020

Validation and development of models using clinical, biochemical and ultrasound markers for predicting pre-eclampsia: an individual participant data meta-analysis.

Health Technol Assess 2020 12;24(72):1-252

Background: Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk is needed to plan management.

Objectives: To assess the performance of existing pre-eclampsia prediction models and to develop and validate models for pre-eclampsia using individual participant data meta-analysis. We also estimated the prognostic value of individual markers.

Design: This was an individual participant data meta-analysis of cohort studies.

Setting: Source data from secondary and tertiary care.

Predictors: We identified predictors from systematic reviews, and prioritised for importance in an international survey.

Primary Outcomes: Early-onset (delivery at < 34 weeks' gestation), late-onset (delivery at ≥ 34 weeks' gestation) and any-onset pre-eclampsia.

Analysis: We externally validated existing prediction models in UK cohorts and reported their performance in terms of discrimination and calibration. We developed and validated 12 new models based on clinical characteristics, clinical characteristics and biochemical markers, and clinical characteristics and ultrasound markers in the first and second trimesters. We summarised the data set-specific performance of each model using a random-effects meta-analysis. Discrimination was considered promising for -statistics of ≥ 0.7, and calibration was considered good if the slope was near 1 and calibration-in-the-large was near 0. Heterogeneity was quantified using and τ. A decision curve analysis was undertaken to determine the clinical utility (net benefit) of the models. We reported the unadjusted prognostic value of individual predictors for pre-eclampsia as odds ratios with 95% confidence and prediction intervals.

Results: The International Prediction of Pregnancy Complications network comprised 78 studies (3,570,993 singleton pregnancies) identified from systematic reviews of tests to predict pre-eclampsia. Twenty-four of the 131 published prediction models could be validated in 11 UK cohorts. Summary -statistics were between 0.6 and 0.7 for most models, and calibration was generally poor owing to large between-study heterogeneity, suggesting model overfitting. The clinical utility of the models varied between showing net harm to showing minimal or no net benefit. The average discrimination for IPPIC models ranged between 0.68 and 0.83. This was highest for the second-trimester clinical characteristics and biochemical markers model to predict early-onset pre-eclampsia, and lowest for the first-trimester clinical characteristics models to predict any pre-eclampsia. Calibration performance was heterogeneous across studies. Net benefit was observed for International Prediction of Pregnancy Complications first and second-trimester clinical characteristics and clinical characteristics and biochemical markers models predicting any pre-eclampsia, when validated in singleton nulliparous women managed in the UK NHS. History of hypertension, parity, smoking, mode of conception, placental growth factor and uterine artery pulsatility index had the strongest unadjusted associations with pre-eclampsia.

Limitations: Variations in study population characteristics, type of predictors reported, too few events in some validation cohorts and the type of measurements contributed to heterogeneity in performance of the International Prediction of Pregnancy Complications models. Some published models were not validated because model predictors were unavailable in the individual participant data.

Conclusion: For models that could be validated, predictive performance was generally poor across data sets. Although the International Prediction of Pregnancy Complications models show good predictive performance on average, and in the singleton nulliparous population, heterogeneity in calibration performance is likely across settings.

Future Work: Recalibration of model parameters within populations may improve calibration performance. Additional strong predictors need to be identified to improve model performance and consistency. Validation, including examination of calibration heterogeneity, is required for the models we could not validate.

Study Registration: This study is registered as PROSPERO CRD42015029349.

Funding: This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in ; Vol. 24, No. 72. See the NIHR Journals Library website for further project information.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.3310/hta24720DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7780127PMC
December 2020

Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small.

J Clin Epidemiol 2021 Apr 8;132:88-96. Epub 2020 Dec 8.

Nuffield Department of Orthopaedics, Centre for Statistics in Medicine, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK, OX3 7LD; NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, OX3 9DU, UK.

Objectives: When developing a clinical prediction model, penalization techniques are recommended to address overfitting, as they shrink predictor effect estimates toward the null and reduce mean-square prediction error in new individuals. However, shrinkage and penalty terms ('tuning parameters') are estimated with uncertainty from the development data set. We examined the magnitude of this uncertainty and the subsequent impact on prediction model performance.

Study Design And Setting: This study comprises applied examples and a simulation study of the following methods: uniform shrinkage (estimated via a closed-form solution or bootstrapping), ridge regression, the lasso, and elastic net.

Results: In a particular model development data set, penalization methods can be unreliable because tuning parameters are estimated with large uncertainty. This is of most concern when development data sets have a small effective sample size and the model's Cox-Snell R is low. The problem can lead to considerable miscalibration of model predictions in new individuals.

Conclusion: Penalization methods are not a 'carte blanche'; they do not guarantee a reliable prediction model is developed. They are more unreliable when needed most (i.e., when overfitting may be large). We recommend they are best applied with large effective sample sizes, as identified from recent sample size calculations that aim to minimize the potential for model overfitting and precisely estimate key parameters.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.jclinepi.2020.12.005DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8026952PMC
April 2021

A note on estimating the Cox-Snell R from a reported C statistic (AUROC) to inform sample size calculations for developing a prediction model with a binary outcome.

Stat Med 2021 Feb 7;40(4):859-864. Epub 2020 Dec 7.

Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.

In 2019 we published a pair of articles in Statistics in Medicine that describe how to calculate the minimum sample size for developing a multivariable prediction model with a continuous outcome, or with a binary or time-to-event outcome. As for any sample size calculation, the approach requires the user to specify anticipated values for key parameters. In particular, for a prediction model with a binary outcome, the outcome proportion and a conservative estimate for the overall fit of the developed model as measured by the Cox-Snell R (proportion of variance explained) must be specified. This proposal raises the question of how to identify a plausible value for R in advance of model development. Our articles suggest researchers should identify R from closely related models already published in their field. In this letter, we present details on how to derive R using the reported C statistic (AUROC) for such existing prediction models with a binary outcome. The C statistic is commonly reported, and so our approach allows researchers to obtain R for subsequent sample size calculations for new models. Stata and R code is provided, and a small simulation study.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8806DOI Listing
February 2021

Protocol for a systematic review on the methodological and reporting quality of prediction model studies using machine learning techniques.

BMJ Open 2020 11 11;10(11):e038832. Epub 2020 Nov 11.

Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.

Introduction: Studies addressing the development and/or validation of diagnostic and prognostic prediction models are abundant in most clinical domains. Systematic reviews have shown that the methodological and reporting quality of prediction model studies is suboptimal. Due to the increasing availability of larger, routinely collected and complex medical data, and the rising application of Artificial Intelligence (AI) or machine learning (ML) techniques, the number of prediction model studies is expected to increase even further. Prediction models developed using AI or ML techniques are often labelled as a 'black box' and little is known about their methodological and reporting quality. Therefore, this comprehensive systematic review aims to evaluate the reporting quality, the methodological conduct, and the risk of bias of prediction model studies that applied ML techniques for model development and/or validation.

Methods And Analysis: A search will be performed in PubMed to identify studies developing and/or validating prediction models using any ML methodology and across all medical fields. Studies will be included if they were published between January 2018 and December 2019, predict patient-related outcomes, use any study design or data source, and available in English. Screening of search results and data extraction from included articles will be performed by two independent reviewers. The primary outcomes of this systematic review are: (1) the adherence of ML-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), and (2) the risk of bias in such studies as assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). A narrative synthesis will be conducted for all included studies. Findings will be stratified by study type, medical field and prevalent ML methods, and will inform necessary extensions or updates of TRIPOD and PROBAST to better address prediction model studies that used AI or ML techniques.

Ethics And Dissemination: Ethical approval is not required for this study because only available published data will be analysed. Findings will be disseminated through peer-reviewed publications and scientific conferences.

Systematic Review Registration: PROSPERO, CRD42019161764.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1136/bmjopen-2020-038832DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7661369PMC
November 2020

Minimum sample size for external validation of a clinical prediction model with a continuous outcome.

Stat Med 2021 Jan 4;40(1):133-146. Epub 2020 Nov 4.

Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK.

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8766DOI Listing
January 2021

External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis.

BMC Med 2020 11 2;18(1):302. Epub 2020 Nov 2.

Department of Epidemiology, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands.

Background: Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting.

Methods: IPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis.

Results: Of 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model's calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%.

Conclusions: The evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice.

Trial Registration: PROSPERO ID: CRD42015029349 .
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1186/s12916-020-01766-9DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7604970PMC
November 2020

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches.

Stat Med 2021 Jan 26;40(2):498-517. Epub 2020 Oct 26.

Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK.

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8787DOI Listing
January 2021

Flaws in the development and validation of a covid-19 prediction model.

Clin Infect Dis 2020 Sep 16. Epub 2020 Sep 16.

Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands.

View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1093/cid/ciaa1406DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7543337PMC
September 2020

Testing small study effects in multivariate meta-analysis.

Biometrics 2020 12 29;76(4):1240-1250. Epub 2020 Aug 29.

Department of Biostatistics, Epidemiology & Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania.

Small study effects occur when smaller studies show different, often larger, treatment effects than large ones, which may threaten the validity of systematic reviews and meta-analyses. The most well-known reasons for small study effects include publication bias, outcome reporting bias, and clinical heterogeneity. Methods to account for small study effects in univariate meta-analysis have been extensively studied. However, detecting small study effects in a multivariate meta-analysis setting remains an untouched research area. One of the complications is that different types of selection processes can be involved in the reporting of multivariate outcomes. For example, some studies may be completely unpublished while others may selectively report multiple outcomes. In this paper, we propose a score test as an overall test of small study effects in multivariate meta-analysis. Two detailed case studies are given to demonstrate the advantage of the proposed test over various naive applications of univariate tests in practice. Through simulation studies, the proposed test is found to retain nominal Type I error rates with considerable power in moderate sample size settings. Finally, we also evaluate the concordance between the proposed tests with the naive application of univariate tests by evaluating 44 systematic reviews with multiple outcomes from the Cochrane Database.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1111/biom.13342DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7736122PMC
December 2020

COVID-19 prediction models should adhere to methodological and reporting standards.

Eur Respir J 2020 09 10;56(3). Epub 2020 Sep 10.

Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK.

View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1183/13993003.02643-2020DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7377211PMC
September 2020

The statistical importance of a study for a network meta-analysis estimate.

BMC Med Res Methodol 2020 07 14;20(1):190. Epub 2020 Jul 14.

Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center - University of Freiburg, Stefan-Meier-Strasse 26, Freiburg, 79104, Germany.

Background: In pairwise meta-analysis, the contribution of each study to the pooled estimate is given by its weight, which is based on the inverse variance of the estimate from that study. For network meta-analysis (NMA), the contribution of direct (and indirect) evidence is easily obtained from the diagonal elements of a hat matrix. It is, however, not fully clear how to generalize this to the percentage contribution of each study to a NMA estimate.

Methods: We define the importance of each study for a NMA estimate by the reduction of the estimate's variance when adding the given study to the others. An equivalent interpretation is the relative loss in precision when the study is left out. Importances are values between 0 and 1. An importance of 1 means that the study is an essential link of the pathway in the network connecting one of the treatments with another.

Results: Importances can be defined for two-stage and one-stage NMA. These numbers in general do not add to one and thus cannot be interpreted as 'percentage contributions'. After briefly discussing other available approaches, we question whether it is possible to obtain unique percentage contributions for NMA.

Conclusions: Importances generalize the concept of weights in pairwise meta-analysis in a natural way. Moreover, they are uniquely defined, easily calculated, and have an intuitive interpretation. We give some real examples for illustration.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1186/s12874-020-01075-yDOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7386174PMC
July 2020

Meta-analysis of continuous outcomes: Using pseudo IPD created from aggregate data to adjust for baseline imbalance and assess treatment-by-baseline modification.

Res Synth Methods 2020 Nov 25;11(6):780-794. Epub 2020 Jul 25.

Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands.

Meta-analysis of individual participant data (IPD) is considered the "gold-standard" for synthesizing clinical study evidence. However, gaining access to IPD can be a laborious task (if possible at all) and in practice only summary (aggregate) data are commonly available. In this work we focus on meta-analytic approaches of comparative studies where aggregate data are available for continuous outcomes measured at baseline (pre-treatment) and follow-up (post-treatment). We propose a method for constructing pseudo individual baselines and outcomes based on the aggregate data. These pseudo IPD can be subsequently analysed using standard analysis of covariance (ANCOVA) methods. Pseudo IPD for continuous outcomes reported at two timepoints can be generated using the sufficient statistics of an ANCOVA model, i.e., the mean and standard deviation at baseline and follow-up per group, together with the correlation of the baseline and follow-up measurements. Applying the ANCOVA approach, which crucially adjusts for baseline imbalances and accounts for the correlation between baseline and change scores, to the pseudo IPD, results in identical estimates to the ones obtained by an ANCOVA on the true IPD. In addition, an interaction term between baseline and treatment effect can be added. There are several modeling options available under this approach, which makes it very flexible. Methods are exemplified using reported data of a previously published IPD meta-analysis of 10 trials investigating the effect of antihypertensive treatments on systolic blood pressure, leading to identical results compared with the true IPD analysis and of a meta-analysis of fewer trials, where baseline imbalance occurred.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/jrsm.1434DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7754323PMC
November 2020

Doug Altman: Driving critical appraisal and improvements in the quality of methodological and medical research.

Biom J 2021 Feb 8;63(2):226-246. Epub 2020 Jul 8.

Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK.

Doug Altman was a visionary leader and one of the most influential medical statisticians of the last 40 years. Based on a presentation in the "Invited session in memory of Doug Altman" at the 40th Annual Conference of the International Society for Clinical Biostatistics (ISCB) in Leuven, Belgium and our long-standing collaborations with Doug, we discuss his contributions to regression modeling, reporting, prognosis research, as well as some more general issues while acknowledging that we cannot cover the whole spectrum of Doug's considerable methodological output. His statement "To maximize the benefit to society, you need to not just do research but do it well" should be a driver for all researchers. To improve current and future research, we aim to summarize Doug's messages for these three topics.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/bimj.202000053DOI Listing
February 2021

The Value of Preseason Screening for Injury Prediction: The Development and Internal Validation of a Multivariable Prognostic Model to Predict Indirect Muscle Injury Risk in Elite Football (Soccer) Players.

Sports Med Open 2020 May 27;6(1):22. Epub 2020 May 27.

Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.

Background: In elite football (soccer), periodic health examination (PHE) could provide prognostic factors to predict injury risk.

Objective: To develop and internally validate a prognostic model to predict individualised indirect (non-contact) muscle injury (IMI) risk during a season in elite footballers, only using PHE-derived candidate prognostic factors.

Methods: Routinely collected preseason PHE and injury data were used from 152 players over 5 seasons (1st July 2013 to 19th May 2018). Ten candidate prognostic factors (12 parameters) were included in model development. Multiple imputation was used to handle missing values. The outcome was any time-loss, index indirect muscle injury (I-IMI) affecting the lower extremity. A full logistic regression model was fitted, and a parsimonious model developed using backward-selection to remove factors that exceeded a threshold that was equivalent to Akaike's Information Criterion (alpha 0.157). Predictive performance was assessed through calibration, discrimination and decision-curve analysis, averaged across all imputed datasets. The model was internally validated using bootstrapping and adjusted for overfitting.

Results: During 317 participant-seasons, 138 I-IMIs were recorded. The parsimonious model included only age and frequency of previous IMIs; apparent calibration was perfect, but discrimination was modest (C-index = 0.641, 95% confidence interval (CI) = 0.580 to 0.703), with clinical utility evident between risk thresholds of 37-71%. After validation and overfitting adjustment, performance deteriorated (C-index = 0.589 (95% CI = 0.528 to 0.651); calibration-in-the-large = - 0.009 (95% CI = - 0.239 to 0.239); calibration slope = 0.718 (95% CI = 0.275 to 1.161)).

Conclusion: The selected PHE data were insufficient prognostic factors from which to develop a useful model for predicting IMI risk in elite footballers. Further research should prioritise identifying novel prognostic factors to improve future risk prediction models in this field.

Trial Registration: NCT03782389.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1186/s40798-020-00249-8DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7253524PMC
May 2020

One-stage individual participant data meta-analysis models for continuous and binary outcomes: Comparison of treatment coding options and estimation methods.

Stat Med 2020 08 11;39(19):2536-2555. Epub 2020 May 11.

Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK.

A one-stage individual participant data (IPD) meta-analysis synthesizes IPD from multiple studies using a general or generalized linear mixed model. This produces summary results (eg, about treatment effect) in a single step, whilst accounting for clustering of participants within studies (via a stratified study intercept, or random study intercepts) and between-study heterogeneity (via random treatment effects). We use simulation to evaluate the performance of restricted maximum likelihood (REML) and maximum likelihood (ML) estimation of one-stage IPD meta-analysis models for synthesizing randomized trials with continuous or binary outcomes. Three key findings are identified. First, for ML or REML estimation of stratified intercept or random intercepts models, a t-distribution based approach generally improves coverage of confidence intervals for the summary treatment effect, compared with a z-based approach. Second, when using ML estimation of a one-stage model with a stratified intercept, the treatment variable should be coded using "study-specific centering" (ie, 1/0 minus the study-specific proportion of participants in the treatment group), as this reduces the bias in the between-study variance estimate (compared with 1/0 and other coding options). Third, REML estimation reduces downward bias in between-study variance estimates compared with ML estimation, and does not depend on the treatment variable coding; for binary outcomes, this requires REML estimation of the pseudo-likelihood, although this may not be stable in some situations (eg, when data are sparse). Two applied examples are used to illustrate the findings.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8555DOI Listing
August 2020

Individual participant data meta-analysis to examine interactions between treatment effect and participant-level covariates: Statistical recommendations for conduct and planning.

Stat Med 2020 07 30;39(15):2115-2137. Epub 2020 Apr 30.

Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK.

Precision medicine research often searches for treatment-covariate interactions, which refers to when a treatment effect (eg, measured as a mean difference, odds ratio, hazard ratio) changes across values of a participant-level covariate (eg, age, gender, biomarker). Single trials do not usually have sufficient power to detect genuine treatment-covariate interactions, which motivate the sharing of individual participant data (IPD) from multiple trials for meta-analysis. Here, we provide statistical recommendations for conducting and planning an IPD meta-analysis of randomized trials to examine treatment-covariate interactions. For conduct, two-stage and one-stage statistical models are described, and we recommend: (i) interactions should be estimated directly, and not by calculating differences in meta-analysis results for subgroups; (ii) interaction estimates should be based solely on within-study information; (iii) continuous covariates and outcomes should be analyzed on their continuous scale; (iv) nonlinear relationships should be examined for continuous covariates, using a multivariate meta-analysis of the trend (eg, using restricted cubic spline functions); and (v) translation of interactions into clinical practice is nontrivial, requiring individualized treatment effect prediction. For planning, we describe first why the decision to initiate an IPD meta-analysis project should not be based on between-study heterogeneity in the overall treatment effect; and second, how to calculate the power of a potential IPD meta-analysis project in advance of IPD collection, conditional on characteristics (eg, number of participants, standard deviation of covariates) of the trials (potentially) promising their IPD. Real IPD meta-analysis projects are used for illustration throughout.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1002/sim.8516DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7401032PMC
July 2020

Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.

BMJ 2020 04 7;369:m1328. Epub 2020 Apr 7.

Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Peter Debyeplein 1, 6229 HA Maastricht, Netherlands

Objective: To review and critically appraise published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of becoming infected with covid-19 or being admitted to hospital with the disease.

Design: Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group.

Data Sources: PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020.

Study Selection: Studies that developed or validated a multivariable covid-19 related prediction model.

Data Extraction: At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

Results: 37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models.

Conclusion: Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

Systematic Review Registration: Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

Readers' Note: This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity..
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1136/bmj.m1328DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7222643PMC
April 2020

Temporal recalibration for improving prognostic model development and risk predictions in settings where survival is improving over time.

Int J Epidemiol 2020 08;49(4):1316-1325

Biostatistics Research Group, Department of Health Sciences, University of Leicester, Leicester, UK.

Background: Prognostic models are typically developed in studies covering long time periods. However, if more recent years have seen improvements in survival, then using the full dataset may lead to out-of-date survival predictions. Period analysis addresses this by developing the model in a subset of the data from a recent time window, but results in a reduction of sample size.

Methods: We propose a new approach, called temporal recalibration, to combine the advantages of period analysis and full cohort analysis. This approach develops a model in the entire dataset and then recalibrates the baseline survival using a period analysis sample. The approaches are demonstrated utilizing a prognostic model in colon cancer built using both Cox proportional hazards and flexible parametric survival models with data from 1996-2005 from the Surveillance, Epidemiology, and End Results (SEER) Program database. Comparison of model predictions with observed survival estimates were made for new patients subsequently diagnosed in 2006 and followed-up until 2015.

Results: Period analysis and temporal recalibration provided more up-to-date survival predictions that more closely matched observed survival in subsequent data than the standard full cohort models. In addition, temporal recalibration provided more precise estimates of predictor effects.

Conclusion: Prognostic models are typically developed using a full cohort analysis that can result in out-of-date long-term survival estimates when survival has improved in recent years. Temporal recalibration is a simple method to address this, which can be used when developing and updating prognostic models to ensure survival predictions are more closely calibrated with the observed survival of individuals diagnosed subsequently.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1093/ije/dyaa030DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7750972PMC
August 2020

Calculating the sample size required for developing a clinical prediction model.

BMJ 2020 03 18;368:m441. Epub 2020 Mar 18.

Julius Center for Health Sciences, University Medical Center Utrecht, Utrecht, Netherlands.

View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1136/bmj.m441DOI Listing
March 2020

Predicting and preventing relapse of depression in primary care.

Br J Gen Pract 2020 02 30;70(691):54-55. Epub 2020 Jan 30.

School of Primary, Community and Social Care, Keele University, Keele.

View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.3399/bjgp20X707753DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7018432PMC
February 2020