The proposed ‘concordance-statistic for benefit’ provided a useful metric when modeling heterogeneous treatment effects

5 years ago

The proposed ‘concordance-statistic for benefit’ provided a useful metric when modeling heterogeneous treatment effects

Clinical prediction models that support treatment decisions are usually evaluated for their ability to predict the risk of an outcome rather than treatment benefit – the difference between outcome risk with versus without therapy. We aimed to define performance metrics for a model’s ability to predict treatment benefit.

Study Design and Setting

We analyzed data of the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) trial, and of 3 recombinant tissue plasminogen activator trials. We assessed alternative prediction models with a conventional risk concordance-statistic (c-statistic) and a novel c-statistic for benefit. We defined observed treatment benefit by the outcomes in pairs of patients matched on predicted benefit but discordant for treatment assignment. The ‘c-for-benefit’ represents the probability that from two randomly chosen matched patient pairs with unequal observed benefit, the pair with greater observed benefit also has a higher predicted benefit.

Results

Compared to a model without treatment interactions, the SYNTAX Score II had improved ability to discriminate treatment benefit (c-for-benefit 0.590 versus 0.552), despite having similar risk discrimination (c-statistic 0.725 versus 0.719). However, for the simplified Stroke-Thrombolytic Predictive Instrument (TPI) versus the original Stroke-TPI, the c-for-benefit (0.584 versus 0.578) was similar.

Conclusion

The proposed methodology has the potential to measure a model’s ability to predict treatment benefit not captured with conventional performance metrics.

Publisher URL: www.sciencedirect.com/science

DOI: S0895435617303037