A comparison of frequentist and Bayesian approaches to the Personalised Randomised Controlled Trial (PRACTical)—design and analysis considerations

Background: Multiple treatment options frequently exist for a single medical condition with no single standard of care (SoC), rendering a classic randomised trial comparing a specific treatment to a control treatment infeasible. A novel design, the personalised randomised controlled trial (PRACTical), allows individualised randomisation lists and borrows information across patient subpopulations to rank treatments against each other without comparison to a SoC. We evaluated standard frequentist analysis with Bayesian analyses, and developed a novel performance measure, utilising the precision in treatment coefficient estimates, for treatment ranking. 

Methods: We simulated trial data to compare four targeted antibiotic treatments for multidrug resistant bloodstream infections as an example. Four patient subgroups were simulated based on different combinations of patient and bacteria characteristics, which required four different randomisation lists with some overlapping treatments. The primary outcome was binary, using 60-day mortality. Treatment effects were derived using frequentist and Bayesian analytical approaches, with logistic multivariable regression. The performance measures were: probability of predicting the true best treatment, and novel proxy variables for power (probability of interval separation) and type I error (probability of incorrect interval separation). Several scenarios with varying treatment effects and sample sizes were compared. 

Results: The Frequentist model and Bayesian model using a strong informative prior, were both likely to predict the true best treatment (Pbest≥80%) and gave a large probability of interval separation (reaching a maximum of PIS=96%), at a given sample size. Both methods had a low probability of incorrect interval separation (PIIS<0.05), for all sample sizes (N=500-5000) in the null scenarios considered. The sample size required for probability of interval separation to reach 80% (N=1500-3000), was larger than the sample size required for the probability of predicting the true best treatment to reach 80% (N≤500). 

Conclusions: Utilising uncertainty intervals on the treatment coefficient estimates are highly conservative, limiting applicability to large pragmatic trials. Bayesian analysis performed similarly to the frequentist approach in terms of predicting the true best treatment.