Indirect treatment comparisons can advance clinical care by offering information to fill gaps in the knowledge of the relative effects of two or more treatment options (Bucher et al., 1997; Jansen et al., 2011). These studies attempt to approximate direct comparisons when head-to-head studies are lacking or have not sufficiently resolved uncertainties about relative treatment effects (Bucher et al., 1997; Jansen et al., 2011). However, even under ideal conditions, indirect treatment comparisons may not definitively resolve these uncertainties, and therefore, their results must be interpreted with caution and considered in the context of the assumptions required to ensure their validity (Kim et al., 2014). These studies are prone to numerous potential methodological challenges, which increase the degree of uncertainty (Jansen et al., 2011).
The International Society for Pharmacoeconomics and Outcomes Research Task Force published a two-part report detailing Good Research Practices when carrying out indirect treatment comparison studies (Hoaglin et al., 2011; Jansen et al., 2011). The report highlights three basic assumptions critical to the validity of these studies: all trials included must be comparable in terms of potential effect modifiers, such as trial or patient characteristics (assumption of similarity); there must be no substantial heterogeneity between trial results in pairwise comparisons (assumption of homogeneity); and there must be no relevant discrepancy or inconsistency between direct and indirect evidence (assumption of consistency) (Hoaglin et al., 2011; Jansen et al., 2011). However, use of a valid random-effects model can appropriately account for the increased uncertainty introduced by heterogeneity, but a heterogeneous result can still be difficult to interpret. Others have weighed in similarly: ‘The results of examination of similarity, homogeneity, and consistency must be handled appropriately. If there has been a major violation of any of these assumptions, this may even mean that no worthwhile indirect comparison can be performed’ (Kiefer et al., 2015).
Pae et al. (2017) report an indirect treatment comparison study of efficacy and tolerability between aripiprazole once monthly (AOM) and paliperidone palmitate once monthly (PP). The authors concluded that the results ‘may suggest that there may be relative advantages for AOM over PP in the short-term treatment of schizophrenia’. However, several methodological issues, beyond those cited by the authors, compromise the validity of the study and highlight more generally the challenges of performing indirect treatment comparisons. The most critical threats to the study's validity arise from violations of the assumption of similarity. Other methodological issues pertain to the selection of trials to be included in the indirect comparisons.
In particular, it is questionable whether the assumption of similarity can be supported, given the differences identified between the placebo-arm patients from PP and AOM studies. Differences in terms of dropout rates, change in symptom severity, and baseline symptom severity are indications that treatment modifiers, such as study populations and settings, were not similar. A large difference was found between placebo-arm patients who dropped out because of treatment-emergent adverse events [14.3% (73/510) for PP and 0.8% (13/172) for AOM] and all-cause early discontinuations [63.5% (324/510) for PP and 50.6% (87/172) for AOM] (Pae et al., 2017; Table 1). Baseline symptom severity, as assessed by the mean Positive and Negative Syndrome Scale (PANSS) total score, in placebo arms was markedly different between the studies [89 (13.5) for PP studies and 104 (11.0) for AOM]. Greater symptomatic improvement, as measured by the PANSS score, occurred for patients treated with placebo during the AOM study compared with those treated with placebo during PP studies [−11.7 (20.9) vs. −2.2 (21.9), respectively] (Table 1). Baseline symptom severity, as assessed by the PANSS total score, is predictive of effect size in response to antipsychotic treatment, with higher scores being associated with greater effect sizes (Furukawa et al., 2015). At baseline, the difference in the mean PANSS scores among the combined PP studies and the AOM study is 14.2 using all subject data (Pae et al., 2017), and the difference is 15.0 units when only the placebo subjects are examined (Table 1). The authors acknowledge this difference, but do not consider it meaningful (Pae et al., 2017). However, smaller differences in total PANSS scores are associated with meaningful effects on functional outcomes, hospitalization duration, and healthcare costs (Glick et al., 2015; Leddy-Stacy and Rosenheck, 2016). In addition, the difference of 14.2 points approaches the minimum PANSS score change of 15.3 points associated with a clinically meaningful difference on the basis of an analysis using Clinical Global Impression score anchoring (Hermes et al., 2012). In other words, this magnitude of difference seems difficult to ignore.
Many AOM and PP randomized-controlled trials were excluded in the analysis by Pae et al. (2017). This is understandable meta-analyses must sacrifice a degree of data completeness for the sake of being able to combine and compare data that would otherwise not be possible. A significant concern with the Pae et al. (2017) analysis is the failure to account for whether the excluded data have clinical significance and whether the included studies and specific study results are representative of the treatment effects of interest.
A total of four PP studies were included in the Pae et al. (2017) analysis. Two of the four studies used low-dose initiation regimens that are not recommended for clinical practice (Gopal et al., 2010; Nasrallah et al., 2010). As the efficacy results from these two earlier, developmental studies were marginal, the initiation regimen was revised to achieve higher plasma concentrations (i.e. two administrations, 1 week apart, in the deltoid muscles, using higher doses). The revised initiation regimen is currently recommended worldwide in product labeling to negate the requirement for additional oral antipsychotic supplementation during initiation, and thus represents the standard of care.
In 2010, Pandina and colleagues published the primary study using a revised two-dose initiation regimen. The study was not included in the Pae et al. (2017) analysis, but was eligible for inclusion: it is a 13-week, double-blind comparison of adults with acutely exacerbated symptoms of schizophrenia who were randomized in a 1 : 1 : 1 : 1 ratio to paliperidone palmitate at 25, 100, or 150 mgEq doses or placebo, assessing the mean change in the PANSS total score from baseline. The study is included in the PP prescribing information and can be readily accessed through PubMed. Instead, the data used were from a post-hoc subgroup analysis of the Pandina et al. (2010) study, which included patients with moderate-severe symptoms (Alphs et al., 2011). Although one could argue that the selection of this post-hoc defined subgroup should make the study population more similar to that of the AOM study population, the disease severity in the included Alphs and colleagues data is still about 10 PANSS points less than that in the AOM study. Furthermore, of the two treatment arms within the post-hoc analysis by Alphs et al. (2011), the higher dose arm (150 mgEq) – the arm showing the greatest effect – was excluded (essentially creating a subgroup of a subgroup) (Pae et al., 2017). The effect of these inclusion and exclusion criteria applied by Pae and colleagues is to lower the overall estimate of PP efficacy, relative to considering a broader set of PP studies.
The PP study in acute schizophrenia by Kramer et al. (2010) was also excluded (Pae et al., 2017). The rationale for this exclusion is not provided, but it also appears to fulfill all eligibility criteria as defined by Pae and colleagues: a 9-week, double-blind, comparison of adults with acutely exacerbated schizophrenia who were randomized to paliperidone palmitate or placebo, assessing a primary endpoint of the mean change in the PANSS total score from baseline. Coincidentally, the Kramer et al. (2010) and Pandina et al. (2010) represent data from the PP development program with the strongest effect sizes. The exclusion of these two PP studies shifts the results in favor of AOM. Inclusion of the studies with the earlier, suboptimal initiation regimens is also relevant to early dropout findings as dropout rates would be inflated in the context of suboptimal efficacy.
In the absence of prospective, randomized, head-to-head clinical trials, a statistical comparison using matched patient-level pooled data may be a feasible and informative technique to provide a preliminary comparison of two medications. Using patient-level data, one may create comparison groups that are similar with respect to the distribution of a number of demographic and baseline characteristics so that indirect comparisons would be valid. Data points from many of the Janssen and colleagues clinical studies and almost all the paliperidone palmitate clinical trial data are available through the Yale University Open Data Access (YODA) Project, http://yoda.yale.edu/johnson-johnson. One of the main intentions of this project is to promote strong scientific collaboration for greater public good.
The indirect treatment comparison study by Pae et al. (2017) sought to answer questions on the comparative efficacy between AOM and PP in the short-term treatment of schizophrenia. They suggest possible advantages of AOM over PP on the basis of their findings. However, our close reading of the paper gave rise to methodological concerns that we believe affect the strength and interpretability of their conclusions.
Conflicts of interest
Jesse Berlin is an employee of Johnson & Johnson. All other authors are employees of the Janssen Pharmaceutical Companies of Johnson & Johnson.
Alphs L, Bossie CA, Sliwa JK, Ma Y-, Turner N (2011). Onset of efficacy with acute long-acting injectable paliperidone
palmitate treatment in markedly to severely ill patients with schizophrenia
: post hoc analysis of a randomized, double-blind clinical trial. Ann Gen Psychiatry 10:12.
Bucher HC, Guyatt GH, Griffith LE, Walter SD (1997). The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 50:683–691.
Furukawa TA, Levine SZ, Tanaka S, Goldberg Y, Samara M, Davis JM, et al (2015). Initial severity of schizophrenia
and efficacy of antipsychotics: participant-level meta-analysis of 6 placebo-controlled studies. JAMA psychiatry 72:14–21.
Glick HA, Li P, Harvey PD (2015). The relationship between Positive and Negative Syndrome Scale (PANSS) schizophrenia
severity scores and risk for hospitalization: an analysis of the CATIE Schizophrenia
Trial. Schizophr Res 166:110–114.
Gopal S, Hough DW, Xu H, Lull JM, Gassmann-Mayer C, Remmerie BM, et al (2010). Efficacy and safety of paliperidone
palmitate in adult patients with acutely symptomatic schizophrenia
: a randomized, double-blind, placebo-controlled, dose-response study. Int Clin Psychopharmacol 25:247–256.
Hermes EDA, Sokoloff D, Stroup TS, Rosenheck RA (2012). Minimum clinically important difference in the positive and negative syndrome scale with data from the Clinical Antipsychotic
Trials of Intervention Effectiveness (CATIE). J Clin Psychiatry 73:526–532.
Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, et al (2011). Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR task force on indirect treatment comparisons good research practices: part 2. Value Health 14:429–437.
Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, et al (2011). Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR task force on indirect treatment comparisons good research practices: part 1. Value Health 14:417–428.
Kane JM, Peters-Strickland T, Baker RA, Hertel P, Eramo A, Jin N, et al (2014). Aripiprazole
once-monthly in the acute treatment of schizophrenia
: findings from a 12-week, randomized, double-blind, placebo-controlled study. J Clin Psychiatry 75:1254–1260.
Kiefer C, Sturtz S, Bender R (2015). Indirect comparisons and network meta-analyses: estimation of effects in the absence of head-to-head trials. Dtsch Ärztebl Int 112:803–808.
Kim H, Gurrin L, Ademi Z, Liew D (2014). Overview of methods for comparing the efficacies of drugs in the absence of head-to-head clinical trial data. Br J Clin Pharmacol 77:116–121.
Kramer M, Litman R, Hough D, Lane R, Lim P, Liu Y, Eerdekens M (2010). Paliperidone
palmitate, a potential long-acting treatment for patients with schizophrenia
. Results of a randomized, double-blind, placebo-controlled efficacy and safety study. Int J Neuropsychopharmacol 13:635–647.
Leddy-Stacy M, Rosenheck R (2016). Obtaining employment as an anchor for estimating the minimum clinically important difference on the Positive and Negative Syndrome Scale (PANSS) in schizophrenia
. Psychiatry Res 238:304–309.
Nasrallah HA, Gopal S, Gassmann-Mayer C, Quiroz JA, Lim P, Eerdekens M, et al (2010). A controlled, evidence-based trial of paliperidone
palmitate: a long-acting injectable antipsychotic
, in schizophrenia
. Neuropsychopharmacology 35:2072–2082.
Pae CU, Wang SM, Han C, Bahk WM, Lee SJ, Patkar AA, et al (2017). Comparison between long-acting injectable aripiprazole
palmitate in the treatment of schizophrenia
: systematic review and indirect treatment comparison
. Int Clin Psychopharmacol 32:235–248.
Pandina GJ, Lindenmayer J-, Lull J, Lim P, Gopal S, Herben V, et al (2010). A randomized, placebo-controlled study to assess the efficacy and safety of 3 doses of paliperidone
palmitate in adults with acutely exacerbated schizophrenia
. J Clin Psychopharmacol 30:235–244.
Takahashi N, Takahashi M, Saito T, Iizumi M, Saito Y, Shimizu H, et al (2013). Randomized, placebo-controlled, double-blind study assessing the efficacy and safety of paliperidone
palmitate in Asian patients with schizophrenia
. Neuropsychiatr Dis Treat 9:1889–1898.