LETTERS TO THE EDITOR
Readers are invited to submit letters for publication in this department. Submit letters online at http://joem.edmgr.com. Choose “Submit New Manuscript.” A signed copyright assignment and financial disclosure form must be submitted with the letter. Form available at www.joem.org under Author and Reviewer information.
To the Editor:
In their recent publication, Steinberg et al report on a randomized controlled trial (RCT) of wellness program to address metabolic syndrome. I congratulate the authors on planning to subject their intervention to such a rigorous evaluation, but wish they had actually applied this rigor to the analysis. In my opinion, they violated two principles of RCT analysis, which casts doubt on the validity of their findings.
The first is that the primary analysis of RCT data should follow the original design, that is, the authors should have reported separate results for the two intervention arms compared with the control group. Pooled (as the authors did) and subgroup analyses are legitimate but only after presenting the main results. Omitting them raises the suspicion that they were inconclusive.
The second is that RCTs should be analyzed on an intent-to-treat basis. Enrollment and engagement for any program will be less than complete under real-world condition. To judge the true program effect, we need to look at its impact on the eligible population, not just on the subset that volunteered to join. The authors focus mostly on the results in the participants, which would be appropriate as a secondary analysis, and play down the finding of no effect based on the intent-to-treat analysis. This is particularly problematic in light of the low response rate: only 212/1890 (11%) of the invitees participated per table 2 (whereas table 3 suggests 264). This highly selected group may differ in important observable and unobservable characteristics from the population, posing a substantial threat to validity.
My two other concerns are as follows. First, the authors did not reveal whether treatment and control groups were balanced with respect to health care cost at baseline. Given the high variability of health care cost, randomization of a small sample does not ensure this balance, so using changes in cost would have been better than using absolute numbers. Second, the authors conclude that their intervention is cost saving and provides return on investment. Setting aside my doubts about the validity of estimates that are based on participants, at best the study could show lower spending on health care, as the authors did not report the cost of the intervention. With $122 per member per month saved in 11% of the participant, the population-level cost reduction would be around $13 per member per month or $161 per member per year. That number would have to be compared with the intervention cost per eligible member to assess effect on cost.
It is entirely possible that a wellness program that targets the higher risks of the spectrum can reduce cost, and our results1 on a wellness program for a broad population does not rule that out. But Steinberg et al have not answered that question convincingly.
1. Caloyeras J, Liu H, Exum E, et al. Managing manifest diseases, but not health risks, saved PepsiCo money over seven years. Health Aff