Data Agnosticism and Implications on Method Comparison Studies

The evolution of method comparison studies has occurred through a series of fits and starts, the most notable of which occurred in the early 1980s as the shortcomings of the traditional, linear regression approach to method comparisons were described and a complementary, “agreement”-based methodology proposed.^1–3 However, the evolution did stop there. As clinicians began to deemphasize absolute values and focus on trend monitoring (cynics argue that the latter is easier, advocates suggest it is more useful), additional techniques were developed, the 2 most notable of which are the 4-quadrant and polar plotting techniques developed by Perrino et al.⁴ and Critchley et al.⁵

In this issue of Anesthesia & Analgesia, Saugel et al.⁶ describe, step by step, how these plots are produced and, most importantly, how they differ. By “transforming” trending data from Cartesian to polar coordinate systems, Saugel et al.⁶ demonstrate, both visually and mathematically, how the selection of a particular analytical technique can impact the results. The fact that highly discordant measurements in which the average change is 0 are excluded from the quantitative estimate of agreement when using the polar technique, but not the 4-quadrant technique, is essential to proper interpretation of these tests.

Why does the polar plotting technique do this? The reason is because the polar plotting technique does not make any assumptions about which technique is better. Ironically, most published studies that use the polar plotting technique compare a new method of measurement with an accepted reference standard, and in these instances, an agnostic approach may not be appropriate.^7–9 This question of “what is truth” may sound philosophical on the surface but in reality has significant mathematical implications. For instance, if changes in cardiac output (ΔCO) are measured by a pulmonary artery catheter and a magic 8 ball, and the pulmonary artery catheter estimates that ΔCO is −2 L/min but the magic 8 ball estimates that ΔCO is 2 L/min, the polar analysis will exclude that data point because “true” ΔCO is 0 L/min. This is, in essence, information loss. The 4-quadrant technique, by contrast, will place this point in quadrant 2, which will negatively affect the estimate of concordance.

Interestingly, the Bland-Altman technique, as it was originally described, takes the same approach to the relative value of data.^1–3 In some sense, this is appropriate. As Altman pointed out, clinicians are not particularly interested in the probability that the slope of a best fit line between 2 outputs is not 0 “when the two variables are obviously associated by their very nature… What we really want to know in these studies is how well the two measures agree.”¹ However, this assumes that 2 variables are actually related. A key element to the agreement strategy, which is often overlooked, is that “good” agreement does not necessarily imply any correlation if the range of data tested is small. From the viewpoint of the clinician, performance of an agreement analysis makes the a priori assumption that a correlation actually exists.

Take, for instance, 2 monitors, X and Y, which estimate cardiac output (Θ). If the slope of the best fit line relating X and Y is m, estimates of Θ can be described as

and

where Θ represents the true value of stroke volume and ε represents normally distributed measurement errors. Now let us assume that m = 0, that is, that there is absolutely no statistical correlation between the 2 devices (i.e., they are completely independent of one another). If X and Y are “tested” over a narrow range of values (i.e., the distribution of Θ is small [Fig. 1, upper left quadrant]), it is possible for a random number generator to produce limits of agreement approaching what would be deemed clinically “ acceptable” by Critchley and Critchley,¹⁰ who in 1999 stated that “Bias and precision statistics has now replaced correlation and regression as the accepted statistical technique of comparing two techniques measuring the same physiological variable, such as cardiac output.”

Figure 1: Limits of agreement, correlation coefficient, and limits of agreement-based P value for 4 cardiac output data sets of varying sample distribution (the distribution of the data is displayed on the lower aspect of each figure) and in whom there is no correlation between X and Y. This data set demonstrates that if σ_Θ is decreased, the limits of agreement will also decrease despite the fact that the intrinsic error has not changed. By contrast, the correlation coefficient is relatively insensitive to σ_Θ.

Clearly, many in the anesthesia community agree, because some investigators have stopped publishing correlation coefficients and even basic scatterplots altogether, relying exclusively on limits of agreement and/or trending analysis to compare methods of measurement,^7,8,11–15 yet this was not the intent of Bland or Altman. Indeed, in their 1983 manuscript, they stated “The first step, one which should be mandatory, is to plot the data… For the purposes of comparing the methods the line of identity is much more informative, and is essential to get a correct visual assessment of the relationship.”² Three years later, in their oft-cited Lancet paper, they reemphasized that “The first step is to examine the data… A simple plot of the results of one method against those of the other though without a regression line is a useful start.”³

Thus, when making method comparison studies, we suggest that the following steps are taken:

1. The authors explicitly state whether or not 1 device is being used as a reference standard.
2. The authors state, a priori, the acceptable range of clinically acceptable values for each statistical test used. For instance, the acceptable limits of agreement for blood pressure monitoring are likely to be narrower during cerebral aneurysm surgery than placement of ear tubes.
3. Data are initially presented in a scatterplot to allow visual assessment of the relationship, as suggested by Bland and Altman.^2,3 This will reduce the probability that the limits of agreement between measures, which have no meaningful relationship, are deemed clinically acceptable. Preiss and Fisher¹⁶ described a random permutation technique, which estimates the probability that a paired data set arose from unassociated measurements, which some investigators may want to consider using.
4. The limits of agreement are calculated and plotted, as described by Bland and Altman, with allowances made for repeated measures when necessary.^2,3,17,18 For repeated measures, Bland and Altman base the estimate of the limits of agreement on both within-subject variances and the variance of the differences between-subject means.¹⁷ Myles and Cui¹⁸ recommend calculating the mean of repeated measures and using a random effects model to account for the reduced variation that occurs with averaging. When the residuals are not normally distributed, a transformation may be appropriate. For instance, some data may be distributed log-normally. In these instances, Dexter et al.¹⁹ have suggested both a nonparametric (based on ranking the observations and selecting cutoff values that remain within the desired percentiles [taking into account sample size and degrees of freedom]) and a parametric approach (based on the Student t distribution) for the calculation of prediction limits, either of which is acceptable. Last, confidence intervals around these limits should be calculated and displayed graphically, ensuring that the limits are interpreted properly (i.e., that the sample size is sufficient to draw meaningful conclusions from the data).^3,17
5. When trending data are analyzed, they should be performed using either the 4-quadrant (when an accepted standard is being used) or a polar plotting techniques (when there is no accepted reference standard).^4,5 If the polar plotting technique is used, particular attention must be paid to the exclusion zone, because the polar plotting technique may exclude important data points that are highly discordant.⁶

Saugel et al.⁶ have pointed out a subtle but extremely important and underappreciated difference in 2 frequently used statistical techniques. They have also helped clarify a concept that has clearly confused many (the original paper describing the polar plotting technique states that “agreement is shown by the angle the vector makes with the line of identity [y x] and magnitude of change by the length of the vector (Fig. 4),” yet in the fifth published figure, and the Appendix, the average value is used, not the length of the vector).⁵ It is our hope that standardizing the presentation of data will improve the inferences derived from these important studies.

DISCLOSURES

Name: Robert H. Thiele, MD.

Contribution: This author helped design the study and write the manuscript.

Attestation: Robert H. Thiele approved the final manuscript.

Name: Timothy L. McMurry, PhD.

Contribution: This author helped design the study and write the manuscript.

Attestation: Timothy L. McMurry approved the final manuscript.

This manuscript was handled by: Franklin Dexter, MD, PhD.