Reconciling Theory and Practice Regarding P Values

To the Editor:

I read the commentary on P values by David Savitz with interest.¹ However, there are some statements that appear to be problematic.

First, the interpretation of P values becomes wrong when Dr. Savitz gives scenarios where “interpretation … can benefit from calculation of null P values.”^{1p. 213} He explains that the P value is able to answer the following question: “How likely is it that an apparent association is truly null but has arisen because of random error?”^{1p. 213} This question is not answered by the P value. Dr. Savitz misses an important property of P values: the P value, a conditional probability (under the null hypothesis), also includes even more extreme unobserved results than the single result at hand.

Second, when Dr. Savitz lists arguments for the null P value, he states that point estimates and confidence intervals for interaction and trends “are less familiar”^{1p. 214} (are they?), making these measures less easily interpretable. He thus implies that less familiarity with point estimates and confidence intervals is an argument for P values. Doesn’t this remind us of Abraham Maslow’s statement: “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”² Should lack of familiarity with statistical concepts prompt us to promote approaches that can be highly misleading?

Third, Dr. Savitz states that we “are much less impressed with non-monotonic associations, whatever the null P value may be.” Unfortunately, this statement promotes the obsession with monotonic dose–response relations in epidemiology, despite the fact that plenty of research informs us that monotonicity in biological systems is not as common as expected.

Fourth, fundamental epidemiologic concepts for study design, conceptualization and assessment of exposure and outcomes, conceptualization of confounding and other biases can be only partially used by studies that simultaneously assessed an “extensive… array of candidate predictors,”^{1p. 213} with genome-wide association studies as the extreme case with “essentially no prior hypotheses.” ^{1p. 213} Without prior hypotheses on underlying causal relations, fundamental epistemological and epidemiologic concepts—especially those related to confounding—cannot be properly applied. Consequently, this data-driven (as opposed to hypothesis-driven) research may deserve a name other than epidemiologic research.

Andreas Stang

Institute of Clinical Epidemiology

Martin-Luther-University Halle-Wittenberg

Halle, Germany

School of Public Health

Boston University

Boston, MA

[email protected]