Limitations of Microarray Studies

Pan, Jonathan Z. MD, PhD; Eckenhoff, Roderic G. MD; Eckenhoff, Maryellen F. PhD

Section Editor(s): Saidman, Lawrence

doi: 10.1213/01.ane.0000260459.35198.c9
Letters to the Editor: Letters & Announcements

Department of Anesthesiology & Critical Care, University of Pennsylvania School of Medicine, Philadelphia, PA,

Article Outline
Back to Top | Article Outline

To the Editor:

Transcription profiling via microarray technology is still evolving, but it is now clear that two features are key to producing interpretable and/or meaningful results. The first is experimental design, and the second is statistical handling of the results.

Principal in the design is studying an adequate number of animals and arrays, and minimizing variance. Variance arises from technical features (RNA isolation and handling, chip-to-chip, hybridization conditions, scanner characteristics, etc) and biological features (animal to animal, experimental conditions, etc). So, because the magnitude of any posited drug effect is at the outset unknown, it is essential to keep variance down and the “n” up. The Rampil et al. study (1) should be commended for a relatively large “n” of animals, and the remarkably low variances reported in their Figure 1. In fact, their reported 2%–5% coefficients of variation (SD divided by mean) is lower than typical centers report for technical variance alone. When combined with the exceptionally robust recovery of RNA from only the basolateral amygdala, this suggests that samples might have been pooled prior to hybridization. Such detail is critical to designing the analysis and interpreting the results, and yet this and other important details (such as the number of chips) are not reported. Further, the rationale for the exposure paradigm, the single late mRNA harvest time point, and the choice of the basolateral amygdala are never made clear. Much needed statistical power (see below) is lost by not incorporating a dose-response or a time series design.

But the real problem is in statistical treatment of the results, the weakness of any microarray approach. In reality, each dot on the chip is an independent experiment, so that a typical gene array can represent over 10,000 individual experiments. Random chance alone would predict that a fraction (dictated by the P value) of these are “significant” by paired t-test. Thus, in the 8800 tests (after the truncation for nonexpression), one would expect that 5%, or 440, of these to be called “altered” by chance alone. Thus, it should not be surprising that the authors find 424. Are these all due to random chance, or could there be a subset of truly changed transcripts? Since it would be heroic to check all 424 with qRT-PCR, this question initially requires a form of multiple testing correction—either Bonferroni (most stringent), or Benjamani–Hochberg—to control for false discovery rate (FDR). This often follows an ANOVA or Principal Components Analysis (PCA) to identify and minimize sources of variation. To be fair, FDR correction is not yet universal in microarray studies, but its lack probably explains the high rate of irreproducible results with this technology. Some authors have gone so far as to conclude that most published gene array work is simply invalid (2). But, like others, Rampil et al. wanted to avoid negative results, so they invoked the “hypothesis-generating” argument, which carries the unproven assumption that a subset of real alterations is within their widely cast net. Yet, also like others, the “hypothesis-generating” aspect is quickly discarded by concluding throughout the remainder of the paper (and in the title) that a large number of genes (all 424?) are regulated in the basolateral amygdala as a result of a 15-min isoflurane exposure, without any form of ranking for fold-change, P value or any other parameter. One cannot have it both ways! In reality their data suggest that isoflurane exposure altered about what would be predicted by chance alone, which could certainly be interpreted to indicate a lack of transcriptional response. Although perhaps surprising, such a result is consistent with other recently published work (3). We studied far more provocative exposures to isoflurane or halothane in the rat or in cultured cells, and found little evidence for transcriptional regulation when the data were analyzed rigorously. Had FDR been neglected, we could also have reported hundreds of hits. It might be interesting to ask whether there is overlap between the Rampil et al. set and ours, a question easily addressed if their data are deposited in the NCBI Gene Expression Omnibus (GEO) database ( (a condition of publication in many journals). For deposit, the data must comply with the MIAME (minimal information about microarray experiments) standard ( Combination of the raw data is unwise, however, since increased variance is likely to dominate any beneficial effect of the increased “n.”

Is there middle ground between stringent FDR and uncorrected paired t-tests? Probably. Many investigators now “relax” the P value after ANOVA and FDR correction until they get hits. These hits, or “hypotheses” are then tested with qRT-PCR. In the Rampil et al. study, it appears that two genes were tested by qRT-PCR, but only one was in their set of 424 hits from the microarray approach. Even in this one, it is not clear if the hypothesis was confirmed, as the PCR data were not provided.

In summary, we certainly sympathize with the authors because negative data, no matter how well-analyzed, are difficult to publish. On the other hand, falsely positive data are, at best, useless. It may be that low affinity drugs like the inhaled anesthetics, at clinical concentration, simply don’t evoke a large enough change in transcription for the undirected microarray approach to detect. Perhaps more focused approaches like qRT-PCR or custom arrays will provide an answer as to whether inhaled anesthetics modulate transcription.

Jonathan Z. Pan, MD, PhD

Roderic G. Eckenhoff, MD

Maryellen F. Eckenhoff, PhD

Department of Anesthesiology & Critical Care

University of Pennsylvania School of Medicine

Philadelphia, PA

Back to Top | Article Outline


1. Rampil IJ, Moller DH, Bell AH. Isoflurane modulates genomic expression in rat amygdala. Anesth Analg 2006;102:1431–8.
2. Nadon R, Shoemaker J. Statistical issues with microarrays: processing and analysis. Trends Genet 2002;18:265–71.
3. Pan JZ, Wei HF, Hecker JG, et al. Rat brain DNA transcript profile of halothane and isoflurane exposure. Pharmacogenet Genomics 2006;161:171–82.
© 2007 International Anethesia Research Society