ABSTRACT Centers for Medicare and Medicaid Services (CMS) estimated that Medicare's Hospital-Acquired Condition Reduction Program (HAC-RP) would reduce hospital payments by $364 million in fiscal year 2016. Although observers have questioned the validity of certain HAC-RP measures, less attention has been paid to the determination of low-performing hospitals (bottom quartile) and the reliability of penalty assignment. This study used publicly available data from CMS's Hospital Compare to simulate the consistency of hospitals' scores and the assignment of penalties under repeated measurement with no change in each hospital's underlying quality. The simulation showed that 64.0% of all hospitals and 40.6% of hospitals subject to payment penalty are statistically significantly different from the penalty threshold at the 95% confidence level. The proportion of hospitals statistically different from the threshold showed significant variation by ownership status, teaching status, bed size, and other factors. The simulation further showed that due only to chance, 18.0% of penalized hospitals would escape penalty on repeated measurement. Policymakers should consider alterations to the HAC-RP to improve its reliability.