Increased physical activity may protect against cognitive decline, the primary symptom of Alzheimer disease. In this study, we examined the relationship between physical activity and trajectories of cognitive functioning over serial assessments. Cognitively normal (Clinical Dementia Rating 0) middle-aged and older adults (N=173; mean age, 60.7±7.8 y) completed a self-report measure of physical activity and a battery of standard neuropsychological tests assessing processing speed, attention, executive functioning, and verbal memory. At baseline, individuals with higher physical activity levels performed better on tests of episodic memory and visuospatial functioning. Over subsequent follow-up visits, higher physical activity was associated with small performance gains on executive functioning and working memory tasks in participants with one or more copies of the apolipoprotein ε4 allele (APOE4). In APOE4 noncarriers, slopes of cognitive performance over time were not related to baseline physical activity. Our results suggest that cognitively normal older adults who report higher levels of physical activity may have slightly better cognitive performance, but the potential cognitive benefits of higher levels of physical activity over time may be most evident in individuals at genetic risk for Alzheimer disease.