Direct application of phosphate rock (PR) may provide the essential phosphorus (P) nutrient for crop production in acid soils. However, the agronomic effectiveness of PR depends on several factors including PR reactivity, soil properties, and crop species. This greenhouse study investigated the effects of PR reactivity, soil pH, Al saturation, and crop species on the initial and residual relative agronomic effectiveness (RAE) of PR compared with water-soluble triple superphosphate (TSP) in two acid soils (Hartsells pH 4.8 and Hiwassee pH 5.4). Three PR sources, Tunisia, Mali, and Togo, representing high, medium, and low PR reactivity, respectively, were used. The soils were treated with P sources at 0, 25, 50, 100, 200, and 500 mg P kg−1 soil. Wheat, ryegrass, and canola were the test crops in the first season, and wheat was used as a residual crop after all three initial crops in the second season. Soil samples were collected for chemical analyses at 0 and 500 mg P kg−1 soil after the first crops. The initial RAE of PR for crop species followed: canola > ryegrass = wheat in the Hartsells soil and canola > ryegrass > wheat in the Hiwassee soil. However, the residual crop of wheat after wheat, wheat after ryegrass, and wheat after canola did not show any significant effect of previous crop. Among P sources, both initial and residual RAE followed: TSP > Tunisia PR > Mali PR ≥ Togo PR for all the crops and soils except for the initial canola crop grown in Hiwassee soil where all PR sources were as effective as TSP. In general, the RAE of PR in Hiwassee soil was higher than that of Hartsells because of the negative effects of soil acidity and Al saturation on crop growth in the Hartsells soil. A significant relationship between available P after first crops and residual wheat grain yield was found in the Hiwassee soil.