Applying excessive amounts of livestock manure to agricultural land may increase the risk of P loading to surface water, particularly through smaller-sized aggregates that are more susceptible to wind and water erosion. The objective of this study was to determine the long-term and residual effects of livestock manure on aggregate size distribution and P content. Soil samples were collected from rain-fed and irrigated blocks of land. Solid cattle feedlot manure had been applied to the rain-fed block at rates of 0 (Treatment M0), 30 (M30), 60 (M60), and 90 (M90) Mg ha−1 yr−1 for 28 years or for 14 years followed by 14 years of no application (R30, R60, and R90) and to the irrigated block at rates of 0 (M0), 60 (M60), 120 (M120), and 180 (M180) Mg ha−1 yr−1 for 28 years or for 14 years followed by 14 years of no application (R60, R120, and R180). Feedlot manure had no significant effect on aggregate size distribution, geometric mean diameter, and wind erodible fraction even after 28 annual applications, but it significantly increased total P (TP), soil test P (STP), and water-soluble P (WP) content in the whole soil and in all aggregate sizes. The TP content increased from <1123 mg kg−1 in M0 to 3278 mg kg−1 for M90 and 4942 mg kg−1 for M180. Similarly, STP content increased from <162 mg kg−1 in M0 to 1178 mg kg−1 for M90 and 1823 mg kg−1 for M180; and WP content increased from <30 mg kg−1 in M0 to 129 mg kg−1 in M90 and 156 mg kg−1 in M180. The increases in TP and STP contents were greater in larger than in smaller (<0.177 mm) aggregates. Irrigation had no significant effect on soil P content, but it did increase the size and reduce the wind erodible fraction of soil aggregate. The effects of manure application on soil P (TP, STP, and WP contents) were long lasting, as indicated by the residual effects still measurable 14 years after manure application was discontinued. When P-enriched soil aggregates are deposited on surface water as a result of erosion, they have an immediate effect through their high WP and STP content and a long lasting effect through the slow release of the remaining TP over time. When using livestock manure on agricultural land, it is necessary to consider application rate, application frequency, residual effects, and farm management practices to protect surface water and to minimize environmental impacts.