The relationship of pulmonary function to exposure to asbestos and radiographic abnormalities has been controversial, especially when smoking is present as a confounder. The aim of the study was to provide further understanding on the radiographic–physiologic associations in nonsmoking and smoking asbestos workers. Radiographic asbestosis, pleural lesion, and pulmonary function were studied in 269 Chinese asbestos workers, with average exposure years of 23 for male workers and 18 for female workers. Their functional data were compared with those of 274 controls without exposure to dust. Although most of the male workers were smokers, none of the female workers smoked. In comparison with controls, asbestos workers had significantly lower lung volume and diffusing capacity, irrespective of gender. Female workers and smoking male workers had lower measurements of forced expiratory volume in 1 second and instantaneous forced expiratory flow at 50% and 25% of forced vital capacity. After adjustment for relevant covariates, asbestos exposure, asbestosis, and pleural abnormalities were associated with decreased parameters of pulmonary function, including lung volume, diffusing capacity, and airway flow. These data indicate that asbestos-related functional defects manifested by lung restriction and mild airway obstruction correlate with exposure to asbestos and with parenchymal and pleural abnormalities, independent of smoking.