Retrospective assessment of occupational exposures for the study of birth defects in offspring presents many challenges with the validity and reliability of the assessment dependent on the method employed.
Within the Baltimore-Washington Infant Study (BWIS), a large population-based case-control study of cardiovascular malformations, we evaluated the inter- and intra-rater reliability associated with the retrospective assessment of parental occupational exposure to nickel and cobalt.
All case and control jobs (n = 2724) were first screened by five professionals with expertise in retrospective assessment of occupational exposures for epidemiologic studies. Jobs for which at least one expert indicated the potential for exposure were further assessed by two industrial hygienists (IHs). The assessment included familiarity with the job, probability of direct and indirect exposure, and frequency, duration, and intensity of exposure. Each assessment was repeated by the same two IHs, three months later. A consensus panel of three IHs (including the two original IHs) reviewed all jobs for which disagreement existed on exposure parameters, as well as, those jobs for which one or both IHs had no familiarity. Percent agreement and Kappa statistic (K) were used to assess reliability between and within IHs. K values of <0.40, 0.40–0.75, >0.75 are traditionally interpreted to represent poor, fair to good, and excellent reliability, respectively.
The screening resulted in 468 and 496 jobs being evaluated by the IHs for potential cobalt and nickel exposure, respectively. Among jobs with which IHs were familiar, IH1 indicated that 55 jobs had potential cobalt exposure and 55 nickel exposure, while IH2 indicated 83 had potential cobalt exposure and 68 nickel exposure. Inter-rater reliability was poor for presence of cobalt exposure (K = 0.21) and nickel (K = 0.27) with IHs agreeing on exposure status for 80% (349/437) and 84% (n = 328/389) of jobs, respectively. Among jobs for which both IHs indicated exposure, inter-rater reliability was excellent for probability of direct exposure (K = 0.79 and 0.83) and good for indirect exposure (K = 0.61 and 0.73) for cobalt and nickel, respectively. Inter-rater reliability was excellent for frequency, duration, and intensity of both direct cobalt and nickel exposures (K > 0.85). Intra-rater agreement for the presence of cobalt exposure was poor (K = 0.35) and good to excellent (K = 0.54–0.92) for probability, frequency, duration and intensity of direct and indirect exposure. Intra-rater reliability for the presence of nickel exposure was fair (K = 0.42) and fair to excellent (K = 0.43–0.88) for probability, frequency, duration, and intensity of exposure. The consensus panel reviewed 140 and 185 jobs for cobalt and nickel exposure, respectively. Among the jobs assessed for cobalt, the consensus panel concluded that 6% had potential exposure to cobalt compared to 42% by IH1 and 59% by IH2. Among the jobs assessed for nickel, the panel concluded 22% had potential exposure compared to 30% by IH1 and 37% by IH2.
While inter- and intra-rater reliability varied by parameter assessed, exposure was overestimated when assessed by individual IHs as compared to the consensus panel. Further attempts need to be made to improve retrospective exposure assessment methods through the use of expert panels and standardization of methodology employed by individual hygienists.