A previous study determined that multiple questions on a 360-degree evaluation instrument were highly correlated, suggesting possible redundancy in what was being measured, and that some questions may be eliminated. The current study uses factor analysis and examines a larger data set to further explore this question.
To evaluate the structure of the questionnaire, a factor analysis was performed on 3 yrs of data from a 19-item resident 360-degree evaluation. The number of factors was determined by examining the scree plot of eigenvalues for each item in the instrument, with a cutoff where the slope changes from rapid to slow decline. A reliability analysis was performed with the indicated number of factors, with deletion of each variable to evaluate its influence on overall reliability (Cronbach alpha).
There were 299 evaluations with complete responses to all 19 questions. The scree plot supported a single factor model. The reliability of the full, single factor survey was excellent (Cronbach α = 0.99). The three items with the highest loading on the factor were retained, which related to humanistic, moral/ethical, and professional responsibility behaviors. The reliability for these final three items remained excellent (Cronbach α = 0.96).
The factor analysis suggests that one overall opinion of the evaluated resident was informing the responses of the evaluator. Shortening the instrument to the three items responsible for the greatest influence on the survey does not result in a large decrease in reliability as measured by Cronbach alpha. It is possible that time limitations prevent residents from putting thought into the evaluation of their peers, which results in unidimensional responses. Shortening the instrument may improve evaluations and should be studied in the future.
From the MetroHealth Rehabilitation Institute of Ohio, Cleveland (JCS, RDW, MMV); Case Western Reserve University School of Medicine, Cleveland, Ohio (RDW, ATP, DT, MMV); Center for Healthcare Research and Policy, Cleveland, Ohio (ATP); and Louis Stokes Veterans Administration Medical Center, Cleveland, Ohio (DT). Current affiliation for JCS: Lovelace Rehabilitation Hospital, Albuquerque, New Mexico.
All correspondence and requests for reprints should be addressed to: Mary M. Vargo, MD, MetroHealth Rehabilitation Institute of Ohio, 2500 MetroHealth Drive, Cleveland, OH 44109.
This work was previously presented in poster format at the 2010 Association of Academic Physiatrists Annual Meeting in Bonita Springs, FL.
Financial disclosure statements have been obtained, and no conflicts of interest have been reported by the authors or by any individuals in control of the content of this article.