Implicit bias among health professionals may have adverse implications for the health of minority, underserved, and stigmatized populations. For example, implicitly held attitudes about certain groups can result in biased clinical decisions,1–3 erosion of trust due to poor interpersonal interactions, and biased behaviors that are perceived as discriminatory or prejudicial by patients and caregivers.4–6 In response to growing recognition that implicit attitudes influence behavior despite consciously formed goals, a nascent literature is emerging on strategies for implicit bias recognition and management in health professions education. Bias management strategies may involve enhancing internal motivation, confidence, and emotional regulation.7,8 Additional authors suggest mitigating the influence of bias through improving perspective taking and empathy,7 identifying common group identities, promoting counterstereotypes,9 and recognizing that the removal of all bias is impossible.10
Before implicit bias can be effectively managed, however, it must be made visible. Among the tools that can facilitate implicit bias recognition, the implicit association test (IAT) is an instrument that measures response time and can be completed online by individuals to measure their degree of implicit bias regarding a specific group or concept. Van Ryn and colleagues11 found that completing an IAT with feedback was a significant predictor of decreases in implicit bias. Other research has suggested that feedback through the IAT can facilitate an individual’s awareness of their implicit attitudes.12–15 While the IAT is useful for revealing deeper biases that shape behavior outside of conscious awareness, there may be immediate defensive responses associated with bringing implicit bias into awareness.16
Several authors have described how confronting individuals with evidence that they are biased can incite a defensive response.17–19 For example, when individuals are provided with feedback about their implicit bias, participants respond by denigrating measures of implicit bias20,21 or avoiding feedback altogether.20 Using such strategies serves to maintain consciously formed egalitarian beliefs despite contrary factual evidence.19,22 Because these defenses can adversely affect internalization of feedback,20,23,24 more research is needed in this area. For example, if an individual is given feedback that demonstrates they hold negative bias toward a marginalized group, this may not align with their self-perception and consciously held beliefs. Because of this knowledge, learners may experience an emotional reaction that strongly influences subsequent efforts to leverage the IAT to foster future change. This resonates with research on feedback in health professions education, which highlights that when encountering negative feedback that is incongruent with self-perceptions of performance, physicians often disagree, experience negative emotional reactions, and have difficulty assimilating this information.25–27 In a series of studies, Sargeant and colleagues28 found that physicians who encountered such feedback experienced intense and long-lasting emotions, stimulating a call for facilitated reflection to reconcile challenging feedback.28–30 A central challenge related to feedback relates to the dilemma that those who need feedback the most may be the least receptive to it and find it the least useful.31 This issue is further compounded by the immense complexity and contextual variation related to feedback and self-assessment.
Unless we develop a sound understanding of how individuals process and integrate their IAT results, we cannot fully appreciate the potential of the IAT and its role in fostering bias management and personal and professional development in health professions education. We set out to study how health professionals perceive the experience of completing the IAT and receiving their results, as a prelude to better understanding the affordances and challenges of this novel educational approach.
We used constructivist grounded theory methodology to explore the influence of promoting conscious awareness of implicit bias on health care professionals using interviews. Constructivism assumes that researchers and participants construct the realities in which they participate, and grounded theory is a method to learn about the worlds under study and develop theories to understand them. Constructivist grounded theory is well suited to explore a social process that is not currently explained by a well-established theoretical construct.32
We defined implicit bias as associations toward a specific social grouping that are outside of an individual’s conscious awareness.33 More specifically, our study explored implicit dangerousness bias toward individuals with mental illness. To our knowledge, existing studies relating to implicit bias recognition and management have not explored implicit bias related to mental illness. We chose implicit dangerousness bias toward individuals with mental illness bias for our research, anticipating that implicit dangerousness bias might be a commonly arising issue that could adversely influence trusting relationships and clinical decisions.
The mental illness IAT is an online measure of response time that measures implicit associations between the concepts of physical illness and dangerousness versus mental illness and dangerousness.33 At the end of the test, participants receive a result that indicates they either hold no dangerousness bias related to mental or physical illness, or that they hold mild, moderate, or strong dangerousness bias. The mental illness IAT is available online to the public at Project Implicit (https://implicit.harvard.edu/implicit/user/pimh/index.jsp).
The IAT measures response time and latency and has typically demonstrated good internal consistency,34–37 insensitivity to procedural variation,33,38 high test–retest reliability,39 and significantly less susceptibility to faking compared with explicit measures such as surveys.40 Criticism of the IAT suggests that instead of reflecting authentic negative attitudes, IAT scores may stem from other associations such as victimization, maltreatment, and oppression.41–45
We invited pediatric physician and nurse participants at the Schulich School of Medicine and Dentistry, Western University, through recruitment notices posted internally and through electronic means to participate in 60 minute, semistructured, one-on-one interviews. Participants were informed that they would be part of a study exploring implicit bias and mental illness. We obtained approval from the Western University Research Ethics Board to conduct the study. Team member composition included the principal investigator (J.S.), a child and adolescent psychiatrist, faculty member, and PhD candidate in health professions education; as well as nursing staff (A.M.); and three experts in health professions education (C.W., L.L., and P.T.).
We initially chose the pediatric emergency department of an accredited Canadian academic health science center to explore health professional responses to feedback from the mental illness IAT. An emergency department is a useful setting to explore feedback about implicit bias due to the fast-paced nature of emergency practice. There is often considerable pressure on providers to make high-stakes clinical decisions within limited time,46,47 and therefore the potential influence of implicit bias on patient–provider relationships and clinical decisions is more overt in this specific type of acute health care context.48–52 A pediatric setting provided a sample with an appropriate size and scope to conduct the study and begin the process of purposeful theoretical sampling while remaining open to recruiting additional participants.
During the first few interviews, participants were asked open-ended questions about the experience of taking the IAT, whether their results were expected/unexpected, and about the relationship between attitudes toward mental illness and implicit bias. We deliberately informed participants that they were not required to share their IAT results with us. This decision was motivated by the desire to foster a nonjudgmental interview and facilitate open discussion. We emphasized that we were interested in participants’ interpretation of and reaction to their result, rather than the result itself.
In our analysis of the first interviews, however, we found a striking range of emotional reactions to the test itself, which we decided required further elaboration. In keeping with a purposeful theoretical sampling strategy, we expanded our recruitment and altered our interview approach to elaborate perspectives on the IAT experience more fully. We broadened our sample to include physicians and nurses who worked outside of the emergency department but still worked in pediatric inpatient and outpatient settings. We chose to expand beyond an emergency setting to consider responses in different clinical contexts with less acuity, but remained restricted to pediatric providers to continue our exploration within a similar social grouping of health professionals.
Interviews were conducted from September 2015 to November 2016 and followed a discussion guide developed from an initial literature review. The discussion guide was adapted iteratively as the study proceeded in accordance with a constructivist grounded theory approach (the final version is available in Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A504).32 We initially explored participants’ understanding of their implicit attitudes about mental illness and responses to the IAT results. Our next interviews expanded to explore more about emotions and cognition both during the administration of the test and the presentation of the result. Discussion expanded to include questions relating to perspective change and whether changing implicit attitudes and associated behaviors was possible and how to achieve it. Each participant was assigned a number based on which stakeholder group they belong to.
Once transcribed, coding and analysis was conducted by a team consisting of J.S. and A.M., beginning with line-by-line coding by J.S. and using constant comparative analysis to develop focused codes and working toward major themes. Key themes were shared with the entire team as analysis shifted to relating codes and categories to one another and the development of an explanatory theory that accounted for possible relationships between themes. Data collection continued until the authors felt that sufficient data had been collected to enable a coherent and logical conceptual understanding of the process under study.53 A synthesis of results was shared in writing with a representative group of participants to ensure that the results were consistent with the research questions and their experience and perspectives.
To achieve adequate elaboration of the core thematic patterns we identified, we completed a total of 21 interviews with 10 nurse and 11 physician participants. Four of 21 participants (19%) were male, and 17/21 participants (81%) were female. Although they were not required to reveal their IAT results, 15/21 (71%) participants shared that their result demonstrated implicit dangerousness bias toward mental illness. Six of 11 physician participants and 2/10 nurse participants classified their results as being what they expected, while 4/11 physicians and 7/10 nurses classified their results as being unexpected. Nine of 11 (81%) of those with an unexpected result also acknowledged that their result demonstrated implicit dangerousness bias toward mental illness. Nineteen of 21 participants (90%) described the experience of taking the IAT as positive, neutral, or interesting, while only 1/21 described the experience as negative.
Acceptance versus justification
Most participants justified or rationalized their results as being due to experience. They described how external influences in both personal and professional domains shaped their implicitly held beliefs over time. One participant who was given a result that described implicit dangerousness bias suggested that their bias is a natural result of experiencing violent patients in their work as a pediatric emergency physician:
Well, for sure, it affects how I treat people. I mean, the 12-year-old kid who comes in snarling at me, spitting at me, lunging at me to bite me, because she has anxiety, depression, et cetera and that’s her way of acting out, is just going to elicit a different response from me than the 12-year-old who is sitting there, you know, cooperative for the exam and has a pneumonia or whatever. It’s just a different … you react to how you’re getting treated. (MD4)
Participants described a continuum of responses ranging from acknowledging their result to accepting it, criticizing it, and denying it. While some participants were generally quite accepting that they held implicit dangerousness bias in congruence with their IAT result, some participants were more critical and denied the validity of their result. Denial and criticism of the IAT were more prominent among individuals who had “unexpected” results. Some participants described the IAT as “silly” and suggested that the test was not helpful (MD5) or accurate (MD1). One nurse stated:
No, like I know who I am and that’s not going to impact anything about me. (RN9)
Idealized versus actual personal and professional identity
There was a prominent sense of tension in responses to the IAT across all groups. Tension was more evident in participants who had unexpected results. In addition to the tension between acceptance and justification, two additional tensions emerged: the difference between participants’ actual and ideal self-perceptions, and the difference between attitudes demonstrated in personal versus professional contexts.
Many participants shared that they were unhappy with their IAT result because it was not consistent with their self-perceptions. Some of these participants described being disappointed and seemed to accept their result while struggling with what their result meant for their desired self-image. Others suggested that while they found the test “disturbing,” their experience “opened some ideas” and caused them to examine some of their “deepest feelings” (RN4). Another acknowledged that they know themselves yet simultaneously acknowledged that they might have negative attitudes toward individuals who are not compliant with their medications (RN3). One participant shared that the result conflicted with a “fantasy” of themselves:
Everyone has expectations, right? And everyone has sort of this fantasy of themselves and they always think, you know, maybe they should be better or they are better than they are, right? In reality they’re aspiring to a higher state of being. (MD8)
Others shared how the experience of the test highlighted the discrepancy between
personal and professional contexts. Specific IAT questions about encountering individuals with mental illness outside of a hospital setting provoked reflection and discussion about the differences between personal and professional experiences, attitudes, and behaviors. Some participants expressed that as a physician or nurse, they are expected not to demonstrate bias and that biased attitudes and behaviors are inconsistent with their professional identity:
I just feel like I’m fairly open-minded and eager to learn. I don’t want to find myself seeing people and prejudging. It’s not a role that I think is healthy in my own personal life, and I know it’s not a healthy role in the work that I do. (RN8)
If I think where I am coming from and how I was raised and now I am a physician, I am sure that I try very hard not to be biased. (MD9)
In contrast with professional identity, most participants acknowledged that their implicit bias was influenced by their personal life experiences and relationships. One participant suggested that they were more “closed” to mental illness because of personal experiences with loved ones who have suffered (MD11). Another suggested that they have actively engaged in egalitarian activities in their personal lives in response to an upbringing that may have fostered implicit bias (MD9). A nurse participant who acknowledged that their result showed a “moderate” degree of implicit dangerousness bias stated that they would not let their children play on the sidewalk if there was a mentally ill patient who lived on their street (RN1). Another acknowledged that after having children, they were more conscious of individuals with mental illness in public and that they would not fear for their own safety, but that of their child (RN4):
In order for me to be a good nurse, to be a good advocate, it’s important for me to examine my deepest darkest feelings of mental illness … there is a part of me having worked in the emergency department that knows that there are people with mental illnesses who could be aggressive and dangerous. You know what I mean it’s hard, because I want to say of course I would let her [my daughter] walk by [people with mental illness] and I mean people with a broken leg or whatever could be aggressive and dangerous and things like that. So yeah so there you go. (RN4)
Desire to change, while acknowledging that change is difficult
When discussing the influence of their IAT result on future attitudes, most participants felt that changing implicit attitudes was necessary but difficult. Several participants suggested that implicit biases about mental illness have become “embedded” through experience and were therefore difficult to overcome:
And part of the unconscious part of it is also, like personally like what you’ve grown up seeing, hearing, you know that kind of gets embedded in it, and I think to overcome that you have to be very active and diligent about it. I think it’s very hard because, especially if you’ve had 25 years, right, from a physician’s perspective before you start training, so to deconstruct all those stigmas, I mean that takes a while, and it has to be done, you have to actively seek it out and do it. (MD1)
Sometimes a certain way of thinking becomes so engrained in our mind and psyches that we don’t question it, you see, and that’s the dangerous bit … when we become so engrained in something that we think this is the only right way … therefore I’m doing it right and my attitude is right and my opinion is the right one and not be willing to change and to, you know, to reconsider your norms, your standards, your values, your beliefs. (MD8)
Changing implicit attitudes was not something participants described as being easily accomplished:
I think there has to be a strong drive to want it to change.… I think there has to be an openness and a willingness to address the fact that you do have them [biases] and that you understand that they might be wrong. And then I don’t know how you would go about changing them but I think addressing them or sort of acknowledging that they’re there and that it’s a problem is probably the first step. (MD11)
Some suggested that bringing implicit attitudes into conscious awareness was the first step which could trigger “constant self-feedback and self-evaluation” (MD8) and that openness to express their feelings would facilitate improved reflection and self-awareness (MD10). Several participants described the experience as positive and a worthwhile endeavor, as one comment illustrates:
It was a little shocking to me.… It was disturbing, because it opened up some ideas about myself that I don’t like to think that are there … So it was very good, it is really worthwhile, because it causes you [to] examine some of your deepest feelings and some of the things that you don’t perhaps like to recognize about yourself. (RN4)
The multiple tensions identified in our analysis of health professionals’ responses to IAT results resonate with existing literature on methods to facilitate the acceptance and integration of emotionally charged feedback. Figure 1 provides a visual depiction of our key findings. Our findings provide insights relating to personal and professional identity that merit further discussion.
Difficult and emotional nature of IAT feedback
Kluger and Denisi’s54 feedback intervention theory emphasizes that the effectiveness of any feedback intervention depends on the level (task learning, task motivation, or self) at which the intervention focuses our attention. In accordance with Kluger and Denisi’s hierarchy principle, if the feedback intervention addresses high-level processes related to the learners’ beliefs about themselves, this may produce a strong emotional reaction such as despair, disappointment, or elation that interferes with task performance.55–59 The work of Sargeant and others28 suggests that, when discerning whether feedback is negative or positive, physicians compare this feedback with global self-perceptions of performance. Because we know that any feedback information about the “self” is emotionally charged and being objective about it is difficult,60 reactions to negative feedback are influenced by perceptions of accuracy and credibility of data and usefulness of feedback. Feedback leads to shame when the recipient internalizes negative evaluation of self and shamed individuals feel inferior and defective.61,62 While existing literature on feedback at the self level would suggest that the IAT may be a particularly challenging tool to provide feedback information, our participants largely engaged with and reflected on their IAT results.
Our participants expressed tension between accepting their bias and justifying attitudes that are embedded through experience. Our participants wanted to believe that they did not hold implicit dangerousness bias; however, they struggled with accepting feedback that conflicted with the professional they wished to be. Unlike previous research that has demonstrated tension when negative feedback does not align with self-appraisal, our participants were challenged with feedback that created tension between their actual and aspirational identities.
Unique findings from IAT provide insights on how to manage reaction to negative feedback
Existing research suggests that emotional reactions to charged feedback need to be managed through guidance and facilitation. Facilitated reflection can contribute to assimilation and acceptance of negative and emotionally charged feedback.29 Reflection can facilitate the difficult process of reconciling tensions within one’s self through explicit recognition of the emotional aspect of receiving feedback and the difficulty of reconciling self-appraisal with contradictory feedback.63
While participants’ responses did not specifically mention guided reflection, the finding of prominent tension related to actual and idealized professional identity provides guidance on how to inform facilitated reflection related to challenging or emotional feedback. Feedback about an individual’s implicit bias is an opportunity to explore how health professionals negotiate and navigate information discordant with a fantasy version of themselves as free from prejudice and stereotyping. Health professional identity views bias and stereotyping as antithetical to the values that comprise the “ideal” health professional.64–66 Indeed, many of our participants described aspiration toward a “fantasy” of themselves as a “higher state of being,” as informing their responses to IAT feedback. Further research is needed to define how health professionals reconcile feedback information discordant from the idealized and unattainable aspects of their professional identity.
Openness to change is challenging but possible
Despite the challenges related to implicit bias recognition, we were struck that only 1 out of our 21 participants found the experience of receiving IAT results to be negative. This finding contrasts with literature on negative and emotional feedback related to the self. We anticipated that more participants would have identified the IAT as distressing or unpleasant. While our sample size was limited, there may be several explanations for this finding that have implications for future research.
Among the many variables associated with feedback such as task nature, source, context, complexity, or form,67 credibility is often a key factor in determining whether feedback is influential.31,68 Our study provided feedback through a computer-based test with a simple message; however, we also immediately provided an opportunity for reflection through a qualitative interview designed to foster open disclosure by a researcher who was explicitly focused on creating a nonjudgmental setting to explore participant responses. Several of our interviews reflected a developmental trajectory from denial to defense, minimization, and eventually acceptance, which has been previously described in the literature.8
Another possibility is that participants were influenced by social desirability bias.69 We found that participant responses reflected a desire to conform with an ideal professional identity. Given the taboo associated with prejudice and bias, there may have been a tendency to appear reflective and accepting of feedback while minimizing defenses. Further research that focuses on how attributes of the feedback source relate to the personal and professional identities of all parties involved in feedback conversations may help elaborate on these possibilities. Perhaps our participants noted that change was possible in the presence of the interviewer, yet did not integrate the feedback information into future behavioral change. Further longitudinal data may also improve our understanding of how feedback related to implicit biases changes over time.
The role of the IAT and controversies regarding its use
Our findings regarding the use of the IAT should be interpreted in the context of mixed views regarding its use and the issue of implicit bias. Critics suggest that an online IAT does not demonstrate an accurate or authentic representation of individually held biases.70 The IAT can only measure relative associations and is not a definite measure of implicit biases.71 For example, our participants were given a measure of their associations between “dangerousness” relative to mental or physical illness, whereas the concept of mental illness has multiple nuances and groups several distinct illness categories into a broad sweeping generalization. While some have proposed the implicit relational assessment procedure (IRAP) as a tool to address flaws in the IAT, the IRAP is strongly influenced by how tasks are framed to participants, and newer procedures are emerging in the social psychology literature.72
Despite these critiques, we did not use the IAT as a definitive metric but, rather, as a tool to prompt reflection and discussion. Therefore, we believe that our study emphasizes that there is potential for the IAT to provide a window into individual biases to facilitate change. As assessment science progresses toward advanced measures of implicit bias, our findings may therefore further the application of implicit bias recognition and management to health professions curricula. Although other types of negative feedback have been found to be perceived as unnecessarily confrontational or challenging, our study demonstrates the potential for feedback related to implicit bias to inform further research and exploration on promoting acceptance and integration of emotionally laden feedback.
Because we explicitly informed participants to not reveal their IAT results, this raised the possibility that our participants were given information that they either had implicit dangerousness bias or did not. We reconciled this information by coding our transcripts according to what participants shared regarding whether they found their results to be expected or unexpected, and conducting a supplemental analysis that confirmed that most participants received an IAT result that revealed implicit dangerousness bias. Further, there is some controversy regarding the use of the IAT and whether results represent authentic biases or associations from experience that are shaped by forces such as victimization or oppression. Our study also drew exclusively from a limited pediatric sample, and we acknowledge that there may be aspects of the professional identity of pediatricians that uniquely contributed to our results. Also, the demographic breakdown of our sample included a limited number of male participants, and therefore further exploration of our findings with demographically diverse participants and medical specialties is needed for the future.
Health professionals’ responses to feedback about their implicit bias reveals insights related to the processing and integration of emotionally charged feedback information that creates tension between actual and ideal professional identity. Reconciling this tension and reflecting on IAT results offers a promising approach to incorporate implicit bias recognition and management into health professions education.
The authors would like to acknowledge first and foremost the study participants. Further acknowledgments include Rod Lim, Paul Atkison, Alicia Cooper, Christina Scerbo, and Kristina Miller for their support related to this work.
1. Green AR, Carney DR, Pallin DJ, et al. Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients. J Gen Intern Med. 2007;22:12311238.
2. Sabin JA, Greenwald AG. The influence of implicit bias on treatment recommendations for 4 common pediatric conditions: Pain, urinary tract infection, attention deficit hyperactivity disorder, and asthma. Am J Public Health. 2012;102:988995.
3. Chapman EN, Kaatz A, Carnes M. Physicians and implicit bias: How doctors may unwittingly perpetuate health care disparities. J Gen Intern Med. 2013;28:15041510.
4. Dovidio JF, Kawakami K, Gaertner SL. Implicit and explicit prejudice and interracial interaction. J Pers Soc Psychol. 2002;82:6268.
5. Fazio RH, Jackson JR, Dunton BC, Williams CJ. Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? J Pers Soc Psychol. 1995;69:10131027.
6. Sukhera J, Miller K, Milne A, et al. Labelling of mental illness in a paediatric emergency department and its implications for stigma reduction education. Perspect Med Educ. 2017;6:165172.
7. Burgess D, van Ryn M, Dovidio J, Saha S. Reducing racial bias among health care providers: Lessons from social-cognitive psychology. J Gen Intern Med. 2007;22:882887.
8. Teal CR, Gill AC, Green AR, Crandall S. Helping medical learners recognise and manage unconscious bias toward certain patient groups. Med Educ. 2012;46:8088.
9. Stone J, Moskowitz GB. Non-conscious bias in medical decision making: What can be done to reduce it? Med Educ. 2011;45:768776.
10. Burgess DJ, Fu SS, van Ryn M. Why do providers contribute to disparities and what can be done about it? J Gen Intern Med. 2004;19:11541159.
11. van Ryn M, Hardeman R, Phelan SM, et al. Medical school experiences associated with change in implicit racial bias among 3547 students: A Medical Student CHANGES Study report. J Gen Intern Med. 2015;30:17481756.
12. Boysen GA. Integrating implicit bias into counselor education. Couns Educ Superv. 2010;49:210227.
13. Hillard AL, Ryan CS, Gervais SJ. Reactions to the implicit association test as an educational tool: A mixed methods study. Soc Psychol Educ. 2013;16:495516.
14. Jackson SM, Hillard AL, Schneider TR. Using implicit bias training to improve attitudes toward women in STEM. Soc Psychol Educ. 2014;17:419438.
15. Morris KA, Ashburn-Nardo L. The Implicit Association Test as a class assignment: Student affective and attitudinal reactions. Teach Psychol. 2009;37:6368.
16. Teal CR, Shada RE, Gill AC, et al. When best intentions aren’t enough: Helping medical students develop strategies for managing bias about patients. J Gen Intern Med. 2010;25(suppl 2):S115S118.
17. Frantz CM, Cuddy AJ, Burnett M, Ray H, Hart A. A threat in the computer: The race implicit association test as a stereotype threat experience. Pers Soc Psychol Bull. 2004;30:16111624.
18. Monteith MJ, Voils CI, Ashburn-Nardo L. Taking a look underground: Detecting, interpreting, and reacting to implicit racial biases. Soc Cogn. 2001;19:395417.
19. O’Brien LT, Crandall CS, Horstman-Reser A, Warner R, Alsbrooks A, Blodorn A. But I’m no bigot: How prejudiced white Americans maintain unprejudiced self-images. J Appl Soc Psychol. 2010;40:917946.
20. Howell J, Ratliff K. Implicit-Explicit Attitude Discrepancy Prompts Defensive Responding to IAT Feedback [doctoral dissertation]. 2014.Gainesville, FL: University of Florida
21. Howell JL, Gaither SE, Ratliff KA. Caught in the middle: Defensive responses to IAT feedback among whites, blacks, and biracial black/whites. Soc Psychol Personal Sci. 2015;6:373381.
22. Crandall CS, Eshleman A, O’Brien L. Social norms and the expression and suppression of prejudice: The struggle for internalization. J Pers Soc Psychol. 2002;82:359378.
23. Epton T, Harris PR, Kane R, van Koningsbruggen GM, Sheeran P. The impact of self-affirmation on health-behavior change: A meta-analysis. Health Psychol. 2015;34:187196.
24. Sherman DK, Cohen GL. The psychology of self-defense: Self-affirmation theory. Adv Exp Soc Psychol. 2006;38:183242.
25. Fidler H, Lockyer JM, Toews J, Violato C. Changing physicians’ practices: The effect of individual feedback. Acad Med. 1999;74:702714.
26. Lockyer J, Violato C, Fidler H. Likelihood of change: A study assessing surgeon use of multisource feedback data. Teach Learn Med. 2003;15:168174.
27. Bennett H, Gatrell J, Packham R. Medical appraisal: Collecting evidence of performance through 360 feedback. Clin Manag. 2004;12:165172.
28. Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract. 2008;13:275288.
29. Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF. Reflection: A link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract. 2009;14:399410.
30. Sargeant J, McNaughton E, Mercer S, Murphy D, Sullivan P, Bruce DA. Providing feedback: Exploring a model (emotion, content, outcomes) for facilitating multisource feedback. Med Teach. 2011;33:744749.
31. Watling CJ, Lingard L. Toward meaningful evaluation of medical trainees: The influence of participants’ perceptions of the process. Adv Health Sci Educ Theory Pract. 2012;17:183194.
32. Charmaz K. Constructing Grounded Theory: A Practical Guide Through Qualitative Research. 2006.London, UK: Sage Publications Ltd.
33. Greenwald AG, McGhee DE, Schwartz JL. Measuring individual differences in implicit cognition: The implicit association test. J Pers Soc Psychol. 1998;74:14641480.
34. Bosson JK, Swann WB Jr, Pennebaker JW. Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? J Pers Soc Psychol. 2000;79:631643.
35. Dasgupta N, McGhee DE, Greenwald AG, Banaji MR. Automatic preference for white Americans: Eliminating the familiarity explanation. J Exp Soc Psychol. 2000;36:316328.
36. Greenwald AG, Nosek BA, Banaji MR. Understanding and using the implicit association test: I. An improved scoring algorithm. J Pers Soc Psychol. 2003;85:197216.
37. Greenwald AG, Farnham SD. Using the implicit association test to measure self-esteem and self-concept. J Pers Soc Psychol. 2000;79:10221038.
38. Nosek BA, Greenwald AG, Banaji MR. Understanding and using the Implicit Association Test: II. Method variables and construct validity. Pers Soc Psychol Bull. 2005;31:166180.
39. Nosek BA, Greenwald AG, Banaji MR. The Implicit Association Test at age 7: A methodological and conceptual review. Autom Process Soc Think Behav. January 2007:265292.
40. Steffens MC. Is the implicit association test immune to faking? Exp Psychol. 2004;51:165179.
41. Blanton H, Jaccard J, Christie C, Gonzales PM. Plausible assumptions, questionable assumptions and post hoc rationalizations: Will the real IAT, please stand up? J Exp Soc Psychol. 2007;43:399409.
42. Andreychik MR, Gill MJ. Do negative implicit associations indicate negative attitudes? Social explanations moderate whether ostensible “negative” associations are prejudice-based or empathy-based. J Exp Soc Psychol. 2012;48:10821093.
43. Banaji MR, Nosek BA, Greenwald AG. No place for nostalgia in science: A response to Arkes and Tetlock. Psychol Inq. 2004;15:279289.
44. Uhlmann EL, Brescoll VL, Paluck EL. Are members of low status groups perceived as bad, or badly off? Egalitarian negative associations and automatic prejudice. J Exp Soc Psychol. 2006;42:491499.
45. Nosek BA, Hansen JJ. The associations in our heads belong to us: Searching for attitudes and knowledge in implicit evaluation. Cogn Emot. 2008;22:553594.
46. Thompson C, Dalgleish L, Bucknall T, et al. The effects of time pressure and experience on nurses’ risk assessment decisions: A signal detection analysis. Nurs Res. 2008;57:302311.
47. Gunnarsson BM, Warrén Stomberg M. Factors influencing decision making among ambulance nurses in emergency care situations. Int Emerg Nurs. 2009;17:8389.
48. Chen EH, Shofer FS, Dean AJ, et al. Gender disparity in analgesic treatment of emergency department patients with acute abdominal pain. Acad Emerg Med. 2008;15:414418.
49. Cone DC, Richardson LD, Todd KH, Betancourt JR, Lowe RA. Health care disparities in emergency medicine. Acad Emerg Med. 2003;10:11761183.
50. Beach C, Croskerry P, Shapiro M; Center for Safety in Emergency Care. Profiles in patient safety: Emergency care transitions. Acad Emerg Med. 2003;10:364367.
51. Croskerry P. Cognitive forcing strategies in clinical decisionmaking. Ann Emerg Med. 2003;41:110120.
52. Lehmann JB, Wehner PS, Lehmann CU, Savory LM. Gender bias in the evaluation of chest pain in the emergency department. Am J Cardiol. 1996;77:641644.
53. Morse JM. The Significance of Saturation. 1995.Thousand Oaks, CA: Sage Publications
54. Kluger AN, Denisi AS. The effects of feedback interventions on performance: A historical review, a meta-analysis and a preliminary feedback intervention theory. Psychol Bull. 1994;119:254284.
55. Carver CS, Scheier MF. Origins and functions of positive and negative affect: A control-process view. Psychol Rev. 1990;97:19.
56. Brett JF, Atwater LE. 360 degree feedback: Accuracy, reactions, and perceptions of usefulness. J Appl Psychol. 2001;86:930942.
57. Atwater LE, Waldman DA, Brett JF. Understanding and optimizing multisource feedback. Hum Resour Manage. 2002;41:193208.
58. Smither JW, London M, Reilly RR. Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Pers Psychol. 2005;58:3366.
59. Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract. 2012;17:1526.
60. Ashford SJ, Blatt R, Walle DV. Reflections on the looking glass: A review of research on feedback-seeking behavior in organizations. J Manag. 2003;29:773799.
61. Bynum WE 4th.. Filling the feedback gap: The unrecognised roles of shame and guilt in the feedback cycle. Med Educ. 2015;49:644647.
62. Tangney JP. Conceptual and methodological issues in the assessment of shame and guilt. Behav Res Ther. 1996;34:741754.
63. Mann K, van der Vleuten C, Eva K, et al. Tensions in informed self-assessment: How the desire for feedback and reticence to collect and use it can conflict. Acad Med. 2011;86:11201127.
64. Notzer N, Soffer S, Aronson M. Traits of the “ideal physician” as perceived by medical students and faculty. Med Teach. 1988;10:181189.
65. Arkes HR, Tetlock PE. Attributions of implicit prejudice, or “would Jesse Jackson ‘fail’ the Implicit Association Test?” Psychol Inq. 2004;15:257278.
66. Westgate E, Riskind R, Nosek B. Implicit preferences for straight people over lesbian women and gay men weakened from 2006 to 2013. Collabra Psychol. 2015;1:110.
67. van de Ridder JM, McGaghie WC, Stokking KM, ten Cate OT. Variables that affect the process and outcome of feedback, relevant for medical training: A meta-review. Med Educ. 2015;49:658673.
68. Watling C, Driessen E, van der Vleuten CP, Vanstone M, Lingard L. Beyond individualism: Professional culture and its influence on feedback. Med Educ. 2013;47:585594.
69. Nederhof AJ. Methods of coping with social desirability bias: A review. Eur J Soc Psychol. 1985;15:263280.
70. Singal J. Psychology’s favorite tool for measuring racism isn’t up to the job. N Y Mag. January 11, 2017. http://nymag.com/scienceofus/2017/01/psychologys-racism-measuring-tool-isnt-up-to-the-job.html
. Accessed October 29, 2017.
71. Blanton H, Jaccard J. Arbitrary metrics in psychology. Am Psychol. 2006;61:2741.
72. O’Shea B, Watson DG, Brown GD. Measuring implicit attitudes: A positive framing bias flaw in the Implicit Relational Assessment Procedure (IRAP). Psychol Assess. 2016;28:158170.