The Institute of Medicine has suggested that a culturally sensitive approach to care that avoids bias, prejudice, and stereotyping may help reduce health care disparities.1 Multiple frameworks have been proposed to improve cultural competence among health care providers.2–10 Cultural competence training has covered many different topics, has taken any number of different shapes, and has been shown to improve provider knowledge, attitudes, and skills, although patient outcomes are not frequently assessed.11,12 However, no clear consensus is available for the organization of content that ought to be included in a cultural competence curriculum for physicians or physicians-in-training.
Processes more frequently employed in the social sciences have been useful in gathering and organizing data that can shape educational products. For instance, nominal group technique (NGT)13,14 offers advantages over traditional focus-group discussions by allowing equal participation, avoiding participants' inhibition in informal group discussion, avoiding focusing on one particular area, reducing apprehension from public speaking, generating a greater number of ideas, and minimizing inefficiencies and process loss.15–17 NGT has been used in medical education to assess trainees' perceptions of successful teaching rounds,18 to evaluate curricula,19 and to identify key attributes of an ideal continuing medical education (CME) institution to create a roadmap for excellence,20 among others. In medical practice, NGT has been used to identify intervention strategies to improve osteoporosis treatment in nursing homes,21 to identify reasons for lack of medication adjustment in patients with seemingly uncontrolled hypertension,22 to identify barriers for diabetes self-management,17 and to achieve consensus among stakeholders in guidelines development.23 Importantly, NGT does not limit participants to a predetermined framework; rather, it forces participants to consider any and all options.
In addition, grouping methods allow the visual representation of mental processes, such as the data gathered during NGT, for instruction. The goal is to reveal the underlying cognitive dimensions, or “cognitive map,” that describes the spatial representation of how the items generated during NGT are related to each other and how ideas are prioritized within each cluster.24–27 Card-sorting, hierarchical cluster analysis (HCA), and multidimensional scaling (MDS) have been used to organize information and understand group similarities,28–32 to understand consumer behavior,33 to design the contextual navigation of Web sites,28,34,35 to provide insight on how journals are cognitively organized,26 and to understand motivation in schools.36 However, to our knowledge, such methods have not been used in medical education research.18,37
We set out to inform the development and organization of a cultural competence curriculum that would increase health care professionals' knowledge of health disparities and fundamental issues of cultural competence as a step in enhancing cardiovascular care for minority groups. We did so by engaging groups of relevant stakeholders (including cultural competence experts, physicians, and physicians-in-training) in a robust, formative assessment to identify, prioritize, and organize the critical components of cultural competence.22 The combined approach we describe is a novel application in medical education research; the qualitative and quantitative approach is specifically designed to guide the process of content development and organization of information.
We report below the development of the Cultural Competence Curriculum program, an Internet-based educational intervention for physicians and physicians-in-training aimed at reducing health disparities in cardiovascular health.
Information gathering: NGT
First, we used NGT to generate and prioritize a list of concepts deemed important to include in the curriculum. As explained earlier, NGT is a highly structured, small-group-meeting approach used to elicit and organize answers to a specific question.18 NGT involves the following steps: (1) silent, written generation of responses to a specific question, (2) round-robin recording of ideas, (3) discussion for clarification only, and (4) voting on the rank order of the top three ideas, usually with respect to importance, feasibility, or other characteristics. (These steps are described in much greater detail elsewhere18.) Figure 1 is a graphical description of the question development, the NGT groups, and how the responses were processed to the question that was developed.
To formulate the central question for our NGT sessions, we convened a panel of clinician–educators, nurses, minority community physicians, minority health researchers, health services researchers, social scientists, and distance learning experts. The panel reviewed the literature and composed and pilot-tested the question among medical students, faculty members, and a community physician.
Between February and March 2006, we conducted four separate NGT sessions with medical students (n = 9), internal medicine residents (n = 7), primary care community physicians (n = 7), and disparities or educational researchers (n = 7). The small sample size was by design and is justified, given NGT methodology where the process allows for idea saturation quickly with smaller numbers of participants, provided the demographic composition of the groups broadly represents the target end-user. We tried to have participants who represented a mixture of age, gender, and ethnicity; however, by design, we did not ask or collect information on such characteristics, because such inquiries could have influenced the responses during the NGT sessions.
We recruited two convenience samples of fourth-year medical students and internal medicine house officers from the University of Alabama at Birmingham, Birmingham, Alabama. We recruited community physicians from the Alabama Practice-Based CME Network, a group in regular communication with the academic medical center, for purposes of exploring educational need in the community and easy delivery of educational materials and activities addressing those needs. Participants were recruited via fax and awarded a $100 honorarium. Finally, we recruited investigators funded by the National Heart, Lung, and Blood Institute.38 NGT participants were informed that the purpose of the session was to generate their ideas concerning curricular content, not format, for a cultural competence program focusing on cardiovascular disease risk and treatment; the program would be available to medical students, residents in training, and practicing physicians.
The final question formulated by the panel and posed to NGT participants was “What sorts of things could be included in a curriculum that focuses on cultural competence training for physicians?” Once all responses to the question had been given, each group was asked to identify the three most important ideas and the three most feasible ideas that the responses contained. Experienced facilitators conducted the sessions.
Organizing the ideas: Card sort
The ideas from the NGT sessions that received at least two votes as “most important” or “most feasible” were then organized by a group consisting of a convenience sample of 45 educators and disparities researchers. These participants physically grouped and ranked the ideas, which were written on index cards. These educators and researchers hailed from 19 cities and 14 states in the United States. There were 29 general internal medicine physicians, 1 pediatrician, 3 specialists, 4 nurses, 4 PhDs, and 4 from other professions. Of these participants, 26 were women and 19 were men; there were 24 Caucasians, 8 African Americans, 6 Hispanics, 3 Asians, 3 unknown, and 1 “other.”
Each participant worked independently for 10 to 20 minutes to
- examine the cards together as a deck,
- sort them into piles based on their own perceptions or criteria for how they should be grouped,
- create at least 2 but no more than 10 piles,
- place cards that were not in some way related to each other into a pile labeled “cards cannot be grouped,”
- note whether two or more ideas reflected the same concept, and
- make no attempt of assigning a label to each pile.
Last, within each pile, participants ranked the ideas, again based on their own criteria, by placing the most important idea at the top of the pile and the least important idea at the bottom of the pile. We did not provide an incentive or honoraria.
Information analysis: Cognitive mapping
To analyze the results of the card sort, we used MDS and HCA.22 The goal of MDS is to reveal the psychological dimensions in the data that can meaningfully describe the underlying cognitive constructs. The result, a “cognitive map,” is a spatial representation of how the ideas are considered to be similar to or different from each other.26,39 The coordinates defining each item's position from the MDS analysis are used as input data in the cluster analysis. The goal of HCA is to facilitate the interpretation of the MDS map by dividing a set of objects into a smaller number of homogeneous groups on the basis of their similarity.40 Enclosed within lines, the smaller homogeneous groups represent “families,” and clusters of groups represent “neighborhoods.”41 The map in our study reflected how the card sorters cognitively organized the areas to include in the cultural competency training curriculum for physicians.
The data from the card-sorting task were first used to create a matrix for each participant. This co-occurrence matrix indicated the number of times each item was sorted with all other items (29 × 29 matrix). The individual co-occurrence matrices were then aggregated across all participants to form a group co-occurrence matrix. This matrix indicated the number of times all participants sorted each reason into similar or dissimilar piles. The group co-occurrence matrix was then analyzed with MDS (using the ALSCAL algorithm) and with cluster analysis to derive the cognitive map. This iterative process derived an optimal geometric solution that reflects the proximity of the data and the relative ordering from the distances observed in the matrix.24
The number of dimensions in an MDS solution is determined by its interpretability and analysis. Perhaps the most important is the interpretability. A two-dimension solution is usually recommended because additional dimensions make interpretation more difficult.41 MDS analysis results in the R-square (RSQ) and stress statistics. The RSQ statistic reflects the extent to which the map corresponds to the actual proximity data, and values of RSQ >.90 suggest high correspondence. The stress statistic reflects the “badness of fit,” with acceptable stress values being >.01 and <.20 (<.01 indicates a degenerate solution, >.20 indicates poor fit).40,41
Although MDS and cluster analysis are computationally sophisticated, they are not based on parametric statistics. Consequently, their validity is not sample-size dependent, but dependent on the representativeness of participants in the sorting task. Whereas the location of items on the map is analytically determined, the meaning of the dimensions (i.e., the two axis labels) must be interpreted. We used a standard approach to interpret the derived dimensions by engaging the entire research team and by examining the substantive differences between items at the extremes of the dimensions.
The institutional review board at The University of Alabama at Birmingham approved this study.
The results of the NGT and the cognitive mapping are shown in Table 1 and Figure 2.
Nominal group technique
During the four independent NGT sessions, a total of 61 ideas were generated. Twenty-nine of these ideas were selected by at least two participants as most important or feasible; they are all listed in Table 1. The other 32 ideas were discarded and not incorporated into the resulting Web-based program's content (a list of these discarded ideas is presented in Supplemental Digital Table 1, which may be found at http://links.lww.com/ACADMED/A41). As shown in Table 1, the top-rated idea among experts was to “include content to help learners become aware of [their] own biases and attitudes …” (item 17, sum of the ratings = 9); among community physicians, the two top-rated ideas were to “describe habits that are unique …” (item 13, sum of the ratings = 7) and “present objective data on the cultures …” (item 12, sum of the ratings = 7); among residents, the top-rated idea was to “provide access to resources …” (item 6, sum of the ratings = 6); and, among medical students, the top-rated idea was to “identify critical questions that would permit taking cultural history …” (item 1, sum of the ratings = 17). Table 1 shows the organization of the 29 ideas selected by at least two participants as most important or feasible, classified by family (i.e., small group).
Organization and analysis of ideas: The cognitive map
The 45 clinician–educators and health disparities researchers organized the 29 ideas in a median of three piles (range 1–11); each pile had a median of 6 ideas (range 3–10).
The MDS and HCA revealed two possible solutions, on two or three dimensions. Although the three-dimension solution better explained the data than did the two-dimension solution (higher RSQ, 0.97 versus 0.91; lower stress measure, 0.07 versus 0.15), we present here the two-dimension solution, because it was the most parsimonious and interpretable.
The cluster analysis revealed 10 small groups, or families, of cognitively most-similar items, each within four larger groups, or neighborhoods, of cognitive similarity. The decision to interpret a 10-cluster solution was informed by examining the pattern of the agglomeration coefficients and by visually inspecting the two-dimension plot and the dendrogram to determine the sequence of cluster formation. The cognitive map resulting from the MDS and cluster analysis is shown in Figure 2.
We propose that the four larger groups can be viewed as neighborhoods of related themes: (1) patient's cultural background (i.e., information on cultures, habits, values), (2) provider and health care (e.g., clinical skills, awareness of one's bias, patient centeredness, and professionalism), communication skills (e.g., history, stereotype avoidance, and health disparities epidemiology), (3) cross-culture (e.g., idiomatic expressions, examples of effective communication), and (4) resources to manage cultural diversity. These neighborhoods are shown in Table 1 and Figure 2, which give a fuller picture of them than could be presented in this paragraph. We base our interpretation of the mapping results on the substantive orientation and thematic consistency of the ideas within these clusters of families and among neighborhoods.
The patient's background neighborhood includes three distinct families: cultural background (e.g., religion, customs, moral values), culture and health (e.g., impact of religion and culture behavior and adherence to cultural standards), and social background (e.g., health patterns, views on complementary and alternative medicine, region of the country).
The provider and health care neighborhood includes three families: clinical skills (e.g., awareness of one's bias, patient centered, professionalism), communication skills (e.g., history, stereotype avoidance), and health disparities epidemiology (e.g., prevalence, causes, medications).
The cross-culture neighborhood includes two families: cross-cultural communication (e.g., idiomatic expressions, barriers, effective communication examples) and cross-cultural resources.
The resources to manage cultural diversity neighborhood consists of two families: translator guides/instructions and community resources.
In Figure 2, we conceptualize the x-axis as a dimension of patient centeredness (Patient Centrism) with a neighborhood of issues more intrinsic to the patient (Patient's Background, on the left side of the figure) and a neighborhood of issues more extrinsic to the patient (Resources, on the right side). Similarly, we conceptualize the y-axis as a dimension of health care centeredness (Provider and Health System Centrism) with a neighborhood of issues more intrinsic to health care (Provider/Health Care, on the upper portion of the figure) and a neighborhood of issues more extrinsic to health care (Resources and Cross-Culture, on the lower portion of the figure).
We arrived at these interpretations by examining the ideas at the extremes on each axis to determine similarities and differences. For instance, with respect to the dimension of patient centeredness (the Patient Centrism axis), differences between the two items farthest apart were relatively obvious. Item 8 (“Provide a list of available translator services”) is not under the control of the patient or would be the same across all patients. Item 15 (“Provide information on religious principles of particular cultures and how these might influence medical care”) is very individualized to the patient and not the same from patient to patient. However, this approach was more difficult in the case of the dimension of health care centeredness (the Provider/Health System Centrism axis), where the variance between extremes was not as great. Consequently, we had to examine other items near the extremes to find meaning. Ultimately, we concluded that the items near the extremes on this axis also related to the level of control of the physician or health care system.
Importantly, the Patient's Background and the Provider/Health Care neighborhoods seem to be the most important, as illustrated by the higher overall sums of ratings of 56 and 55, respectively, compared with the Cross-Culture or Resources neighborhoods, with sums of ratings of 25 and 23, respectively.
As explained earlier, in a formative evaluation of concepts important and feasible for inclusion in a cultural competence curriculum, medical students, residents, community physicians, and academic physicians identified and organized 29 ideas along the two axes of patient centeredness and provider/health care system centeredness (illustrated in Figure 2). The resulting cognitive map was interpreted to include four neighborhoods of concepts: patient's background, provider/health care, cross-culture, and resources to manage cultural diversity; each of them contained a family of ideas related to the neighborhood's concept. We extended the methods and analytical approach from the social sciences to effectively outline a curriculum for cultural competence training. Our goal with this process was not to identify new areas of content in cultural competence (although, in our study, religion emerged as an important dimension not specifically defined in prior frameworks42) but, rather, to understand in a more structured fashion the ways in which target groups conceptually framed and organized existing cultural competence content.
Implications for curriculum development and organization
The organization and prioritization of information is critical for mental processing, learning, and problem solving, that is, learning with understanding. The concept of learning with understanding is fully supported by a body of evidence summarized in the National Research Council's seminal document How People Learn:Brain, Mind, Experience, and School.43 Learners gain a deeper understanding of a theme when they organize ideas, understand facts and ideas in the context of a conceptual framework, and organize knowledge in ways that facilitate retrieval and application.
An approach similar to ours can guide the process of content development, which includes the identification of learning needs as well as organization and prioritization of the content. Our study does not address which modality is best suited for curriculum delivery. However, the product of this formative assessment was, in fact, an online curriculum that our institution created entitled “Cultural Competence Online for Medical Practice” (www.c-comp.org), which is available in MedEdPortal.44 Within the Web site, teaching points are organized according to the dimensions and neighborhoods we discovered through this process. Furthermore, the overall importance of the neighborhoods (expressed as the overall sums of ratings) also informed the Web design. The size of the visual display of the content of the Patient's Background and the Provider/Health Care neighborhoods was twice as large as the Cross-Culture or Resources neighborhoods. Though we focused on the topic of cultural competence, a similar strategy may work well for others developing curricula in other topic areas, especially in areas that are ill-defined. Similarly, cognitive mapping can allow the inclusion of perspectives of critical stakeholders such as patients, providers, or health care systems.
We noted significant differences among groups with regard to the ideas rated highest in importance, perhaps stemming from the group members' own skills at their individual career stages. However, we recognize these differences as potential strengths for curriculum developers. These differences present an opportunity for tailoring content to particular learners or groups of learners based on their professional level. Their unique responses may also provide a foundation from which to review, refocus, or update materials, especially as culture shifts, people change, the health care system evolves, and resources become more or less available.
Uses of the knowledge gained in this study
The use of the NGT in curriculum development is emerging in medical education; however, the use of cognitive mapping methods is a novel application in medical education research. Building from the body of knowledge from this study, our group has used cognitive mapping methods, MDS, and HCA as key processes in medical education.18,45 The attributes that contribute to successful ward-attending teaching rounds identified by Castiglioni et al18 were organized using cognitive mapping methods from the faculty perspective45 and from the medical students' and residents' perspectives37—both published in abstract form. For example, Salanitro et al45 found that faculty grouped the attributes of successful rounds into six unique clusters, which were learning climate, explaining the clinical rationale, respect, managerial skills, teaching style and content, and setting expectations. Roy et al37 reported that trainees grouped the same attributes into five unique clusters, which were role modeling, learning environment, teaching process, team management, and expectations. The results of those two studies are guiding an ongoing faculty development program. A review of cognitive mapping methods in the social sciences is available elsewhere.25,27
We acknowledge that there may be other, even more meaningful, interpretations of our cognitive map that we did not consider. Though we convened a multidisciplinary group, the sample was relatively small and may not be representative for all U.S. trainees or practicing physicians, and there were certainly perspectives that were not brought to the table or may have been excluded from our expansive discussions. However, our data show great diversity of ideas among groups and sufficient overlap to conclude that we reached some level of saturation. We felt that the final agreed-on interpretation of the cognitive map best embodied the purpose and outcomes of our formative method.
Curriculum development is difficult, particularly in an area like cultural competence, where multiple frameworks exist. In this study, we set forth one approach for engaging learners, elucidating important topical domains, reaching some level of consensus, and organizing the information in a way that is both logical and helpful in curriculum development in cultural competence. Our approach illustrates one method, technique, or process that may be used by educators to identify content for initial development in any number of clinical or practice areas. It can also be used to revisit or reconsider that content over time for purposes of update or revision in light of new advances or improvements in care.
The authors thank Michael Schoen, PhD, Trudi Horton, PhD, Analia Castiglioni, MD, and Brita Roy, MD, MPH, for critical revisions of the manuscript.
This study was funded by a National Heart, Lung, and Blood Institute (K07 HL081373-01) grant to Dr. Estrada as part of the National Consortium for Multicultural Education for Health Professionals. The funding source did not make any restrictions on the scope or reporting of the study.
This study was approved by the institutional review boards of the University of Alabama at Birmingham.
The opinions expressed in this article are those of the authors alone and do not reflect the views of the Department of Veterans Affairs.
The abstract of an earlier version of this article was presented at the 2007 Southern Society of General Internal Medicine, New Orleans, Louisiana.
1Smedley BD, Stith AY, Nelson AR, eds; Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Washington, DC: National Academies Press; 2002.
2Lie D, Boker J, Cleveland E. Using the tool for assessing cultural competence training (TACCT) to measure faculty and medical student perceptions of cultural competence instruction in the first three years of the curriculum. Acad Med. 2006;81:557–564. http://journals.lww.com/academicmedicine/Fulltext/2006/06000/Using_the_Tool_for_Assessing_Cultural_Competence.11.aspx
. Accessed November 19, 2010.
3Betancourt JR, Green AR. Commentary: Linking cultural competence training to improved health outcomes: Perspectives from the field. Acad Med. 2010;85:583–585. http://journals.lww.com/academicmedicine/Fulltext/2010/04000/Commentary__Linking_Cultural_Competence_Training.14.aspx
. Accessed November 19, 2010.
4Smith WR, Betancourt JR, Wynia MK, et al. Recommendations for teaching about racial and ethnic disparities in health and health care. Ann Intern Med. 2007;147:654–665.
5Betancourt JR. Cross-cultural medical education: Conceptual approaches and frameworks for evaluation. Acad Med. 2003;78:560–569. http://journals.lww.com/academicmedicine/Fulltext/2003/06000/Cross_cultural_Medical_Education__Conceptual.4.aspx
. Accessed November 19, 2010.
6Betancourt JR. Cultural competence and medical education: Many names, many perspectives, one goal. Acad Med. 2006;81:499–501. http://journals.lww.com/academicmedicine/Fulltext/2006/06000/Cultural_Competence_and_Medical_Education__Many.3.aspx
. Accessed November 19, 2010.
7Weech-Maldonado R, Dreachslin JL, Dansky KH, De Souza G, Gatto M. Racial/ethnic diversity management and cultural competency: The case of Pennsylvania hospitals. J Healthc Manag. 2002;47:111–124.
8Teal CR, Street RL. Critical elements of culturally competent communication in the medical encounter: A review and model. Soc Sci Med. 2009;68:533–543.
9Saha S, Beach MC, Cooper LA. Patient centeredness, cultural competence and healthcare quality. J Natl Med Assoc. 2008;100:1275–1285.
10The Office of Minority Health. National Standards on Culturally and Linguistically Appropriate Services (CLAS). http://minorityhealth.hhs.gov/templates/browse.aspx?lvl=2&lvlID=15
. Accessed November 13, 2010.
11Kutob RM, Senf JH, Harris JM Jr. Teaching culturally effective diabetes care: Results of a randomized controlled trial. Fam Med. 2009;41:167–174.
12Beach MC, Cooper LA, Robinson KA, et al. Strategies for improving minority healthcare quality. Evid Rep Technol Assess (Summ). January 2004:1–8.
13Centers for Disease Control and Prevention. Brief 7: Gaining consensus among stakeholders through the nominal group technique. http://www.cdc.gov/HealthyYouth/evaluation/pdf/brief7.pdf
. Accessed November 13, 2010.
14Paulus PB, Huei-Chuan Y. Idea generation in groups: A basis for creativity in organizations. Organ Behav Hum Decis Process. 2000;82:76–87.
15Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62:337–342.
16Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: A research tool for general practice? Fam Pract. 1993;10:76–81.
17Miller D, Shewchuk R, Elliot TR, Richards S. Nominal group technique: A process for identifying diabetes self-care issues among patients and caregivers. Diabetes Educ. 2000;26:305–310, 312, 314.
18Castiglioni A, Shewchuk RM, Willett LL, Heudebert GR, Centor RM. A pilot study using nominal group technique to assess residents' perceptions of successful attending rounds. J Gen Intern Med. 2008;23:1060–1065.
19Mihalic AP, Morrow JB, Long RB, Dobbie AE. A validated cultural competence curriculum for US pediatric clerkships. Patient Educ Couns. 2010;79:77–82.
20Kristofco R, Shewchuk R, Casebeer L, Bellande B, Bennett N. Attributes of an ideal continuing medical education institution identified through nominal group technique. J Contin Educ Health Prof. 2005;25:221–228.
21Levine DA, Saag KG, Casebeer LL, Colon-Emeric C, Lyles KW, Shewchuk RM. Using a modified nominal group technique to elicit director of nursing input for an osteoporosis intervention. J Am Med Dir Assoc. 2006;7:420–425.
22Safford MM, Shewchuk R, Qu H, et al. Reasons for not intensifying medications: Differentiating “clinical inertia” from appropriate care. J Gen Intern Med. 2007;22:1648–1655.
23Raine R, Sanderson C, Hutchings A, Carter S, Larkin K, Black N. An experimental study of determinants of group judgments in clinical guideline development. Lancet. 2004;364:429–437.
24Schiffman SS, Reynolds ML, Young FW. Introduction to Multidimensional Scaling. New York, NY: Academic Press Inc.; 1981.
25Whaley AL, Longoria RA. Preparing card sort data for multidimensional scaling analysis in social psychological research: A methodological approach. J Soc Psychol. 2009;149:105–115.
26Shewchuk RM, O'Connor SJ, Williams ES, Savage GT. Beyond rankings: Using cognitive mapping to understand what health care journals represent. Soc Sci Med. 2006;62:1192–1204.
27Mugavin ME. Multidimensional scaling: A brief overview. Nurs Res. 2008;57:64–68.
28Pretlow C. 10 Web tools to create user-friendly sites. Comput Libr. 2008;28:14–17.
29Tobar LM, Andres PML, Lapena EL. WebA: A tool for the assistance in design and evaluation of websites. J Univers Comput Sci. 2008;14:1496–1512.
30Villa J. Time for a marketing facelift? Mark Health Serv. 2009;29:30–31.
31Giguere G. Collecting and analyzing data in multidimensional scaling experiments: A guide for psychologists using SPSS. Tutor Quant Meth Psychol. 2006;2:27–38.
32Frisby CL. The use of multidimensional scaling in the cognitive mapping of cultural difference judgments. School Psych Rev. 1996;25:77–93.
33Desarbo WS, Kim J, Choi SC, Spaulding M. A gravity-based multidimensional scaling model for deriving spatial structures underlying consumer preference/choice judgments. J Consum Res. 2002;29:91–100.
34Bussolon S. Card sorting, category validity, and contextual navigation. J Inform Architecture. 2009;1:5–29.
35Harper S, Michailidou E, Stevens R. Toward a definition of visual complexity as an implicit measure of cognitive load. ACM Trans Appl Percept. February 2009;6. http://doi.acm.org/10.1145/1498700.1498704
. Accessed November 13, 2010.
36McInerney DM, Ali J. Multidimensional and hierarchical assessment of school motivation: Cross-cultural validation. Educ Psychol. 2006;26:717–734.
37Roy B, Salanitro AH, Willett LL, et al. Using cognitive mapping to identify attributes contributing to successful ward-attending rounds—A resident and student perspective. J Gen Intern Med. 2010;25(suppl 3):S424–S425.
38Carter-Pokras O, Bereknyei S, Lie D, Braddock CH; National Consortium for Multicultural Education for Health Professionals. Surmounting the unique challenges in health disparities education: A multi-institution qualitative study. J Gen Intern Med. 2010;25(suppl 2):108–114.
39Shewchuk R, O'Connor SJ. Using cognitive concept mapping to understand what health care means to the elderly: An illustrative approach for planning and marketing. Health Mark Q. 2002;20:69–88.
40Aldenderfer MS, Blashfield RK. Cluster Analysis. Newbury Park, Calif: Sage University Press; 1984.
41Kruskal JB, Wish M. Multidimensional Scaling. Newbury Park, Calif: Sage University Press; 1978.
42Association of American Medical Colleges. Tool for Assessing Cultural Competence Training (TACCT). https://www.aamc.org/initiatives/tacct/
. Accessed January 20, 2011.
43Bransford JD, Brown AL, Cocking RR; Committee on Developments in the Science of Learning and Committee on Learning Research and Educational Practice, Commission on Behavioral and Social Sciences and Education, National Research Council. How People Learn: Brain, Mind, Experience, and School. Expanded ed. Washington, DC: National Academy Press; 2004.
44Association of American Medical Colleges. MedEdPortal. http://services.aamc.org/30/mededportal/servlet/segment/mededportal/information
. Accessed November 13, 2010.
45Salanitro AH, Castiglioni A, Willett LL, et al. Using cognitive mapping to identify faculty perspectives for successful ward-attending rounds. J Investig Med. 2009;57:382–383.