Secondary Logo

Journal Logo

Innovation Reports

Evaluating the Impact of the Medical Education Partnership Initiative at the University of Zimbabwe College of Health Sciences Using the Most Significant Change Technique

Connors, Susan C. PhD; Nyaude, Shemiah MS; Challender, Amelia MA; Aagaard, Eva MD; Velez, Christine MA; Hakim, James MD

Author Information
doi: 10.1097/ACM.0000000000001519

Abstract

Problem

In 2010, in response to the HIV/AIDS crisis, the Medical Education Partnership Initiative (MEPI), funded by the National Institutes of Health, awarded grants to 13 medical schools in sub-Saharan Africa. The University of Zimbabwe College of Health Sciences (UZCHS) received three grants; the primary grant was the Novel Education Clinical Trainees and Researchers (NECTAR) program. The UZCHS also received two linked (or supplemental) awards. The goals of MEPI were to increase training and retention of health care workers through faculty development in research and medical education and enhancement in health professions training programs.1

Because of the complex and experiential nature of medical education and academic training programs, meaningful outcomes are often difficult to evaluate. Typical methods of monitoring focus on accountability to funders and quantitative data in the form of participation numbers, surveys, etc. To complement the conventional methods, we need effective evaluation tools that provide a more comprehensive understanding of programs, especially programs intended to transform attitudes or influence career trajectories.

To address this problem, we chose to evaluate the impact of MEPI at the UZCHS using an adaptation of the most significant change (MSC) technique, which was initially developed to evaluate a program in Bangladesh2 and is often used in international development evaluation.3–5 We could find no literature applying the MSC to the evaluation of faculty development programs in health professions education. We believe our interpretation of the MSC fits with the developers’ invitation to innovate and refine the technique in other fields and local contexts.6,7 We also believe this innovative application of the MSC addresses the Best Evidence Medical Education (BEME) Collaboration’s recommendation for greater use of qualitative and mixed methods in the assessment of programs designed to prepare and support medical educators.8

Approach

The grants at the UZCHS supported multifaceted programs and initiatives designed to address MEPI goals in sub-Saharan Africa.9 Unpublished evaluation reports (2011–2015) on the grants by The Evaluation Center, School of Education and Human Development, University of Colorado Denver, used conventional methods that yielded important formative feedback and summative data related to the institutional impacts at the UZCHS. For example, on exit surveys following 15 faculty development workshops, respondents consistently indicated that sessions were relevant (average ratings of 4.52 on a 5-point scale, where 1 = “not relevant” and 5 = “very relevant to my work”). A review of attendance records showed that 115 of 166 (69%) UZCHS faculty members attended one or more of these workshops, and 40 of 42 (95%) faculty members successfully completed the requirements of a yearlong advanced faculty development program. In surveys and interviews, faculty and students noted that MEPI programs had improved Internet connectivity, and faculty credited MEPI with serving as a catalyst to the establishment of a department of health professions education. Additionally, analysis of institutional data showed that UZCHS faculty increased from 128 to 166 (30%) over five years (2010–2015), and medical student admissions increased from 126 to 203 (61%) over the same period, although other economic factors may have been influential.

However, we wanted to systematically examine the personal significance and level of positive transformation that individuals attributed to their MEPI participation. For this purpose, we selected and adapted the MSC approach because it differs from conventional methods in four ways. First, with the MSC, interviewers ask broad, open-ended questions rather than focusing on progress toward predetermined goals and objectives. Therefore, the MSC is an ideal method to understand both the expected and unexpected outcomes and to gain more in-depth understanding of a program’s impact. Second, because the MSC uses a highly participatory analysis and interpretation process, we saw the MSC as a useful way to foster participants’ self-reflection and promote discussions among MEPI program leaders.

Third, we selected the MSC because of the large and complex nature of the MEPI programs at the UZCHS. The programs had 11 initiatives (Figure 1) and 5 international partner institutions (the University of Colorado, Stanford University, King’s College London, University College London, and University of Cape Town). Over five years (2010–2015), virtually all UZCHS students and faculty members had the opportunity to engage in initiatives that ranged from minimal exposure (e.g., using the improved Internet access) to intense involvement (e.g., international field experiences). The MSC allowed a more holistic evaluation of these experiences, acknowledging the variety of subjects and participation levels, and complemented the conventional evaluations. Finally, we selected the MSC as a means to enhance the sustainability of program accomplishments by publicly highlighting both individual and institutional impacts.

Figure 1
Figure 1:
The 11 Medical Education Partnership Initiative (MEPI) grant and awards initiatives at the University of Zimbabwe College of Health Sciences, 2014 to 2015. The initiatives are organized under guiding themes (related to MEPI’s overall goals); intended outcomes and impacts are shown. aOne linked award was related to cardiovascular health (initiatives 4.1 and 4.2); the second linked award was related to mental health (initiatives 5.1 and 5.2).

The MSC technique

Using the MSC, evaluators collect stories of change and engage in participatory interpretation of those stories.6,7 The steps for this process include identifying the domain of change of interest, the reporting period, and the sample population. Although the MSC can be used to explore negative impacts, the technique is most often used to make sense of the experiences of those who have been the most successful.6 That is, evaluators look at exemplary cases, rather than average ones. Individual or group interviews are conducted to collect stories of change. Then stakeholders are convened to review the stories and select those that they believe show the most significant change. This process is gently guided by facilitators to allow stakeholders to determine their own criteria for significance; the actual selection may be done through a formal rating process or informal discussions. Facilitators record the identified cases and the reasons for selection, which may provide evaluators and leaders with insights into the underlying values of the stakeholders. In some MSC studies, it may be important to verify stories and identify emerging themes.

Our application of the MSC

Our interpretation of the MSC included seven steps (List 1). In April 2014, we identified our domain of change of interest as the positive program impacts on individuals resulting from participation in MEPI programs. In July 2014, we (S.C.C., S.N.) asked local MEPI program leaders to nominate individuals whom they viewed as benefiting from the programs; they nominated 31 individuals who were all invited to participate in an interview. To ensure transparency, nominees were told the purpose of the interviews, that their stories would be shared, and that they would have an opportunity to edit any quotes attributed to them. The UZCHS ethics committee approved the study as evaluation.

List 1

The Most Significant Change (MSC) Process Steps Used to Evaluate the Impact of the Medical Education Partnership Initiative at the University of Zimbabwe College of Health Sciences, 2014 to 2015a

  1. Identify the domain of change to be examined
  2. Identify the sample populationb
  3. Collect stories of change
  4. Summarize the prevalent themes in the stories and select representative quotes
  5. Verify the stories
  6. Convene stakeholders to select the most significant changes and note criteria they used
  7. Disseminate the results of the selection process

aThe authors used the MSC process6,7 with some adaptations.

bAs with this study, this usually comprises the most successful cases.

In September 2014, evaluators from Zimbabwe and Colorado (S.C.C., A.C., S.N., C.V.) conducted interviews with 28 of the 31 (90%) nominees (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A414 for the interview protocol); 3 had prior commitments. Four (14%) interviewees were senior faculty, 8 (29%) were junior faculty, and 16 (57%) were graduate students, whom we included because they participated in MEPI workshops designed to prepare them for roles as educators. Interviews lasted from 30 to 60 minutes and were digitally recorded and transcribed. To facilitate the time-consuming review process for the stakeholders including UZCHS faculty members and MEPI leaders from the partner institutions, we modified the MSC technique described above by coding results inductively for prevalent themes before the stakeholder discussions. We (S.C.C., S.N.) also prepared a brief profile of each interviewee that included representative quotes to illustrate the themes, which we sent to each interviewee to edit for accuracy. We hoped these profiles would serve to place the changes within a personal and locally relevant context for the stakeholders.

Using the MSC method, a critical next step was to engage stakeholders in discussion to select the most significant changes. In March 2015, five stakeholders in Zimbabwe (including J.H.) and two in Colorado (including E.A.) were asked to select the themes perceived to be the most important and stories that represented those themes.10 Profiles and themes were shared in advance via e-mail, followed by in-person discussions. By consensus, the stakeholders selected 9 themes and 18 stories to illustrate the most significant changes; the stories represented individuals from 11 UZCHS departments and all MEPI programs. Evaluators shared the criteria that stakeholders used: evidence of action (e.g., implementing a new teaching method), evidence of results (e.g., improved student evaluations), and alignment with MEPI goals. In July 2015, evaluators shared the results with UZCHS faculty and staff through a magazine publication and a slide show at a faculty development workshop; publicizing program impacts was intended to improve their sustainability.

Outcomes

Our stakeholders identified nine themes (or outcomes) of significant personal and professional changes; these themes and their prevalence are reported in Figure 2. Six were themes we expected, as they aligned with MEPI goals—becoming a better teacher, becoming a better clinician, increased interest in teaching, increased interest in research, new career pathways (including a commitment to practice in Zimbabwe), and improved research skills. However, there were also three we did not expect—increased confidence, expanded interprofessional networks, and improved interpersonal interactions.

Figure 2
Figure 2:
Frequency of 9 themes from 18 interviewees’ answers to the question “How has your participation in the MEPI programs benefitted you personally and professionally?” in a most significant change study of the impact of the Medical Education Partnership Initiative (MEPI) at the University of Zimbabwe College of Health Sciences, 2014 to 2015.

Becoming a better teacher

The most frequently mentioned benefit was the perception that MEPI programs improved teaching abilities. One senior faculty member described this change as follows: “I attended that team-based learning workshop. My life was never the same in terms of teaching … I made an overhaul to the whole course … and it is now team-based learning.… I have grown professionally.”

Increased confidence

The second most prevalent theme was that MEPI programs increased participants’ confidence across their various roles as teachers, clinicians, and researchers. A senior faculty member explained:

Before I started the MEPI program, I was very apologetic. I would always feel other people were more knowledgeable.… When I participated in the MEPI program, I got encouragement from my mentors and from my peers, and people would actually applaud the things that I would say.… Now … I talk with confidence.

Expanded interprofessional networks

The third most dominant theme was an appreciation for the expanded interprofessional networks (both internationally and within the UZCHS) developed through MEPI. Speaking about networks within the UZCHS, one junior faculty member said:

I learned the importance of networking. Before I used to more or less live in a cocoon.… I used to think I could only benefit from people within my discipline, and I realized I could benefit from anyone. I remember … a hands-on writing workshop.… What surprised me the most was … [a participant from another department] picked up a lot of things … people from my department had overlooked.

And speaking about international networks, a graduate student described a new sense of connection from MEPI participation: “I think there was a time when we felt we were not in a global village. Now I feel we are part of what is going on in the world.”

Becoming a better clinician

Some interviewees reported that their MEPI participation contributed to their effectiveness as clinicians, particularly noting the contribution of new technology and e-resources. One junior faculty member explained:

You now hear a lot more up-to-date discussions and literature being quoted.… They now have Internet access [and] infrastructure to allow the ideas to be implemented. The combination has really had a significant impact … translating into how well our students are doing and … it actually translates into how well our patients are doing.

Increased interest in teaching

Some interviewees reported that MEPI programs increased their interest in teaching as a career. A junior faculty member shared:

I didn’t think I was going to join UZCHS because [I thought] teaching must be really boring because I was thinking of teaching as in lectures only. When we were introduced to these staff development workshops, I realized there are more enjoyable methods of teaching. That is when I actually said, “Okay. I am going to join UZCHS.”

Increased interest in research and improved research skills

Similarly, some interviewees also described a new interest in research careers and/or reported improved research skills. For example, one graduate student said:

Professionally, I think now I can do research. I have more confidence.… I am inspired and I have […] passion from knowing that we need to have research from our own setting so that we can make our own decisions based on our own population.

New career pathways

Some interviewees also described how MEPI participation supported their career development, including a commitment to practice in Zimbabwe, with two interviewees describing a new interest in medical education as a career. For example, one faculty member said:

[My MEPI participation] … helped me to realize my interest in medical education. I had never thought about being a lecturer until I started being a lecturer.… Education as a career had never been emphasized.… It made me realize that maybe this is what I would like to do.… It was a time when I was having a lot of doubts in my teaching capabilities, … then [MEPI] came up. It turned out to be what I was looking for.

Improved interpersonal interactions

Finally, a few interviewees reported that their MEPI participation positively influenced their interpersonal interactions, both professionally and personally. A senior faculty member shared:

I am using the skills that I got from these courses to handle affairs [both inside and] outside my profession. Personally, I have grown. The way I am managing my family now is different.… The skills from the faculty development workshops … are applicable to both professional work and personal growth.

Next Steps

We found the MSC to be a useful and systematic evaluation approach for large, complex, and transformative initiatives like MEPI. Our results provided evidence of the achievement of MEPI goals and revealed some unexpected outcomes, such as increased confidence and improved interpersonal interactions. Additionally, in debriefing meetings conducted at the conclusion of the grants and awards, most MEPI program leaders reported that our results gave them a greater understanding of the personal and cultural impacts of the MEPI programs than they would have gotten from conventional methods alone.

We also noted limitations of this approach. The MSC is a time-intensive method requiring at least twice the time typical for conventional qualitative analysis. The method entails active participation by stakeholders, a challenge when stakeholders have many demands on their time and attention. Additionally, the MSC is likely not a cost-effective method for use when expected outcomes are known, when understanding typical experience is desirable, or in groups not open to alternative methods of evaluation. It is also not an efficient method of program monitoring for accountability purposes.

Given these strengths and limitations, we believe the MSC technique warrants additional trial applications in academic medicine. By encouraging participants to reflect on their experiences, supporting values inquiry by program leaders, and providing insights into the personal and cultural impacts of the program as it seemed to do in our study, the MSC technique can contribute to meaningful program evaluation. Additionally, it appears to be especially suitable to assess “hard to measure” outcomes (e.g., interprofessional collaboration, leadership, empathy, professionalism, cultural change). MSC results, such as ours, have the potential to contribute to greater understanding of the effective preparation and support of medical educators.

Acknowledgments: The authors wish to thank Midion Chidzonga, dean of the University of Zimbabwe College of Health Sciences, for his ongoing support.

References

1. Medical Education Partnership Initiative Principal Investigators. Foreword. Acad Med. 2014;89(8 suppl):S1S2.
2. Davies RJ. An evolutionary approach to facilitating organizational learning: An experiment by the Christian Commission for Development in Bangladesh. www.mande.co.uk/docs/ccdb.htm. Revised June 25, 1996. Accessed October 18, 2016.
3. Willetts J, Crawford P. The most significant lessons about the most significant change technique. Dev Pract. 2007;17:367379.
4. Shah R. Assessing the “true impact” of development assistance in the Gaza Strip and Tokelau: “Most significant change” as an evaluation technique. Asia Pacific Viewpoint. 2014;55:262276.
5. Kraft K, Prytherch H. Most significant change in conflict settings: Staff development through monitoring and evaluation. Dev Pract. 2016;26:2737.
6. Davies RJ, Dart J. “Most significant change” (MSC) technique: A guide to its use. www.mande.co.uk/docs/MSCGuide.htm. Published 2005. Accessed October 18, 2016.
7. Dart J, Davies RJ. A dialogical, story-based evaluation tool: The most significant change technique. Am J Eval. 2003;24:137155.
8. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Med Teach. 2006;28:497526.
9. Ndhlovu CE, Nathoo K, Borok M, et al. Innovations to enhance the quality of health professions education at the University of Zimbabwe College of Health Sciences–NECTAR program. Acad Med. 2014;89(8 suppl):S88S92.
10. Choy S, Lidstone J. Evaluating leadership development using the most significant change technique. Stud Eval. 2013;39:218224.

Supplemental Digital Content

Copyright © 2016 by the Association of American Medical Colleges