Systematic evaluation of both learners and teachers is at the core of the medical educational process.1 Researchers and educators have described a broad range of modalities for evaluating students, teachers, courses, and programs.2,3 Although many forces drive the operationalization of these evaluative processes,1 they almost all share, in almost every context, rigorous efforts to define measurable outcome parameters that are external to the learner.3 This generally means comparing a given student, teacher, course, or program to a set of externally generated benchmarks to ensure both safety and quality. This assessment approach is both vital and helpful, but its limitations may lead to unwanted consequences or impediments to individual adult learners in the medical setting as detailed below. Current approaches to assessment do not incorporate vital aspects of adult learning,4 nor do they fully utilize the highly specific, labor-intensive, and expensive one-on-one teaching that is characteristic of most undergraduate and graduate clinical medical education settings. At Children’s Hospital Boston, we have attempted to address some of the drawbacks to traditional assessment processes by incorporating known principles of adult learning and drawing on best practices from industry. In the following sections, I outline the underlying influences of our innovative approach to assessment and describe how the new approach will be implemented.
Elements of Adult Learning
Educators have shown several elements and principles to be critically important in promoting optimal and effective adult learning.4,5 Adult learners should:
- be actively involved in constructing individual educational goals and meanings.
- activate and build on their prior knowledge.
- own their learning (intrinsic motivation) as opposed to only responding to their teachers and/or evaluators (extrinsic motivation), since the latter is associated with more superficial learning.
- take increasing responsibility for being their own teachers.
Effective adult learners in the medical setting are independent, self-motivated, and self-directed.4 By increasing educational cross-links across different learning contexts and integrating them with adult learners’ past experiences, educators can significantly enhance learning. At the same time, adult learner education should ultimately be directed at making these same educators unneeded, over time fostering individual autonomy and intellect. Adult learners need to be actively involved in the learning process from initial self-appraisal to setting goals, from developing mastery to participation in the evaluative process.
According to Ludmerer,6 institutions “have yet to create true learner-centered environments that make active, self-directed learning under the close tutelage of interested faculty members the core of the experience.” Without this faculty-assisted, learner-centered approach, initial student self-appraisals or faculty-aided appraisals are unlikely. This presents a challenge to educational effectiveness and runs counter to the key elements of adult learning. To foster maximum growth, to measure progress most accurately, and to provide effective and efficient teaching, both trainees and their educators must know each trainee’s initial strengths and vulnerabilities.
Not starting an educational process with self-appraisal can lead directly to ineffective goal setting. For instance, teachers of residents are mandated by the Accreditation Council for Graduate Medical Education (ACGME) to provide trainees with the goals and objectives of each specific rotation, yet simply informing trainees of goals rarely strikes an educational contract that encourages them to “buy in” to the goals outlined. Trainees are more likely to invest themselves in goals established through a collaborative discussion that is built on trainees’ past experiences, strengths, needs, and intended accomplishments and is aligned with the specifics of the rotation. A lack of initial appraisal and collaborative goal setting can limit the development of mastery, since, without these things, there is no way to focus the joint resident–teacher efforts to efficiently and effectively promote specific mastery for the specific trainee. Ironically, although most clinical teaching occurs in a highly individualized context, without the critically needed initial appraisal the actual teaching effort cannot be individualized, thus missing a major opportunity for trainee-specific learning that would improve both the quality and efficiency of teaching. Rethinking and then reworking the evaluative process can address these challenges.
Limitations of the Current Individual Evaluation Process
The limitations of the most common types of individual evaluations become clear from an adult learner perspective. There is a tension between the time and expense required for creating, administering, and scoring assessment tools and the tools’ meaning and adult learner value to individual trainees.2,5 In response, evaluative tools (i.e., multiple-choice examinations) have been developed that most often rely heavily on approaches that are concise and readily scored, measures that provide summative feedback.2 While important and even necessary, the simple feedback of “you’ve passed and you’re competent” or “you’ve failed and you’re not” ignores key elements of the adult learner’s role (e.g., activating prior knowledge, cross-linking, and intrinsic motivation) and may limit teachers to being gatekeepers between passing and failure rather than participatory educators. Efforts to improve the evaluative process in medical education7–9 have led to more complex assessment tools (e.g., OSCEs, ACGME core competencies, and the educational portfolio). Although they have an important role in medical education and enhance the evaluative process, these tools still leave several key problems relatively unaddressed.
First, most traditional evaluations are experienced by the majority of trainees as externally regulated rather than internally regulated.10 They do not feel “self-directed,” nor do they experience the motivation for mastery as intrinsic as they would if the assessment tools were informed by adult learning theory.5,9,11 Quite often, learners report that they see these evaluations as fences to get over or around. They view such evaluations as measures used and owned by programs to weed out stragglers. The failure to incorporate basic principles of adult learning into assessment tools makes it far less likely that trainees will feel responsibility or ownership for their work and less likely that they are seen as a vital part of the collaborative effort at directing their education.5 Responsibility for trainees’ education and improvement is thus externalized, leading even top-quality, hard-working residents to wonder what would happen if, for example, they didn’t achieve 70% attendance as the ACGME requires. This also explains endemic problems with resident attendance at seminars in general. We expect trainees to be responsible and learn like adults, but we fail to engage them like adult learners.
The process-oriented nature of medical education is also generally underaddressed and underappreciated in traditional assessment. According to Wilkes and Bligh,1 clinical education is a “complex combination of systematic teaching and learning activities (clinical work) within a professional environment …. [H]ow [trainees] learn is as important as what they learn, and understanding how they learn can contribute much to what they learn.” Armstrong et al12 describe the common limitations of medical education as a process management problem. Although there is much professed interest in how learning occurs for an individual on a specific rotation or over time, there is relatively little actual effort directed towards actively managing that educational process.12 Even though much of medical education is one-on-one, few teachers assess and then tailor learning to specific clinical settings and learners. Fewer still address an individual’s learning from one context to the next.12 The inherent problems for both learning and efficiency are obvious. The traditional medical education process neither builds on the specific trainee’s experiences, skills, knowledge, and needs nor attempts to cross-link or activate learning experiences from one context to the next.5,9 An instructor may teach a specific student about child development, for example, without knowing and building on the fact that the student had been a preschool teacher prior to medical school or may reteach the basic evaluation of delirium to residents in the emergency room after they’ve just finished two months on the neurology consult service. In both cases, simple initial inquiry would have allowed the linkage of one context and knowledge set to the next and allowed the specific teaching to be taken to a higher level. We set up labor-intensive medical education systems and then too often do not fully utilize them, wasting vital opportunities for improvement.
Failing to examine the ongoing educational process and progress while in a specific clinical setting further compromises learning, thus missing the chance for “real-time” learning collaboration, corrections, and confirmation that educational goals are being met.13 More troubling still is that when trainees move from one setting to the next, neither the teachers’ nor the trainees’ approach in the new rotation is informed by the trainees’ prior educational experiences. Without remembering and building on a specific trainee’s learning history, teachers and students may find themselves facing an educational “Groundhog Day” where the same material or problems are repeated again and again. This markedly decreases teaching and learning effectiveness and efficiency.
Insights From a Manufacturing Model
Business manufacturing focuses on producing the highest quality product at the best possible price and offers insights useful for educators.12,14 If teachers think of students as “products” and students’ knowledge, skills, and attitudes as “quality measures,” then the goal of teachers is to efficiently and effectively produce the highest-quality clinicians. Ideally, teachers in the medical setting provide trainees with the right balance of experiences to target their individual windows of need, bringing them to market as the best, most efficiently produced “product” possible. In doing so, teachers would create well-informed, clinical problem solvers who are equipped to effectively fill the needs of patients and health care systems (by analogy, the “consumers”).
Most manufacturers—and, by analogy, educators—use what is described in manufacturing as a “push” system of production. This means that well in advance, upper management, with all due diligence, sets the best estimate of anticipated product need (what and how much the consumer will buy or what every student should be taught). The materials, labor, and assembly processes all follow, and there is a downstream flow of materials and production, ultimately leading to the product’s manufacture. In this style of production, changing market situations, emerging production problems, and individual worker or student skills, needs, or productivity are generally not factored in. Each of these variables then has the potential to compromise productivity (both quality and cost). From the corporate perspective, this has all too often led to wide-ranging problems in efficiency and quality control. Excess capital is tied up in anticipated supplies and production lines but is not keyed to actual consumption or rapidly changing markets. Without the appropriate feedback loops, production problems or misdirection can go unnoticed until too late and can result in scenarios like the 2008 dealer surplus of formerly profitable sport utility vehicles as gas prices skyrocketed and consumer demand plummeted.
In this model, workers feel disconnected from the production process because they have no real voice in the matter and tend to feel that the set process and product are out of their control and responsibility. Without personal input or ownership for productivity, workers’ and trainees’ roles as responsible adults are eroded and there is little sense of personal efficacy, often resulting in the passivity that is a frequent complaint of management and educators. Workers and trainees perceive problems with production and quality control as external and as problems for supervisors to fix. It is common for workers and trainees to think that they are doing all that they can “under the circumstances,” and they are often right, because the “push” system does not include them in process development and management. What may also emerge under such circumstances is an inadvertent adversarial relationship between management or educators and workers or students, with each side pointing the finger at the other, failing to remember that they actually are working towards the same goal: high-quality products that are efficiently produced.
The Toyota Motor Company uses an alternative manufacturing system.14 Toyota employs a “pull” system of production and clear, uniform, and evaluated approaches to each job task. The “pull” system establishes an ongoing loop, which matches production to customer demand, and production line management with actual worker experience and productivity. The system monitors real-time production and consumer demand and then uses the data to “pull” or trigger adjustments in supplies and labor to match actual production need. This offers a far more efficient way to optimize the use of resources. It strives for continuous improvement by matching supply to production to product demand, all in real time.
Toyota’s system both regulates and empowers workers. Toyota has explicit and detailed expectations, based on worker-conducted trials that are uniformly applied and carried out. The expectation is that if workers have tested and validated a production plan, then production is more likely to go smoothly. It presumes that, through their participation in its development, workers have “bought in” to the optimized production paths and, through this ownership, are far more likely to carry those paths out.
Because Toyota’s management and labor jointly own the production plan, the “us/them” split is less pronounced when problems occur. Observations of Toyota13 find that, when problems arise, they are seen first as a result of faulty execution of the plan, then as a potential plan design problem, and lastly as due to faulty workers or management. Joint ownership of the plan encourages both labor and management’s responsibility for solutions and avoids placing them in an adversarial position. This Total Quality Management (TQM) approach has been identified as an integral part of Toyota’s success. By analogy, trainees’ struggles would be seen as educational system/process problems that need to be collaboratively solved, rather than a result of incompetence on the part of trainees or faculty.
Toyota’s workers are empowered to have personal responsibility for monitoring their productivity and their product. To carry this out, the company has implemented an extensive signal system in which each worker is continuously self-monitoring and can “pull” the supplies or support needed to carry out each task. Toyota (and others) carry out this signal system through a Kanban, which is Japanese for card. Each worker has a Kanban that is specific to that worker and his or her tasks. The Kanban serves as a tangible signaling device that can immediately propagate up or down the production chain in real time, triggering immediate corrections. For instance, a Toyota worker whose job is to place and tighten bolts on an assembly line would have a “bolt Kanban.” As the worker’s supply of bolts started to dwindle, the worker would hand his bolt Kanban to the supplier, who would in turn pass it on. Eventually, the individual worker’s bolt Kanban reaches the distribution center and is then returned to the worker along with another bucket of bolts. The one-to-one matching of supply to demand throughout the production process promotes both worker efficiency and management’s conservation of capital by allotting the appropriate amount of materials and support needed to accomplish the task at hand in real time.
It is the combination of tangible personal responsibility and system support that is crucial to success in this type of manufacturing model. The Kanban’s focus is process-oriented promotion of the workers’ and systems’ efficiency and quality; its goal is to facilitate continuous success and improvement, not to judge. The educational equivalent of the Kanban system would be a process that monitors the trainee and educational context to develop explicit goals for each student on each rotation, coupled with the real-time collaborative monitoring needed to direct teaching resources to ensure a well-educated trainee.
The Educational Kanban
The Educational Kanban (EK) is an innovative assessment tool that pairs the pull system approach to manufacturing with key educational elements for adult learners. It was developed and preliminarily piloted at Children’s Hospital Boston in 2008 and will be fully implemented in the Child and Adolescent Psychiatry training program in July 2009. Its name reflects its focus on dynamically monitoring educational progress (production) and allowing educational supplies (teaching) to be specifically “pulled” by the needs of the individual trainee. The EK is an electronic document that’s owned and accessed exclusively by the trainee. It is designed specifically to not be part of the trainee’s “permanent record,” instead traveling exclusively with the trainee from one context to the next. It serves both as a personal log of what the trainee has accomplished and as a map for what he or she must master in the future. Continuously tracking improvement through a tool that travels with the trainee across educational experiences addresses the educational process and systems problems noted by Armstrong et al.11 This will be particularly helpful to medical school clerkship directors and students where process or systems problems between rotations are marked.11
Under the EK system, trainees begin each learning experience with a self-appraisal of their current progress by reviewing with their new attending their ongoing EK to date (Appendix 1). Trainees and attendings will jointly identify areas of mastery and need based on their past individual experiences. This is followed by collaboratively developing a set of learning goals for the new experience that builds on and is linked to their earlier work. This set of goals is based on a combination of rotation-specific goals (mandated by the ACGME) combined with the trainee’s actual prior experiences, both covering the general rotational goals and being tailored to each resident.
The initial EK meeting differentiates it from other assessment tools in that, from the start, it establishes a collaborative framework providing both the continuity and specificity to efficiently focus teaching and learning effectiveness. In this way, trainees and teachers can specifically identify and address changing “market needs” (the way trainees differ from one another), problems with production (what trainees have learned) or supply (what they have been taught), or problems with “upper level management” (a priori problems with goals or planned experiences). Each EK will have both common and specific elements. The “ownership,” continuity, and collaboration involved in the EK process provide the framework for activating, cross-linking, and building on prior experiences and promoting the adult learning that makes trainees more responsible (and intrinsically motivated) for their own education. It is then designed to track a trainee’s efficiency and production at every step, with ongoing meetings (at least monthly) throughout the rotation. This tracking provides the educational equivalent to Toyota’s one-to-one handoff. This ownership, continuity, and real-time collaborative monitoring both promote adult learning and will minimize the all-too-common “us/them” disconnect at play in traditional assessment approaches.5,10 Ultimately, the EK provides real-time individual quality control that is learner-experience based, within a specific learning framework, which the trainee carries from one context to the next. The dynamic yet archival nature of the document will aid the trainee and the program in integrating their experiences over time and context, providing real continuity across experiences, ultimately fostering a more holistic educational approach. For the program directors, the process-oriented systems approach integrated over time will continuously identify recurrent problems, clarifying whether educational goals are being met12 and facilitating ongoing individual and programmatic improvement. Thus, the EK employs a TQM approach to optimize clinical teaching and learning or, by analogy, “productivity.”
Anticipating the Change
As with every new system, certain types of pushback are expected and can be anticipated. Among the most apparent will be objections about the time needed to undertake the initial and then monthly reviews. This would translate into four, rather than the currently mandated two, meetings. Because staff teaching time is specifically “pulled” to where it is needed with the EK system, we expect that the gains in effectiveness and the elimination of redundancies should compensate for any initial additional time required. Ultimately, the work of education will be far more time-efficient and effective for both the trainee and the teacher, with the quality improvement justifying time spent.
A second area of pushback may come from the difficulties many faculty members have in providing effective feedback. The two, two-hour, mandatory faculty development sessions needed to implement the new system will provide educators with a chance to actively address this. Additionally, the EK’s trainee-as-adult-learner approach will promote evaluative collaboration between trainees and teachers throughout the process and minimize the often unpleasant us/them tension. This shift towards constructive collaborative correction and away from end-of-rotation summary judgment will make the feedback process more productive and valuable for teachers and trainees. Finally, because of the required initial self-appraisal review with their attendings, trainees will be primed to initiate ongoing feedback, and this, too, will address the feedback avoidance that some staff display.
A third area of anticipated pushback may be around “ownership” of the EK. Although educators may argue that most evaluative tools (evaluation forms, tests, journals, logs) are for the resident’s “benefit,” these tools, once used, are kept in the institutional files and used as measures for promotion or reference (by requirement). Thus, trainees experience traditional evaluative tools as belonging to the institution and not really for the trainees’ “benefit” at all. For most trainees, this externalizes responsibility, discourages real honesty in formal self-reflection, and at times fosters an adversarial relationship between student and educator. Institutional files must remain, but the EK is different in that trainees maintain their individual EKs in their own files, promoting the vital real ownership that empowers trainees to be adult learners.
The EK approach emphasizes and examines the learning process while making the trainee and teacher a truly collaborative educational dyad. It is designed to target each trainee’s learning needs and to provide the specific necessary learning resources to successfully fill the programmatic and trainees’ targeted goals while empowering the trainee to be responsible for the tasks at hand. The EK has the potential to markedly diminish the us/them and labor/management divides as well as to promote the development of trainees into effective and optimal adult learners while “pulling” the educator’s efforts to where they are most needed.
Demonstrating the effects of the EK will have several components over the next years. For residents and educators, using their prior experiences for comparison, we will track competence- and process-related changes. This study will center on the anticipated changes associated with an adult-centered learning approach. It will include questionnaires and narrative self-reports from both trainees and faculty looking at performance, goal achievement, locus of responsibility, degree of perceived collaboration, and the elimination of identified problems across contexts.
Implementing the EK
Table 1 compares the current resident evaluation system with the new EK evaluation system. In this section, I illustrate how the EK system will be applied, using a child and adolescent psychiatry residency as an example.
Residents and staff will be introduced to the EK at the beginning of residents’ training through staff and resident development sessions. The EK’s electronic pages will be sets of anchored competencies derived from the six core ACGME competencies, covering both the overall program and rotation-specific elements as seen in Appendix 1. Traveling from rotation to rotation, the EK will offer the continuity needed to become a dynamic document that reflects the archived history of what each resident has done and a template for what he or she will need to do to continuously improve while moving through the program.
The EK will not replace summative evaluations completed by the resident’s supervisor. The EK will serve as a new real-time formative device that will complement the standard summative forms. The ongoing EK process should make completing the summative forms far more straightforward, because faculty and residents will have been considering performance-related elements throughout the educational experience. Given the EK’s process-oriented approach, there should be no surprises or conflicts (as there have been in the past) as the trainees sign off on their summative evaluations. The real-time collaborative monitoring identifies and initiates solutions, allowing the summative evaluations to reflect the residents’ progress and achievements rather than their deficiencies. Shifting the perspective from recognizing problems to identifying solutions represents another vital change.
At the beginning of each teaching rotation, individual trainees will meet with their designated supervisor. In this first, as in all meetings, the trainee takes the lead and begins with a self-appraisal of his or her strengths and vulnerabilities, based on the ongoing real-time record that the EK offers. The trainee and supervisor will then develop specific learning goals and objectives for the rotation. These goals are collaboratively developed from considerations of existing EK documentation, the trainee’s self-assessment, the rotation’s and program’s objectives, and the supervisor’s observations as they arise. The goals will reflect the individual trainee’s experiences and needs in the specific context, allowing the supervisors and trainees to focus their energies and efforts on the appropriate areas. It will both “pull” teaching where it is needed and identify problems when they occur, all in the context of the specific trainee and rotation. This trainee-carried, process-oriented, specific matching is analogous to the one-to-one manufacturing handoffs described for Toyota. In this manner, teaching “capital” is more efficient, because it is targeted, and more effective, because the outcomes are specifically monitored. Appendix 1 provides an example of the initial EK for a resident beginning the consultation service.
At one-month intervals, the resident and supervisor will review progress, make corrections, and formally fill out the next month’s plan during a resident-led meeting. Results of the meetings may vary from taking on major new directions to simply affirming what has transpired since the last meeting. The EK will always document where the resident has been and where he or she needs to go. It is always process-centered and evolving.
At the end of the rotation, there will be a final joint review, again with the resident taking the lead. The EK is designed so that, at the end of each rotation, the EK will reflect an individual’s evolving progress and future plans that are carried to the next rotation. It will be a far more detailed and dynamic track record of what specifically has happened and what must happen on the next rotations to ensure the resident’s success. Although the EK will clearly reflect the resident’s performance, it is not designed to provide summative evaluations; it is designed to promote continuous individual growth and improvement. This differentiates it from the more traditional end-of-rotation supervisor–resident meetings.
At the end of each quarter of the training year, the resident and mentor or faculty advisor will review the EK over the course of their ongoing two-year relationship. These meetings will provide both an invaluable overview and additional resource for the resident, identifying early on the problems or processes that need attention. These meetings will also serve as roadmaps for improving effective “mentorship” by providing a tangible record to organize the mentor’s efforts at helping the resident efficiently and effectively optimize training.
By its dynamic nature and its focus on the individual, the EK will provide a highly valuable educational tool for both trainees and teachers. It will reduce unneeded redundancies, target specific needs, and focus educational activities, ultimately leading to more efficient and effective development of all the competencies and skills vital to producing a quality graduate. It will foster integration across contexts for the trainees and across trainees for the program, highlighting both strengths and weaknesses. The EK builds on the principles of adult learning and cycles from self-assessment to goal setting to developing mastery, and ultimately returns to evaluation and self-assessment. Its process-oriented, solution-focused nature brings the TQM approach to clinical education for both trainees and educators. The EK is trainee-owned and is designed to optimize their education. Trainee ownership ensures that the EK becomes a personal roadmap and a monitoring device without the threat of going into their “permanent record,” minimizing the “us/them” phenomenon. In doing so, it complements and enhances other modes of assessment, but it is not a substitute for the summative evaluations that every program must have. Eventually, the ever-growing EK becomes a log of where trainees have been and where they need to go, serving as a compass for the trainees and their teachers to plot the optimal academic course. Ultimately, we hope that the EK fosters the adult learning, solution-focused approach that trainees will carry through out their professional careers.
The author would like to acknowledge the manuscript review and assistance of Dr. David R. DeMaso.