Today’s health care environment has seen numerous changes in the way in which care is provided to patients. A function of these changes is the inclusion of outcomes as a means to provide evidence for effective quality health care. 1 Medical education has also felt the impact of outcomes-based assessment as evidenced by a shift towards outcomes-based standards by accreditation bodies. Commission on Accreditation of Allied Health Education Programs in the year 2007 will move primarily to outcomes-based standards as a method for program accreditation. Because O & P educational standards and guidelines are governed by the Commission on Accreditation of Allied Health Education Programs, it is necessary to examine and understand outcomes as they apply to O & P education.
Traditionally, achievement and/or accreditation for educational programs has been assessed solely on institutional organization and function and the results are consistent with the Flexner model. 2,3 An institution’s admission criteria, institutional objectives, faculty credentials, and resource provisions determine, to a large degree, a program’s accreditation status. These elements are necessary to ensure an appropriate teaching environment, but they fall short in the areas of evidence and achievement. Are educational objectives met? What evidence is there to support these findings? These questions can be answered when the institution looks more closely at its product: the student. The learner becomes the true measure of the effectiveness of an educational program in outcome-based assessment. 4,5 In this paradigm, the institution or program must demonstrate that learning is taking place. To ensure learning is taking place, the intended outcome and appropriate measure of that outcome should be clearly defined. In addition, measurement instruments need to be responsive or sensitive to changes in learner performance. Instruments that are responsive in detecting change allow for stronger inferences about outcome achievement. 6 One final element that can be easily overlooked is how or why an outcome is achieved. A lack of control of confounding variables limits a true cause-and-effect relationship from being established and makes determination of how or why an outcome is achieved difficult. The remainder of this paper will focus on specific examples of educational outcomes, measurement instruments, outcome achievement, outcome evidence, and the use of standardized outcomes.
DETERMINING THE OUTCOME
Establishing outcomes is based largely on the educational programs’ desire to see that learning is taking place. Outcomes may include, but are not limited to, clinical knowledge, clinical skills, attitudes, problem solving, program admission, program graduation, and faculty-student interaction. Outcomes also provide a means to evaluate program or institutional success. The focus in medical education outcomes is learner performance because it is now emphasized as the measure of program success. The evaluation of the program is in fact a result of not only summative considerations but also formative assessment as well. Summative evaluation is used to look at the end product while formative assessment “is an ongoing process aimed at understanding and improving student learning” (Figure 1). 7 In any case, both the context and content of the outcome must be clearly defined up front. Consider the following scenario: A passing grade in biomechanics may signify successful learning outcomes on one level, but not necessarily on another. For instance, if the outcome in this case is strictly knowledge based, strong inferences can be made about successful learning. If, on the other hand, an outcome is deemed successful based on a student’s ability to apply biomechanical knowledge in a clinical setting, a passing grade alone is little evidence to support such success. It is therefore essential that the process of framing outcomes is clearly defined and established before inferences can be made about achievement.
To insure the intended outcome is truly achieved, the selection of an appropriate measure is critical. Selection can either be from pre-existing instruments/models or from the development of new ones. Development of measurement instruments can take place on a smaller scale, such as focus groups, or on a larger one, such as institution collaboration. Results from survey data, as in the case of the practice analysis, can provide additional information and a foundation for content development. 8 Another consideration is the solicitation of organizations primarily involved in the development process and assessment of quality instruments. 6 Two such organizations are The Medical Outcomes Trust and The Joint Committee on Standards for Educational Evaluation. Reasonable measures of outcomes include: (1) faculty member assessment; (2) written examinations; (3) standardized patients; (4) preceptor assessment; (5) and objective structured clinical exams. Kassebaum effectively illustrates the use of outcome measures and provides a framework from which to include additional examples. 2
• An outcome goal for a program is to have students develop exceptional clinical skills. An effective measure to assess performance achievement is the use of objective structured clinical exams or standardized patients. Students demonstrate their ability to perform certain clinical skills under observation in this measurement domain. The observer has a list of desired clinical skills in the form of a checklist and/or global rating scale. Student performance is then recorded and clinical skills assessed. Assuming that additional confounding variables are controlled, strong inferences can be made about the relationship between the instrument and outcome.
• An institution’s faculty members feel strongly that students should develop exceptional problem-solving skills. It is advantageous to measure in the context of problem-based learning or case-based learning when isolating and assessing problem solving. The student must navigate through a case or series of cases and solve various problems for each encounter when using either learning strategy. This method encourages problem solving to occur. The instrument, if responsive, will determine to what extent problem solving has taken place. In this scenario, some inferences can also be made about student knowledge. We assume that in order to problem solve, it is necessary to have some prior knowledge on a subject. The uncertainty that comes with using the instruments for the purpose of knowledge assessment is present because there are two known variables: knowledge and problem solving. Poor performance on either exam may be a function of poor problem-solving ability and not a function of knowledge. In this case, a written examination is more desirable. In this same scenario, we change the outcome now to exceptional clinical skills as seen in the previous vignette. Use of these same instruments to measure exceptional clinical skills may reveal a student’s ability to select appropriate skills, but the student may not necessarily translate those same skills into successful performance…. To be sure, selection of any one instrument should have a stated outcome as its basis.
• The American Board for Certification examination and the United States Medical Licensure Exam are intended to assess longitudinal knowledge and experience with the ultimate goal of determining if individuals are competent to see patients. These examinations are used when an assessment of institution performance provides an element of uncertainty if the connection between academic performance and certification are too far removed. Furthermore, poor exam performance, if used to assess institutional performance, does not answer the questions of how and why an institution is deficient. Measures of institutional performance based on learner performance should be more closely connected so as to elicit a closer cause-and-effect relationship. Using examination results on a larger scale, however, could merit some consideration. A continuum of strategically placed assessment instruments during primary education, clinical preceptorship, residency, certification, and continuing education may provide long-term data to identify trends in clinical competency and performance. 9 Unfortunately, longitudinal studies in this area are sparse throughout the medical education community. Additionally, cost and individual exam anonymity become greater concerns.
• A learning institution’s goal is to develop appropriate student-patient interactions. The burden on the faculty is finding adequate time to assess each student-patient encounter and providing quality feedback to the student. The faculty-student ratio may be such that it is impractical for faculty members to fairly assess each student-patient interaction. The selected outcome measure must then occur from a different reference axis to reduce faculty burden and at the same time measure the intended outcome. A first encounter between patient models and students provides a framework to assess the outcome. Patient models are given a global rating scale or checklist instrument after the encounter to assess student performance. 10 Students are either rated on communication skills or with a task specific checklist. For example, did the student introduce him or herself? Did the student adequately describe tasks or evaluations to be performed? The same outcome could be measured during the 250 hour clinical affiliation (separate from one year residency) as well. In this environment the preceptor would assess student-patient interaction and provide feedback to the program director. Both examples are advantageous in that the first encounter is measured in a controlled environment and the second in a real-time clinical setting. In addition, the first encounter could be measured without prior student knowledge in order to limit observer effects on student performance.
ACHIEVEMENT AND EVIDENCE OF OUTCOMES
Achievement and evidence of outcomes is based largely on a clear definition of the desired outcome performance and the attributes of the intended measurement instrument. Bordage et al. discusses these attributes as a result of The Medical Outcomes Trust advisory committee’s criteria for quality design. 6,11 The criteria for quality instrument attributes are as follows: (1) concept and measurement model; (2) instrument reliability/validity; (3) responsiveness sensitivity to change; (4) interpretability-qualitative meaning to quantitative scores; (5) burden time and demands; (6) alternative forms/modes of administration; and (7) cultural and language adaptations. Once in place, strong inferences about achievement and evidence are possible. It is also important that any decision-making based on outcome achievement is not determined in isolation but assessed through multiple outcome results. “Final judgments about student progress should be made on the basis of multiple assessments, obtained on different occasions, using a variety of methods based on stated goals”. 12 To be of value, the decision-making process should entail longitudinal outcomes as well for purposes of predictability and identification of trends in clinical performance. Collectively, the outcome achievements detail data that is more suggestive of a program’s true achievement-learner performance (Figure 2).
SHOULD MEASUREMENT OUTCOMES BE STANDARDIZED?
The question of whether to use standardized measurement outcomes is not new. Many people have a feeling of trepidation at the inclusion of standardized measurement outcomes for use in program evaluation. The concern is that standardized measurement outcomes “might fail to reflect the distinctive products of individual medical schools”2. In other words, do standardized measurements become too restrictive? From the standpoint of comparing different program philosophies, yes, that is a possibility. For example, a standardized measurement in the form of a knowledge-based examination would bode well for an institutional philosophy deeply seeded in knowledge-based curriculum. Students at an institution with a combined knowledge-based and problem-based curriculum have the potential to do worse on the examination but perform better clinically during their residency. Some would argue that the latter is better because it more closely reflects clinical performance. Advocates of the predominantly knowledge-based curriculum would argue that the student has a stronger foundational base from which to develop other skills. Nonetheless, at the conclusion of both primary education and residency it is quite possible that both sets of students are now equally competent. Adopting a standardized measurement examination may have some merit if it is included on a continuum of outcome measures to evaluate educational programs. There becomes a concern, however, that programs will lose their distinctive quality if the standardized examination is to stand alone as the only outcome achievement criteria
Another possibility is that current standards already adopted and recognized by an accreditation body remain essentially the same. The evaluation of any program should then be in the context of “whether the program accomplished its objectives.”13 Programs would have to provide evidence that objectives were achieved. This method tailors the evaluation process to the individual institution and its effect on the student. It is less likely to focus on distinctive organizational qualities of the institution.
It can be said with some certainty that outcomes will remain an integral part of the health care system for some time. The effects of outcome-based standards on O & P education remain unknown at this time but may signify a greater emphasis on learner performance and less on institutional function and organization. The inclusion of outcome measures does not necessarily remove functional and organizational evaluation of institutions but merely changes the reference point. Effectiveness of institutional make-up, learning strategies, and learning performance are based primarily on assessment of the learners themselves. Inferences about institutional success cannot effectively be determined if requirements only necessitate the need to identify coursework and stated objectives. This may fulfill a set of requirements, but it does not function to measure coursework effectiveness based on learner performance. It therefore becomes necessary to identify clear and specific outcomes to ascertain performance. Additionally, selected outcome measures should be appropriate, implemented in a timely manner, and responsive to changes in learner performance. To avoid decision-making that is unfounded, attention should be given to multiple measures of outcomes, and a student’s longitudinal performance should be tracked. The aggregate response provides a framework from which to make more reasonable and accurate inferences about student learning and program performance.
Implementing outcome measurement in O & P education should be anticipated with a cautious optimism. Used appropriately, the outcome model can provide valuable information about the institution, faculty, learning strategies, and the student.
1. Dickerson FB. Assessing clinical outcomes: The community functioning of persons with serious mental illness. Psychiatr Serv. 1997; 48: 897–902.
2. Kassebaum DG. The measurement of outcomes in the assessment of educational program effectiveness. Acad Med. 1990; 65: 293–296.
3. Flexner A. Medical education in the United States and Canada. A report to the Carnegie Foundation for the Advancement of Teaching. Bulletin No.4. Boston, MA: Updyke, 1910.
4. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988; 260: 1743–1748.
5. Stone SL, Qualters DM. Course-based assessment: Implementing outcome assessment in medical education. Acad Med. 1998; 73: 397–401.
6. Bordage G, Burack JH, Irby DM, Stritter FT. Education in ambulatory settings: Developing valid measures of educational outcomes, and other research priorities. Acad Med. 1998; 73: 743–750.
7. Angelo TA. Transforming assessment: High standards for higher learning. American Association of Higher Education Assessment Bulletin. 1996; 48: 1–3.
8. American Board for Certification. Report of the American Board for Certification Task Force on the practice analysis of the disciplines of the orthotics and prosthetics. Official Records: Alexandria, 2000.
9. Iler E, Slater M. The Gateway Program: Ten-year lessons about outcomes and admission measures. Acad Med. 1998; 73: 1169–1171.
10. Wolf FM, Sisson JC, Zweifler AJ. A standardized patient program to evaluate summarization skills in patient interviews. Acad Med. 1995; 70: 443.
11. .Medical Outcomes Trust. Instrument Review Criteria. Medical Outcomes Trust Bulletin. 1995; 2: I–IV.
12. Shannon S, Norman G, Falconer JG. Evaluation Methods: Reference Handbook, 2nd Ed. Hamilton: McMaster University, 1995; 119–131.
13. Teherani A, Hitchcock MA, Nyquist J. Longitudinal outcomes of an executive-model program for faculty development. Acad Med. 2001; 76: S68–S70.