Preliminary Evaluation of the Learning Outcome Achieved by a Nursing Research Seminar Course for Doctoral Students

Abstract

 

Educational evaluation is a priority policy of the Ministry of Education and student learning outcome is an important criterion used in educational evaluation work. The purpose of this study was to evaluate the learning outcome of a newly developed course for doctoral students entitled, Nursing Research Seminar. The course was one semester in length and required students to attend 2 hours of class per week. Student learning outcome was evaluated based on the level of understanding students had of course objectives. The six objectives of this course were: evaluating and integrating research papers; enhancing critical thinking skills; gaining an in-depth understanding of the literature related to topics of interest; enhancing ability to construct research proposals; guiding student dissertation work; and refining critical research skills. Data were collected from the responses provided by 25 students on a 5-point Likert-type evaluation form based on course objectives filled out during the last class of the semester. Descriptive and non-parametric statistics were adopted. Results showed: (1) The average post-course score (24.76 ± 2.89) was significantly higher than the pre-course score (18.40 ± 5.52); (2) Students realized significant improvements in all six objectives at the end of the course; (3) There were statistically significant differences in improvement scores in all six objectives for students in different years of their doctoral program; (4) The lower the year in the program, the higher the improvement scores for each course objective; (5) The two objectives of the six that saw the most significant improvements were “gaining an in-depth understanding of the literature related to topics of interest”, and “enhancing critical thinking skills”. Because of the small sample size, conclusions drawn from this study should be treated as tentative. Findings provide preliminary information supporting the importance and necessity of offering the Nursing Research Seminar course to nursing doctoral students. The course provided different benefits to students at different years in the program and, overall, research results encourage that this course continue to be offered.

Introduction

Training outstanding and competent professionals depends on retaining and supporting high quality educators. Educational evaluation has been a priority policy of the Ministry of Education in recent years, with teaching quality comprising an important aspect of program evaluations. Education is fundamental in the training of professionals and expansion of professional knowledge. One way to expand professional knowledge is to enhance the quality of education at the higher education levels. Nursing education in Taiwan, first begun in 1944, originated as part of the vocational education system. Nursing education at the university level began to be offered in 1956. A masters program in nursing began in 1984 and the first nursing doctoral program in Taiwan was established in June 1997. Both were inaugurated at the National Taiwan University (NTU)'s College of Medicine. The mission of the nursing doctoral program is “to prepare scientists with advanced nursing knowledge, practice and education through research and scholarly activity to create multidisciplinary and international leaders in the nursing field.” After 53 years, Taiwan finally has a complete nursing education system. As of August 2003, nine institutions in Taiwan offered nursing doctoral programs, with three to eight students enrolled at each (^{Chen, 2004}).

Thirty-one countries around the world, including Taiwan, currently offer nursing doctoral education programs (^{Chen, 2004; Ketefian, Davidson, Daly, Chang, & Srisuphan, 2005)}. The first nursing doctoral program in Taiwan began enrolling students in 1997, with three students the first year, four the second year, and five students each year thereafter. The design of doctoral program courses in Taiwan is based primarily on the model of Doctor of Philosophy, Ph.D. programs in the United States. The training process emphasizes research and technical innovation in order to develop professional nursing knowledge. Besides core nursing knowledge, content includes relevant scientific and philosophical knowledge, theory development process, research methodology, statistical analysis, leadership skills, and dissertation writing (^{Chen, 2004; Ellis, 2005)}. This gives students a total of 36 credit hours of coursework, 18 for core courses (Table 1), 18 for electives, and 12 credit hours for their doctoral dissertation. To expand their knowledge base, students can also take courses related to their research topic at other universities. Most students completed their course work in 2-3 years.

Table 1: Core Courses of This Nursing Doctoral Program

Course planning, designation of objectives, and implementation of teaching strategies are extremely important in facilitating the achievement of educational goals. Because the NTU nursing doctoral program was the first of its kind in Taiwan, a “Doctoral Program Curriculum Affairs Committee” was established in the department to perform annual assessments and revise course design. NTU faculty members noticed that after students had completed their required doctoral coursework, they often did their doctoral dissertations on their own, with guidance only from the supervising committee. This indicated a need for periodic monitoring of student progress during their dissertation work, which would also allow students to learn from each other and provide mutual support. Therefore, the Curriculum Affairs Committee suggested including adding in 2003 a required course, Nursing Research Seminar, for all doctoral students. Course objectives, teaching strategy, and grading criteria were all designed following committee discussion, and it was determined that this course aimed to assist students achieve the following course objectives: (1) evaluating and integrating research papers; (2) enhancing critical thinking skills; (3) gaining an in-depth understanding of the literature related to topics of interest; (4) enhancing ability to construct research proposals; (5) guiding student dissertation work; and (6) refining critical research skills. It was hoped that mutual learning in the class would increase student research capabilities and help them better appreciate research from different fields.

After determining objectives, the next step was to determine which teaching strategy would result in the highest learning efficacy. A study by ^{Hsu (2004)} led the Committee to decide that interactive discussions were most effective at facilitating student learning. The teaching strategies of this course included: (1) oral presentation, (2) peer critique, (3) special lectures, and (4) written term paper. This strategic approach, described below, provided students with multiple learning opportunities.

1. Oral presentation: Each student was required to give a 1-hour oral presentation, the topic of which depended on the stage of the student's dissertation work. The presentation should include a review of the literature, an explanation of the student's dissertation proposal or a dissertation progress report. After a student performs his or her oral report, two other students are responsible to critique the contents.
2. Peer critique: Each student is assigned to critique peer reports to help develop their skills of analysis and integration skills.
3. Special lectures: Each semester, two newly graduated nursing doctors met with students to discuss their dissertation process and findings.
4. Written term paper: At the end of each semester, students are required to submit a publishable manuscript of their work, the focus of which is on enhancing student training in dissertation and manuscript writing.

This course allows students to develop self-directed learning capabilities, trains students to accept different opinions with an open mind, affords opportunities to read and review research papers from different fields, provides familiarity with the focus of current research and important findings within each major nursing field and helps students grasp the salient points of dissertation writing. These multiple approaches assist students to construct a framework for their own dissertation work. Teaching strategies and teaching methods of this course are consistent with “student-centered” interactive teaching ideals, which enhance students' grasp of relevant information, ability to analyze and appraise the accuracy of acquired knowledge, and ability to integrate information through interactive cooperation among student peers (^{Boswell, 2006; Hsieh, 1997; Hsu, 2000, 2001; Hsu & Yao, 2003; Lin et al., 2005)}. These teaching ideals are more beneficial to students than traditional didactic lectures (^{Baker, 2000; Harden, Crosby, Davis, Howie, & Struthers, 2000; Hsu, 2001, 2004)}. Development of these skills is especially important in Taiwan, where the emphasis of education has always been rote memorization of course material (^{Hsu, 2004}).

In order to understand whether the objectives of this course were achieved, course evaluation was performed after course completion. Literature references point to using the educator teaching methods and skills, student learning behavior and learning outcomes, and course design and implementation of the course as the basis of evaluation work (^{Lin et al., 2005)}. The main purpose of course evaluation was to understand student experiences. Such can be reinforced, if required, to allow students to understand their own progress, identify students' professional skills, and determine teaching efficacy (^{Hsieh, 1997)}. Traditional evaluation methods consist primarily of evaluating knowledge obtained by students from teaching material, which emphasizes the use of standardized written examinations. Students are viewed as passive learners, with less focus on evaluating students' perceptions during the teaching and learning process. This method was deemed inappropriate for evaluating this course. Education in the 21st century places greater emphasis on “student-centered” learning approaches that allow students to participate actively in learning. In such, the teacher is a facilitator in the student learning experience, emphasizing cooperative learning (so-called heuristic teaching). Course design and evaluation, therefore, must be consistent with teaching strategies. Evaluations must reflect the application of student knowledge and emphasize the relevance of courses for students. This involves assessing the subjective understanding of students in addition to considering the objective evaluation of teachers (^{Drake, 1998/2001)}. Course objectives are most commonly employed as the basis of evaluation and are especially useful to help understand the actual extent to which course objectives have been realized through teaching activities. At this point, it is necessary to understand student capabilities prior to commencing the course (i.e., pre-teaching evaluation). Comparing differences between pre- and post-teaching evaluations is useful in evaluating teaching efficacy and identifying points of learning interest and difficulty. Results can be used to adjust the teaching process. Another course evaluation method is to understand student responses to the course, focusing mainly on the effects of student learning experiences and learning context. The current trend is to use multimodality teaching strategies and evaluation methods. Besides evaluation of both learning processes and outcomes, it also provides more opportunities for critical thinking and creative expression (^{Ellis, 2005; Graff, Russell, & Stegbauer, 2007; Holzemer, 2005; Hsieh, 1997; Huang, 1990; Drake, 1998/2001; Huang & Tsai, 1999; Lin et al., 2005; Maki & Borkowski, 2006}).

The current literature includes overall program and single course evaluations of different nursing education systems (^{Hsu, Tang, & Huang, 2005; Hung & Lee, 2001; Liu, 1999; Wang, 1996)}. After 10 years of nursing doctoral education in Taiwan, no course evaluations similar to this study have been conducted. The purpose of our study was: (1) to understand the learning efficacy of doctoral students with regard to the Nursing Research Seminar course and (2) to understand differences in the perceived level of competency in terms of each course objective between doctoral students at different year levels in the program. This helped researchers assess the value and necessity of this course to doctoral students and can help serve as a basis for modifying the course in the future.

Methods

This study utilized a cross-sectional design facilitated by the use of a questionnaire survey. Study participants included 26 doctoral students enrolled in a newly opened Nursing Research Seminar course offered by the Department of Nursing at a national university in Taiwan. Data were collected during the final course session (in January 2004). Questionnaires, distributed anonymously to the doctoral students enrolled in this course, were collected by a nominated representative and returned to course faculty. A total of 25 questionnaires were distributed and collected from a total potential participant pool of 26 (one potential participant was on maternity leave and was not included).

The evaluation form used in this study was constructed by two course faculty. The evaluation form achieved face validity as items targeted were based on course objectives established by a consensus of the department's Doctoral Program Curriculum Affairs Committee. The evaluation form consisted of two parts, the first of which was a self-evaluation of pre- and post-course competencies along each of the six course objectives using a 5-level Likert scale. Student subjective evaluations of their own change between pre- and post-course competencies were used to measure course learning efficacy. Total possible scores ranged from 6 to 30 points. Part two consisted of open-ended questions, which prompted students to detail the most beneficial objectives and provide additional suggestions. Findings from part one only are presented in this paper.

After coding, data from the evaluation forms were entered into SPSS 10.0 statistical software for statistical analysis. In addition to using descriptive statistics, results were presented using non-parametric analysis to compare differences between doctoral students in a manner that accounted for the fact that most participants were part-time students and at different stages in their doctoral degree program. Most of the required course curriculum could be completed in 2-3 years, leaving the fourth year and beyond for dissertation work. Students were further divided into four groups during data analysis: first year (Group 1), second year (Group 2), third year (Group 3), and fourth year and above (Group 4).

Results

The 25 evaluation forms collected included four from Group 1, three from Group 2, five from Group 3, and 13 from Group 4. All were valid. Of the 25 students, 24 were part-time students, and one was a full time student. The mean total pre-course score of the six objectives was 18.40 ± 5.52, and the mean total post-course score was 24.76 ± 2.89; the post-course score was significantly higher than pre-course score (Z = −4.30, p < .001). Overall, this course was beneficial to students (Table 2).

Table 2: Pre- and Post-Evaluation Scores for Each Course Objective (N = 25)

In terms of perceived level of competency along each course objective, the mean pre-course score of each objective ranged between 2.92-3.12, and the mean post-course score of each objective ranged between 4.04-4.28. The difference between pre- and post-course scores was statistically significant for each objective, indicating that postcourse scores were significantly higher than pre-course scores (Table 2). Ranked in decreasing pre-course score order, defined objectives were “refining critical research skills” (3.12 ± 0.88), “evaluating and integrating research papers” (3.12 ± 1.01), “enhancing ability to construct research proposals” (3.12 ± 1.17), “gaining an in-depth understanding of the literature related to topics of interest” (3.08 ± 1.04), “guiding student dissertation work” (3.04 ± 1.10), and “enhancing critical thinking skills” (2.92 ± 0.86). Post-course, in decreasing order of scores, the objectives were, respectively, “gaining an in-depth understanding of the literature related to topics of interest” (4.28 ± 0.54), “evaluating and integrating research papers” (4.16 ± 0.47), “enhancing critical thinking skills” (4.12 ± 0.53), “enhancing ability to construct research proposals” (4.08 ± 0.81), “refining critical research skills” (4.08 ± 0.70), and “guiding student dissertation work” (4.04 ± 0.73) (Table 2).

In terms of post-course rank changes in terms of each course objective, greatest rank changes occurred in “gaining an in-depth understanding of the literature related to topics of interest” (from 4th to 1st), and “enhancing critical thinking skills” (from 6th to 3rd). After using Wilcoxon matched paired signed ranks test to analyze post-course changes in scores, differences between results all reached statistical significance in terms of both the course as a whole and the six course objectives individually. In other words, post-course scores were significantly higher than pre-course scores (Table 2).

Pre- and post-course scores as segregated by doctoral student group (by year in program) are shown in Table 3. Pre-course scores were lowest in Group 1 students, with mean scores for each objective ranging between 1.25 and 1.75. These were followed by Group 3 and Group 4 students, with mean scores for each objective ranging between 2.60 and 3.69. Group 2 students had the highest score, with mean scores for each objective ranging between 3.00 and 4.00. In post-course results, the lowest scores still occurred in Group 1 students, with mean scores for each objective ranging between 3.25 and 4.00. The highest scores were still from Group 2 students, with mean scores for each objective ranging between 4.00 and 4.67 (Table 3).

Table 3: Pre- and Post-Evaluation Scores for Each Course Objective by Group (N = 25)

In terms of improvement scores (post-course minus pre-course) (Table 4), overall, the most improvement occurred in “gaining an in-depth understanding of the literature related to topics of interest” (mean improvement score 1.20 ± 0.82) and “enhancing critical thinking skills” (mean improvement score 1.20 ± 0.91), followed by “evaluating and integrating research papers” (mean improvement score 1.04 ± 0.84), “guiding student dissertation work” (mean improvement score 1.00 ± 0.76), “refining critical research skills” (mean improvement score 0.96 ± 0.68), and “enhancing ability to construct research proposals” (mean improvement score 0.96 ± 0.68).

Table 4: Comparisons of Improvement Scores of Each Course Objective by Groups (N = 25)

In terms of improvement scores and student year levels, Group 1 students had the highest total improvement score (mean improvement score 12.50 ± 4.80) and the highest improvement score in each course objective. These were followed by Group 3, Group 2, and Group 4, with mean improvement scores of 5.80, 5.33, and 4.92, respectively. In terms of each course objective, Group 1 students experienced the greatest improvement in “enhancing critical thinking skills” (mean improvement score 2.50 ± 1.29), followed by “gaining an in-depth understanding of the literature related to topics of interest” (mean improvement score 2.25 ± 0.96). Group 2 students experienced the greatest improvement in “evaluating and integrating research papers” (mean improvement score 1.67 ± 1.53), followed by “enhancing critical thinking skills” (mean improvement score 1.33 ± 1.15). Group 3 students experienced the greatest improvement in “gaining an in-depth understanding of the literature related to topics of interest” (mean improvement score 1.60 ± 0.55), followed by “refining critical research skills” (mean improvement score 1.20 ± 0.45). Group 4 students experienced the greatest improvement in “enhancing critical thinking skills” (mean improvement score 0.92 ± 0.49), followed by “enhancing ability to construct research proposals” (mean improvement score 0.85 ± 0.38), and “guiding student dissertation work” (mean improvement score 0.85 ± 0.69). When a Kruskal-Wallis one-way analysis of variance by rank was used to analyze post-course improvement scores at different year levels, statistically significant differences were found in improvement scores for each course objective between doctoral students at different year levels, with year level corresponding negatively to improvement score for each objective (Table 4).

Discussion

Study results demonstrated that, upon completing the Nursing Research Seminar course, doctoral program students realized significant improvements in each ability through an examination of pre- and post-course evaluations. Such demonstrates the efficacy, value and necessity of this new course. Overall, this course had the greatest efficacy for the largest number of students in terms of three objectives, namely “gaining an in-depth understanding of the literature related to topics of interest,” “enhancing critical thinking skills,” and “evaluating and integrating research papers.” This was especially true for the lowest pre-course objective, “enhancing skills in critical thinking,” which showed the greatest improvement between pre- and post-course evaluations both in terms of overall improvement score (improvement score 1.20 [1st], Table 4) and rank (from 6th to 3rd, Table 2). Critical thinking ability is a skill critical to thinking independently and solving problems successfully (^{Alfaro-LeFevre, 2005; Barnes, 1992; Boswell, 2006)}. Some researchers (e.g., ^{Glasman, Koff, and Spiers [1984}]) define critical thinking as a deductive ability encompassing the four areas (1) ability to clarify the problem and propose possible solutions, (2) ability to recognize errors, (3) ability to discover and describe a reasonable conclusion using different sources of information, and (4) ability to discriminate between fact and opinion. Therefore, the essence of critical thinking is analytical and reflective. Students who have critical thinking ability are able to use problem solving processes to connect and evaluate each aspect of a problem and also to collect and analyze data to determine a reasonable conclusion. In practice, nursing often involves complex situations and decisions, requiring continuous updating of knowledge and techniques. If nursing personnel are not equipped with critical thinking ability, they may become part of the clinical problem themselves (^{Alfaro-LeFevre, 2005; Boswell, 2006}). Therefore, critical thinking ability is especially important in the nursing field, explaining why “improving independent thinking and critical thinking abilities” is listed as one of the main teaching objectives in nursing doctoral education. The other two objectives, “gaining an in-depth understanding of the literature related to topics of interest” and “evaluating and integrating research papers,” are also very important abilities in nursing doctoral education. However, only by applying critical thinking can related literature be integrated, and, similarly, the same skill is required to obtain a more thorough understanding of how to conduct research. Therefore, critical thinking capability is essential to improve the research abilities of students. Besides this, students who believed these three abilities as beneficial were roughly distributed among program year levels. No year level specificity was observed, further demonstrating that the contribution of this course to “enhancing critical thinking skills” was endorsed by all student groups.

The mission of nursing doctoral education is to prepare leaders and researchers in each nursing specialty. Research ability is a core skill required of every nursing researcher because it not only allows the application of evidence-based knowledge in each field, but many unsolved problems can be answered only by research. The literature has pointed out that during doctoral education learning, in addition to training in research ability, there is also a need to provide a stimulating learning environment for students, designing courses to meet the needs of students, and to provide mutual support (^{Barnes, 1992; Chen, 2004; Graff et al., 2007)}. This new course was established to provide a “stimulating learning environment for furthering student study and research,” focusing on student research interests and using methods such as integrating literature and research findings, and providing teacher and colleague feedback to improve student research competencies. Overall, learning efficacy was demonstrated for this course mainly because the following assistance was provided to students: (1) Support and guidance - all faculty participated in and led discussions, providing students with a supportive learning environment while also allowing faculty members to understand student learning progress and difficulties, (2) Maintaining content heterogeneity - all department faculty and students participated, allowing students to access research findings from students in different specialty areas, (3) Guiding the research interests of students - students gained a better understanding of faculty and colleague research interests through weekly discussions that facilitated mutual learning and encouraged possible future research collaborations, and (4) Practicing critical thinking - this produced different levels of efficacy in doctoral students at different year levels. Students believe this is a practical course that provides training in various skills.

Is this course beneficial for doctoral students at all year levels? The results of this study showed significant differences between the improvement scores of doctoral students at different year levels for each course objective. An improvement in critical thinking ability was seen in all groups, but the practical contributions of this course differed among students at different year levels. Among first year doctoral students, the objective that improved the most was “gaining an in-depth understanding of the literature related to topics of interest.” Among second year doctoral students, the objective that improved the most was “evaluating and integrating research papers.” Among third year doctoral students, the objective that improved the most was “gaining an in-depth understanding of the literature related to topics of interest.” And among doctoral students in the fourth year and above, the objectives that improved the most were “enhancing ability to construct research proposals” and “guiding student dissertation work” (Table 4). These results show that this course had specific levels of efficacy for doctoral students at different year levels, and that these specific benefits were consistent with doctoral student needs at different year levels. Because students were at different stages of their studies, different results were obtained due to differences in the extent of prior knowledge. Students at higher year levels had already completed required courses, and had more prior knowledge. Specifically, doctoral students in the fourth year and above felt that “understanding knowledge and research from other fields” was beneficial, which was an objective not mentioned by doctoral students from other year levels. This was likely because higher year level students had better prior knowledge of other objectives and had also acquired sufficient understanding of their own field of research. As such, they were more able to appreciate research from other fields with an open mind and utilize a wide range of available information, understanding contributions from other fields more readily. Conversely, lower year level students gave lower scores to learning objectives such as “guiding student dissertation work” and “enhancing ability to construct research proposals,” indicating that these objectives had lower benefit. However, although they had a lower rank, the total improvement scores for these objectives ranged between 1.75 and 2.00 among first year doctoral students, which was still relatively high when compared to the improvement achieved by students at other year levels. Overall, the degree of improvement of first year doctoral students in each course objective was still higher than that at other year levels, demonstrating that this course benefited first year doctoral students in all areas.

Pre- and post-course student evaluation rankings for “guiding student dissertation work” were almost always last, showing that there is room for improvement in this course in regard to guiding doctoral dissertation work. Results also indicate that guiding doctoral dissertation work is a long-term process and cannot be achieved in a short period of time or in a single lecture. In designing the nursing doctoral education course, questions such as “how to supervise doctoral students” and “how well prepared is the supervising faculty” are issues worthy of thought by nursing educators.

Overall, the results of this study showed the learning efficacy of doctoral students in the new Nursing Research Seminar course to be significant, which also demonstrated the necessity of this course.

Limitations and Suggestions

The results of this study faced the following limitations: (1) Evaluation was performed on a single course taught during the nursing doctoral program at a single national university. Although the results should be useful when applied to the design of future courses, lacking a comparison group and with its small sample size, generalizability is limited. Any conclusions drawn from this study should be treated as tentative. In the future, more in depth evaluation for nursing doctoral programs or for other related courses should be conducted. (2) The evaluation tool used in this study only focused on the six course objectives, and did not include all evaluations of learning efficacy. In the future, besides developing more comprehensive evaluation tools, evaluations of other courses should also be included. It is also suggested that evaluation mechanisms should be constructed for Taiwan's nursing doctoral education system.

Conclusion

The purpose of this study was to understand the learning efficacy of doctoral students at the first doctoral nursing program in Taiwan as it applied to the new Nursing Research Seminar course and to evaluate the effects of this new course on doctoral students at different program year levels. Twenty-five doctoral students self-evaluated change in pre- and post-course level of competency along each of the six course objectives. The results of student evaluations showed significant improvements in course objectives observed among all program year groups. Overall, the highest improvements in scores were observed in “gaining an in-depth understanding of the literature related to topics of interest” and “enhancing critical thinking skills.” Analyzing improvement scores by students at different year levels showed significant differences between year levels in terms of course objective improvement scores. Generally, the lower the year level, the higher the degree of improvement. The three most beneficial objectives of this course for doctoral students were: (1) “gaining an in-depth understanding of the literature related to topics of interest,” (2) “enhancing critical thinking skills,” and (3) “evaluating and integrating research papers.” The results of this study demonstrated the efficacy, value, and necessity of this new course. The course was beneficial in different areas for doctoral students from different program year levels and is worth continuing. The format of the course has been revised based on findings and student comments. The presentation time for each student was shortened to 20 minutes, followed by critiques by two peers. More time was also left for free discussion and comments from the class and faculty. Each student must register for this course 4 times to fulfill program requirements. First-year-students must audit the class, with no credit granted. Most students registered for this course during their second to third years of doctoral study. Students at higher year levels who had completed the course requirement were scheduled on a periodic basis to present on dissertation work progress. This strategy not only allows for periodic monitoring of student progress, but also lets students learn from one another. The outcomes of the revised format of the course, such as shortened the presentation time, more time for critiquing and discussion, and timing and frequency of taking this class by students, need to be investigated further in the future.

Acknowledgments

We are grateful for the participation of and the extensive information provided by the 25 nursing doctoral student participants. We also wish to thank all the faculty members at the department of nursing for their participation in this course; without them this study could not have been completed.