Several recent crises have affected the global community and highlighted the importance of health information exchange for managing infectious disease outbreaks and other widespread health challenges.1,2 Healthcare professionals with competence in health informatics play a key role in building the international capacity to improve knowledge regarding prevention and treatment of a variety of threats to worldwide health. International sharing of health information will increasingly become the currency of healthcare.2
In 2010, the International Medical Informatics Association’s (IMIA’s) working group recommended the development of international, interdisciplinary educational opportunities in health informatics.3 Educational programs developed in this manner allow students to be exposed to different ways of organizing healthcare services and health data and an understanding of global variations in health education and research.4
Despite encouragement from IMIA, challenges in delivering adequate interdisciplinary health informatics education remain.3,5,6 The need to merge information science and technology disciplines with clinical health science has been emphasized.7 To address these challenges we developed a graduate International Interdisciplinary Health Informatics (IIHI) course for future healthcare informaticists based on experiences from health informatics courses regularly offered at two US universities and one Norwegian university involved in the project. In all of the topics covered in the course, an international focus was included in the literature, lectures, and exercises assigned. Only students already enrolled in graduate programs at the three universities were eligible to participate in the course. Seven Norwegian healthcare professional graduate students (a mix of five nurses and two healthcare professionals), 16 US graduate nursing students, and two graduate biomedical informatics students participated in the course. The backgrounds of the students were similar to typical informatics courses offered at the universities. The students earned three US credit hours, corresponding to six European Credit Transfer and Accumulation System8 credit points in Norway over one semester in the spring of 2014. No specific curriculum standard was used, but a literature review was conducted as a part of the planning process. We utilized the educational expertise of the four professors from nursing informatics, health informatics, and health information systems. The course built on a virtual learning environment used earlier with students from Norway and the United States.9 The three universities applied for funding from the Norwegian Centre for International Cooperation in Education (SIU) in 2012. The $20 000 grant funded the development of the IIHI course. The book by Amatayakul10 was used together with relevant research articles published in high-quality journals. The weekly individual and group assignments were weighted, and the team of professors jointly did the grading.
The aim of this article is to present a cross-countries and interprofessional novel approach for teaching a graduate IIHI course online. We describe the innovations needed to develop content and structure for the graduate IIHI course, which was taught across two universities in the United States and one in Norway. The article addresses challenges, motivators and enablers, lessons learned, pedagogical recommendations, and future research directions.
Setting and Participants
A collaboration between one of the US universities and the Norwegian university was established in 2002, when a memorandum of understanding (MOU) was jointly endorsed by the US university chancellor and the Norwegian university president. The MOU addressed ongoing faculty and student exchanges and shared online courses in various fields. In 2012, an informatics professor in a college of nursing from a state university medical center was invited to join the team in an application for funding from the SIU. The grant funded the development of a joint course in IIHI for spring 2014.
The content expertise of the four professors included information security, clinical informatics, clinical database design, government health policy, information system analysis and design, human-computer interaction, and distance education. The class consisted of full-time and part-time students. The IIHI course was compulsory for 16 nurses enrolled in the master’s-level program in leadership/administration and an elective for the other students. Some students had extensive experience in online courses, and for others, this was their first course taught through distance technology.
Development of the Online Course
The IIHI course was designed to be part of the curriculum for all three graduate programs (nursing, biomedical informatics, and health informatics). A graduate-level-course offering was selected for this collaboration for two reasons: the undergraduate degrees have fewer electives and the learning objectives of the course developed were aimed at higher-level students. Curriculum advisors for all institutions were involved and assisted in obtaining curricular approval.
The professors met six times using Adobe Connect (Adobe Systems, San Jose, CA) and/or Skype (Skype Technologies, Luxembourg City, Luxembourg) approximately once a week during summer 2013, concluding with an on-site planning session in Norway in August 2013. Planning continued during the fall, and the syllabus was completed in November 2013. One university was selected to host the course management system, Blackboard, (Washington, DC) for all students. Because of a longstanding collaboration, one of the universities had routines to allow guest professors and guest students on Blackboard.
The syllabus structure was based on a syllabus used by faculty in the US college of nursing who taught an introduction to health informatics. This syllabus followed an existing framework for online course delivery and the college guidelines. Details specific to the other two universities were added as necessary. The course was taught in English, and all the readings were in English.
Content for each week was assigned to one or two professors based on his/her specific areas of expertise and interest. The professor(s) assigned to a given week was independently responsible for selecting readings, designing assignments, participating in the discussion board, and grading activities for the week. All team members reviewed the content and were welcome to participate in weekly discussions and did so on a regular basis. Each professor retained accountability to his/her own university/college for course quality. Two professors had extensive online teaching experience and served as mentors to the other two professors in regard to use of the online methodology and delivery.
The main goals of the course were for students to have a broad understanding of international issues in interdisciplinary healthcare informatics and to appreciate the differences and similarities between the two countries healthcare delivery systems. Table 1 provides an overview of specific course objectives.
The weekly schedule of topics is shown in Table 2.
The assigned professor prepared and posted the following materials weekly:
- learning objectives for the weekly topic as listed in the syllabus and an outline describing the most important materials for the course;
- reading assignments (scholarly articles and chapters from a textbook);
- links to online materials/videos, discussion materials, and readings (some custom-made materials and some Internet resources); and
- discussion questions/task for the groups to discuss on the discussion board, including student expectations (developed by the assigned professor).
Students were divided into groups of four to five students for the weekly discussion boards. Student groups included students from all three universities with both clinical and technical perspectives. The course was compulsory for one of the groups (nursing) and elective for two of the groups (biomedical and health informatics). Students were asked to discuss and respond primarily in their designated group. In addition, they were also encouraged to read communication exchanges that occurred in other groups. One example from Week 12 security/privacy was to “Describe a specific instance where a privacy or security violation occurred. Describe why this is a violation, what response you would take, and how you would investigate.”Another example from Week 6 reporting/data output/user-interface design: “For this assignment you have two options: (1) identify a reporting need for your organization and create a dashboard to meet that reporting need or (2) identify a personal healthcare-monitoring tool that would benefit patients and create a dashboard to meet that need.”
Most weeks students were expected to contribute to the discussion at least two different times. Students were responsible for completing the assignment or discussion on Thursdays (official class day). Students were then expected to read the contributions of their group and to respond directly to at least one contribution in their group by the next Monday. Professors contributed to the discussion throughout the week. On occasion, professors directed students to specific posts in another group when a student made a particularly astute observation or when the professor had made an important comment. This process of sharing expertise and insights across groups resulted in minimal redundancy for professors and students and increased efficiency.
Student Expectations on Discussion Board Contribution
The discussion board rubric provided criteria for the evaluation of initial student comments on Blackboard assignments and questions. The rubric also provided criteria for evaluating student responses to posts from peers regarding both the quality and the quantity of responses. The quality of student responses was based on issues such as if the student made a suggestion for an enhancement, related the dashboard to his/her own work and how it might be useful, and mentioned how the dashboard related to the readings for the week. The assignments and responses were based on the context of the workplace. The students were expected to write professionally but were not expected to follow academic or publication formatting requirements. The focus was on practice-based, open-format learning as discussed in IMIA’s recommendations.3
Online Synchronous Sessions
In the first, the 10th, and the final week of the term, voluntary 45 minutes of nonrecorded online synchronous sessions was offered during the lunch hour, US time. This time corresponded to early evening in local Norwegian time. Adobe Connect was used, but a Web camera was not required. No option to attend in person was offered. Students and teachers introduced themselves, and the course outline was presented in the first week. Students took advantage of the time to interact synchronously.
Much of the time during the synchronous classes was used to discuss the differences among the three universities, as well as expectations on grading and examinations. The sessions were optional but strongly recommended. A midterm meeting was scheduled to check in with the students. The last week of the semester was a closing session and included a discussion on the final examination.
Student assessment was a shared responsibility between two professors. All professors graded the first assignment as an internal consistency test. Variance in grading was discussed in the weekly Skype call, and adjustments in grades were made as indicated. A grading rubric was provided to the students. Many of the students had not previously experienced such high expectations for discussion board participation. Professors gave extensive feedback to explain the rationale for their grading and to assist students in adjusting to course expectations. After the first discussion board, the professor(s) responsible for the week’s curriculum did the grading.
Discussion board grading can be a time-consuming endeavor with an online class of 25 students. To minimize faculty workload while maintaining timely feedback, the weekly discussion board tasks were graded in a random fashion. This grading methodology had been created and used successfully for more than 5 years by one faculty member. This approach provided more time for the faculty member to participate in the conversation, helping all students, rather than concentrating on feedback to one student at a time.
In using this methodology students were graded for only 3 to 5 weeks of the semester for each student. Students did not know which weeks would be graded. Approximately six students were graded each week. At the end of the course, the grades for the graded weeks were averaged. Students who knew that they would be having a particularly rough week or who became ill could request that a given week not be graded.
In addition to the weekly assignments, students completed a course project and wrote a final examination. The course project consisted of an informatics-related proposal that would theoretically require administrative or legislative approval. The article was to be written in a style consistent with institutional standards for similar “proposals.” Students were specifically instructed that this was not a research article. Students proposed project plans for system implementation, new or revised informatics-related policies/procedures, development of a decision support system or other clinical tool, or a white paper in support of proposed legislation.
For the final examination, a health informatics initiative proposal for a healthcare organization from a previous year’s course (with author permission) was provided to the students. The open-book final examination consisted of the student evaluating the proposal as if he/she was on a healthcare organization committee tasked with evaluating institutional proposals. The evaluation was to consist of a discussion of the strengths and the weaknesses of the proposal, as well as a decision whether to fund this proposal. A discussion of the rationale for the decision was also required.
As a part of the grant, six US students were randomly selected and awarded scholarships to cover their travel expenses to visit Norway. The students who did not receive awards had the option to self-pay, and all the US students were informed about the scholarships in advance and that they needed to organize vacation time for the traveling. The US students and professors and the Norwegian students were offered an intensive and enriching week at the Norwegian University. The goals for the visit were to expose the students to clinical, social, and technology differences between the two countries. Under the guidance of the Norwegian professors, US students and professors visited information communication technology (ICT) projects in two different municipalities, a vendor of the ICT and ICT support to one health region in Norway, the local public hospital, a nursing home, and an independent-living facility, which is a “living-lab” with welfare technology. Professors and students participated in multiple discussion meetings for sharing ideas and for reflections during the week in Norway. The sites visited and the topics discussed were selected to provide the students and professors an opportunity to further discuss cross-cultural opportunities and challenges in the use of ICT in the healthcare services in Norway.
Students shared these experiences with those not traveling through the discussion board and a midsemester synchronous class using Adobe Connect. A small group, less than half of the class, participated in the synchronous class because they were all part-time students. The US students highlighted the fact that in Norway they used only one information system for the health record for all inpatients and one of three alternative systems for all outpatient services. The limited vendor competition was a significant difference from the US system. Students noted the distinct separation of the inpatient system from the outpatient system in both health record and infrastructure to support the software. Other differences experienced are listed in Table 3.
Extensive curriculum planning allowed the professors to deliver culturally rich and diverse health informatics content, although a few challenges were encountered during the course delivery. Some of these issues were both a challenge and strength. One noticeable challenge was the magnitude of the differences in professional backgrounds and educational and learning expectations of the students. This diversity challenged the faculty in areas of vocabulary and baseline knowledge. At the same time, it significantly enriched the course as students shared their experiences and asked clarifying questions, helping to broaden the knowledge of all participants. These experiences are similar to results from a case study of students in an undergraduate online course with interuniversity teaching. The participants in this case study recommended the type of teaching to prepare students for a global work life.4
Another challenge experienced was that some of the Norwegian students had difficulty in the extensive use of English as a study language, especially for written assignments and communication within the discussion board. In contrast, the discussion board promoted the use of English and facilitated an improvement in English writing skills for the Norwegian students. While written skills are crucial to the learning experience and may have disadvantaged the Norwegian students,7 most of the research literature in the field of health informatics is in English, so the Norwegian students gained a necessary skill. The professors worked to minimize the disadvantage by recognizing issues of using a second language and not penalizing the Norwegian students for their English usage.
Another issue was the uneven distribution of students across the universities. The majority of the students were nursing students. The US nursing students had more prior experience with online courses because their university used more online courses in nursing than did the Norwegian university students, and they used Blackboard regularly. The Norwegian students used Blackboard for the first time. Students with online learning experience had better time management and navigated through Blackboard more easily. Not all students embraced the teaching modality.
Student engagement with the content varied across programs. Students for whom the course was an elective were often more engaged and excited about the content. Overall, the students adjusted by the third week of the semester and realized the value of the international and global perspectives.
Creating a learning community has been discussed in earlier research as more challenging in online teaching than in traditional teaching.11–14 However, most of the research has taken place in North America, and the applicability of the theoretical ideas and prescriptions may be limited to cross-country learning environments.12 Creating communities among learners is aligned with definitions of high-quality reflective teachers who adjust their instruction to the students.13 The concept of developing learning communities among students is based on a pedagogical foundation, which emphasizes the importance of collaborative learning and the social construction of knowledge.14
Faculty invested a lot of time in establishing a learning community among the group. However, establishing an online learning community may have been more challenging in this course than in other online courses. Several of the students had already established face-to-face learning communities with some of their fellow students; hence, there may have been little incentive to include online students from the other universities.
Another disadvantage for some of the US students was that not all of them could travel to Norway because of limited funding from the project grant. The students knew from the beginning that only some would receive travel stipends. The grant covered the traveling expenses for the professors. The professors were sensitive to this and made sure that student interaction was not hindered. Although we had a few challenges, overall, the course was very well received, and the students’ evaluations validated how much the diversity of the students and professors enriched their learning.
Pedagogy Lessons Learned
Not only did the students experience an enriched education, but faculty likewise benefited. Sharing instruction provided a way for the professors to evaluate and discuss personal teaching strategies and may have improved and expanded their ability to conduct subjective grading. All faculty graded the initial discussion board assignments and shared grading on the research articles. When there were questions, they were discussed for consistency.
Faculty understanding of perspectives on health informatics was expanded. For example, the broad acceptance of and techniques for ubiquitous sharing of data across healthcare providers and facilities in Norway expanded the perspectives of the US faculty.
Weekly Skype discussions on the upcoming course material ensured that all professors had a good understanding of the purpose of the assignments. The opportunity to give and receive critique from peers strengthened the quality of the professors’ teaching.
The course was enriched by incorporating nonclinical material and perspectives. In the same way, the clinicians enriched the perspective of those coming from a more technical background.
Grade Expectations and Grading Criteria
The three universities had slightly different course expectations and grading criteria. The students did not share the same “definition” of a good grade, and this proved to be a key cultural difference. In the United States, graduate education students are expected to achieve a B or better, equating to approximately an 80% or higher grade in graduate studies. In Europe, graduate grades are based on a bell curve centered around 50%, the equivalent of a C in the United States.
The US college of nursing professor had noticed in past years that the grading for the first week of the discussion board was key. If expectations were set high, students adjusted and performed better for the rest of the semester. Students were prone to not put forth maximum effort the first week, and thus grades on the initial discussion board ranged from 60% to 90%, averaging approximately 80%. The US students were not pleased with these grades, and most put forth increased effort in subsequent weeks. In contrast, the Norwegian students who generally received grades between 40% and 60% were pleased with grades averaging 80%. The quality of their submissions was similar to the US students, but they lacked the perception that they needed to work harder in order to increase their grades. The professors carefully clarified the grading interpretations to the students, which resulted in increased participation by all students.
Course Delivery Lessons Learned
The difference in time zones was challenging, but not impossible. US graduate students often work full time and take courses in the evenings. Evening in the United States is in the middle of the night in Norway. Therefore, the US lunch hour/Norway early evening proved to be the only time when both groups were likely to be awake and available. Most US students could arrange to have time free over their lunch hour to participate in synchronous class activities. Communicating this information to the US students before class began was essential as this scheduling did not follow US tradition.
US graduate courses are typically 3 credit hours. The expectations on the amount of course work, readings, activities, and student involvement were carefully communicated to all students. Courses in the United States follow the requirements of the Department of Education,15 where the expectation is that 1 credit hour consists of 2 hours in class and approximately 3 hours of work outside the classroom. Thus, for this 3-credit-hour online course, US students expected to put in approximately 9 hours each week on class work. In contrast, the Norwegian students expected more flexibility on managing the class work. They generally focused their efforts at the end of the course.
Course Management Lessons Learned
A major difference was the expectation for weekly assignments. Norwegian students were more accustomed to smaller amounts of work during the course and an intense period of activity at the end of the course when major projects and examinations were due. Many of the students, both Norwegian and American, tended to struggle with the expectation of meeting the two-times-a-week discussion board deadlines for posting blogs. The dialogue resulting from interacting with peers twice a week increased the richness of the discussion, before moving on to the next topic. To communicate this expectation, a more in-depth introduction and emphasis on time commitments and involvement are suggested for future course offerings.
Motivators and Enablers
It was observed that the students successfully mastered the content. This was evident in their discussion boards, short assignments, and examinations. The professors also noted that teaching the course internationally did not disadvantage the students. Norwegian students (five nurses and two other healthcare professionals) were able to adapt to the US driven course organization, and the students without clinical backgrounds (biomedical informatics) were able to interact and learn from the healthcare professionals and vice versa. Although the underlying course structure was from a nursing informatics program, the expanded interdisciplinary curriculum in information systems and international healthcare issues proved to be beneficial. Professors thoughtfully selected the additional course content to extend beyond a clinical perspective.
Another factor that contributed as motivator was that the professors were available and very involved in the discussion boards. Faculty were contributors and asked probing questions when necessary. The discussion board organization helped everyone to get up to speed and learn other disciplines, as well as to build a sense of community. Many students shared their experiences. They recognized the similarities and the differences.
Highly motivated students and faculty, an online technology infrastructure that supported asynchronous communication and delivery, course management technology, an ability to adapt the curriculum to meet the pedagogy requirements for all universities involved, and the support of higher administration for global collaboration in education were critical to the successful outcomes of the IIHI course. Teaching this course would have been more difficult without faculty advocates from each institution who had diverse expertise and knowledge.
This project demonstrated the feasibility and advantages of an interdisciplinary interprofessional and cross-countries approach in teaching health informatics online. Students were able to establish relationships and conduct professional conversations across disciplines and international boundaries using content management software. This graduate course can be used as a part of informatics, computer science, and/or health science programs.
The IIHI course was a tremendous opportunity for professors and students to learn more about the healthcare systems and healthcare informatics in the two countries, and we will continue to explore opportunities to offer this joint course. We were able to do this because of support from the host university to use its course management system, the successful earlier collaboration between two of the universities, the curriculum planning process, and the funding from the SIU.
The authors thank the SIU for funding this project.