Secondary Logo

Journal Logo

Original Study

Survey-based Evaluation of the Use of Picture Archiving and Communication Systems in an Eye Hospital—Ophthalmologists’ Perspective

Lam, Thomas Chi Ho MRCSEd(Ophth)∗,†; Lok, Jerry Ka Hing MRCSEd(Ophth), FCOphthHK; Lin, Timothy Pak Ho MBChB; Yuen, Hunter Kwok Lai FRCOphth, FRCSEd∗,†; Wong, Mandy Oi Man FCOphthHK, FRCSEd(Ophth)∗,†

Author Information
Asia-Pacific Journal of Ophthalmology: May-June 2022 - Volume 11 - Issue 3 - p 258-266
doi: 10.1097/APO.0000000000000467

Abstract

Picture archiving and communication system (PACS) is a medical imaging system that provides storage and access of images from multiple modalities. It consists of image acquisition devices, storage archiving units, display stations, computer processors, and database management systems, which are integrated by a communications network system.1 Since the introduction of the concept by Lemke et al in 1979,2 PACS has been widely adopted in the field of radiology worldwide, in the hope of achieving “filmless radiology”.3–6

In recent decades, the rapid evolution of ocular imaging technology brought about increasing reliance on graphical information to guide the diagnosis and treatment of eye diseases.7,8 In Hong Kong, with an annual attendance of over 1 million in the public ophthalmology service, around 1.7 million ophthalmology images were handled every year.9,10 Before 2018, common ophthalmic investigations, eg, visual field (VF), optical coherence tomography (OCT) of retinal nerve fiber layer (RNFL) and macula, fluorescein angiography (FA), and indocyanine green angiography (ICG), were printed and filed into paper medical records manually. An electronic imaging system is imminently needed to reduce filing errors, facilitate retrieval, reduce storage space, and to prepare for the advent of artificial intelligence.11 In the 2000s, a large-scale web-based electronic patient record (ePR) system with image distribution based on enterprise PACS was developed in all public hospital services in Hong Kong.12 Since 2018, Hong Kong Eye Hospital (HKEH), which had near 250,000 ophthalmic outpatient attendances per year, was chosen as 1 of the 2 pilot hospitals to incorporate common ophthalmic investigations into 3 types of image archiving and display systems, namely ePR and 2 commercially available PACS platforms. The investigation results were uploaded onto PACS and viewed through designated workstations in other clinical areas. According to the plan, PACS will be implemented in the remaining 9 ophthalmic centers in Hong Kong from 2021 onwards.

Similar to other electronic health information systems, PACS is often installed at the enterprise level, involving significant investment and change in workflow.13,14 Timely evaluation of PACS at an early stage, with end-user perspective in particular, would be beneficial for informing potential users of obstacles in implementation. Currently, there is a lack of evidence on clinicians5 view on the use of PACS in ophthalmology. Our study aimed to identify ophthalmologists’ view on the use of PACS, with a comparison of different image management systems following their recent implementation in our hospital.

METHODS

This is a cross-sectional survey among ophthalmology specialists and trainees to evaluate the use and satisfaction of PACS in HKEH. The study was approved by the Research Ethics Committee of Hospital Authority, Hong Kong Special Administrative Region, and followed the tenets of the Declaration of Helsinki.

PACS Setup in Hong Kong Eye Hospital

PACS in HKEH includes 3 systems: ePR, Heidelberg Eye Explore (HEYEX, Heidelberg Engineering, Switzerland), and FORUM (Zeiss, US). ePR is a pre-existing system implemented in all workstations in HKEH since 2000 s, alongside all other public hospitals and clinics in Hong Kong. Since December 2018, investigation reports of OCT RNFL, OCT macula, VF, FA, and ICG performed at HKEH can be accessed in ePR in Portable Document Format (PDF), instead of Digital Imaging and Communications in Medicine (DICOM) format, which is a commonly used standard for communication in biomedical digital imaging systems.15 HEYEX, a commercial eye care data management solution, is installed in all outpatient consultation rooms and selected inpatient areas in HKEH, with a total of 29 viewing stations for investigations including OCT RNFL, OCT macula, VF, FA, and ICG in DICOM format. Aside from displaying investigation reports in the same manner as ePR, comparison between reports, eg, VF/OCT RNFL reports can be achieved using “layer” function. Interpretation of OCT macula (Heidelberg Engineering, Switzerland) is facilitated by various built-in functions, eg, volume scan, star scan, thickness map progression, and various overlay tools. Similarly, interpretation of FA/ICG is also facilitated by functions including operator-generated reports, individual series, magnification, and image inversion, etc. Another commercial platform, FORUM (Zeiss, US), has 5 viewing licenses for OCT RNFL and VF in DICOM format. It is mainly designated to enhance the management of glaucoma by providing Guided Progression Analysis (GPA) of both VF and OCT RNFL (Zeiss, US).

Image quality of the investigations in ePR is comparable to the paper printout, while that in HEYEX and FORUM would be the same as that in the respective original imaging modality. Currently, all images uploaded to ePR can be read by all public hospitals in Hong Kong, which are managed by the statutory body of the Hospital Authority (HA). Likewise, those uploaded to the ePR by other public hospitals under HA can be read at our hospital. On the contrary, for HEYEX and FORUM, images uploaded at a particular HA hospital can only be viewed within that hospital, but not by any other hospitals within HA. ePR can be reached by corporate-provided personal home devices with special registration, while PACS cannot be reached by all home devices. In compliance with corporate policy, images will be deleted from the system if the patient did not attend our service for more than 6 years. The training was provided by information system personnel in form of lectures and onsite support during the early stages of implementation.

Subjects

We included all HKEH ophthalmologists or ophthalmology resident trainees, who had utilized patients’ ophthalmic imaging for clinical care from December 2018 to September 2020 and had undergone the transition from paper-based to electronic ophthalmic imaging system. Staff members who had not experienced paper-based systems were excluded.

Survey Design/Instrumentation

The survey is designed with the framework developed from DeLone and McLean model of information systems (IS) success, which identified 7 domains for assessing the success of IS, including systems quality, information quality, service quality, intention to use/use, user satisfaction, and net benefits.16 While this model is widely applied in studying the success of information systems or electronic medical record systems,17 we identified no study that focused on ophthalmic imaging. The survey was therefore modified to fit into the context of our study. Intention to use was not assessed as the use of system was compulsory. Participants were asked to choose from a 10-point Likert scale for items regarding ePR, HEYEX, and FORUM respectively in 4 domains, namely system quality and overall satisfaction, information quality, service quality/service support, and the use of PACS. In the “use” session, participants were asked to indicate the frequency of use for the individual functions of each system. An “others” session was added to allow patients to choose from a list of net benefits and issues commonly encountered, and to allow for open-ended answers. After pilot testing among 5 ophthalmologists of different subspecialties and years of experience for clarity and usability, the link for the online questionnaire (Supplementary Digital Content 1, https://links.lww.com/APJO/A120) was formally sent out to all relevant staff through staff email list. The survey was self-administered and took approximately 45 minutes to complete. Anonymity was maintained without the need for signature or identifiable personal information.

Study Outcome

The primary outcome of this study was strengths and limitations of PACS compared to paper-based system, in terms of system quality, information quality, and service quality/support. Secondary outcomes included pattern of use of PACS and comparison of different electronic systems.

Data Analysis

Statistical analysis was performed with IBM SPSS Statistics v27 (SPSS Inc, Chicago, IL). Continuous variables were presented in median and interquartile range (IQR). Paired and independent continuous variables were compared with Wilcoxon signed-rank test and Mann-Whitney U test, respectively. Correlation between continuous variables was studied with Spearman correlation. Sensitivity analysis was performed for scores in system quality and information quality to identify the effects of the subspecialty of the respondent. P value < 0.05 was considered statistically significant.

RESULTS

Demographics

Twenty-eight out of 37 (75.7%) ophthalmologists or ophthalmology resident trainees responded to the survey. The median age group was 31 to 40 years old (14/28, 50.0%), with 23/28 (82.1%) having less than or equal to 15 years of experience in ophthalmology. Further demographic data were listed in Table 1.

TABLE 1 - Demographics of Respondents
Age, y ≤30: 9/28 (32.1%)
31 to 40: 14/28 (50.0%)
41 to 50: 3/28 (10.7%)
51 to 60: 1/28 (3.6%)
≥60: 1/28 (3.6%)
Ophthalmologic experience, y 0 to 5: 11/28 (39.3%)
6 to 10: 6/28 (21.4%)
11 to 15: 6/28 (21.4%)
16 to 20: 1/28 (3.6%)
21 to 25: 2/28 (7.1%)
26 to 30: 1/28 (3.6%)
>30: 1/28 (3.6%)
Position Resident (basic trainee): 2/28 (7.1%)
Resident (higher trainee): 10/28 (35.7%)
Associate consultant: 12/28 (42.9%)
Consultant or above: 4/28 (14.3%)
Subspecialty Vitreo-retina: 6/28 (21.4%)
Pediatrics and squint: 6/28 (21.4%)
Cornea/external eye disease: 3/28 (10.7%)
Glaucoma: 5/28 (17.9%)
Oculoplastics: 3/28 (10.7%)
Neuro-ophthalmology: 1/28 (3.6%)
Uveitis: 2/28 (7.1%)
General ophthalmology: 5/28 (17.9%)
Two members worked in both uveitis and vitreo-retina team, with one of them working also in neuro-ophthalmology.
% time at HKEH per week 40.0 to 60.0%: 3/28 (10.7%)
70.0%: 1/28 (3.6%)
80.0%: 7/28 (25.0%)
90.0%: 16/28 (57.1%)
100.0%: 1/28 (3.6%)
No. y on paper system 0 to 5: 12/28 (42.9%)
6 to 10: 7/28 (25.0%
11 to 15: 5/28 (17.9%)
≥16: 4/28 (14.3%)
No. y on electronic imaging system 0 to 5: 26/28 (92.9%)
6 to 10: 1/28 (3.6%)
11 to 15: 1/28 (3.6%)
Use of FORUM in reading VF/OCT RNFL 14/28 (50.0%)
HKEH indicates Hong Kong Eye Hospital; OCT RNFL, optical coherence tomography of retinal nerve fibre layer; VF, visual field.

Use of System

Images were accessed through ePR for a median of 80% (IQR 50 to 90%) of time, while HEYEX was used for 20% (10 to 50%) of time. Fourteen doctors (14/28,50.0%) had used FORUM, yet the majority (71.4%) of them used it as route of access for <10.0% of the time. 82.1% (23/28) and 42.9% (12/28) used the electronic imaging system for >30 and >50 cases per week respectively. The use of various functions of ePR, HEYEX, and FORUM for different investigations were listed in Table 2. Among them, PDF reports were the most commonly used, for a median of 100%, 100%, and 90% of time for VF, OCT RNFL, and OCT macula respectively. Display for star scan (70.0% of time) and volume scan (75.0%) for OCT macula in HEYEX were also commonly used.

TABLE 2 - Use of System
No. staff aware of the function Frequency of use (median, IQR)
1. VF
 ePR/HEYEX PDF generated by operator 27/28 (96.4%) 100.0% (80.0 to 100.0%)
 HEYEX comparison of multiple test mode 22/28 (78.6%) 10.0% (0.0 to 32.5%)
 FORUM overview and browser 13/14 (92.9%) 0.0% (0.0 to 25.0%)
 FORUM GPA 13/14 (92.9%) 0.0% (0.0 to 10.0%)
2. OCT RNFL
 ePR/HEYEX report generated by operator 28/28 (100.0%) 100.0% (90.0 to 100.0%)
 HEYEX multiple display using “layer” function 21/28 (75.0%) 0.0% (0.0 to 40.0%)
 FORUM structure-function GPA 13/14 (92.9%) 0.0% (0.0 to 5.0%)
3. OCT macula
 ePR/HEYEX report generated by operator 28/28 (100.0%) 90.0% (50.0 to 100.0%)
 HEYEX overlay tools (eg, calipers, area measurement, etc) 25/28 (89.3%) 30.0% (0.0 to 75.0%)
 HEYEX magnification tool 24/28 (85.7%) 15.0% (0.0 to 50.0%)
 HEYEX volume scan 27/28 (96.4%) 70.0% (0.0% to 100.0%)
 HEYEX star scan 28/28 (100.0%) 75.0% (0.0 to 100.0%)
 HEYEX thickness map progression 26/28 (92.9%) 30.0% (0.0 to 92.5%)
 HEYEX thickness profile 28/28 (100.0%) 15.0% (0.0 to 50.0%)
 HEYEX 3D view 26/28 (92.9%) 0.0% (0.0 to 27.5%)
4. FA/ICG
 ePR/HEYEX PDF generated by operator 28/28 (100.0%) 100.0% (57.5 to 100.0%)
 Individual series on HEYEX 27/28 (96.4%) 20.0% (0.0 to 80.0%)
 HEYEX overlay tools (eg, calipers, measure circle, etc) 26/28 (92.9%) 0.0% (0.0 to 27.5%)
 HEYEX magnification tool 26/28 (92.9%) 10.0% (0.0 to 20.0%)
 HEYEX image inversion 25/28 (89.3%) 0.0% (0.0 to 55.0%)
 HEYEX image modification (eg, contrast, brightness, etc) 26/28 (92.9%) 0.0% (0.0 to 15.0%)
5. Other functions 1/28 (3.6%)
 Using “previous” arrow to compare with previous OCT macula scan n/a
ePR indicates electronic patient record; FA, fluorescein angiography; GPA, guided progression analysis; ICG, indocyanine green angiography; IQR, interquartile range; OCT, optical coherence tomography; PDF, portable document format; RNFL, retinal nerve fiber layer; VF, visual field.

Sensitivity Analysis

Subspecialty members working with posterior segment (vitreo-retina and uveitis, VRU) used HEYEX significantly more than non-VRU members [% of time using HEYEX: VRU: 90.0% (60.0 to 90.0%) vs non-VRU: 15.0% (10.0 to 30.0%), P < 0.0001], but not the other 2 systems (P > 0.475).

There was no difference in use between the 3 systems by glaucoma/neuro-ophthalmology (GNO) members (P > 0.106).

System Quality

All 3 systems received positive feedback on system quality, with ePR having a significantly higher score in all categories than HEYEX, and in all except login response time when compared with FORUM. In comparison to paper system, all 3 electronic systems scored highly (median of 7.0 to 9.0) in reducing patient identification error in filing the investigations and in image retrieval during consultation, with ePR performing better than HEYEX and FORUM (Table 3A).

TABLE 3 - System Quality, Overall Satisfaction, and Information Quality
Median score
Items ePR (n = 28) HEYEX (n = 28) P value (HEYEX compared with ePR) FORUM (n = 8) P value (FORUM compared with ePR) P value (HEYEX compared with FORUM)
TABLE 3A. System Quality and Overall Satisfaction
About the system:
 1. Easy to learn 9.0 (8.0 to 10.0) 6.0 (6.0 to 8.0) <0.001∗∗∗ 6.0 (5.0 to 7.0) 0.010 0.403
 2. Workstations easily available 9.0 (8.0 to 10.0) 7.5 (6.0 to 8.0) <0.001∗∗∗ 5.0 (5.0 to 7.0) 0.003∗∗ 0.032
 3. Satisfactory response time to log in 8.0 (8.0 to 9.0) 6.0 (4.0 to 8.0) <0.001∗∗∗ 6.0 (5.0 to 8.0) 0.081 0.204
 4. Stable and seldom crash 8.0 (7.0 to 8.8) 7.0 (5.0 to 8.0) 0.001∗∗ 6.0 (5.0 to 7.0) 0.022 0.230
 5. Reduce patient identification error in filing the investigations (compared to paper system) 8.0 (8.0 to 9.0) 8.0 (7.0 to 9.0) 0.025 7.0 (7.0 to 8.0) 0.014 0.102
 6. Reduce patient identification error in file retrieval during consultation (compared to paper system) 9.0 (7.0 to 9.0) 8.0 (7.0 to 8.0) 0.039 8.0 (7.0 to 8.0) 0.031 0.317
TABLE 3B. Information Quality
About information from the system:
 1. Display investigations in right chronological order 8.0 (8.0 to 9.0) 8.0 (7.0 to 10.0) 0.841 8.0 (6.0 to 8.0) 0.719 0.892
 2. Reduce loss of investigation records (compared to paper system) 9.0 (8.0 to 9.8) 8.0 (7.3 to 9.8) 0.072 8.0 (7.0 to 9.0) 0.121 0.516
 3. Display all important information one looks for from the reports/images
  VF 8.0 (7.0 to 9.0) 8.0 (7.0 to 9.0) 0.467 8.0 (7.0 to 9.0) 0.396 0.796
  OCT RNFL 8.0 (7.0 to 9.8) 8.0 (7.0 to 9.8) 0.299 8.0 (5.5 to 9.5) 0.131 0.131
  OCT macula 8.0 (7.0 to 9.0) 9.0 (8.0 to 10.0) 0.041
  FFA/ICG 7.5 (7.0 to 8.0) 8.0 (7.0 to 10.0) 0.016
 4. The display is of sufficient quality for interpretation
  VF 8.0 (7.0 to 9.8) 8.0 (7.0 to 9.0) 0.732 8.0 (8.0 to 9.0) 0.795 0.434
  OCT RNFL 8.5 (8.0 to 9.8) 8.0 (6.3 to 9.8) 0.032 8.0 (6.0 to 9.0) 0.169 0.932
  OCT macula 8.0 (6.0 to 9.0) 9.0 (8.0 to 10.0) 0.009∗∗
  FFA/ICG 7.5 (5.3 to 8.8) 8.0 (6.0 to 10.0) 0.284
 5. More efficient in retrieving investigation reports/images (compared to paper system)
  VF 6.5 (4.3 to 7.8) 6.5 (2.0 to 8.0) 0.986 6.0 (3.0 to 7.0) 0.798 1.000
  OCT RNFL 7.0 (6.0 to 8.8) 7.5 (4.3 to 8.0) 0.344 7.0 (5.0 to 9.0) 0.866 0.865
  OCT macula 6.5 (3.0 to 9.0) 9.0 (6.0 to 9.8) 0.027
  FFA/ICG 7.0 (3.3 to 8.0) 7.0 (3.0 to 9.0) 0.878
 6. Make the interpretation easier (compared to paper system)
  VF 4.0 (3.0 to 7.0) 6.0 (2.0 to 8.0) 0.098 6.0 (3.0 to 8.0) 0.199 0.205
  OCT RNFL 5.0 (4.0 to 8.0) 6.0 (3.3 to 8.0) 0.780 5.0 (2.0 to 8.0) 0.546 0.916
  OCT macula 7.0 (3.5 to 8.8) 9.0 (7.3 to 10.0) 0.002∗∗
  FFA/ICG 6.0 (4.0 to 8.0) 7.5 (3.3 to 9.0) 0.356
 7. Facilitate comparison with previous results (compared to paper system)
  VF 3.0 (2.0 to 6.8) 5.0 (2.0 to 7.0) 0.086 5.0 (3.0 to 9.0) 0.051 0.041
  OCT RNFL 4.0 (2.0 to 7.0) 6.0 (3.0 to 8.0) 0.193 5.0 (3.0 to 9.0) 0.629 0.523
  OCT macula 4.5 (2.0 to 7.8) 8.5 (6.3 to 10.0) 0.001∗∗
  FFA/ICG 4.0 (3.0 to 6.8) 6.5 (3.0 to 9.0) 0.023
ePR indicates electronic patient record; FFA, fundal fluorescein angiography; ICG, indocyanine green angiography; OCT, optical coherence tomography, RNFL, retinal nerve fiber layer; VF, visual field.
∗∗∗P < 0.05.
∗∗P < 0.05.
P < 0.05.

Sensitivity Analysis

For HEYEX, VRU members gave higher scores for satisfaction for login response time [VRU: 8.0 (6.8 to 9.0), non-VRU: 6.0 (4.0 to 7.0), P = 0.020], and reduction in patient identification error in filing the investigation during consultation [VRU: 9.0 (8.0 to 10.0), non-VRU: 8.0 (6.5 to 8.0), P = 0.017] (Table 4A).

TABLE 4 - Sensitivity Analysis for System Quality and Information Quality Between Different Subspecialties
TABLE 4A.
Items in HEYEX Scores by VRU members (n = 6) Scores by Non-VRU members (n = 22) P value
System quality
 Easy to learn 8.0, 6.0 to 9.3 6.0, 6.0 to 8.0 0.107
 Workstations easily available 8.0, 6.8 to 8.3 7.0, 6.0 to 8.0 0.356
 Satisfactory response time to log in 8.0, 6.8 to 9.0 6.0, 4.0 to 7.0 0.020
 Stable and seldom crash 6.5, 2.5 to 7.3 7.0, 5.0 to 8.0 0.318
 Reduce patient identification error in filing the investigations (compared to paper system) 9.0, 8.0 to 10.0 8.0, 6.5 to 8.0 0.017
 Reduce patient identification error in file retrieval during consultation (compared to paper system) 7.5, 7.0 to 8.3 8.0, 6.8 to 8.0 0.748
Information quality
 1. Display all important information one looks for from the reports/images
 OCT macula 10.0, 9.8 to 10.0 8.0, 8.0 to 9.0 0.004∗∗
 FFA/ICG 10.0, 8.5 to 10.0 8.0, 6.0 to 9.0 0.033
 2. The display is of sufficient quality for interpretation
 OCT macula 10.0, 9.8 to 10.0 9.0, 7.8 to 10.0 0.022
 FFA/ICG 10.0, 7.5 to 10.0 8.0, 5.8 to 9.0 0.055
 3. More efficient in retrieving investigation reports/images (compared to paper system)
 OCT macula 10.0, 9.0 to 10.0 8.0, 5.0 to 9.0 0.006∗∗
 FFA/ICG 9.5, 7.5 to 10.0 5.5, 3.0 to 8.0 0.013
 4. Make the interpretation easier (compared to paper system)
 OCT macula 10.0, 9.8 to 10.0 8.5, 6.0 to 9.0 0.005∗∗
 FFA/ICG 10.0, 7.5 to 10.0 6.5, 3.0 to 8.0 0.012
 5. Facilitate comparison with previous results (compared to paper system)
 OCT macula 10.0, 9.8 to 10.0 8.0, 5.8 to 9.0 0.002∗∗
 FFA/ICG 10.0, 7.5 to 10.0 5.5, 3.0 to 8.0 0.007∗∗
TABLE 4B.
Items in FORUM Scores by GNO members (n = 5) Scores by non-GNO members (n = 6) P value
System quality
 Easy to learn 5.0, 2.5 to 7.5 6.0, 5.0 to 7.5 0.405
 Workstations easily available 5.0, 3.0 to 6.5 5.5, 5.0 to 7.5 0.302
 Satisfactory response time to log in 5.0, 4.0 to 6.5 7.5, 6.0 to 8.3 0.040
 Stable and seldom crash 5.0, 2.5 to 7.0 6.5, 6.0 to 7.5 0.187
 Reduce patient identification error in filing the investigations (compared to paper system) 8.0, 6.0 to 9.0 7.0, 6.8 to 7.5 0.443
 Reduce patient identification error in file retrieval during consultation (compared to paper system) 8.0, 6.0 to 8.0 7.5, 6.8 to 8.3 0.847
Information quality
 1. Display all important information one looks for from the report/images
 VF 9.0 (7.5 to 10.0) 8.0 (5.6 to 8.3) 0.133
 OCT RNFL 8.0 (6.5 to 10.0) 6.5 (5.3 to 8.5) 0.264
 2. The display is of sufficient quality for interpretation
 VF 8.0 (8.0 to 9.5) 8.0 (5.6 to 8.3) 0.162
 OCT RNFL 8.0 (7.5 to 9.0) 7.5 (5.8 to 9.3) 0.643
 3. More efficient in retrieving investigation reports/images (compared to paper system)
 VF 6.0 (2.5 to 7.5) 5.5 (3.0 to 7.5) 1.000
 OCT RNFL 6.0 (2.5 to 7.5) 8.5 (5.6 to 9.0) 0.096
 4. Make the interpretation easier (compared to paper system)
 VF 7.0 (2.5 to 9.0) 5.5 (3.0 to 8.3) 0.783
 OCT RNFL 7.0 (2.5 to 8.0) 5.0 (2.0 to 6.8) 0.713
 5. Facilitate comparison with previous results (compared to paper system)
 VF 5.0 (2.5 to 9.5) 5.5 (3.0 to 9.3) 0.927
 OCT RNFL 7.0 (2.5 to 9.5) 5.0 (3.0 to 6.8) 0.520
FFA indicates fundal fluorescein angiography; GNO, glaucoma/neuro-ophthalmology; ICG, indocyanine green angiography; OCT, optical coherence tomography; RNFL, retinal nerve fiber layer; VF, visual field; VRU, vitreo-retina and uveitis.
∗∗P < 0.01.
P < 0.05 (asymptotic 2-tailed).

For FORUM, score for response time to log in was significantly lower among GNO members compared to non-GNO members [GNO: 5.0 (4.0 to 6.5), non-GNO: 7.5 (6.0 to 8.3), P = 0.040] (Table 4B).

Information Quality

All 3 systems showed satisfactory feedback in most of the items in display of information, including correct chronological order, display of all important information and of sufficient quality for interpretation of data in all types of investigations (median scores 7.5 to 9.0 for VF, OCT RNFL, OCT macula, and FFA/ICG for ePR and HEYEX, VF and OCT RNFL for FORUM) (Table 3B).

When compared with paper system for the efficiency of use of information, the results were more diversified. ePR was inferior to paper in facilitating comparison with previous results in all 4 investigations (median scores 3.0 to 4.5). It also failed to make interpretation easier than paper system in VF and OCT RNFL (median score 4.0 and 5.0, respectively). Scores for making retrieval of investigation more efficient than paper were moderate across all 4 types of investigations (median scores 6.5 to 7.0). For HEYEX, except failing to facilitate VF comparison better than the paper system, it scored moderately (median score 6.0 to 7.5) in comparison to paper system in being more efficient in retrieving investigations and making the interpretation easier for VF, OCT RNFL, and FFA/ICG, and in facilitating comparison with previous results for OCT RNFL and FFA/ICG. For OCT macula, HEYEX scored highly for all 3 categories comparing with paper system, with median score of 8.5 to 9.0, which were significantly higher than ePR. HEYEX also scored higher mark than ePR in facilitating comparison with previous results for FFA/ICG. For FORUM, scores were modest (median scores 5.0 to 7.0) when compared to paper system. Although FORUM scored signficantly higher than HEYEX in facilitating comparison with previous results when compared to paper system, the median score of 5.0 reflected the worse performance of both FORUM and HEYEX when compared to paper.

Sensitivity Analysis

For HEYEX, VRU members gave significantly higher scores than non-VRU members for displaying investigations in right chronological order [VRU: 10.0 (8.0 to 10.0), non-VRU: 8.0 (6.75 to 9.25), P= 0.042], as well as for most items in information quality session for both OCT macula and FFA/ICG (Table 4A). There was no significant difference in scores between GNO members and non-GNO members for all items in VF and OCT RNFL in FORUM (P ≥ 0.096) (Table 4B).

Service Quality/Support

Twenty-three out of 28 (82.1%) respondents received 0 to 2 hours of training. 92.6% (25/28) needed technical support 0 to 2 times per month in the first 14 days of use. Issues encountered when seeking support were listed in Table 5A. Overall, sufficiency of training before implementation and technical support were moderate [score 7.0 (6.0 to 8.0), 7.0 (5.0 to 7.8), respectively]. Confidence in using the system was good for ePR and HEYEX, scoring 9.5 (8.0 to 10.0) and7.0 (7.0 to 8.0)respectively, but suboptimal for FORUM [score 5.0 (2.5 to 7.3)] (Table 5A).

TABLE 5 - Service Quality/Service, Benefit of PACS and Issues Encountered
TABLE 5A. Service Quality/Service Support
1. Need for technical support for the first 14 days of use (times/month) 0–2: 23 (82.1%)
2–4: 5 (17.9%)
2. Need for technical support is needed for the first 14 days of use (times/month) (n = 27) 0–2: 25 (92.6%)
2–4: 2 (7.4%)
3. Issues encountered while seeking technical support
 a. Lack of time to seek technical support 14/28 (50.0%)
 b. Technical support staff cannot be reached 5/28 (17.9%)
 c. Assistant/other staff not being able to help 7/28 (25.0%)
4. Training provided before implementation is sufficient 7.0 (6.0 to 8.0)
5. Technical support provided before implementation is sufficient 7.0 (5.0 to 7.8)
6. Confident in using the system
 a. ePR 9.5 (8.0 to 10.0)
 b. HEYEX 7.0 (7.0 to 8.0)
 c. FORUM (n = 10) 5.0 (2.5 to 7.3)
TABLE 5B. Benefits of PACS
Shorten time needed for making clinical decisions compared with paper system 8/28 (28.6%)
Enhance quality of clinical decisions compared with paper system 13/28 (46.4%)
Enhance doctor-patient communication compared with paper system 9/28 (32.1%)
Enhance communication with colleagues compared with paper system 15/28 (53.6%)
Nil 5/28 (17.9%)
Other: Easier to review report by other clusters 1/28 (2.9%)
Issues Encountered When Using PACS
ePR HEYEX P value (ePR compared with HEYEX) FORUM P value (ePR compared with FORUM) P value (HEYEX compared with FORUM)
Unable to identify location of investigations 8/28 (28.6%) 4/28 (14.3%) 0.102 5/14 (35.7%) 0.564 0.157
Failure to log in 4/28 (14.3%) 8/28 (28.6%) 0.157 3/14 (21.4%) 0.655 0.317
Unacceptable login time 7/28 (25.0%) 14/28 (50.0%) 0.052 3/11 (21.4%) 0.414 0.102
Unacceptable loading time 6/28 (21.4%) 11/28 (39.3%) 0.132 4/14 (28.6%) 0.083 0.414
Poor resolution of investigations 5/28 (17.9%) 0/28 (0.0%) 0.025 0/14 (0.0%) 0.317 1.000
Nil 9/28 (32.1%) 7/28 (25.0%) 7/14 (50.0%)
Other comments 0/28 (100.0%) 1/28 (2.9%): system “not responding” 1/14 (7.1%): need training for FORUM
Overall comments about electronic imaging system.ePR indicates electronic patient records.1/28 (3.6%): No marking of right or left eye for visual fields in e-systems making retrieval inefficient.1/28 (3.6%): Prefer a comparison button so that visual fields can be displayed in chronological order rather than creating a layout by the user.
P < 0.05.

Other Benefits and Issues of PACS

Other benefits and issues of PACS were listed in Table 5B. The most agreed benefits included enhancing communication with colleagues (15/28, 53.6%) and quality of clinical decisions (13/28, 46.4%) compared to paper system. For the issues encountered, unacceptable login time were noted in both ePR (25.0%) and HEYEX (50.0%), while 28.6% of ePR users noted diffculty in identifying location of investigations, and 39.3% of HEYEX users noted unacceptable loading time. ePR had significantly more users reporting poor resolution of investigations (17.9%) than HEYEX (P = 0.025).

DISCUSSION

Our study evaluated the ophthalmologists’ view on 3 recently implemented image archiving systems in an eye hospital, demonstrating overall positive feedback and identifying areas of limitation. To our knowledge, there had been no published report on PACS from ophthalmologists’ perspective.

Compared to paper system, PACS gained most recognition on the improvement of data archiving, reflected by high scores in reducing patient identification error in filing and retrieving investigations, displaying in right chronological order, and reducing loss of investigation records. This was echoed by reported outcome on the implementation of a DICOM-compatible workflow in an ophthalmology clinic, which showed ≥50% reduction in need to enter or edit patient information into the testing device, and reduced need to manage misfiled images by 76% in a survey among technicians.14 Limitations identified in our study involving technical aspects, such as unsatisfactory login time, were similar to those reported in radiology, for example, low speed of network.18 Same as physicians,5 training was a challenge in our hospital, with nearly half of the respondents reporting a lack of time to seek technical support. On the other hand, certain limitations of PACS identified in our study were not shared among radiologists or physicians. While the latter reported improved productivity and reduced reporting time with PACS,6 in our study, except among VRU subspecialty members, there was no significant advantage of PACS over paper system on the overall perceived efficiency of data usage, for example, in being more efficient in retrieving investigation report or images, making interpretation easier, or in facilitating comparison with previous results, despite relatively high awareness of the multiple display or comparison mode of HEYEX (≥75%). This study highlighted some of the unique challenges faced by ophthalmologists when implementing PACS, despite its numerous evaluation in radiology and physicians.5,19,20

This study brought important insights about factors to consider when choosing different types of PACS to implement in ophthalmology. First, in a general ophthalmology setting, the frequency of use of an individual system seemed to depend on system quality rather than image display quality. ePR, while having a lower score in information quality session, achieved the highest scores in system quality compared to HEYEX and FORUM. The superior system stability may be explained by the long duration of use of ePR in the public health system of nearly 2 decades. When the image viewing function was added to ePR, the adaptation needed was minimal, making it easiest to learn and use. Together with fast login response time, system quality probably contributed to the high usage of ePR, accounting for 80% of route of access. Second, the pattern of use and perception of the system were subspecialty-dependent. In our hospital, OCT macula and FA/ICG were performed mainly with Heidelberg SPECTRALIS, using the same platform as HEYEX, while OCT RNFL was performed with Cirrus HD-OCT by Carl Zeiss Meditec, using the same platform as FORUM. In our study, the information quality of OCT macula and FFA/ICG in HEYEX gained higher scores among VRU doctors compared to other colleagues, demonstrating a high demand for higher quality image display in aiding clinical diagnosis and treatment in vitreoretinal diseases. On the other hand, FORUM did not achieve significantly higher scores among GNO doctors. Apart from limited viewing licences, another explanation was that only few patients had multiple VF or OCT RNFL test results uploaded to the system, especially at the early stage of implementation, limiting the availability of Guided Progression Analysis, which is the major advantage of FORUM over paper/PDF format for VF/OCT RNFL interpretation. Overall, the hybrid system of using both PDF format incorporated into an existing ePR and DICOM format in platform-specific PACS (HEYEX and FORUM) gave us the advantage of catering for both busy general ophthalmology clinics, when response time is crucial, and advanced subspecialty needs, where image quality is more important. The choice of PACS would depend on the volume of practice and the degree of subspecialization.

Currently, there is no well-recognized tool for assessing the effectiveness of PACS. Evaluations of PACS were done through heterogeneous methods, ranging from self-designed questionnaires5,19,21 to summarizing comments from professionals in online discussion groups,22 making direct comparison across studies difficult. In our study, DeLone and McLean model of IS success was chosen as it was well-validated and provided a comprehensive framework to assess IS effectiveness,16 with past applications on health IS, eg, electronic health record system.17 Although this model did not provide specific parameters under each domain, it allowed for a systematic approach to report and compare research work on IS success, and for study of interrelationships between domains,16 which facilitated the comparison between various PACS in our study. However, one should take caution that not all aspects of the model were covered in our study. Net benefit, in particular, would require a much more comprehensive survey covering the entire workflow of archiving and retrieval of the ophthalmic imaging apart from the current end-user experience.

Ophthalmology involves substantial utilization of different imaging modalities and evaluation of image-based data. With the rapid advancement in artificial intelligence, deep learning, and big data, which have fueled mankind's fourth industrial revolution,23 it is now possible to harness the vast volume of ocular images for analysis to yield highly accurate diagnostic information.24–29 Nevertheless, to meaningfully utilize big data in ophthalmology, in which images play an essential role, developing the necessary softwares and hardwares to handle, share, and analyze image-based data from different sources would be crucial.30 Understanding the strengths and deficits of the different PACS from the perspective of ophthalmologists would be pivotal in aiding developers to standardize, refine, and optimize such systems to efficiently facilitate patient care and research to fit the needs of the information age.

Our study had several limitations. First, the small sample size and single-center design would limit the generalizability of our results to ophthalmology services worldwide. However, the inclusion of 3 types of PACS and sensitivity analysis among different subspecialty members, which was made possible as our center provided high-volume service in both general and subspecialized ophthalmology, would enhance the relevance of our results to a broader range of ophthalmic practices. Second, our survey assessed only respondents’ subjective estimation of events, which would lead to recall bias. More objective evaluations including measurement of consultation time both pre-implementation and post-implementation of PACS may be considered in future studies. Although login time and image loading time were not measured for each respondent, an estimation by 2 authors (TL and MW) showed results consistent with the questionnaire response (average login time: ePR: 2.9 s, HEYEX: 10.9 s, FORUM: 4.1 s; average loading time of all types of images available in the platform: ePR: 0.8 s, HEYEX: 1.8 s, FORUM: 1.7 s). However, one should be cautious that numerous factors, for example, hardware, image size, etc, may affect the direct application of these results to PACS installed in other settings. Third, our study did not assess the benefits or costs of PACS from the health system and hospital management perspective, which is a crucial component for understanding the cost-effectiveness of PACS in health systems, and therefore informing policy planning. The cost and expertise required in setting up the systems, which may affect the ophthalmologists’ perception of the system, were not evaluated in this study as the systems in HKEH were set up by respective management and information technology teams and information system personnel. Fourth, factors affecting the acceptance rate of PACS among staff were not discussed in this study, as paper printouts for images were no longer available soon after implementation of PACS, making use of the latter compulsory. Lastly, built-in functions of PACS are numerous and being upgraded from time to time, and the results of our survey may not be applicable to future versions of the systems.

In conclusion, our results demonstrated overall positive feedback from ophthalmologists as end-users of PACS in system quality and display of information, with major limitations being the inefficiency in the use of information and a lack of time in accessing technical support. In particular, ophthalmologists working in posterior segment had differential use pattern and feedback on information display quality compared to those in other sub-specialties, making subspecialty of an ophthalmology service a consideration for choosing PACS.

REFERENCES

1. Tong CKS, Wong ETT. Picture Archiving and Communication System for Public Healthcare. Pennsylvania, US: IGI Global; 2011.
2. Lemke HU. A Network of Medical Work Stations for Integrated Word and Picture Communication in Clinical Medicine, Technical Report. Berlin: Technical University; 1979.
3. Meyer-Ebrecht D. Picture archiving and communication systems (PACS) for medical application. Int J Biomed Comput 1994; 35:91–124. doi:10.1016/0020-7101(94)90061-2.
4. Madapana N, Gonzalez G, Rodgers R, et al. Gestures for picture archiving and communication systems (PACS) operation in the operating room: is there any standard? PLoS One 2018; 13:e0198092doi:10.1371/journal.pone.0198092.
5. Alalawi ZM, Eid MM, Albarrak AI. Assessment of picture archiving and communication system (PACS) at three of ministry of health hospitals in Riyadh region - content analysis. J Infect Public Health 2016; 9:713–724. doi:10.1016/j.jiph.2016.09.004.
6. Mackinnon AD, Billington RA, Adam EJ, et al. Picture archiving and communication systems lead to sustained improvements in reporting times and productivity: results of a 5-year audit. Clin Radiol 2008; 63:796–804. doi:10.1016/j.crad.2007.12.015.
7. Mwanza JC, Budenz DL. New developments in optical coherence tomography imaging for glaucoma. Curr Opin Ophthalmol 2018; 29:121–129. doi:10.1097/ICU.0000000000000452.
8. Al-latayfeh MM, Sun JK, Aiello LP. Ocular coherence tomography and diabetic eye disease. Semin Ophthalmol 2010; 25:192–197. doi:10.3109/08820538.2010.518107.
9. Hospital Authority. Hospital Authority Major Statistics. 2019.
10. Hospital Authority. HA Quality and Safety Annual Report. 2019.
11. Balyen L, Peto T. Promising artificial intelligence-machine learning-deep learning algorithms in ophthalmology. Asia Pac J Ophthalmol (Phila) 2019; 8:264–272. doi:10.22608/AP0.2018479.
12. Huang HK. From PACS to web-based ePR system with image distribution for enterprise-level filmless healthcare delivery. Radiol Phys Technol 2011; 4:91–108. doi:10.1007/s12194-011-0122-5.
13. Chantler C, Clarke T, Granger R. Information technology in the English National Health Service. Jama 2006; 296:2255–2258. doi:10.1001/jama.296.18.2255.
14. Pandit RR, Boland MV. Impact of digital imaging and communications in medicine workflow on the integration of patient demographics and ophthalmic test data. Ophthalmology 2015; 122:227–232. doi:10.1016/j.ophtha.2014.08.036.
15. (MITA) TMITA. DICOM Strategic document. Update on March 8, 2017.. Available from https://www.dicomstandard.org Accessed May 3, 2021.
16. DeLone W, McLean E. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst 2003; 19:9–30. doi:10.1080/07421222.2003.11045748.
17. Bossen C, Jensen LG, Udsen FW. Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: approach, results, and success factors. Int J Med Inform 2013; 82:940–953. doi:10.1016/j.ijmedinf.2013.05.010.
18. Khajouei R, Jahromi ME, Ameri A. Challenges of implementing picture archiving and communication system in multiple hospitals: perspectives of involved staff and users. J Med Syst 2019; 43:182doi:10.1007/s10916-019-1319-0.
19. Carlin L, Smith H, Henwood F, et al. Double vision: an exploration of radiologists’ and general practitioners’ views on using picture archiving and communication systems (PACS). Health Informatics J 2010; 16:75–86. doi:10.1177/1460458210361935.
20. Arora D, Mehta Y. Use of picture archiving and communication system for imaging of radiological films in cardiac surgical intensive care unit. J Anaesthesiol Clin Pharmacol 2014; 30:447–448. doi:10.4103/0970-9185.137306.
21. Abbasi R, Sadeqi Jabali M, Khajouei R, et al. Investigating the satisfaction level of physicians in regards to implementing medical Picture Archiving and Communication System (PACS). BMC Med Inform Decis Mak 2020; 20:180doi:10.1186/s12911-020-01203-0.
22. Alhajeri M, Shah SGS. Limitations in and solutions for improving the functionality of picture archiving and communication system: an exploratory study of PACS professionals’ perspectives. J Digit Imaging 2019; 32:54–67. doi:10.1007/s10278-018-0127-2.
23. Ting DSW, Pasquale LR, Peng L, et al. Artificial intelligence and deep learning in ophthalmology. Br JOphthalmol 2019; 103:167–175. doi:10.1136/bjophthalmol-2018-313173.
24. Abramoff MD, Lou Y, Erginay A, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci 2016; 57:5200–5206. doi:10.1167/iovs.16-19964.
25. Breusegem C, Fieuws S, Stalmans I, et al. Agreement and accuracy of nonexpert ophthalmologists in assessing glaucomatous changes in serial stereo optic disc photographs. Ophthalmology 2011; 118:742–746. doi:10.1016/j.ophtha.2010.08.019.
26. Goh JHL, Lim ZW, Fang X, et al. Artificial intelligence for cataract detection and management. Asia Pac J Ophthalmol (Phila) 2020; 9:88–95. doi:10.1097/01.Apo.0000656988.16221.04.
27. He M, Li Z, Liu C, et al. Deployment of artificial intelligence in real-world practice: opportunity and challenge. Asia Pac J Ophthalmol (Phila) 2020; 9:299–307. doi:10.1097/apo.0000000000000301.
28. Huang SS. Future Vision 2020 and beyond-5 critical trends in eye research. Asia Pac J Ophthalmol (Phila) 2020; 9:180–185. doi:10.1097/apo.0000000000000299.
29. Ruamviboonsuk P, Cheung CY, Zhang X, et al. Artificial intelligence in ophthalmology: evolutions in Asia. Asia Pac J Ophthalmol (Phila) 2020; 9:78–84. doi:10.1097/01.APO.0000656980.41190.bf.
30. Cheng CY, Soh ZD, Majithia S, et al. Big data in ophthalmology. Asia Pac J Ophthalmol (Phila) 2020; 9:291–298. doi:10.1097/apo.0000000000000304.
Keywords:

electronic ophthalmic imaging; ophthalmology diagnostic tests/investigations; optical coherence tomography; picture archiving and communication system; visual field

Supplemental Digital Content

Copyright © 2022 Asia-Pacific Academy of Ophthalmology. Published by Wolters Kluwer Health, Inc. on behalf of the Asia-Pacific Academy of Ophthalmology.