Secondary Logo

Journal Logo

Original Article

Establishment of a Quantitative Medical Technology Evaluation System and Indicators within Medical Institutions

Wu, Suo-Wei; Chen, Tong; Pan, Qi; Wei, Liang-Yu; Wang, Qin; Li, Chao; Song, Jing-Chen; Luo, Ji

Author Information
doi: 10.4103/0366-6999.232804
  • Free



The development and application of medical technologies reflect the medical quality and clinical capacity of a hospital. Meanwhile, the operation of advanced medical technologies is also an effective approach in upgrading medical service and core competitiveness among medical institutions.[1] Therefore, the regulation on medical technologies is a crucial part of medical quality management and hospital management, and it is the primary task for the medical administrative department to ensure the safety and effectiveness of medical technologies in clinical applications.[2] To be specific, the key contents of medical technology management are the evaluation and assessment of the operation procedures and protocols, the technical competency of medical staffs and medical units as well as the external conditions such as the equipment and materials required in the process to determine the safety, effectiveness as well as the adaptability in clinical practice.[3] Thus, the establishment of a comprehensive evaluation system and related indicators of clinical technology are vital in enhancing the management and supervision within medical institutions.[4] According to recent studies, some domestic and foreign medical administrative departments and medical institutions have already established relatively mature admission procedures and management regulations as well as assessment methodology of medical technology.[5] However, researchers and outcomes on the accurate evaluation systems and standards, especially on quantitative assessment systems that can assess to an objective, unified evaluation are still rare, which make it almost impossible for different medical technology for comparison.[6]

On this basis, the current study aimed to build a quantitative medical technology evaluation system through questionnaire survey using the Delphi method and set index weight coefficient to each indicator through analytical hierarchy process within medical institutions. Moreover, it is hoped with the establishment of the evaluation system and further verifications in clinical practice, medical institutions can make a better assessment to clinical technologies, promote the management of medical quality technologies, and ensure the medical safety of various operations among the hospitals.[7]


Selection of questionnaire survey experts

According to the principle of the Delphi method, a certain number of experts were selected to participate in the questionnaire survey. In general, the accuracy and constancy of the results are related to the number of experts joining the investigation.[8] Normally, to ensure the credibility and authority of the results, the optimum number of experts investigated was among 15–50.[9] All the experts selected should be involved in the related specialty fields and that are acquainted with the contents of the survey. To ensure the accuracy and objectivity of the results, the professional levels, specialty majors as well as the length of occupations of the experts should be evenly distributed.[10]

Establishment of the evaluation system and two-leveled indicators

According to previous literature reviews and expert consultations, it is generally suggested to build a medical technology evaluation system of the two-leveled indicators.[11] Based on the current reports and studies, practical experiences of medical technology operations in combination with the results of the expert consultation, the research drafted the initial evaluation system of four primary level indicator and 14 secondary indicators. Then, the first round of survey questionnaires was developed according to the system and indexes we drafted and distributed to experts selected. Based on the principle of the Delphi method, experts would be responding to the questionnaire and grade each indicator based on the Likert scale (containing 5 scales, which are “strongly agree”, “agree”, “uncertain”, “disagree”, and “strongly disagree”, with the grade of 5, 4, 3, 2, and 1, respectively, in statistical analysis) by the importance of the subjects according to their expertise and experience.[12] Besides, suggestions of the experts in adjusting the structure of the evaluation system through adding, deleting, or merging indicators would also be faithfully recorded. The results of the first round questionnaires were collected and processed, and the outcomes and suggestions of statistical significance were summed up in rebuilding the second round of survey questionnaires for the same expert team of the same procedures.

Allocation of the index weight

To set the allocation of index weight, analytical hierarchy process was adopted in determining the weight coefficient of each indicator. In the second round of the survey, experts were asked to assign weights to each level of indicators according to their experience and expertise, and the matrix was built in the principle of analytical hierarchy process.[13] In this period, experts were supposed to make comparisons of specific indicators in each judgment matrix, and Saaty 1–9 scale relative materiality table was applied in the grading the importance of the indicators between one and another, the matrix was completed based on the value of relative importance of each subject.[14] Calculate the primary and secondary index weight coefficient by the survey results of the matrix, and the index weight coefficient was used in judging the importance of the indicators. The judgment of the experts on primary level indicators was input into Microsoft Excel 2010 (Microsoft Corporation, Washington, USA) to build the matrix so that the weight coefficient and maximum eigenvalue (λ max), consistency index (CI), and random consistency ratio (CR) were obtained and collected.[15] Consistency test was performed according to the calculated value, as the results of CR <0.1 was considered adaptable, and the average value of the results was calculated to determine the index weight primary level indicators.[16] Then collected the weight coefficient of secondary indicators according to the same principle and with the index weight obtained, calculated the integrated index weight of secondary indicators according to the formula Nij = XiYij (i = 1, 2…; j = 1, 2,…). As Xi refers to the primary level index weight, and Yij is the j level index weight under the i primary level index.[17] All the data and material collected were entered into Excel 2010 software for Microsoft (Microsoft Corporation), and statistical analyses were performed using SPSS version 14.0 (SPSS Inc., Chicago, IL, USA).


General situation of experts and the distribution of questionnaires

In this study, 26 experts of different occupational lengths and professional levels majoring in clinical medicine, hospital management, nursery as well as pharmacy management were invited in the research, as the general situation of the experts are shown in Table 1. According to the initial index system as well as the evaluation indicators we drafted, the first round of survey questionnaires was designed as shown in Supplementary Table 1. In the first round of investigation, 26 questionnaires were distributed and 25 copies were recovered, with a recovery rate of 96.15%. While in the second round, a total of 25 questionnaires were allocated to the 25 experts responded in the last round, and all the questionnaires returned were examined to be effective.

Table 1
Table 1:
General situation of the surveyed experts (n = 26)
Supplementary Table 1
Supplementary Table 1:
The contents of first round survey questionnaires

Establishment of the evaluation system and indicators

The first round of survey results questionnaires of the 25 survey experts were collected, analyzed and verified through consistency tests. According to the grading of experts, it was suggested that the initial structure of the evaluation system are basically recognized. Meanwhile, modifications on some indicators were made according to suggestions given by the experts [Table 2].

Table 2
Table 2:
The adjusted evaluation system and indicators

Indicators suggested to be added

During the survey, 6 experts out of 25 (24.0%) suggested that the indicator “the priority of the technology in professional field” were of great importance that should be included in the system. Over careful consideration, we decided to include the subject as the secondary indicator under the primary column of “innovativeness”.

Indicators suggested to be combined or separated

Four experts out of 25 (16.0%) proposed the combination of secondary indicators “the establishment of practice guideline and operation regulations” and “the formulation of risk disposal plans” under the primary column of “safety” since they could all be classified as the preoperational systematic requirements of medical technologies. Besides, 3 experts (12.0%) suggested the indicator “definitions on indications and contraindications” be separated in consideration for the aptness of grading. After further discussion, we separated the original indicator into “definitions on indications of diseases” and “occurrence of contraindications” two parts.

Indicators suggested to be deleted

According to the survey results of 3 experts (8.0%), the secondary indicator “the origination of innovations” under the primary column “innovativeness” was thought hard to be quantitatively analyzed and evaluated, and hence that it was suggested to be deleted, and the proposal was accepted by the research group.

With the adjustments made of the evaluation system and the indicators, the second round of survey questionnaires was developed.

Allocation of the index weight

The aim of the second round survey was carried out to identify the index weight of each indicator as we distributed the questionnaires to the 25 experts responded in the first round. After the results were collected and analyzed, verification on the consistency of the experts was made, and results showed that all the CR values were below 0.1, suggesting the opinions of experts were relatively stable. Besides, the experts had no revisions on the setting of the indicators in the second round of survey. Thus, the assignments of primary and secondary indicators were analyzed and calculated by building the matrix according to the grading of experts on each individual indicator by Saaty 1–9 scale relative materiality table, as the results of index weight of indicator are calculated as shown in Table 3. According to the results, the maximum index weight of primary indicator was the safety of the medical technology, with the coefficient of 0.33, followed by effectiveness and innovativeness, with the index weight coefficient of 0.28 and 0.27 respectively, benefits of medical technology ranked last in the survey, with the index weight coefficient of 0.12. The index weight of each secondary indicator was conducted according to the formula Nij = XiYij (i = 1, 2…; j = 1, 2,…), and the results are shown in Table 4.

Table 3
Table 3:
Results of experts on index weight of primary indicators
Table 4
Table 4:
Results of experts on index weight of secondary indicators


Selection of initial indicators of the system

According to general consensus, the basis and core elements of medical technologies management within medical institutions are consist of safety, effectiveness, innovativeness, and the benefits.[18] In the light of the principle, the study initially set the four basic subjects as the primary indicators of the evaluations system. For the record, since all the medical equipment and material applied in hospitals went through the safety and effectiveness assessment by medical administration departments, the evaluation on safety and effectiveness of medical technologies mainly refers to the operations performed by medical staffs. Similarly, the secondary indicators are drafted according to clinical experience and preliminary research reports, and the acquisition of all the indicators should be linear indexes (such as the secondary indicator “quantity of operated cases” that could be rated by number and “recovery rate” that could be marked by ratio under the column of “effectiveness”) or that could be valued through unified and objective grading standards (such as the secondary indicator “the priority of the technology in professional field” that could be graded by “globally advanced”, “domestic advanced”, “regional advanced”, “horizontal advanced”, “well-developed” as well as “under-developed” and marked by arithmetic sequence number). On this basis, some initial indicators that are hard to be quantitatively estimated (such as “the origination of innovations”) are eliminated or replaced by other indexes.

Application and advantages of the evaluation system and its indicators

The evaluation system we built provided a methodology for medical institutions to analyze the comprehensive adaptability of clinical technologies since all the indicators could be measured quantitatively that made it possible for different medical technologies comparable with one and another.[19] Compared with previous studies, the advantage and innovativeness of the system lie in its accuracy and objectiveness. Besides, the evaluation system along with the indicators enable medical institution to make assessment in medical technologies of all fields (including surgical operations, noninvasive operations, laboratory operations as well as image inspection operations and so on), all classes ( first-, second-, and third-class technologies), and all phases (preadmission and postadmission phase), which provided a standardized, comparable platform in medical technology management for medical institutions.[20] Moreover, the results could be adopted in the assessment of individuals, departments, and institutions of medical fields.

Verification of the evaluation system and adjustment of indicators

To better apply the evaluation system into practical operation, further verification of the indicators should be made in the clinical application. Besides, since the availability and suitability of all the indicators might be changing with time, the reassessment and adjustment of all the indicators should be dynamically made over a certain period.[21]

Study limitations

The accuracy and objectiveness of the evaluation system and the indicators are closely related to the practical situation of medical institutions among different regions.[22] The degree of development, structure, and scale of medical institutions might all have a direct impact on indicators.[13] Since the research was performed within a regional medical institution and all the experts we surveyed are from medical fields in Beijing, considering the different situation in various regions, as well as the limitation of the data and material, the generalization of the results should be more careful in other circumstances.[23]

In conclusions, though a two-round questionnaire survey of experts and statistical analysis, along with the verification on the credibility of the results through consistency evaluation test, the study established a quantitative medical technology evaluation system model and assessment indicators within medical institutions based on the Delphi method and analytical hierarchy process.[24] It is believed that the system could be better applied to clinical practice through continuous revalidation and dynamic adjustment with time. Furthermore, the methodology of the study could be extended to other subjects of hospital evaluation.

Supplementary information is linked to the online version of the paper on the Chinese Medical Journal website.

Financial support and sponsorship


Conflicts of interest

There are no conflicts of interest.


1. Wu LJ, Pan YL. Problem and countermeasures of clinical application management of medical technology (in Chinese) Chin Health Qual Manage. 2015;22:29–31 doi: 10.13912/j.cnki.chqm.2015.22.3.11
2. Wu SW, Chen T, Wu M, Pan Q, Xuan Y, Wei LY, et al Management of medical technology under the new medical policy background in China Chin Med J. 2016;129:2745–8 doi: 10.4103/0366-6999.193453
3. Lu B, Lin FF, Li ZY. Reflection and management after the canceling of non-administrative licensing approval to the third rank medical technology (in Chinese) Chin Hosp Manage. 2015;35:28–9 doi: 10.01-5329(2015)10-0028-02
4. Indira C. Medical technology in India: Tracing policy approaches Chin Med J. 2013;57:197–202 doi: 10.4103/0019-557X.123240
5. Wu S, Ge Y, Xu X, Xuan Y, Chen T. Research of admission management standards and working patterns of first class medical technologies Chin Med J. 2014;127:2860–2 doi: 10.3760/cma.j.issn.0366-6999.20140948
6. Ogunmola OJ, Oladosu OY. Pattern and outcome of admissions in the medical wards of a tertiary health center in a rural community of Ekiti state, Nigeria Chin Med J. 2014;13:195–203 doi: 10.4103/1596-3519.142291
7. Zechmeister I, Schumacher I. The impact of health technology assessment reports on decision making in Austria Int J Technol Assess Health Care. 2012;28:77–84 doi: 10.1017/S0266462311000729
8. Han L, Yan SJ, Jia YB, Cai J, Zhang C. Practice and exploration of medical technology management in Shanghai (in Chinese) Chin Health Resour. 2014;17:169–71 doi: 1007-953X(2014)03-169-03
9. Raghav PR, Kumar D, Bhardwaj P. Experience of Delphi technique in the process of establishing consensus on core competencies Int J Appl Basic Med Res. 2016;6:191–4 doi: 10.4103/2229-516X.186966
10. Weill C, Banta D. Development of health technology assessment in France Int J Technol Assess Health Care. 2009;25(Suppl 1):108–11 doi: 1010.1017/S0266462309090503
11. Sakineh S, Asghar E, Zohreh M. Toward a framework of statistical information system 1 for Iranian hospital Int J Health Syst Disaster Manage. 2015;3:103–8 doi: 10.4103/2347-9019.151327
12. Yousefy A, Shayan S, Mosavi A. Developing a clinical performance logbook for nursing students receiving cardiac care field training J Educ Health Promot. 2012;1:7 doi: 10.4103/2277-9531.94415
13. Xie F, Bowen JM, Sutherland SC, Burke N, Blackhouse G, Tarride JE, et al Using health technology assessment to support evidence-based decision-making in Canada: An academic perspective Expert Rev Pharmacoecon Outcomes Res. 2011;11:513–21 doi: 10.1586/erp.11.60
14. Kanoute A, Faye D, Bourgeois D. Strategies to promote better research on oral health in Africa: A Delphi consensus study Contemp Clin Dent. 2014;5:13–9 doi: 10.4103/0976-237X.128654
15. Paulo F, Mohammed D. Barriers to immunization coverage in DRC: An analysis of the GAVI-Alliance cash-based support Ann Trop Med Public Health. 2013;6:401–7 doi: 10.4103/1755-6783.127773
16. José M, Carmen A, Valero C, José C. Pooling expert opinion on environmental discounting: An International Delphi Survey Conserv Soc. 2016;14:243–53 doi: 10.4103/0972-4923.191162
17. Perleth M, Gibis B, Göhlen B. A short history of health technology assessment in Germany Int J Technol Assess Health Care. 2009;25(Suppl 1):112–9 doi: 10.1017/S0266462309090515
18. Somayeh H, Shaghayegh V, Zohreh S. Balanced scorecard method: The success factor in achieving medical tourism objectives in hospitals Int J Health Syst Disaster Manage. 2013;1:115–23 doi: 10.4103/2347-9019.128129
19. Mohammadi E, Raissi AR, Barooni M, Ferdoosi M, Nuhi M. Survey of social health insurance structure in selected countries; providing framework for basic health insurance in Iran J Educ Health Promot. 2014;3:116 doi: 10.4103/2277-9531.145919
20. Menon D, Stafinski T. Health technology assessment in Canada: 20 years strong? Value Health. 2009;12(Suppl 2):S14–9 doi: 10.1111/j.1524-4733.2009.00554.x
21. Kuchenbecker R, Polanczyk CA. Institutionalizing health technology assessment in Brazil: Challenge ahead Value Health Reg Issues. 2012;1:257–61 doi: 10.1016/j.vhri.2012.09.009
22. Huang JF, Wang HB, Zheng SS, Liu YF, Shi BY, Shen ZY, et al Advances in China's organ transplantation achieved with the guidance of law Chin Med J. 2015;128:143–6 doi: 10.4103/0366-6999.149183
23. Wang W, Jiang B, Sun H, Ru X, Sun D, Wang L, et al Prevalence, incidence, and mortality of stroke in China: Results from a nationwide population-based survey of 480 687 adults Circulation. 2017;135:759–71 doi: 10.1161/CIRCULATIONAHA.116.025250
24. . The Lancet. The best science for achieving healthy China 2030 Lancet. 2016;388:1851 doi: 10.1016/S0140-6736(16)31842-6

Edited by: Qiang Shi


Evaluation System; Medical Management; Medical Technology

© 2018 Chinese Medical Association