Once the lower confidence limit for MTBF is estimated, the reliability, R, can be calculated for a given mission duration by using the expression for R given above. For a preclinical reliability demonstration test goal, the test durations for a decision to begin clinical trials will be much shorter than the anticipated wear-out durations with the newer MCS devices. The reliability goal will then be met with the assumption of an exponential distribution (given that the test will not give data to infer a Weibull distribution). The assumption of an exponential distribution gives a very conservative estimate of the lower confidence limit of reliability. See Appendix C for a discussion on the reliability statistics.
Conduct of the Reliability Study
The study should be performed per the study protocol. The study protocol includes a description of the test articles (the MCS device under test) and the test system (the mock circulatory loop). In addition, the study protocol prescribes the operating modes, operating conditions (pressures, flows, run times at conditions, etc.), data collection requirements, data analysis procedures, and study success/failure criteria. It is a good practice to include a study description, study design, and study justification narrative in the protocol. The narrative helps to inform the reviewer of the study protocol to understand the study and the reasons for the study methods.
The reliability goal is dependent on the intended use of the MCS device being tested. The goal is normally given as a reliability, R, for a given mission duration, t, and a given confidence, c. For some devices, the intended use duration may be relatively short and so the mission duration should be appropriate for the intended use. For an intended use such as a MCS device used for “destination therapy,” a goal of R = 0.8 for a mission duration of 1 year, with a confidence of 60% has been considered appropriate for the start of a clinical trial (Investigational Device Exemption [IDE]).
The study protocol should prospectively define the reliability goal for the different stages of clinical introduction of the device (i.e., IDE and premarket approval [PMA] in the United States). An initial goal is stated for the start of a clinical trial (preclinical goal for IDE application), and another goal is stated for the introduction of the device as an approved product (goal for PMA application). It is recommended that the study includes a long-term phase that is a failure terminated study; the study is not ended until a certain number of failures have been accumulated. Without having measured failures, the actual reliability of the device is not known, and the study only demonstrates that the device does not possess an unacceptably low reliability.
System Reliability Demonstration Test
There are several premises associated with reliability testing. One is that the test set up (such as a mock circulatory loop) and test conditions are adequately representative of the anticipated clinical environment (simultaneously includes all the important, clinically relevant loads). As will be discussed below, it may require judgment to accomplish this. For example, it may be possible to simulate corrosion, afterload fluctuation, and body temperature simultaneously. However, it may be impractical to produce a working fluid that simulates both corrosion and the viscous properties of blood because of the difficulties associated with maintaining this type fluid for extended periods of time. The mock circulatory loop must be sufficiently robust but maintainable for reliability testing, so that the test article is evaluated not the test loop.
Another important premise is that the device being tested is adequately representative of the final configuration and orientation of the device that will ultimately be used clinically. Device changes during reliability testing may require a protocol amendment providing the reason that the test results are still applicable with the device change or the change may require retesting of the device.
The system reliability test should start with a protocol prospectively signed-off by a cross-functional team that includes both engineers and clinicians. Protocol review by the appropriate regulatory agency is encouraged before initiation of the study to increase the probability of acceptance of the results.
In Vitro Test Environment.
Test conditions should model the applicable patient population. The mock circulatory loop test conditions should simulate anticipated conditions that have been derived from known pathologic findings and the clinical database. These conditions will vary depending on the design of the device. However, all conditions chosen must be justified in relation to the specific device design, the intended patient population (e.g., adult or pediatric), and anticipated clinical use.
Reliability testing under “worst case(s)” physiologic conditions that have been determined by device characterization testing to place the most stress or load on the device design (i.e., hypertension) should also be conducted. If these test conditions do not fall within the test conditions described in the paragraph above, then this testing should be performed separately.
As an example, one approach to the reliability demonstration for a rotary pump with a magnetically levitated impeller is discussed in Appendix D.
Mock Circulatory Loop.
The design of the mock circulatory test loop for the device reliability demonstration depends on the device design (e.g., volume displacement and rotary), its application (e.g., left heart support, right heart support, and biventricular support), and the patient population to be treated (e.g., adult or pediatric patients).
There are different approaches to the design of a test loop. A common approach is to emulate the physiologic system with representation of resistance and compliance, systolic, and diastolic function. With this approach, the test loop simulates the circulatory system with the device placed in the human simulated anatomical position and functioning as intended, i.e., TAH, bi-VAD, or VAD.
An alternative approach is to design a test system that applies the anticipated loads onto the device. The device is exposed to the same loading it will be exposed to in the intended clinical application including anticipated extreme conditions; however, the loop itself does not necessarily emulate the physiologic system.
Appendix E contains examples illustrating different approaches for developing reliability demonstration mock loops.
A comprehensive analysis to determine the relevant loads to be imposed by the mock circulatory loop may include a physiologic in vivo study and/or in vitro experiments, i.e., characterization tests, physical, and mathematical modeling. This analysis must consider pulsatile loads associated with residual ventricular function and pulsatile loads created by the device itself. The loop parameters and test conditions selected should be justified based on the analytical results.
The following features simulate the conditions relevant to the patient population and should be considered in designing the reliability test loop:
- A ventricle pump model that can simulate the effect of residual ventricular function (i.e., varying degrees of failure) on the device performance anticipated with the intended patient population.
- Appropriate heart valves, e.g., aortic and pulmonary (for biventricular or RV systems).
- A tubing circuit to simulate the aortic pressures and/or pulmonary pressures, compliance, and systemic resistance.
- Conduits that adequately represent the device's respective inflow and outflow grafts from a fluid dynamics perspective.
Variables to consider in the design of the mock loop and the selection of test conditions include but are not limited to the following (see Appendix B for tables of hydrodynamic load ranges experienced with circulatory support systems.):
- Systemic vascular resistance.
- Pulmonary vascular resistance.
- Systemic arterial compliance.
- Pulmonary arterial compliance.
- Systemic venous compliance.
- Pulmonary venous compliance.
- Left inlet or left atrial pressure.
- Right inlet or right atrial pressure.
- Left outlet or aortic pressure.
- Right outlet or pulmonary artery pressure.
- Flow rates (e.g., pump inflow and outflow, total systemic, and/or pulmonary flow).
The final design of the mock circulatory test loop will depend on the device design (e.g., volume displacement and rotary) and intended application (e.g., left heart support, right heart support, and biventricular support) and the patient population to be treated (e.g., pediatric and adult).
The immersion fluid should be normal saline (0.9% NaCl), which simulates the corrosive nature of blood and the interstitial fluids surrounding the implant. The working (i.e., pumped) fluid should adequately represent the hydraulic loads and corrosion properties of blood with respect to with the pump surfaces. When it is not feasible to simulate all conditions simultaneously, the choices made should be based on the risk analysis (such as the FMECA and FTA referred to in Figure 1) specific for the device.
Saline can be used to evaluate corrosion. Historically, the effect of viscosity on reliability was determined to be negligible with pulsatile pumps and testing with saline has been justified. However, hydrodynamic-based designs may have issues that are affected by viscosity. To determine whether a viscous solution should be used, acute characterization testing that compares the effect of a viscous blood analog to saline should be conducted. If viscosity is determined to be important, saline will not adequately represent the viscosity, and a glycerin solution may be used to simulate the viscosity of blood.
The parameters of the mock loop can be adjusted to compensate for the fluid dynamics of saline as opposed to blood (i.e., viscosity), which may be essential for the appropriate testing of a particular blood pump system. The justification for these adjustments should be provided.
Note that saline does not adequately represent the thermal conductivity of the implanted condition and may not account for the effects of thermal cycling associated with clinical load variations, e.g., circadian cycles. Separate, additional testing such as HALT and environmental stress screening may be useful to address this issue.
The implanted components of the test article should be submerged in saline simulating the normal body temperature of 37°C ± 3°C.
The test article includes all implantable components, representative of the final design configuration to be used clinically. Deviations from the clinical configuration of the device must be justified. For example, if, for practical reasons, certain items are excluded (such as conduits and valves) functional equivalents should be substituted. Any excluded items should be evaluated separately through subsystem testing. External components, such as the controller, should also be representative of the clinical configuration. If prototype external hardware must be used, the unit must be equivalent to the anticipated clinical configuration functionally and electrically at the implant interfaces, to maintain integrity of the simulation. Similarly, software should be clinically representative. The software version used in the test may also include additional provisions for the test protocol execution. In general, it will be advisable to include software upgrades for reliability test units when available to maximally simulate ultimate interface conditions.
Preconditioning of the test articles should be considered. For example, if it is thought that sterilization, and perhaps other factors, might affect long-term performance, then the test articles should be preconditioned accordingly.
Test Operation Profile.
The device operation should be cycled to simulate periods of normal activity, exercise, and rest,10 as appropriate. Test conditions (e.g., cycling intervals and flow rates) can be based on the data provided in Appendix B or individually justified by the sponsor based on their device and its intended use.
The physiological states of all anticipated patient conditions should be considered when defining worst-case conditions. The reliability test may not be able to accommodate all worst-case conditions in the same test. Furthermore, testing all devices under conditions that may reflect a small portion of the patient population or temporary conditions in patients may not produce the reliability results that reflect what is expected clinically.
Therefore, the impact of the worst-case conditions should be evaluated for its impact on the failure rates of the MCSD components. If the worst case condition significantly alters the test conditions defined for the majority of the expected patients, or conflicting conditions exist, the sponsor should consider additional testing to evaluate device performance under these extraordinary circumstances.
If necessary, condition durations may be extended beyond normal daily durations or averaged over a period of time to limit the number of switches between test modes. However, averaging needs to be justified because it will reduce the number of diurnal cycles and therefore may not adequately represent this variable loading profile.
The physiologic and device parameters of interest may include:
- Device inlet pressure.
- Device outlet pressure.
- Flow rate (i.e., flow at the pump inlet, flow at the pump outlet, flow at the outflow anastomoses, total systemic, and/or pulmonary flow).
- Simulated patient activity level.
- Device power consumption.
- Device orientation.
- Motor waveform.
- Ejection fraction, etc.
The Role of Animal Testing in Reliability Assessment
Animal testing provides a reasonable simulation of some aspects of the clinical application, enabling preclinical characterization of system performance in an in vivo environment. Although the purpose of the animal testing is to provide initial safety data, it may also provide additional insight regarding system reliability. However, the statistical analysis that is needed for the device reliability assessment cannot use this data because of obvious differences in the test duration and conditions. It may be most appropriate to use in vivo test data to assess load-strength margin, for example. Any failures that occur during animal testing need to be carefully evaluated for the possible need to implement a device design or operation change or to indicate the need to change the reliability demonstration test protocol.
The reliability of the device is established by more than a single system test. A series of tests that include characterization tests, HALT, subsystem, and component testing are used to demonstrate device performance when exposed to a range of extreme clinical conditions. It is anticipated that with the evolution of the circulatory support systems, the reliability issues may be more limited by the subsystems (those that are not realistically loaded by the system testing). For example, with the rotary devices, the predominant failure mode could end up being failure of the electrical circuits within the implanted part of the percutaneous lead. With that in mind, the subcommittee has compiled a list of subsystems that may be considered for load-strength margin analysis and subsystems that may require separate reliability testing beyond the system test. Table 1 in Appendix F presents possible subsystems, some of the relevant loads for consideration, potential failure modes, and test loads that may be important in the design of the subsystem or subsystem test. Where test methods have been identified or recognized, they have been included in the table.
As an example of how the table may be useful, consider a manufacturer attempting to demonstrate the reliability of the inflow and outflow conduits. The manufacturer should consider using the maximum clinical pressures with an accelerated flex, pulse, and/or torsion test and look for leakage over the length of the conduit, at the connector and the conduit's termination to its connector.
Guidelines for Qualifying Design Changes
Information from in vitro, in vivo, and/or clinical testing may require that design changes be made to the system, while reliability testing is in progress. All design changes must be made in accordance with the sponsor's design control procedures under ISO 13485 Quality Management Systems for Medical Device Manufacturers and the US FDA Quality System Regulation, 21 CFR Part 820. All design changes must be evaluated using risk analysis and management techniques for their potential impact on the reliability estimate of the system on test. If a design change is implemented in the system(s) undergoing reliability testing, the sponsor must provide a sound engineering rationale justifying any pooling of life test data across system configurations. If design changes are not implemented in the system(s) undergoing reliability testing, the sponsor must provide a sound engineering rationale justifying the application of the previous configuration life test data to the new design configuration intended for clinical use. It should be noted that a design change that results in a significant alteration from the original configuration or the configuration to be used clinically may generate the need for a new reliability study.
System Reliability Information for Surgeons and Prospective Patients
The Informed Consent, a document intended to inform prospective patients of the risks associated with a device (regarding health, quality of life, potential risks, and otherwise), should refer to reasonably foreseeable risks.11–13 The current preference is to favor more, rather than less, disclosure. Examples of items commonly disclosed include risk or hazard probability assessment based on results of the in vitro reliability tests (systems and subsystems), description of backup systems (if any), and possibilities of repair and replacement. More detail may be found in the Ref. 1.
Similar information should appear in product labeling, such as the instruction for use.14
The Investigators Brochure, a document intended to inform investigators and others about the study and its rationale, should also provide the investigator with appropriate information to allow his/her own unbiased risk-benefit assessment of the proposed trial and analysis of results. There should be a layman's summary that can inform patients about the value of the reliability studies in making their decisions about treatment options. Information about in vitro reliability results may include the following:
- The number of systems.
- Test time (range, minimum, and maximum durations).
- Reliability statistics.
- Number of catastrophic and critical failures, including the types of failures in these two categories.
- If the units are continuing on test at the time of device approval, that information should be so stated, and periodically revised to reflect updated information.
- Design differences between the units that were bench tested and those in clinical use, and how those differences may affect reliability.
This recommendation suggests that the reliability study should be designed as a failure terminated study, so that true reliability is ultimately measured. It is recommended that the device developer consider, at appropriate intervals, as the reliability data is accumulated, updating the information provided for surgeons and prospective patients.
Listing of Appendices
Appendix A: Example of reliability test design methodology.
Appendix B: Clinical experience—relevant loads.
Appendix C: Reliability, durability, clinical trial, and clinical use.
Appendix D: Discussion of reliability demonstration for a rotary pump with a magnetically levitated impeller.
Appendix E: Examples of test loops for VAD reliability demonstration.
Appendix F: Subsystem reliability, tables of possible considerations.
National Clinical Trial Initiative
This recommendation was developed by the Reliability Subcommittee of the National Clinical Trial Initiative. Subcommittee approval of the recommendation does not necessarily imply that all subcommittee members voted for its approval. At this time, the Reliability Subcommittee of the NCTI has the following members:
Subcommittee chair: James Lee
Members: Michael Berman, U.S. Food and Drug Administration; David Borzelleca, Evaheart Medical USA, Inc.; Carl Botterbusch, Arrow International, Inc.; Kevin Bourque, Thoratec Corporation; Gary Cederwall, Thoratec Corporation; Eric Chen, U.S. Food and Drug Administration; Scott Corbett, Abiomed, Inc.; Scott Fitzgerald, SynCardia; Mike Geringer, Thoratec Corporation; Guruprasad A Giridharan, PhD, University of Louisville; Jonathan Grashow, Evaheart Medical USA, Inc.; Steven C. Koenig, PhD, University of Louisville; Robert T.V. Kung, PhD, Abiomed, Inc.; Jeff Larose, HeartWare; James Lee, World Heart Inc.; James Long, MD, LDS Hospital; Steve Marshall, Ventracor Limited; Alex Medvedev, Terumo Heart, Inc.; Don Middlebrook, Thoratec Corporation; Phil Miller, World Heart Inc.; David Munjal, PhD, Terumo Heart, Inc.; Chisato Nojiri, Terumo Heart, Inc.; Kathryn O'Callaghan, U.S. Food and Drug Administration; Raj Pandey, HeartWare; George Pantalos, PhD, University of Louisville; Sonna Patel, U.S. Food and Drug Administration; Jane Reedy, HeartWare; Jean Rinaldi, U.S. Food and Drug Administration; Rhona Shanker, Z & B Enterprises, Inc.; Alex Stonehouse, Ventracor Limited; Dan Tamez, HeartWare; John W. Toigo R.A.C., Terumo Heart, Inc.; John T. Watson, PhD, University of California, San Diego; Ray Wong, PhD, Abiomed, Inc.; Hong Zhang, PhD, Abiomed, Inc.
Participation by federal agency representatives in the development of this recommendation does not necessarily constitute endorsement by the federal government or any of its agencies.
1. Altieri F, Berson A, Borovetz H, et al
: Long-term mechanical circulatory support system reliability recommendation American society for artificial internal organs and society of thoracic surgeons: Long-term mechanical circulatory support system reliability recommendation. ASAIO J
44: 108–114, 1998.
2. O'Conner PDT: Practical Reliability Engineering
, 2nd ed. Hoboken, NJ, Wiley, 1985.
3. Dietrich DL: Reliability from Design Inception to Product Retirement, 2006 Proceedings Annual Reliability and Maintainability Symposium, IEEE. Newport Beach, CA, 2006.
4. Nelson W: Accelerated Testing: Statistical Models, Test Plans, and Data Analysis, 2nd ed. Hoboken, NJ, Wiley, 2004.
5. Hobbs GK: HALT and HASS, the New Quality and Reliability Paradigm
. CO, Hobbs Engineering, 2002.
7. Hasdai D, Berger P, Battler A, Holmes DR Jr: Cardiogenic Shock, Diagnosis and Treatment
. Totowa, NJ, Humana Press, 2002, p. 12.
8. Mielniczuk L Mussivan T, Davies R, et al
: Patient selection for LVAD. Artif Org
28: 152–157, 2004.
9. Aaronson KD, Patel H, Pagani FD: Patient selection for left ventricular assist device therapy. Ann Thorac Surg
75: S29–S35, 2003.
10. Yingjie L, Allaire P, Wood H, Olsen D: Design and initial testing of a mock human circulatory loop for left ventricular assist device performance testing. Artif Org
29: 341–345, 2005.
11. 21CFR50.25 Protection of Human Subjects: Elements of Informed Consent, Guidance on Medical Device Patient Labeling; Final Guidance for Industry and FDA Reviewers issued on: April 19, 2001.
12. ISO14155–1: 2003 Clinical Investigations of Medical Devices for Human Subjects. Part 1: General Requirements Section 6.7.3.
13. ICH Harmonized Tripartite Guideline. Guideline for Good Clinical Practice E6(R1), Technical Report, 1996.
Appendix A: Example of Reliability Test Design Methodology
A simplified hypothetical example illustrates the application of the methodology. The information from the clinical database is used to determine the appropriate systemic compliance and resistances to use as models of the circulatory system lumped parameters to use on a mock circulatory loop From the device description and the risk analysis results, we identify a bearing as a component with a potential failure mode of concern. It is determined that the load on the bearing is proportional to the pressure load within the pump. Since the pressure pulse is a function of the flow rate generated by the MCS as well as the systemic resistance and compliance of the circulatory system, a realistic pressure load must be determined by a measurement. Utilizing simulated use testing and active loop characterization, we quantify the pressure load within the pump that ultimately defines the load applied to the bearing of concern. The pressure within the MCS pump chamber is measured over the range of operation defined by the design description. The result of the hypothetical study is summarized in Figure A1 which gives the diastolic and systolic pressure, within the pump, over the range of mean aortic pressures (MAP) – as measured in the mock loop test. From the clinical database, we can see that a small percentage of the data points are at the worst case Mean Aortic Pressure load conditions (90–99%). By correlating the probabilities, we can derive a probability distribution curve for the bearing load. For example, the clinical database shows that the 99th percentile mean aortic pressure is at 110 mmHg. The corresponding systolic pressure is at 133 mmHg. So the 99th percentile pump pressure is 133 mmHg. The cumulative distribution is shown in Figure A2. The derivative of the cumulative distribution (Figure A3) gives the probability distribution function (pdf). The pdf gives the frequency of occurrence for the systolic pressure range. Since the pump pressure is proportional to the load on the bearing, the distribution of the load on the bearing is estimated.
From the manufacturer's data (or test data), the strength of the bearing is determined and compared to the load distribution (Figure A4 is a hypothetical example). Depending on the strength, one can rank the design margin into a range of categories (for example: minimal margin, acceptable margin, wide margin, very wide margin). If the device design does not have a wide or very wide margin, the device developer may chose to run the test with an accurate simulation of the systemic pressure loads when running the in-vitro reliability demonstration test.
Figure A4 shows three bearings. Bearing 1 has a strength that is above the 99th percentile load (15 lbs), but the strength is near the higher 99th percentile load (about 13.5 pounds). One may expect this bearing to exhibit high cycle life, but with such a narrow margin, the reliability demonstration test would best accurately model the load. Bearing 2 is less than the 99th percentile load, so one would expect this bearing to wear rapidly. If for good reasons, this is the case, one would clearly want to accurately model the load to get a good estimation of the bearing reliability. Bearing 3 is a distribution of strengths that may have come from actual testing of bearings. The distribution is narrow and is far away from the distribution of loads, so one would expect this to exhibit high cycle life as well. Since the strength distribution is much greater than the load distribution, accuracy of the load in the reliability demonstration test need not be as much of a concern. Note that there is no rule for how much margin is necessary, so the decision for accurate modeling of the load with the reliability demonstration test is not a prescriptive process.
Appendix B. Clinical Experience – Relevant Loads
The reliability subcommittee has collected the following information derived from clinical trial records of adult human VAD patients to help the reliability test developer design a test system that will apply clinically relevant loads for adult patients to their circulatory support system. Tables B1–B6 represent aggregated clinical data from several different device types. This comprehensive set of data gives the test system developer a resource that they may use to design the appropriate tester for their device. Note that the values are representative of the range of data provided for the different sources of clinical data. Data is stratified for volume displacement devices and rotary devices. The percentile data points are given for different timeframes after implantation of the circulatory support system (for example POD1-7 represents data collected between post operative day 1 to post operative day 7 and POD8-30 represents data collected between post operative day 8 to post operative day 30).
Data derived from Physiologic and Hemodynamic Data while on VAD therapy Waveform data was available for a limited number of clinical cases. The data was analyzed to estimate ranges for clinically relevant values of a lumped parameter circulatory model. Tables B7a and B7b summarize the model data.
The reliability subcommittee recognizes that the clinical data, although comprehensive by being inclusive of experience with many device recipients and many types of devices, is still insufficient to derive all of the relevant parameters for a reliability test loop. One area that is important is the value used to model arterial compliance. This parameter is important because the arterial compliance, along with the circulatory support 'systems flow pulse, defines the pulse pressure. To expose the unit under test to a realistic systolic and diastolic pressure, the arterial compliance must be clinically relevant. This is particularly important for a device that has a minimal load-strength margin for peak afterloads or pulse pressure at the afterload. Historically, the arterial compliance has been modeled with a typical value of 1.8 to 1.9 ml/mmHg. Arterial compliance is known to stiffen with age (typically starting at age 20). An allometric equation gives a sense for the range of values that is expected in the normal population. (Li, J, The Arterial Circulation Physical Principles and Clinical Applications, Humana Press, New Jersey 2000).
Another factor is that patients with end-stage heart disease are known to have a stiffer arterial walls resulting in lower compliance due to the underlying disease (atherscerosis). Using the above expression, a normal value for a 65 year old would be 0.99 ml/mmHg. Comparing to the data in Tables B7a and B7b, one can see that the median arterial compliance is in the order of 0.5 ml/mmHg.
Based on the above, the subcommittee recommends that the reliability test developer consider a more accurate arterial compliance value consistent with the intended use of their circulatory support system.
Appendix C. Reliability, Durability, Clinical Trial and Clinical Use
To understand the relationship between device reliability characteristics and its clinical applications, we first need to examine the nature of MCS device reliability. Figure C1 shows the typical bath-tub curve of MCS device's hazard rate (this curve applies to most products). The high early hazard rate (due to a weak population that may be caused by production variability) is followed by a relatively low and nearly constant hazard rate. The magnitude of this nearly constant hazard rate region determines whether the device is suitable to start a clinical trial. The rising hazard rate over time is caused by wear-out.
The high early hazard rate failures should be resolved by either eliminating its causes or using burn-in methods to filter such devices. The bath-tub curve with early failures removed corresponds approximately to the fully characterized reliability of the device as shown by the solid reliability curve in Figure C2. As an example, a Weibull curve with β = 4 is used for illustration, β is determined by the test results and varies widely depending on each system's wear out mode. Durability is the information about how long it takes for the device to wear out. To obtain the durability, we need to continue the reliability test until most, if not all, devices fail. Once the devices have failed, a Weibull estimate of reliability can be obtained (as an example, the “β = 4, Final results curve” in Figure C2).
If the device wear out mode is in the 4 or 5 year timeframe, then there is a low probability of failures due to wear out with a one-year test. However, the one-year test does give an estimate for a lower confidence limit (lower bound) for constant hazard rate failures. Following 1 year of testing, we know that it is probable that if the constant hazard rate reliability was at or below the specified level, we would experience one or more failures. Following 2, and then 3 years of operation, if there are no failures or very few failures with larger sample sizes, then the lower confidence limit for constant hazard rate failures is “lifted up” as shown in Figure C2.
We cannot obtain the ultimate one-year durability with a one-year test, but by assuming β = 1, we can conservatively estimate the one-year reliability from a one-year test result. Since no failure is expected or observed during a one-year test for a device expected to survive to three years, one can only assume β = 1. This is demonstrated by the W (β = 1, 1-year test) curve in Figure C2 that shows a one-year test results assuming β = 1. This curve is a conservative lower confidence interval limit based on a test with no failures and a small sample size. Since the final β will most likely be much greater than 1 thus lifting the curve up, assuming β = 1 gives us the most conservative estimation of reliability.
When reliability tests are carried out, determining the sample size is a challenging task because to demonstrate certain reliability, the larger the sample size, the greater the probability of success as demonstrated by Figure C3. Each curve in Figure C3 shows the probability of successfully demonstrating 80% reliability at 83% confidence level versus the true reliability of the device for a specific sample size. Different curves correspond to different sample sizes. Suppose the true reliability is 90%, the test success probability significantly increases from 0.43 to 0.79 respectively when the sample size increases from 8 to 40. When a device is being tested, its true reliability is unknown. This is why determining the optimum sample size is crucial. For a given reliability goal with specified use duration and confidence level, one knows the minimum sample size that requires no failure. Using the minimum sample size (n = 8) and supposing the true reliability is not much greater (e.g. 85%) than the reliability to be demonstrated (i.e. 80%), the chance of having a successful outcome is very low (i.e. 27%). Thus using the minimum sample size can have significant risk of test failure unless there is high confidence that the true reliability of the device is much higher than the target reliability.
Considering only the issue of reliability and durability, in the ideal world of development of a MCS, it would be desirable to fully characterize the reliability of such a device (i.e., to obtain the Weibull curve in its entirety, hence the information regarding durability) before making any clinical evaluation of the device. However, for initiating clinical trials, this is impractical. It is impractical because to fully characterize a MCS device aimed at, for example, a 5-year durability, 5+ year's time for the test is needed to measure the durability data. Initial clinical trials often reveal new issues that lead to device design change. The change or changes may be significant enough to mandate another round of reliability test. A decade or so would easily pass before the device reaches the market to start benefiting patients if full reliability data were required for every new device design. Due to the small sample sizes that are practical for the reliability demonstration tests, for the test durations that are shorter than the wear-out intervals, the lower confidence limit proven by the reliability demonstration test is necessarily conservatively low even if the true reliability is very high. It is not until the test yields failure data that the reliability estimate becomes a reasonable predictor of the actual reliability/durability of the MCS. Given that the clinical trial yields important information other than that of device reliability, it is important to facilitate clinical trials to evaluate the medical benefits of a given MCS. With continued reliability demonstration testing (in parallel with the clinical trial and beyond), when actual failure data is collected, a more realistic estimate of the reliability curve can be determined.
Appendix D. Discussion of Reliability Demonstration for a Rotary Pump with a Magnetically Levitated Impeller
Many mechanical circulatory support (MCS) systems that have been developed over the past 40 years including first and second generation (pulsatile and continuous flow with mechanical bearings, respectively), and third generation pumps which include continuous flow, magnetically suspended devices. Additional devices include IABP and total artificial hearts (TAHs). The differences between generational devices are highlighted by the specific components of each generation of device. For example, the inlet and outlet cannulae are similar for most devices, however, 3rd generation devices have magnetic or hydrodynamic suspension system and other devices have a mechanical bearing system. Further, 1st generation devices have a polyurethane blood sac as the blood flow through region, while in continuous flow devices it is titanium and the blood is in contact with an impeller.
The following example shows how the reliability of a rotary device with a magnetically levitated impeller could be established. This example follows the recommendation that clinically relevant loads should be the basis for the real-time reliability test design. It also addresses the concern that for some device designs these realistic clinical loads will not significantly challenge the particular device's reliability. This concern is addressed through reliability tests of the subsystems.
The valve-less design of the rotary pump with the magnetically suspended impeller and actively controlled gaps eliminates high levels of hemolysis within the blood chamber. It is also expected that such a pump, if properly designed, will not be subjected to failures associated with mechanically deteriorated parts of the blood pump, such as a bearing or sac failure.
Magnetic levitation robustness (impeller stability) should be demonstrated by applying a series of disturbing loads to the operational pump: pulsing pressure, drop shock, vibration, angular acceleration, etc. These tests need to be designed to show that the impeller is not in contact with the blood chamber and that the pump's hydraulic performance is not affected. If the impeller does come into contact with the blood chamber under some disturbing loads, then the possible damages should be assessed, and device operating limitations, if any, should be identified through appropriate labeling.
The structural integrity of the pump blood chamber and laser welds can be tested by overstressing, e.g., applying a pulse pressure that exceeds the chamber design limits. Recommendations for the overstressing level and test duration have been published.1,2
Other system components such as inflow/outflow subsystems, percutaneous cables and connectors are subject to flex, pulse, pull, bend and other mechanical stresses. These subsystems can be tested separately, using real-time or accelerated tests. For example, an outflow conduit — consisting of the vascular graft, a protective cover and a pump connector — can be exposed to pulsatile pressures from the residual ventricular function and a periodic movement associated with the patient's breathing. In this case, the test samples should be placed into the saline tank, where they are connected to the pulsing pressure source and where they can be moved appropriately. The test design should consider the expected number of pulses and movements for the period of time required for immobilizing the implanted graft by surrounding tissues. After the test completion, the subsystem must also pass the conduit tensile and burst strength test according to design specifications.
After reliability at the subsystem level is demonstrated, the real time system reliability test can focus on the stressing of electromechanical components (motor, drivers, hermetic connectors, solder joints, etc.). In the intended use these components of an MCS system are subjected to changing electrical and mechanical loads and associated temperature changes. Important parameters to consider in designing the system reliability test should include (along with hydraulic parameters) power consumption and RMS currents. These parameters are influenced by the pump load changes associated with the patient's activity levels and circadian cycles. Therefore, the pump load conditions simulating various patient activity levels should be included in the mock loop reliability test. The magnitude and duration of load variations should reflect clinically relevant conditions, i.e., sleeping, moderate activity, light exercise. For example, the following load cycling can be used: 8 hours of a minimum load (minimum pump speed, flow, etc.), 4 hours of a maximum load, 8 hours of a moderate load, 4 hours of a maximum load, 8 hours of minimum load, and so on.
In the test loop, thermal and mechanical loads should simulate clinically relevant loads that may include worst-case conditions. To determine worst-case conditions for the ventridular assist device (VAD) system with a rotary pump, an analysis of the pump design should be conducted. In the case of centrifugal pumps, for example, the higher flows at a given speed mean greater motor torques and currents and thus the temperature of components; for the magnetically suspended impeller, an increased pump pressure pulse produces a greater impeller displacement force; the higher rotary pump speed results in faster wear of the motor's mechanical bearings; etc. The stresses produced by corresponding clinically relevant loads can be analyzed by performing the system characterization test using a pulsatile loop that simulates a wide range of clinical conditions: an impaired or a healthy ventricle, hypotension or hypertension, pump flow obstructions, etc. In general, the system characterization test is used to study the effects of physiologically relevant conditions on the system performance, thus determining the long-term reliability test design (i.e., the mock loop hydraulic and pump operational parameters).
The following example (see Table D1) shows the results of a study conducted to evaluate the effects of residual ventricular function on the rotary pump performance. The conditions of the real patient were re-created in the mock loop, consisting of the left ventricle model and the systemic circulation model. The patient was 63 years old, with BSA = 2.2m2, weight = 97 kG, NYHA Class I, after one year of MCS by the rotary pump with a magnetically levitated impeller. The pump's performance was evaluated in the presence of the left ventricle model and after the artificial ventricle was clamped out. The test results show that in this case, the patient's LV pulse did not affect the motor and levitation coil RMS currents, impeller position, pump performance, power consumption, and the temperatures of either the pump or the electronics.
If the blood-chamber structural integrity (under pulsing pressures) is tested by overstressing, and if the reliability of the inflow and outflow conduits is confirmed at the subsystem level, then the real-time reliability test mock loop may not need to include the pulsing ventricle model. Such a design decision has to be supported by evaluation of the system performance at the wider physiologic conditions, including the fully recovered healthy ventricle that may generate larger pulse loads.
Appendix E. Examples of Test Loops for VAD Reliability Demonstration
A common approach to the test design for the device reliability demonstration employs a test loop that emulates the physiologic system with lumped parameter surrogates of the actual physiologic system. With this approach, the test loop simulates the clinical use of the MCS placed in the appropriate position to support functioning ventricles. Figure E1 illustrates an left ventricular assist device (LVAD) reliability test set up that includes the model of a systemic circulation, i.e., mock ventricle, valves, systemic resistance and compliance, etc.
An alternative approach is to design a test system that applies the anticipated loads onto the device, i.e., the device is to be exposed to the same loading as it would when implanted as intended. The loop itself does not need to emulate the physiologic system. The mock loop in this case (Fig. E2) consists of the clinical load emulator (CLE) that may include power and control units, the test articles, appropriate instrumentation and a computerized data acquisition system (DAQ). CLE should be able to respond to the outflow of the VAD to create pressures and flows that are relevant for the device intended application including the anticipated extreme conditions. The physiologic and device parameters monitored in the test may include: 1) device inlet pressure, 2) device outlet pressure, 3) flow (waveform, rate), 4) simulated patient activity level, (i.e. CLE parameters for walking, standing, sleeping, etc.), 5) device power consumption, 6) etc.
Appendix F: Subsystem Reliability, Tables of Possible Considerations
Tables F1–F7 provide a checklist of possible considerations for setting up tests of subsystems. Since there are many types of subsystems and possible methods to test them, this appendix should not be considered a comprehensive source of considerations and test methods for the subsystems.
Test Methodology – Accelerated Subsystem Testing, Comments
The subcommittee recognizes that accelerated testing is an important test methodology for the development of reliable systems that should be considered during the development phase of an MCS program and may be used in the verification phase of subsystems that the system test does not sufficiently address. Testing of subsystems using accelerated techniques can help to identify potential failure modes that may not be apparent during pre-clinical real-time testing. Given the complexity and range of potential acceleration techniques, the subcommittee cannot recommend a particular method. However, if accelerated testing is utilized, the following considerations are important.
When designing accelerated test systems, consideration of the acceleration method, the method(s) for the detection of fault conditions, and rationale for the validity of the acceleration method (as well as the acceleration factor model) are important factors to consider. If and when a failure mode is precipitated, the validity of the failure mode must be considered as well. With accelerated testing, the failure mode may be a mode caused by the acceleration process and the failure might be clinically irrelevant. As lab testing and clinical experienced is gained, and failure modes are identified, the accelerated test methodology should be re-evaluated for validity and revised as needed.
Some forms of acceleration are intuitively simple to understand. For example, failure modes that are sensitive to cyclic loading can be accelerated by cycling the subsystem at a higher rate than in the intended use. Parts that normally cycle at 100 bpm may be accelerated 3X by cycling the part at 300 bpm. The tester must be designed to apply a realistic cyclic load and the acceleration should not significantly affect the environment of the subsystem (such as the temperature of the subsystem).
It is important to realize that the results of some types of accelerated testing can be ambiguous and should be carefully considered. For instance, some failure modes may be accelerated by testing at elevated temperatures. With this type of acceleration technique, the failure modes detected by the test may not be the same kinds of failure modes that would be seen with testing at normal operating temperatures. So the test result could be misleading, causing a device manufacturer to focus on the wrong failure mode. With temperature acceleration, the acceleration factor is always not well known. Use of simple rules of thumb for acceleration factors may not give sufficient scientific validity. On the other hand, the literature may give realistic ranges for activation energy which may give a good range for an acceleration factor.
Where there is good rationale for designing and performing the accelerated test, the performance of accelerated testing is encouraged.Copyright © 2009 by the American Society for Artificial Internal Organs