The monitoring of basic cardiorespiratory vital signs, such as heart rate (HR), heart rate variability (HRV), and respiratory rate (RR), is essential in clinical care.1 For this purpose, various techniques are available in, for example, intensive care unit monitors and anesthesia equipment. The use of an ambient and unobtrusive measuring strategy is a major advantage in current medical diagnostics. Novel sensor concepts and computer-aided measurement modalities for clinically relevant, cost-effective, noninvasive/nonobtrusive monitoring of cardiorespiratory functions, and peripheral vascular hemodynamics offer important advantages for early-stage diagnostics.2 From this viewpoint, optoelectronic sensors are an especially helpful and intriguing bridge between the human body and biomedical technology devices. These advantages, combined with their miniaturization and integration, together with new functional imaging techniques have led to the rapid growth of biomedical optics as an independent field of research. As an example of the progress in biomedical optics diagnostic systems, the following section describes recent developments in classical photoplethysmography (PPG) and the role played by their novel camera-based variant, photoplethysmography imaging (PPGI).
From Hertzman’s Discovery in 1938 to the Photoplethysmographic Imager 2015
Classical PPG has for many years been one of the most popular methods for functional monitoring of dermal perfusion status. In 1935, the groundbreaking work of the German pharmacologist and physiologist, Karl Matthes (1905–1962), resulted in the first publication on the use of optical sensor-based assessment of arterial blood saturation level.3 Then, in 1938, Alrick B. Hertzman (1898–1991), physiologist at the St. Louis University School of Medicine (St. Louis, MO), found a relationship between the intensity of backscattered polychromatic light and blood volume in the skin. His instruments consisted of 3 main components that are still found in modern systems: a light source, a light detector (Figure 1A), and a registration unit. He called the device a “photoelectric plethysmograph” and wrote the following about his findings (Ref 4, p 336):
The volume pulse of the skin as an indicator of the state of the skin circulation at rest and
Amplitude of volume pulse as a measure of the blood supply of the skin.
The basic principle behind measurement of changes in blood volume in skin by means of PPG is the simple fact that hemoglobin in blood absorbs infrared light many times stronger than the remaining skin tissue (Figure 2).5–9 Figure 2 shows typical reflection spectra of anemic skin, of a 0.12-mm-thick layer of blood layer, and the extinction spectrum of a 0.3-mm epidermal layer. There is a clear difference in reflectivity between skin tissue and blood, and this leads to high optical contrast between skin and dermal vessel plexus. Optical attenuation of the epidermis is very high in the ultraviolet wavelengths (our natural protection against ultraviolet sun radiation) and lowest at wavelengths of ±930 nm. For example, as blood pressure in skin vessels decreases, so does blood volume in the transilluminated vessels. Thus, the surface area of the vessels is reduced leading to increased average reflection in the measuring window under the sensor, which results in an increase in the PPG signal.
In contrast to the conventional PPG variants with skin-attached discrete sensors (Figure 1B),6 which can also be applied in new body regions (with possibility for long-term monitoring in, for example, an in-ear channel; Figure 1C),10 PPGI operates remotely, that is, not in contact with tissue (Figure 1D).11–13 The use of a highly sensitive camera as a detector array (Figure 1D) offers exceptional advantages over current state-of-the-art devices. The large area of the skin to be measured is illuminated by quasi-monochromatic light-emitting diodes (LED) of selected wavelengths and is filmed by the camera from a distance of (generally) about 50 cm. This allows detection of small fluctuations in tissue brightness, which is synchronous with the venous and/or arterial blood volume dynamics. The fluctuations are caused by parts of the light beam that are reflected or transmitted toward the camera by passing through skin tissue.
Detecting Changes in Dermal Light Attenuation as a Function of Blood Volume
Concerning electromagnetic radiation at a frequency of about 300 THz (near-infrared, approximately 1000 nm), biological tissue can be viewed as a highly scattering, nonhomogeneous material. The spreading of photons injected into tissue (depending on the wavelength) is of special interest in many therapeutic and diagnostic applications of medical optoelectronics. PPG is currently widely used in many medical disciplines and is highly appreciated because of its simple design and relatively low cost per examination.
The absorption and scattering coefficients vary with skin depth and tissue. Typical values are about 0.05 to 0.15 mm−1 (μa) and 3 to 10 mm−1 (μs) in the 800- to 1000-nm wavelength range. Monte Carlo simulations show that the mean scattering-to-absorption probability ratio is about 50, and the free photon path between 2 collisions is about 0.24 mm.6 This leads to an optical attenuation in skin of about 7 dB/mm. Because of this, most optical sensors work in reflection mode, that is, the light source and the detector are placed next to each other in a single encasement on the surface of the skin. Transmission mode sensors can only be used on a few body locations, such as a fingertip or earlobe. PPG systems with active optical sensor technology in the near-infrared and/or visible light range can noninvasively detect changes in blood volume (related to arterial and venous hemodynamics) in the dermal blood vessel network, by registering the optical attenuation A of transilluminated tissue according to the Beer-Lambert law14:
where μ is the extinction coefficient of the absorbing tissue species i; c, its concentration; l, the path length through this tissue; and I 0 and I, the intensity of the injected and transmitted light, respectively. The sensitivity and effective measurement depth of the PPG sensors can be adjusted (as can the measuring wavelength used) by a variation of sensor geometry (distance a) and axis alignment of both components, as well as by the beam angles of the opening (numerical aperture [NA]). With a = 6 mm, NA = 0.09 (α =±5°) and right-angled positioning, the main detection area of the standard reflective PPG sensor for skin perfusion studies lies between 0.1 and 3.1 mm skin depth (decrease of maximal sensitivity to 1/e,6) with a 900-nm wavelength, giving a measurement volume of about 100 mm3. Of every million injected photons, only about 120 reach the detector and can be used for further signal processing. A sensor sensitivity profile can also be calculated when the light intensity at different depth locations is regarded as
with x and y as the area coordinates of the active sensor area, and z as the distance from the skin surface at z0. I d is the space-variant detector sensitivity and I e the space-variant intensity distribution of the emitter. I tot is the measured resulting detector signal.
Last but not least, knowledge of the basic optical tissue parameters (absorption coefficient μa(λ), scattering coefficient μs(λ), and anisotropy factor g(λ)) enables modeling and optimization of the light penetration depth in tissue.6,12,15
Following the basic working principle, the PPG signal reflects changes in blood volume in the cutaneous and also (partially) in the subcutaneous vessel plexus. Figure 3 shows that the PPG signal is first composed of a very large static component (S T) that is due (for a large part) to measuring light passing only through skin, tissue, or bone (without interaction with blood vessels). The second largest part of the detected light is attenuated/modulated by venous blood volume changes in the transilluminated tissue volume (S V). This component slowly varies because of respiration, vasomotor and vasoconstriction activity, and also because of thermoregulation. The smallest PPG signal component is proportional to the number of photons passing arterial and terminal microvessels (S A); this component will mainly possess peripheral blood volume pulse dictated by the heartbeat (central oscillator in our cardiovascular system). The last 2 PPG signal components depend on (possible) illumination artifacts (eg, ambient light coupling into the sensor) and quantization noise in the sensor close to the A/D signal conversion:
Therefore, this noninvasive technique allows acquisition of functional data from the dermal venous and/or arterial circulation. The classic/conventional skin-attached PPG sensor systems allow only a selected view into global hemodynamics from the measuring location. Applications on wounded or sensitive skin (eg, preterm infants) are a challenge. In addition, the sensor housings and cable connections restrict the patient’s freedom of movement, especially during 24/7 monitoring.
Photoplethysmographic Imager and Typical Records Today
The first PPG imagers were developed at RWTH Aachen University (Germany) as early as 199711–15; this was a computer-based system containing both hardware and software (Figure 4).
PPGI is a strategy that allows contactless recording and the processing and displaying of sequences of the selected skin area to visualize skin vessels and analyze dermal perfusion. The selected body area is illuminated by monochromatic light (multiple LED panels). In contrast to Hertzman’s device and all other commercial PPG devices, we generally use green light to visualize arterial skin perfusion. The rationale for this is that, in this spectral range, there is good optical contrast between blood and skin tissue; moreover, limited penetration of this light results in a targeted collection of perfusion dynamics in the microcirculation area of the skin.16,17
The size of the observed skin/body region and the spatial resolution can be arbitrarily chosen, depending on the camera lens used and the distance between the camera and the measured object. To minimize motion artifacts (possibly caused by the camera sensor separating from the skin), we prefer measurements in the recumbent patient. A foam cup is placed on the selected body region to minimize movement artifacts. This means that the validity of the PPGI technology is strictly linked to the quality of the video signal postprocessing (this is also an important research topic). Because of the high data volume recorded by the PPGI cameras, data reduction algorithms and multidimensional perfusion visualization tools are required to enable the use of this promising technology in routine medical practice. Another limitation of camera-based PPG is the high cost of the scientific cameras.
To detect the weak light modulation backscattered from the skin, which is caused by dermal perfusion (dermal blood volume pulse), a sensitivity camera has to be used. Initially (1997–2010), we used an UltraPix FE 250 camera from Life Science Resources (Cambridge, UK) because of its high dynamic range of 14 bits and high readout speed of 5.5 MB/s. The sensor, a silicon frame-transfer black/white CCD chip with a pixel volume of 512 × 512, is sensitive in the visible and near-infrared range of the spectrum. More recently, we have used the Pike F-210B CCD camera from Allied Vision Technologies (Stadtroda, Germany). Compared with the first-generation system, the Allied Vision Technologies sensor chip (Truesense KAI-2093) has a higher spatial resolution of 1920 × 1080 pixels and a faster readout speed of 31 frames per second (fps) at full-frame resolution; this value can be increased by reducing the spatial resolution. Temporal resolution can be increased by trading in spatial resolution.
We implemented 2 modes of operation: interactive offline and automatic. In the interactive operator system mode, before or after recording of video sequence, the examiner can select one or more arbitrary regions of interest (ROI) in the test field. For these marked regions, the mean backscattered light intensity is calculated and displayed as a perfusion-sensitive parameter over time.
Figure 5 presents an example of 3 PPGI registrations on the back of the left hand (ROI 3) and the inside of the right hand (ROI 2) of a 69-year-old patient with arrhythmia. The signal detected in ROI 1 (dark background area) was used to minimize the influence of ambient light and drifts in the PPGI illumination unit. The entire video stream had a length of 1500 images, which corresponds to a period of 30 seconds (at 50 fps).
At this point, it should be noted that PPGI enables contactless robust detection of HR and HRV. The accuracy is comparable to that of an electrocardiogram (ECG) if the PPG frame rate is >50 fps. In PPG and PPGI, the local maximum in the filtered transient signal (Figure 5: see * in plots 6 and 7) is suitable for such analysis, whereas in the ECG, it is usually the R-peak. However, extra systoles, arrhythmic episodes (in the presented example, 3 within 30 seconds) or measurement artifacts (mainly because of position changes between the contactless PPGI sensor and the skin), can lead to misinterpretation. Our recent study15 compared the consecutive beat-to-beat intervals (measured with ECG as the “gold standard” and with reflective in-ear PPG) in healthy subjects in a steady state (ie, without physical stress). The results are shown in Figure 6 as regression analysis and Bland-Altman error analysis. For simple interpretation, all detected heartbeats were scaled as beats per minute (bpm). The correlation coefficient between detected beat pairs in the range from 48.4 to 116.4 bpm was 0.9987; the sum of the squared errors for all compared heartbeats (n = 16,813) was 1.2 bpm.18
A particular advantage of the PPGI systems is that several areas of the skin can be selected in the same measurement scenario to analyze and visualize the spatial resolution in perfusion dynamics. Figure 7 shows a typical recording of 3 adjacent PPGI signals obtained from a hand with a small, fresh wound on the middle finger: there is a significant difference in the perfusion patterns from healthy skin and from the wound. When looking only at the heartbeat-related signal amplitude, it is slightly increased inside the wound. However, the high autonomous slow perfusion rhythms of about 0.1 Hz19,20 are strongly reduced inside the wound. Comparison of the 3 regions not only allows differentiation between the wound and healthy skin, but also shows that the low perfusion patterns have strong local variations. Even in the healthy case, 2 different ROIs (some millimeters apart) show different phases and amplitudes in the 0.1-Hz band.
Further analysis of the perfusion patterns with the classical fast Fourier transform yields very little new information. Even though it is possible to recognize differences at low frequencies, the resolution is limited. The frequency spectrum cannot reveal additional information because the fast Fourier transform is not suitable for analysis of transient perfusion signals.
Instead, additional insight into the spatiotemporal distribution of perfusion patterns is obtained when applying the Wavelet transform (Figure 7, lower panel). Here, the evolution of different frequency components (x-axis) can be recognized versus time (y-axis). The generated advanced PPGI signal visualization reveals (for the first time) that the slow rhythms in the skin perfusion are not stationary but fluctuate in amplitude and also (slightly) in frequency.
CONTACTLESS ARTERIAL PULSE OXIMETRY
Using multiwavelength illumination and the PPGI setup, spatially resolved arterial skin oxygen saturation (SpO2) can be monitored by applying the classical PPG pulse oximetry method, first presented by Aoyagi21 in 1972. To prove the feasibility of this contactless sensing concept, preliminary validation of the PPGI setup was performed with 13 volunteers. For this, their hands and lower arms were positioned on an armrest at a distance of 20 to 30 cm in front of the camera. To ensure valid SpO2 evaluation, the signal quality of the PPG time series extracted from the PPGI video sequence must be adequate. In particular, this means that the arterial blood volume pulse signal portion must be clearly separable from the remaining signal on both working wavelengths, because the downstream algorithms are based on estimation of the AC/DC and R values.
For illumination, a combination of red (630 nm) and infrared (905 nm) light was used in time-multiplex mode. For each subject, a video sequence of about 30-second duration was performed.22–24 In this study, a minimum of 20 frames per second and at least 600 PPG images were analyzed for each sequence. For reference, oxygenation was monitored by means of a commercial finger clip pulse oximeter (Choice Medical Systems, South Pasadena, FL). For all volunteers and all measurements, the gold standard SpO2 value was ≥95% (mean 98%). For further analysis, PPG signals were extracted from each small sliding ROI (a square with 6 pixels per side length) from the whole image frame. On the basis of the classical skin-attached pulse oximetry procedure, the related R value was calculated in a functional representation with spatial resolution.
Figure 8 shows a representative example of the calculated SpO2 mapping (automatic operation mode). As expected, for physiological skin perfusion, a homogeneous coloring of practically all skin areas is visible, representing the same level of arterial blood oxygen saturation (98% at 78 bpm). Exceptions to this are only on high-contrast edges in the PPGI image (eg, at the edge of the hand), the background, and on the mirroring surface of the reference pulse oximeter. Figure 8 (right) shows the results of the 13 volunteers in our study (mean R = 0.514 ± 0.05 corresponding with mean SpO2 = 97.85% ± 1.14%). In addition to these results, the feasibility of contactless SpO2 analysis has also been demonstrated by others.25,26 However, all groups confirmed that the accuracy of the estimated oxygenation level is strongly dependent on the accuracy of the measured amplitude of the PPG heartbeat-related signal.
First Step Toward Contactless Assessment of Venous Oxygen Saturation and Local Oxygen Consumption
Contrary to the advances in pulse oximetry, the monitoring of venous oxygen saturation measurement continues to be conducted invasively (invasively measured venous oxygen saturation [SvO2]). Of the available methods, the invasive in vitro measurement of the extracted venous blood is the gold standard in venous oxygen saturation measurement. The in vivo monitoring of blood gas through an intravascular catheter (eg, in pulmonary artery) is another technique that is clinically accepted.27,28 A noninvasive PPG-based solution is also available, using a small finger cuff.29 With this cuff, external pressure is applied to modulate venous flow in the vascular segment downstream of the cuff. From the pulsatile venous blood flow produced by this occlusion maneuver, venous oxygen saturation can be calculated. A major drawback of this method is that the cuff allows only very small changes in blood volume in venous blood flow, which limits the ability to detect venous oxygen saturation. Our proposed method to measure venous oxygen saturation and arterial oxygen saturation uses the venous muscle pump test (VMPT).30 During this test, the patient is advised to perform dorsal extensions of the ankle in sitting position.6,8 After this active exercise, venous blood is pumped from the periphery (in this case, the leg) back to the heart by contraction of different muscle groups. Subsequently, the “venous refilling time” and “muscle pump activity” are measured. Using the standardized leg exercise (muscle pump) performed during a VMPT, apart from determining venous refilling time and muscle pump activity, arterial and venous oxygen saturation levels at the test site can also be determined.15 An advantage of this method is that it is possible to calculate oxygen intake
and, hence, a measure of metabolic activity of the tissue surrounding the test site. However, a combined measure of arterial and venous saturation and, thus, determination of oxygen consumption (providing insight into metabolism) is difficult. Therefore, the presented method is considered pioneering. Here, the term “oxygen saturation” means the ratio of oxygen present in the blood to the maximum oxygen-carrying capacity of the blood. Based on our preliminary results, under physiological conditions, arterial oxygen saturation is approximately 98% and peripheral venous saturation is significantly lower (around 75%). Invasively measured central venous saturation levels range from 53% to 80%,31,32 whereas noninvasive measurement of venous oxygen saturation (based on analysis of the physiological pulsatility of the complex PPG signal) clusters around 80%.33
Local disturbances in the human oxygen consumption chain are very important because, without oxygen, the cells cannot assimilate nutrients and cannot survive. Thus, without cells burning oxygen, there is no energy infused into the cells and, thus, no healthy metabolism. Usually, by puncturing a central artery and vein and performing invasive blood gas analysis on the samples collected from vein and artery, both arterial saturation and global (central) venous saturation can be measured.
Noninvasive arterial pulse oximetry evaluates local arterial oxygen saturation, using the heart-synchronous pulsation in arterial blood, by analyzing at least dual-wavelength PPG signals from that local site. However, a pulse oximeter does not provide information on local hypoxia in terms of oxygen consumption because the method of estimating oxygen saturation with a pulse oximeter cannot be used to detect venous saturation; therefore, local oxygen consumption (difference between arterial/venous oxygen saturation) cannot be determined with a pulse oximeter. The difference between arterial and venous oxygen saturation determines the metabolism “on site”; diagnostically, this is very important, especially in patients with peripheral vascular diseases, diabetes mellitus, and/or in patients with wounds that are difficult to heal (eg, ulcers).
In this case, arterial oxygen saturation is determined from arterial pulsation; in addition, by applying active leg exercise (the VMPT), local venous oxygen saturation can be assessed from the resulting venous pulsation. This is the first time that local oxygen differences between arterial and venous saturation (oxygen consumption) can be measured. After autocalibration of the PPG measurement system, the patient performs a standardized leg exercise (15 dorsal extensions in 2-second intervals). In this time sequence, 2 recorded perfusion signals are available, one at λ1 (preferably 940 nm) and another at λ2 (preferably 660 nm); the combined arterial and venous blood volume variations in the measurement area at the extremity are recorded. These signals are forwarded to 2 digital filter groups with different spectral signal characteristics. One of the filters (or filter groups) calculates DC and AC components of the arterial signal component, the other filter then calculates DC and AC signal components of the venous signal component (Figure 9). The resulting signal components at the output of both filter groups are added to further signal processing to calculate the saturation values. Depending on the blood oxygenation level and the wavelengths used, R values (the ratio of the ratios of AC to DC of the 2 wavelengths) for arterial and venous oxygen calculation can be evaluated from the PPG transient signal (Figure 9).
To verify the introduced measurement, a commercial pulse oximeter (transmissive photoplethysmography [tPPC] mode) was used and attached to the second toe (volunteer in sitting position). The arterial SpO2 obtained was 98%, whereas the HR was 84 bpm (Figure 10, left). Then, blocking the arterial inflow to the foot using a wide tourniquet, the VMPT was performed as described earlier. Because the arterial pulsation was not available, even the commercial pulse oximeter correctly identified the venous blood volume pulse generated by the exercise. The venous saturation of 82% and the venous pulse rate (resulting from 15 active leg movements; 1 every 2 seconds) of 29 bpm was detected (Figure 10, right). By calculating the difference between arterial and venous oxygen saturation, our method can also quantify local oxygen uptake (peripheral arteriovenous oxygen consumption/oxygen metabolism) by tissue surrounding the site where the sensor is placed.
On the basis of 20 consecutive measurements in a 67-year-old subject (2 measurements per day over 10 days), local mean arteriovenous oxygen consumption of 16.8% ± 1.83% was determined; this value is smaller than expected. Although the final setting for the experimental system (currently planned with an Austrian company) and clinical testing of the new PPG approach is not yet finalized, our preliminary results indicate that the expected results provide clinically relevant differences in saturation between local arterial and venous blood supply.
New PPG Horizons for Detecting Pain and Stress
Pain is difficult to classify as subjective feelings may influence a patient’s well-being. Depending on the patient’s level of consciousness, pain assessment is also often a stress assessment. Unfortunately, a reliable objective measurement of pain/stress intensity is not currently available because these phenomena are individual sensations leading to large individual variability. Awake patients are usually asked to estimate their pain intensity on a given scale; however, this approach is not objective and is not feasible for uncooperative patients.
A key task of anesthesiologists during surgical interventions is to maintain adequate depth of anesthesia. The dosage of drugs used to suppress the patient’s consciousness and sensation of pain must be adapted to the surgical procedure and to each individual patient. Depending on the depth of anesthesia, even anesthetized patients can react to pain, for example, an increase in HR or blood pressure. Strong reactions to intraoperative pain must be prevented because they can cause postoperative stress and complications. In the recovery room and during postoperative care, pain therapy is also important. Optimized pain therapy improves patients’ recovery from surgical interventions and increases their well-being.15,34
Several methods are being investigated to quantify the depth of analgesia. Currently, during surgical interventions and/or in the recovery room, algorithms for pain assessment are used, such as the Analgesia Nociception Index (ANI) or the Surgical Stress Index, that analyze HR (and its amplitude and HRV) based on the ECG or PPG. In 2014, Koeny et al35 were the first to propose the use of PPGI data for calculation of the ANI.
However, different pulse wave-based indexes derived from PPG/PPGI recordings (eg, the Oliva-Roztocil Index36–40) also seem to be sensitive to pain. PPG or PPGI-based ANI seems to improve the assessment of analgesia during surgical interventions and postoperative care when the ECG signal is not available or valid, for example, during electrosurgical interventions or during contactless monitoring. Furthermore, in the future, both indices (ECG-ANI and PPG-ANI) might be combined in a new index to improve the robustness of ANI, if both sources (ie, ECG and PPG) are available.
Infrared Thermography Imaging
Similar to PPGI, infrared thermography imaging (IRTI) is a contactless but passive imaging technique for spatial and temporal registration of an object’s surface temperature and time-dependent thermal body signs in selected areas, for example endogenous thermoregulation, or respiration.
In 1800, the English astronomer W. Herschel discovered infrared radiation. In 1935, M. Melloni built the first thermopile infrared detector, and in 1965, the world’s first infrared scanning camera system was constructed (ThermoVision from Agema, Agema Infrared Systems, Stockholm, Sweden), it weighed 60 kg. In the 1980s, focal plane array infrared cameras were developed. This camera system comprises cooled detector systems with a resolution of 10,000 pixels, at that time weighing 10 kg, and was as expensive as a family house. Using the latest technological provisional status, a 32-g thermal imager for the smartphone was launched in 2014 under the motto “plugging instead dragging.”
In addition to military applications, infrared astronomy was largely responsible for the first developments of thermography. The medical use of infrared thermography (IRT) began around 1950 in Germany, while in 1928, the first infrared image of a human subject was documented by M. Czerny in Frankfurt. In 1952, the physician E. Schwamm, together with the physicist J. Reeh, introduced a single detector infrared bolometer for sequential thermal measurement of defined regions of the human body surface for diagnostic purposes in 1952. Subsequently, in 1980, improved IRT technology became available, mostly based on nitrogen-cooled HgCdTe sensor structures, which since then have been used for medical studies worldwide. Today, the IRT technology has >30 years of successful clinical applications and is established as an effective and useful diagnostic tool in many medical disciplines.41,42
To understand the basic principle of IRTI, we need to recall that any physical object with a temperature above the absolute zero point (0K) emits radiation. Physically, the specific spectral radiance M(λ, T) of ideal black bodies is described by Planck law43:
In this equation, T is the black body’s temperature; h, the Planck constant; c, the speed of light in vacuum; λ, the wavelength, and k, the Boltzmann constant. The maximal emitted intensity for a fixed size wavelength interval is strongly dependent on the body’s temperature. In fact, according to the Stefan-Boltzmann law, the complete emitted radiation power, P, in the whole spectrum is proportional to the body’s temperature to the power of 4:
where, σ is the Stefan-Boltzmann constant and A is the object’s surface. A shift of the maximum to longer wavelengths for lower temperatures can also be observed. This effect is described by Wien displacement law:
where, b stands for the Wien’s displacement constant.
For a typical skin temperature (approximately 35°C), this maximum shifts to approximately 9.5 μm lying in the infrared C-band. To draw conclusions about an object’s temperature by measuring its passive radiant emittance, the sensing device should be most sensitive in this wavelength region.
Therefore, at MedIT, we have been using the VarioCam HD 820S head (InfraTec GmbH, Dresden, Germany), which has an uncooled microbolometric array with a spatial resolution of 1024 × 768 pixels, thermal sensitivity better than 0.03 K at 30°C, and 16-bit signal resolution.
In addition, thermal video sequences can be recorded with a rate from 30 fps (by full resolution) up to 240 fps (by 1024 × 96 pixel resolution); measurement data may be transferred via Ethernet to a PC. The recorded signal intensity correlates with the incident radiation weighted with the sensor’s sensitivity characteristic. With respect to internal calibration of the camera with regard to damping of the object lens, sensor and camera temperature, and duration of exposure, this intensity can be correlated with the black body temperature. The distance of the observed object relative to the camera (50–200 cm in our experiments) does not affect the detected temperature because the radiation density of a fixed-sized object decreases with the square of the distance; but, at the same time, the size of an object observable with a single sensor pixel increases with the square of the distance.
Unfortunately, at the moment, high-resolution IRT cameras are still too expensive for widespread clinical diagnostics. Also, because the calibrations of measurements are case-specific, standardized protocols are not yet available. However, ThermoVision system manufacturers are working on the removal of these particular drawbacks.
The following sections present examples of possible uses of IRTI for diagnostic purposes.
Remote Detection of Breathing Patterns Using IRTI
RR in line with HR is an essential vital sign. An abnormal respiration dynamic is one of the earliest and strongest markers of physiological instability and disorder. However, RR is considered one of the most frequently undocumented variables,44 because state-of-the-art measuring modalities require contact with the patient’s body, often causing both discomfort and stress (eg, in neonatal care). This emphasizes the need for unobtrusive and feasible monitoring alternatives for routine medical care.
For robust detection of breathing patterns from an IRTI video sequence, it is necessary to identify the ROI (mostly the nose) in the first frame of the sequence. Thereafter, it must be tracked to compensate the motion of the subject and, finally, the obtained breathing signal (related to the mean temperature signature in the selected ROI) must be postprocessed and analyzed.
To track the nose, an algorithm proposed by Mei et al45 can be used. This is based on sparse representation, where the L1-regularized least-squares problem is used to achieve sparsity. First, for each frame, the mean signal value of the tracked ROI is calculated as
where x and y are the coordinates in the image plane. To estimate the instantaneous frequencies of breathing dynamics, the approach proposed by Brüser et al46 was adopted. It consists of an adaptive analysis window w k[v] interactively sliding across the breathing signal. For each window location n k, the interval between 2 breaths is computed by using the estimator autocorrelation given by:
As an example, Figure 11 shows the performance of our remote breathing measuring strategy in 1 subject. From our current database (3-phase respiration study on 11 healthy subjects, the total monitored respiration duration was 2.94 hours and the mean correlation coefficient between RRref and RRIRT pairs was 0.959747), excellent agreement was found between the estimated and the “gold standard” RR value (piezoplethysmographically measured thoracic effort).
Analysis of Facial Infrared-Thermographic Sequences for Continuous Monitoring of Anesthetized Patients
“The face is a picture of the mind with the eyes as its interpreter” (Cicero, 106–43 BC). A close study of the face and facial expressions can be, among other things, used to draw conclusions about emotions and, especially, the sensation of pain. Thus, characteristic facial features have been identified and classified into several scientific works. In the 1980s, the Facial Action Coding System of various action units was used to quantify the level of facial (mimic) activity. With respect to the expression level of stress/pain, systems for the automated analysis of facial expressions based on such an evaluation concept have been developed. Face detection algorithms based on physiological information have increasing reliability and performance.48 However, conventional camera images require relatively good and constant lighting conditions for reliable facial recognition or segmentation. In most cases, these can only be realized through active illumination. The IRT camera technique offers experimental advantages as it does not need external illumination. Moreover, the eyes are usually intraoperatively taped with plaster strips to protect against dryness, which is a further challenge in the image analysis.
The aim of our current project is to develop an objective technical process that supports clinical evaluation of the patient by the anesthesiologist during anesthesia, to optimize patient safety and quality of medical care. The contactless method of IRTI can be used for long-term monitoring of contours, and the thermal profile of the face of anesthetized patients, to evaluate possible responses to pain stimuli (changes in facial expression, lacrimation, and perspiration). For comparison of the spatially distributed temperature profiles, several symmetrically mirrored face ROIs have been defined manually; here, collected signal intensities are summed and normalized according to the normal state (no pain). In our case studies, it is noteworthy that pain leads to a decrease in the average temperature by almost 20% in the paranasal region, while it eventually increases by about 8% periorally.
In a first step for automatic evaluation of the pain/stress level, we follow the simplified approach depicted in Figure 12: IRTI video sequences are acquired and then prepared to be used for classification: face images are segmented from the background. Then, facial regions are identified and temporal features can be extracted from them for the classification stage. Finally, the classification process is planned to determine an index for the current pain level.
Although the patient’s face can easily be spotted as the hottest region in the image, facial regions have to be identified by their morphology and temperature pattern; the position of the eyes is determined by the inner canthi, which show as warm spots in thermal imagery because of the blood transported by the ophthalmic arteriovenous complex. After detecting the eyes, the nose can be identified by using geometrical constraints between nose and eyes. Similarly, the positions of mouth, cheeks, and forehead can be obtained. Traditionally, changes in facial expression are analyzed but, with IRTI, of more interest is, what is invisible to the human eye: local changes in temperature. Mean temperature changes are evaluated over time in the obtained regions, to identify which regions qualify for stress detection; such regions will show measurable changes.
For example, we identified that the segmented region of the nose (Figure 12, right) cools over time during situations with induced stress. Remarkably, this region does not contain the nostrils where breathing patterns are normally extracted from (eg, in Ref. ). We aim to identify more of such regions, as well as establish the related physiological causes.
For temporal analysis, normalization is necessary to gain consistent data, for example the ROI needs to be of constant size over time or, alternatively, properties of the region are normalized to the amount of pixels belonging to the object. Otherwise, the extracted 1-dimensional time series (temperature sequence) will have unwanted signal components not related to changes within the region but to, for example, movement. For our analysis, we use both approaches.
Other temporal changes in the face can also be used to help us identify the level of stress. For example, lacrimation, eye blinking, and the state of openness of the mouth, all of which might change in the presence of pain. To evaluate lacrimation and eye blinking, we concentrate on the 9 × 9 regions around the eyes, whereas for openness of the mouth, we concentrate on the rectangular region around the mouth where we expect temperature changes.
Therefore, a combination of both the morphological information and the thermographic signature seems to be a promising approach for identification of stress and/or pain.48,49 Also, in postoperative clinical care, pain control is an important diagnostic item.
Hybrid PPGI/IRTI Solutions and Future Applications
As described previously, PPGI as an active optical imaging method is the preferred choice for remote detection of HR and spatially distributed lower dermal blood perfusion patterns, and passive IRTI is preferred for remote detection of RR and spatially distributed thermal signatures. Therefore, PPGI and IRTI complement each other by covering different vital signs. Thus, a combination of both systems would be advantageous and is currently on our agenda.43,50,51 To create such hybrid imaging, both cameras must be synchronized for data recording. In addition, their positioning and the reduction of movement artifacts are important, and exactly the same body size must be selected for both recordings. Examples of the potential of our combined PPGI/IRTI camera modality are shown in Figures 13 to 15.
Figure 13 shows selected results from an animal trial conducted in RWTH Aachen University hospital (Germany). Physiological and pathological skin perfusion phenomena (like sepsis or centralization) can be analyzed, while both local temperature signature and dermal blood perfusion patterns are available.50 Other applications can be found in the contactless long-term monitoring of neonates, in the prediction and quantification of wounds (eg, decubitus monitoring in geriatric patients), or in early detection of inflammation/sepsis.
Figure 14 clearly quantifies mechanical overloading in the right wrist/forearm of a 69-year-old man after a fall during gardening. The patient reported significant pain; a swollen right hand is visible (Figure 14, top left image). The radiograph (bottom left) showed no fracture of structures. Even with PPGI, no local inhomogeneity of skin perfusion is apparent in the injured area. However, IRTI shows a significant inflammatory process inside the joint, in the center of which the temperature increases by ≥2°C.
However, inflammatory reactions do not only arise because of external tissue lesions or irritations. This is demonstrated by the example in Figure 15: this is an endogenous-initiated thrombosis in a superficial vein (varicophlebitis) in the lower leg of a patient with double-sided problems of varicose veins. When getting up, he noticed an unusual pain on the inside of the lower leg below the knee (Figure 15, top left). Visually no changes on the leg were noticeable and no hardening of the affected vein segment was palpable. An IRTI sequence recorded on the same day showed an area at this position with considerable but inhomogeneously increased local temperature of ≥2.5°C. In the following days, the pain slowly decreased. Three days later, again when getting up in the morning, the patient noticed a change in skin color at the same position (Figure 15, top right). An immediate phlebological examination confirmed an approximately 5-cm-long thrombus inside a segment of a side branch of the great saphenous vein (see duplex ultrasound image, bottom right). A VMPT contactlessly recorded by PPGI (Figure 15, bottom left) showed a shortening of the venous refilling time down to 18 seconds (the accepted standard value is minimally 25 seconds6). The pumping power of the extremity (increased PPG signal during the leg maneuver) was still present. Therefore, we conclude that the IRTI provides valuable indications for early phlebologic/angiologic diagnosis. Initially, inflammation does not represent a disease itself. However, in this case, the IRT recording clearly visualizes the onset of inflammation and signals that the body has started a defense reaction and initiated the healing process. Therefore, the recording can be used to initiate further diagnostic and/or therapeutic actions.
Evidence-based medical diagnostics is based on both structural and functional information. PPGI and IRTI can provide both; they are contactless camera-based measurement methods for monitoring a wide range of vital variables with spatial resolution. In particular, PPGI enhances the classical contact-based PPG. Approved evaluation algorithms of the PPG method can easily be adapted for detection of HR, HRV, and vasomotional activity with PPGI. Use of the multiwavelength VMPT also seems possible to allow remote assessment of both arterial and venous blood oxygen saturation levels in the extremities with PPGI.
Although the IRTI method primarily records temperature distribution of the observed object, information on RR and respiratory variability can also be derived by analyzing the course of temperature distribution, for example, in the nasal region. The main advantages of both methods are unobtrusive data acquisition and the possibility to assess spatial assignment between vital parameters and body region. In addition, both methods can extract individual features for recognition of facial expressions, for example, during anesthesia.
IRTI has an “unobtrusive” advantage because it can also be a passive method used in total darkness. Our preliminary results show that the advanced tracking algorithms work at least as well in the IRTI as in the black/white PPGI image. However, the remote measuring principle can lead to disturbing artifacts when the measured object is moving rapidly relative to the camera. Therefore, sophisticated approximation methods are required to detect and minimize existing movement artifacts in the PPGI and IRT raw data in the hybrid imager.
These methods enable long-term monitoring and the monitoring of effects with special local characteristics. Because the 2 systems supplement one another, we proposed a combined hybrid application and have discussed its feasibility for analysis of facial expression, detection of pain and stress, and functional description and containment of local inflammatory processes.
Name: Vladimir Blazek, Dr.-Ing. Dr. h.c.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Conflicts: Vladimir Blazek reported no conflicts of interest.
Name: Nikolai Blanik, Dipl.-Ing.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Conflicts: Nikolai Blanik reported no conflicts of interest.
Name: Claudia R. Blazek, MD.
Contribution: This author helped write the manuscript.
Conflicts: Claudia R. Blazek reported no conflicts of interest.
Name: Michael Paul, MSc.
Contribution: This author helped write the manuscript.
Conflicts: Michael Paul is a PhD student currently working on a research grand funded by Philips Chair for Medical Information Technology and is loosely related to the content of this article.
Name: Carina Pereira, MSc.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Conflicts: Carina Pereira reported no conflicts of interest.
Name: Marcus Koeny, Dr.-Ing.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Conflicts: Marcus Koeny reported no conflicts of interest.
Name: Boudewijn Venema, Dr.-Ing.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Conflicts: Boudewijn Venema reported no conflicts of interest.
Name: Steffen Leonhardt, Dr.-Ing. MD.
Contribution: This author helped write the manuscript.
Conflicts: Steffen Leonhardt reported no conflicts of interest.
This manuscript was handled by: Maximes Cannesson, MD, PhD.
The authors gratefully acknowledge financial support from the German Federal Ministry of Science and Education and the German Federal Ministry of Economic Affairs and Energy. The authors also thank Dr. med. Ursula Schmeink, Gefäßzentrum Aachen, Germany, for providing the ultrasonic duplex image (Figure 15; bottom right). Michael Paul acknowledges Philips Chair for Medical Information Technology for supporting his PhD work. Carina Pereira acknowledges the Foundation for Science and Technology (FCT) in Portugal for her PhD grant SFRH/BD/84357/2012.
1. Task Force of the European Society of Cardiology and The North American Society of Pacing and Electrophysiology. Heart rate variability—Standards of measurements, physiological interpretation, and clinical use. Eur Heart J. 1996;17:354–381.
2. Hamilton MA, Cecconi M, Rhodes A. A systematic review and meta-analysis on the use of preemptive hemodynamic intervention to improve postoperative outcomes in moderate and high-risk surgical patients. Anesth Analg. 2011;112:1392–1402.
3. Mathes K. Untersuchungen über die Sauerstoffsättigung des menschlichen Arterienblutes. Arch Exper Pathol Pharmakol. 1935;179:698–711.
4. Hertzman AB. The blood supply of various skin areas as estimated by the photoelectric plethysmograph. Am J Physiol. 1938;124:329–340.
5. Barnes RW. Noninvasive diagnostic assessment of peripheral vascular disease. Circulation. 1991;83:I20–I27.
6. Blazek V, Schultz-Ehrenburg U.Quantitative photoplethysmography. Basic facts and Examination Tests for Evaluating Peripheral Vascular Functions. 1996.Düsseldorf, Germany: VDI Verlag.
7. Belcaro G, Veller M, Nicolaides AN, et al Noninvasive investigations in vascular disease. St Mary’s Fellows. ISVI (Italian Society for Vascular Investigations). Angiology. 1998;49:673–706.
8. Schultz-Ehrenburg U, Blazek V. Value of quantitative photoplethysmography for functional vascular diagnostics. Current status and prospects. Skin Pharmacol Appl Skin Physiol. 2001;14:316–323.
9. Allen J. Photoplethysmography and its application in clinical physiological measurement. Physiol Meas. 2007;28:R1–R39.
10. Venema B, Blanik N, Blazek V, Gehring H, Opp A, Leonhardt S. Advances in reflective oxygen saturation monitoring with a novel in-ear sensor system: results of a human hypoxia study. IEEE Trans Biomed Eng. 2012;59:2003–2010.
11. Wu T, Blazek V, Schmitt HJ. Photoplethysmography imaging: a new non-invasive and non-contact method for mapping of the dermal perfusion changes. Proc SPIE. 2000;4163:62–70.
12. Hülsbusch M, Blazek V. Contactless mapping of rhythmical phenomena in tissue perfusion using PPGI. Proc SPIE. 2002;4683:110–117.
13. Hülsbusch M.Ein bildgestütztes, funktionelles Verfahren zur optoelektronischen Erfassung der Hautperfusion [PhD thesis]. 2008.Aachen, Germany: RWTH Aachen University.
14. Goebrecht Hed. Lehrbuch der Experimentalphysik, Band III Optik. 1974.Berlin: Walter de Gruyter.
15. Blazek V, Kumar VJ, Leonhardt S, Rao MM. Studies in Skin Perfusion Dynamics–Photoplethysmography and Its Applications in Medical Diagnostics. New Delhi: Springer (India).
16. Maeda Y, Sekine M, Tamura T, Moriya A, Suzuki T, Kameyama K. Comparison of reflected green light and infrared photoplethysmography. Conf Proc IEEE Eng Med Biol Soc. 2008;2008:2270–2272.
17. Lee J, Matsumura K, Yamakoshi K, Rolfe P, Tanaka S, Yamakoshi T. Comparison between red, green and blue light reflection photoplethysmography for heart rate monitoring during motion. Conf Proc IEEE Eng Med Biol Soc. 2013;2013:1724–1727.
18. Venema B, Blazek V, Leonhardt S. In-ear photoplethysmography for mobile cardiorespiratory monitoring and alarming. Proc BSN IEEE Conf. 2015;2015:1–5.
19. Schmid-Schönbein H, Ziege S, Grebe R, Blazek V, Spielmann R, Linzenich F. Synergetic interpretation of patterned vasomotor activity in microvascular perfusion: discrete effects of myogenic and neurogenic vasoconstriction as well as arterial and venous pressure fluctuations. Int J Microcirc Clin Exp. 1997;17:346–359.
20. Perlitz V, Cotuk B, Blazek V, Krautstrunk G, Ziege S, Schmid-Schönbein H. Blazek V, Schultz-Ehrenburg U. A self-organised rhythm in the autonomous nervous system: a preliminary interpretation of the ca. 0.15 Hz-Band activity prevailing in neuronal centres and peripheral effectors. Computer-aided Noninvasive Vascular Diagnostics. 2005:Düsseldorf, Germany: VDI Verlag; 45–56.
21. Aoyagi T. Discovery of pulse oximetry. Anesth Analg. 2007;105(6 suppl):1–4.
22. Blazek V, Hülsbusch M, Herzog M. Berührungslose, ortsaufgelöste Erfassung der dermalen Sauerstoffsättigung–Konzept eines bildgebenden Pulsoxymeters. Biomed Tech. 2007;52 (Erg.-B.):1–2.
23. Blanik N.Konzept und Realisierung eines kontaktlosen Messsystems für die ortsaufgelöste Erfassung der Sauerstoffsättigung der Haut [diploma thesis]. 2010.Aachen, Germany: RWTH Aachen University.
24. Blanik N, Venema B, Blazek V, Leonhardt S. Remote pulse oximetry imaging–fundamentals and applications. Clinician Technol. 2014;44:5–11.
25. Wieringa FP, Mastik F, van der Steen AF. Contactless multiple wavelength photoplethysmographic imaging: a first step toward “SpO2 camera” technology. Ann Biomed Eng. 2005;33:1034–1041.
26. Zheng J, Hu S. The preliminary investigation of imaging photoplethysmographic system. J Phys Conf Ser. 2007;85:012031.
27. Fahey PJ, Harris K, Vanderwarf C. Clinical experience with continuous monitoring of mixed venous oxygen saturation in respiratory failure. Chest. 1984;86:748–752.
28. Rivers EP, Ander DS, Powell D. Central venous oxygen saturation monitoring in the critically ill patient. Curr Opin Crit Care. 2001;7:204–211.
29. Chan FCD, Hayes MJ, Smith PR. Venous pulse oximetry. US Patent 2008.7 263 395 B2.
30. Blazek CR, Blazek V. Bewegungskorreliertes Verfahren und opto-elektronische Vorrichtung zur nichtinvasiven Bestimmung der dermalvenösen Sauerstoffversorgung peripherer Beingebiete. Europäische Patentanmeldung EP 2609854 A1. 2012.
31. Keys A. The oxygen saturation of the venous blood in normal human subjects. Am J Physiol. 1938;124:13–21.
32. Reinhart K, Bloos F. The value of venous oximetry. Curr Opin Crit Care. 2005;11:259–263.
33. Walton ZD, Kyriacou PA, Silverman DG, Shelley KH. Measuring venous oxygenation using the photoplethysmograph waveform. J Clin Monit Comput. 2010;24:295–303.
34. Koeny M. Blazek V, Kumar VJ, Leonhardt S, Rao MM. Analyzing pain and stress from PPG perfusion patterns. Studies in Skin Perfusion Dynamics–Photoplethysmography and Its Applications in Medical Diagnostics. 2016.New Delhi: Springer (India).
35. Koeny M, Blanik N, Yu X, Czaplik M, Walter M, Rossaint R, Blazek V, Leonhardt S. Using photoplethysmography imaging for objective contactless pain assessment. Acta Polytechnica. 2014;54:275–280.
36. Oliva I, Roztoĉil K. Toe pulse wave analysis in obliterating atherosclerosis. Angiology. 1983;34:610–619.
37. Navratil K, Halek J, Havranek P, Binder S. Pulse wave analysis in objective evaluation of pain–a preliminary communication. Cesk Slov Neurol N. 2008;71:303–308.
38. Korpas D, Hálek J, Dolezal L. Parameters describing the pulse wave. Physiol Res. 2009;58:473–479.
39. Venema B. Photonische Sensorkonzepte für ein mobiles Gesundheitsmonitoring. [PhD thesis]. 2015.Aachen, Germany: Shaker Verlag.
40. Borik S.Elektrické modelovanie a meranie vybraných fyziologických funkcií2014.Zilina, Slovakia: University Press.
41. Berz R, Sauer H. The medical use of infrared-thermography. history and recent applications. Proc DGZfP IRT-2007, Stuttgart. 2007:1–12.
42. Abbas AK, Heimann, K, Blazek V, Orlikowsky T, Leonhardt S. Neonatal infrared thermography imaging: analysis of heat flux during different clinical scenarios. Infrared Phys Technol. 2012;55:538–548.
43. Blanik N, Abbas AK, Venema B, Blazek V, Leonhardt S. Hybrid optical imaging technology for long-term remote monitoring of skin perfusion and temperature behavior. J Biomed Opt. 2014;19:16012.
44. McGain F, Cretikos MA, Jones D, Van Dyk S, Buist MD, Opdam H, Pellegrino V, Robertson MS, Bellomo R. Documentation of clinical review and vital signs after major surgery. Med J Aust. 2008;189:380–383.
45. Mei X, Ling H. Robust visual tracking and vehicle classification via sparse representation. IEEE Trans Pattern Anal Mach Intell. 2011;33:2259–2272.
46. Brüser C, Winter S, Leonhardt S. Robust inter-beat interval estimation in cardiac vibration signals. Physiol Meas. 2013;34:123–138.
47. Pereira CB, Yu X, Czaplik M, Rossaint R, Blazek V, Leonhardt S. Remote monitoring of breathing dynamics using infrared thermography. Biomed Opt Express. 2015;6:4378–4394.
48. Paul M, Blazek V, Leonhardt S. An efficient method for facial components detection in thermal images. Proc SPIE. 2015;95340P:1–7.
49. Kopaczka M, Blanik N, Czaplik M, Hochhausen N, Paul M, Pereira CB, Leonhardt S, Blazek V, Merhof D. A thermal infrared face database and active appearance model based face detection in a system for pain assessment in sedated patients. Proc 11th German-Russian Conf Biomed Eng. 2015:33–36.
50. Blanik N, Paul M, Blazek V, Leonhardt S. Detection and analysis of temperature-sensitive dermal blood perfusion dynamics and distribution by a hybrid camera system. Conf Proc IEEE Eng Med Biol Soc. 2015:2383–2386.
© 2017 International Anesthesia Research Society
51. Leonhardt S. Blazek V, Kumar VJ, Leonhardt S, Rao MM. Concluding remarks and new horizons in skin perfusion studies. Studies in Skin Perfusion Dynamics–Photoplethysmography and Its Applications in Medical Diagnostics. 2016.New Delhi: Springer (India).