Haptic feedback systems are designed to sense tactile and kinesthetic information about an object and convey that information to a user. This type of feedback is becoming increasingly relevant for the development of advanced human-machine interfaces across a broad spectrum of fields, from virtual simulations to medical devices.1–3 The incorporation of haptic feedback is often credited with functionally improving user interaction with human interfaces and enhancing the sense of realism.
Prosthetic technology is a prime candidate for haptic feedback integration. Users of conventional myoelectric prostheses must operate the mechanism, relying solely on the visual feedback of their prosthetic hand. The myoelectric prosthetic hands do not have a mechanism to convey any sensory information, making it difficult for users to feel connected to their hand and to engage in active grasping and exploration tasks. This lack of closed-loop control is a common reason for amputees to choose body-powered prostheses with cable-driven mechanisms that convey some useful force feedback.4,5 Bringing a form of supplementary sensory feedback to myoelectric prostheses may provide amputees with a means to receive force information while leveraging the improved aesthetics and power of a myoelectric hand. This article presents a haptic feedback simulator to test a vibrotactile representation of force and presents work describing its functional use in the performance of a grasping task.
Previous researchers have investigated the use of direct force feedback for closed-loop prosthesis control. Meek et al.6 presented a proportional force feedback system that used a motor-driven pusher placed against the user’s skin. Patterson and Katz7 tested a pressure cuff system that demonstrated noticeable improvement over a vibrotactile system. Furthermore, they consistently found that the addition of supplemental haptic information channel could improve user control of their grasping force.
Other works have used sensory substitution of vibration for force to avoid the integration problems with large and cumbersome force feedback systems.8–11 The systems operated under the premise that a vibratory response that varied in intensity contiguously with a sensory stimulus such as grasping pressure could be a useful feedback channel and that training would improve the association between the disparate stimuli. Vibrotactile stimulation is a popular choice for a surrogate force feedback system because the motors are small and unobtrusive, allowing for easy integration. Market research indicates that users are highly sensitive to the aesthetic appearance of their prosthetic devices and would be unlikely to accept bulky attachments.12 Pylatiuk et al. have presented work showing how a small vibration motor can be placed between the skin and silicon liner of a prosthesis to provide force feedback.13 Furthermore, numerous studies have explored the psychophysics of pure-tone vibration14–17 as well as complex vibratory signals.18 These studies report positive findings in both functionality improvements and user interest and suggest further work is desirable.
Even with the wide variety of surrogate force feedback schemes available, these systems are normally not integrated into commercial prosthetic systems. Part of the reason is that prosthesis users do not have the opportunity to experience what a haptic feedback system would feel like during their fitting and training sessions. Furthermore, there are still questions over the use of surrogate force feedback and whether the technical challenge of integrating these systems is warranted. It is well understood how a haptic feedback channel can allow users to sense force without visual attention.6,13,19 However, it is less clear whether the addition of a surrogate force feedback system to a myoelectric prosthesis would lead to functional improvements in control of grasp force. This issue is relevant for prosthesis users who must grasp objects with varying weights and compliances without damaging or dropping them.
To help address these needs, a haptic feedback simulator is developed that can train subjects to control their grasping force through an interactive force matching task. The simulator is capable of providing users with a visual indicator as well as a modulated vibrotactile stimulus to feedback grasping force. In the described study, both feedback methods are tested using a task that can quantify performance improvements at a series of force levels. This type of test serves as a framework that prosthetists can use to train their patients to interpret a sensory feedback system and evaluate their performance under various conditions.
A haptic feedback simulator was used to train subjects in a vibrotactile stimulation protocol and test their ability to perform force-matching tasks. A total of eight able-bodied subjects participated in the study. These subjects used a harness that allowed them to wear a prosthetic hand over their limb, simulating the conditions an actual amputee would face. The subjects activated the open and close functions of the hand using electromyogram (EMG) signal control that was built into the harness. The prosthetic hand itself was fitted with strain gauges that measured the grasping force and relayed that data to a computer for recording.
The hardware setup used a customized myoelectric hand-shell designed for use by an able-bodied individual, shown in Figure 1. A customized socket was fabricated so that the subject could grip the inside with his anatomical hand. Empty space at the proximal end of the hand-shell provided room for flexing and extending muscles. The prosthetic hand was a modified Otto Bock Electric Right Hand, 7¾ inch size, a standard male hand size. To wear the prosthesis, the user slid his arm inside the shell and a strap was used to hold it snugly in place. The setup was fixed using a benchtop vice to ensure that the angle at which the object was grasped always remained constant throughout all trials.
A myoelectric circuit was used to control hand movement. Standard stainless steel electrodes were used to pick up motor signals from the skin surface. The electrodes were placed on top of wrist extensor (extensor carpi radialis longus) and wrist flexor (flexor carpiulnaris) muscle pair for all subjects. The force sensor was a 120 [Omega] strain gauge mounted at the carpal-metacarpal thumb joint where maximum bending moment is observed. The configuration chosen was a quarter bridge temperature compensated circuit. The strain gauge circuitry consisted of a differential amplifier with variable gain and offset potentiometers. The output was conditioned with a two-pole low-pass filter for noise reduction. This circuit ensured a robust signal output while providing the option for tuning signal offset and range via the potentiometers.
An information flow chart of the entire system is shown in Figure 2. The output from angle and force sensors was recorded by a National Instruments 12-bit data acquisition card. The vibrotactile feedback hardware consisted of a C2 tactor (Engineering Acoustics, Inc., Casselberry, FL) placed on the biceps. The tactor was placed at least 2 inch from EMG electrodes, a distance which was empirically determined to not interfere with any myoelectric signals. The feedback simulator was enabled on a central controller personal computer (PC) responsible for
- Recording all force and angle data from the myoelectric prosthesis
- Displaying force information to the subject through a visual bar indicator
- Controlling the modulation of the vibrotactile feedback
The control software was written in National Instruments LabView 8.0 and was designed to allow the study operator to toggle easily between multiple haptic feedback modes and to adjust thresholds such as the minimum force required to register contact with the object. During the execution of the experiment, key control parameters were hidden from the subject to avoid confusion. A screenshot of the visual display is shown in Figure 3.
The controller PC interfaced with the C2 tactor via a custom board with digital input/output and a dedicated PIC18f2520 microprocessor to output the vibrotactile waveform. The chip output was a pulse width-modulated square wave, which was fed to the tactor through a low-wattage amplifier circuit.
The vibrotactile feedback used a C2 tactor (Engineering Acoustics, Inc.) that was mounted to the upper arm with an elastic band. This device uses voice coil technology, allowing for the independent control of frequency and amplitude through a microprocessor. A series of discrete vibration pulses comprised the vibratory waveform, which was modulated by varying the pulse width and pulse rate. Shorter, more rapid pulses indicated an increase in perceived stimulus intensity, and longer, less rapid pulses indicated a decrease in perceived stimulus intensity (Figure 4). A 200-Hz carrier frequency was used, because frequencies in this range have been shown to strongly stimulate Pacinian skin mechanoreceptors.15 The subject was asked to reach preset force levels with the vibrotactile feedback. The force information measured by the strain gauge was related to the pulse rate delivered to the subject. Higher pulse rate correlated with higher grasping force.
A common issue with vibrotactile feedback is that the perceived strength of a vibration varies with factors such as contactor area and mounting pressure.20,21 Therefore, modulating the strength of the vibration through changes in amplitude or frequency may be an inconsistent feedback method. This study uses a stimulus that represents force using pulse rate, which is independent of any mounting effects. Factors such as the carrier frequency and the range of pulse rates are adjustable in the feedback simulator software, making it customizable for individual preferences.
The grasping task tested how well a subject could grasp an object with the prosthetic hand and keep the applied force as close as possible to a target force value. Each task used one of three selectable grasping force levels that were established at 40%, 60%, and 80% (low, medium, and high force levels, respectively) of maximum grasping force. The task began when the subject grasped the object (contact determined by force value crossing a “touch” threshold), upon which a cue for the desired force level was presented. The subject was directed to squeeze the object enough to reach the required force level and hold it there until given an automated release command (10 seconds after initial contact). Afterwards, the object was released and the system was reset for a new trial. The object used was a plush ball that exhibited some noticeable visual deformation. This object was chosen to investigate whether haptic feedback of force introduced any performance benefits in the presence of visual feedback of object deformation.
All subjects had an initial training period when they tested their EMG control of the prosthesis and learned the grasping task. During this time, the positions of the EMG electrodes were adjusted until the subjects were comfortable with their control of the prosthesis. Part of the training period included several cycles of fully grasping the ball and releasing it. This was used to determine level of the maximum grasping force and subsequently the three target force levels.
After initial setup and training, each subject completed two separate phases: a “vibrotactile” phase and another “no feedback” phase. The no feedback phase was designed to mirror the vibrotactile phase exactly, except the output to the vibrotactile motor was disabled. Each phase was broken into three distinct sets:
- Set I: Subject performs the task receiving only visual feedback of grasping force through a bar graph indicator (12 trials).
- Set II: Subject learns to map vibratory feedback with the visual representation of force. For this set, a visual indicator of grasping force is provided and vibrotactile feedback may be provided or it may not be if the phase is the no feedback phase (12 trials).
- Set III: Subject performs task receiving only vibrotactile feedback, or no feedback if the phase is the no feedback phase (12 trials).
Within each phase, the order of the sets is as is presented here (I to III) because subjects are more easily introduced to the task using a visual feedback system. However, the order of the vibrotactile and no feedback phases were varied across the subjects to avoid introducing a systematic learning bias in the measurements between the feedback and no feedback cases. Set III in the no feedback phase gives the user no additional sensory information, as it would be with a normal myoelectric prosthesis. Data from this set were used as controls to compare the effects of visual and vibrotactile feedback.
The output of the strain gauge was recorded for every trial and used to generate trajectory plots (a graph of grasping force vs. time). This type of analysis is useful in examining the quality of control at different force levels. Significant fluctuations in force are observed in some trajectories, most commonly when subjects were attempting a match at high force. In addition, force values were relatively stable for all trajectories by the last few seconds of the trial. This confirmed that subjects had enough time in the trial to match the target value and were generally not readjusting up to the last second.
For the trials using vibrotactile feedback only, the mean proportional error was calculated by taking the difference of the target force value and the mean force reading from the last second of the trial and normalizing it with respect to the target value. Nonparametric Mann-Whitney U tests were used to test for significant differences between the vibrotactile feedback and control cases. A comprehensive plot showing the means and standard errors is shown in Figure 5, separated by feedback mode and force level.
The comprehensive plot confirms the expected result of significantly lower errors during visual indicator trials than in the control case (p < 0.01 for all force levels). On average, visual indicator trials yielded an average error reduction of 65% from the control case. Furthermore, the mean error value for visual indicator trials was 0.045, and no significant difference was found across the three force levels using the nonparametric Kruskal-Wallis test (p = 0.39).
The vibrotactile feedback results were not statistically separable from the controls. It seems that the addition of vibrotactile feedback had very little effect on performance error in the low and medium force cases and may have even worsened performance in the high force cases. Although these results are not statistically significant, some useful information can be extracted by separating the study population by their prior experience. Four of the eight subjects tested had previous experience interpreting feedback from a C2 tactor with the type of pulse rate modulation used in this study. The remaining four subjects were completely naive users, experiencing this type of feedback for the first time. Figures 6 and 7 compare vibrotactile feedback errors with control errors for these experienced and novice subgroups, respectively.
Figure 6 shows that experienced users displayed a statistically significant decrease in error rate of 25% with the addition of vibrotactile feedback (p = 0.0293) in the medium force trials. In addition, the mean error of vibrotactile feedback trials was 0.101, which was statistically consistent across all force levels (p = 0.951).
A comparison between Figures 6 and 7 shows that novice users displayed higher mean errors and higher variance than the experienced subjects. None of the force levels yielded statistically significant results. In summary, experienced users were able to use the vibrotactile feedback to lower their error rate at the medium force level whereas novice users seemed to be negatively influenced by the feedback.
To explore the effect of multiple testing sessions on error rates, data from one subject are presented in Figure 8 showing average error rates over three consecutive experiments. As the subject performed subsequent tests, he demonstrated significant improvement in performing the task with no feedback and smaller improvements performing the task with the visual indicator and with the vibrotactile feedback.
The haptic feedback simulator used in this study was designed to train subjects in grasping objects at particular force levels by using a force-matching task. This type of training can teach prosthesis users to control their grasp force better, which will help them grasp objects more securely and confidently. Human haptic feedback is an essential part of this process because automatic local control strategies are dependent on some kind of object recognition, which is imperfect (consider distinguishing an empty paper cup vs. a full paper cup). Haptic feedback will allow prosthesis users to close the control loop themselves and improve their grasping capabilities with training. This simulator allows for the comparison of two different feedback modalities: a visual feedback system that enables highly accurate matching but is impractical for everyday use, and a vibrotactile feedback system that is more practical for prosthetic integration, but uses a tactile feedback method that is not perceived with the same resolution as visual feedback. Furthermore, the inclusion of visual feedback in the simulator allows the subject to initially train on a straightforward visual interface and then transition to the more unfamiliar vibrotactile system.
The approach described in this study is limited by the availability and experience of the subjects. The experimental setup and testing protocol were designed for able-bodied subjects; however, it is reasonable to expect that subjects with upper-limb amputations who are experienced with myoelectric prostheses would interact with the setup differently. Furthermore, separating out the subject pool between experienced and naive subjects reduces the statistical power of the resulting plots. The results should not be considered as generally representative of any particular population, but rather as a guide to what types of analyses could be performed with these types of simulators.
The comprehensive plot of errors across the feedback modalities shows that the visual indicator enabled a great reduction in mean error. Because of the high resolution and information content of the visual feedback modality, the subjects were able to perform the task much more effectively than during control cases. The error level for visual indicator trials was still significantly above zero, which indicates that even with a high-resolution surrogate force feedback system, subjects were still susceptible to control issues that prevented perfect force matching.
The vibrotactile feedback errors did not show any significant change from control errors, indicating that a significant conclusion could not be reached from the average data. Experienced subjects showed a statistically significant drop in error in the medium force case, which indicates that the supplementary haptic information was most useful at mid-level grasping force ranges. It is reasonable to expect that these subjects were able to use visual cues from the object deformation to judge when they had lightly grasped or heavily grasped the object, thus rendering the supplementary haptic information somewhat redundant for the low and high force levels. One explanation for this may be that subjects became aware of consistent endpoints that could be easily determined visually (such as lightly touching the object or grasping the object as hard as possible). In the absence of any sensory feedback, subjects may have used a strategy of visually referencing the target level to one of these easily reached endpoints. However, in the medium force case, visual cues alone may not have provided sufficient information for consistent force matching, so the additional sensory feedback led to improved performance. This behavior is consistent with findings reported in studies that suggest that the haptic modality becomes more dominant when visual information is unreliable.22,23 Further testing would be required to determine how performance is affected by haptic feedback when subjects are faced with unfamiliar tests.
Just as myoelectric control is tested and practiced during the prosthesis fitting process, haptic feedback systems should also be tested if they are to be successfully integrated into prosthetic devices. A multifunctional haptic feedback simulator will be a useful tool for prosthetists and their patients to evaluate a haptic feedback system in two important contexts: how the system feels and how well the user performs with the feedback. As with everything else in prosthetics, a haptic feedback system must be tailored to meet each user’s individual needs, so it is important for the simulator to be extremely flexible in how it provides and modulates the sensory feedback. This type of system will allow users to train with the feedback and will provide quantitative assessments of their performance for clinical evaluation.
This study used a haptic feedback simulator to compare the effects of two different surrogate force feedback strategies on grasp force control. The results show that the vibrotactile feedback system did not yield comparable error reduction to a visual feedback system. However, the vibrotactile feedback did lead to improved performance for experienced users at the medium force level. The inherent difficulties in controlling a prosthesis coupled with the unfamiliar nature of the vibrotactile feedback may have contributed to the poorer performance noted among naive users. Experience and training may allow users to incorporate the vibrotactile information input effectively, which would be desirable for situations where a visual feedback system is inconvenient or impractical. Because users practice with simple feedback systems such as the vibrotactile representation of grip force used in this study, they will be able to explore the functional benefits of haptic feedback themselves.
The authors thank Ander Ramos for assistance in formulating the study design and hardware setup.
1.McLaughlin M, Hespanha JP, Sukhatme GS. Introduction to haptics. In: Touch in Virtual Environments.
Upper Saddle River, NJ: MISC Press Multimedia Series (Prentice Hall); 2002:1–31.
2.Stone RJ. Haptic feedback: a brief history from telepresence to virtual reality. Presented at Proceedings of the First International Workshop on Haptic Human-Computer Interaction; Glasgow, UK; August 31 to September 1, 2000.
3.Benali KM, Hafez M, Alexandre JM, Kheddar A. Tactile interfaces: a state of the art survey. Presented at International Symposium on Robotics; Paris, France; 2004.
4.Weir RF. Design of artificial arms and hands for prosthetic applications. In: Kutz M, ed. Standard Handbook of Biomedical Engineering and Design.
New York: McGraw-Hill; 2003:32.1–32.61.
5.Roeschlein RA, Domholdt E. Factors related to successful upper extremity prosthetic use. Prosthet Orthot Int
6.Meek SG, Jacobsen SC, Goulding PP. Extended physiologic taction: design and evaluation of a proportional force feedback
system. J Rehabil Res Dev
7.Patterson PE, Katz JA. Design and evaluation of a sensory feedback system that provides grasping pressure in a myoelectric hand. J Rehabil Res Dev
8.Shannon GF. A comparison of alternative means of providing sensory feedback on upper limb prostheses. Med Biol Eng
9.Shannon GF. A myoelectrically-controlled prosthesis with sensory feedback. Med Biol Eng Comput
10.Sueda O, Tamura H. Sensory device for the artificial arm. Presented at Proceedings of the Eighth International Conference on Medical and Biological Engineering; Chicago, IL; 1969.
11.Pylatiuk C, Mounier S, Kargov A, et al. Progress in the development of a multifunctional hand prosthesis. Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, California, 1–5 September 2004.
New York, NY: IEEE; 2004;2:4260–4263.
12.Atkins DJ, Heard DCY, Donovan WH. Epidemiologic overview of individuals with upper-limb loss and their reported research priorities. J Prosthet Orthot
13.Pylatiuk C, Kargov A, Schulz S. Design and evaluation of a low-cost force feedback
system for myoelectric prosthetic hands. J Prosthet Orthot
14.Kaczmarek KA, Webster JG, Bach-y-Rita P, Tompkins WJ. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Trans Biomed Eng
15.Mountcastle VB, LaMotte RH, Carli G. Detection thresholds for stimuli in humans and monkeys: comparison with threshold events in mechanoreceptive afferent nerve fibers innervating the monkey hand. J Neurophysiol
16.LaMotte RH, Mountcastle VB. Capacities of humans and monkeys to discriminate vibratory stimuli of different frequency and amplitude: a correlation between neural events and psychological measurements. J Neurophysiol
17.Cholewiak RW, Collins AA. Vibrotactile localization on the arm: effects of place, space, and age. Percept Psychophys
18.Weisenberger JM. Sensitivity to amplitude-modulated vibrotactile signals. J Acoust Soc Am
19.Shannon GF. Sensory feedback for artificial limbs. Med Prog Technol
20.Verillo RT. Effect of contactor area on the vibrotactile threshold. J Acoust Soc Am
21.Cohen J, Niwa M, Lindeman RW, et al. A closed-loop tactor frequency control system for vibrotactile feedback. Presented at ACM CHI 2005; Portland, OR; 2005.
22.Ernst M, Banks M. Does vision always dominate haptics. Presented at Conference on Touch in Virtual Environments; Los Angeles, CA; 2001.
23.Feygin D, Keehner M, Tendick R. Haptic guidance: experimental evaluation of a haptic training method for a perceptual motor skill. Presented at Haptic Interfaces for Virtual Environment and Teleoperator Systems; Orlando, FL; 2002.