Paradoxically, larger characters are slightly more difficult for this volunteer because they extend well beyond the limits of his visual “tunnel.” The rapid fall–off with characters smaller than 20/1200 is also quite reproducible, but the explanation is uncertain. In the future, more sophisticated psychophysical experiments may compare this volunteer with normal patients, separating effect due to processing at the retina and lateral geniculate from those occurring at cortical levels or beyond.
Similar acuity results have been achieved with the television/computer/Internet interface replacing the camera, although scanning is slower because a keypad is currently used for control, rather than neck movements. The volunteer believes that his performance will continue to improve with additional experience, particularly practice in scanning. The resolution of the system itself is ultimately limited by the analog-to-digital conversion in the NTSC link between the camera or other source and the computer, and thus can be improved by a better link, a different camera, or both.
Of course, visual acuity is normally measured with optimal correction. Adding a lens to the existing camera is one possibility; however, because of size, weight, and cosmetic considerations, we have chosen to accomplish magnification “correction” in software, which proved very difficult to write and is still being debugged. In addition, we are exploring use of image processing techniques, including edge detection. This additional computer processing required for edge detection slows the frame rate to approximately 1 per second, but the volunteer is practicing use of such displays for mobility. In a larger (benchtop) development system, with a different camera, no NTSC link, and a 300 MHz processor of slightly different design, frame rates can be increased up to 7 per second.
We had expected that the patient might have trouble with apparent changes in size or shape of the phosphene image, particularly because the electrodes seem to be on visual association cortex. However, at this point there are no signs of either metamorphopsia or dysmetropsia, and corrective image processing has not been necessary.
As we have reported with earlier volunteers, 2 brightness can easily be modulated by changes in pulse amplitude. 11 However, provision of “gray scale” has not proven very valuable so far, probably because of the combination of tunnel vision and limited resolution. The phosphene display is planar, but is of uncertain distance, like the stars in the sky. We, therefore, plan to add an ultrasonic or infra-red “rangefinder”12 in which the brightness of an easily identifiable phosphene, probably the one produced by electrode number 14 in this volunteer, is a function of distance. This is analogous to the “heads up” displays used by military pilots.
Although stimulation of visual cortex in sighted patients 2 frequently produces colored phosphenes, the phosphenes reported by this volunteer (and all previous blind volunteers to the best of our knowledge) are colorless. We speculate that this is the result of postdeprivation deterioration of the cells and/or synaptic connections required for color vision. Consequently, color vision may never be possible in this volunteer or in future patients. However, optical filters could help differentiate colors, and it is also conceivable that chromatic sensations could be produced if future patients are implanted shortly after being blinded, before atrophy of the neural network responsible for color vision.
Contrast is entirely a function of the software, with adjustment by the experimental team depending on the experimental situation. The system also allows “reversals” in which the world looks much like a black and white photographic negative. Reversal is particularly useful when presenting black characters on a white background. These characters are then reversed by the computer so they appear as a matrix of white phosphenes on the patient’s (otherwise dark) visual field.
The phosphene map is not congruent with the center of the volunteer’s visual field. Phosphenes also move with eye movement. However, the volunteer’s ability to fixate with this artificial vision system is a function of aiming the camera using neck muscles, rather than eye muscles. It helps that the camera image is displayed on the remote video screen for monitoring by the experimental team. In addition, we use a laser pointer in the temple-piece of the volunteer’s glasses so the experimental team can tell at any moment where the camera is aimed by looking for the red dot.
Low vision patients often follow lines, including the junction between the wall and the floor, and/or lines of lights on the ceiling, and this volunteer has been practicing this approach. People with very limited vision can also achieve excellent mobility by following people. The volunteer has been practicing use of the system for this purpose as well, and can easily follow an 8 year old child.
The volunteer frequently travels alone in the New York metropolitan area, and to other cities, using public transport. He believes that one of the most dangerous errors in mobility is to mistake the space between subway cars for an open car door. He has been using the artificial vision system to practice this differentiation, while we monitor his performance with the remote VCR and viewing screen.
Discussion
In the United States, there are more than 1.1 million legally blind people, including 220,000 with light perception or less. 13 Similar statistics are thought to prevail in other economically developed countries. Unlike some other artificial vision proposals, such as retinal stimulators, cortical stimulators are applicable to virtually all causes of blindness. Our device may also help some legally blind low vision patients because the cortex of sighted people responds to stimulation similarly to the cortex of blind people. We believe that some blind children will be particularly good candidates for this new artificial vision system, because of their ability to quickly learn to use the system. In addition, without visual input, the visual cortex of blind children may not develop and this would prevent their use of artificial vision in the future. For example, the second patient implanted on the same day in 1978 as the volunteer reported here, was blinded in an accident at age 5 and implanted at age 62. Although he has retained his implant for more than 20 years, he has never seen phosphenes. However, our device is contraindicated in the very small number of blind people with severe chronic infections and the even smaller number blinded by stroke or cortical trauma.
None of the seven blind volunteers in our series have ever exhibited epileptic symptoms or other systemic problems related to the implant. Based on our clinical experience during the last 30 years, implanting thousands of patients in more than 40 countries with other types of neurostimulators (to control breathing, pain, and the urogenital system), 14 we believe the principal risk of our artificial vision device is infection, which might require removal of the implant in addition to antibiotic therapy.
To control costs and ensure easy maintenance, our design uses commercial off-the-shelf (COTS) components. The computer, stimulating electronics, and software are all external, facilitating upgrades and repairs. However, despite ongoing software improvements and use of larger numbers of electrodes in the future, it is unlikely that patients will be able to drive an automobile in the foreseeable future, much less get legal approval to do so.
Development of implanted medical devices such as this artificial vision system progresses in three stages. First there is speculation, 15 then there is hope, 1 and finally there is promise.
Given our considerable experience with neurostimulator implantation, we believe that we can promise a 512 electrode system that will be cost–competitive with a guide dog. More important, that cost can be expected to drop dramatically in the future, while performance should continue to improve.
Acknowledgments
I thank W.J. Kolff, the mentor and friend who enabled me to begin this project from 1968 to 1976. I also thank this blind volunteer and his family, as well as the more than 50 other sighted, blind, and deaf volunteers who have been involved as surgical subjects in our sensory prostheses research, as well as the thousands of patients in more than 40 countries who have been implanted with our clinical neurological stimulators since 1969. I also thank (alphabetically) R. Avery, G. Brindley, M. Dobelle, † M. D. Dobelle, C. Eyzaguirre, D. Evans, † H. K. Hartline, † B. Lisan, E. F. McNichol, Jr., W. Partridge, W. Penfield, † K. Reemstma, D. Rushton, and T. Stockholm, for reasons best known to each of them. J. Girvin has been our principle neurosurgical collaborator since 1970. He implanted all seven of our blind patients assisted by J. Antunes, D. Fink, M. McDonald, D. Quest, T. Roberts, and T. Stanley among others. D. Dohn, C. Drake, † P. Gildenberg, M. G. Yasagil, and many others have also provided neurosurgical advice and assistance. M. Mladejovsky and, more recently, P. Ning have guided our computer engineering efforts during the past 30 years with programming assistance from a group including D. Eddington, J. Evans, A. Halpert, M. O’Keefe, and J. Ochs. More than 300 other scientists, physicians, engineers, and surgeons have been involved in our experiments since 1968, including K. Aron, B. Besser, M. D’Angelo, G. Dulmage, S. Fidone, B. Goetz, R. Goldbaum, J. Hanson, D. Hill, R. Huber, D. Kiefer, G. Klomp, T. Lallier, L. Pape, B. Seelig, K. Smith, L. Stensaas, S. Stensaas, and M. Womack III. † J. Andrus and L. Homrighausen (Surdna Foundation, New York, NY), Max Fleischman Foundation (Reno, Nevada), H. Geneen † (IT&T Corp., New York, NY), E. Grass (Gross Instruments, Boston, Mass.), Wm. Randolph Heart Foundation (San Francisco, CA), E. Land † (Polaroid Corp., Cambridge, Mass.), S. Olsen (Digital Equipment Corp., Maynard, Mass.), D. Rose † (New York, NY), M. Shapiro † (General Instrument Co., New York, NY), Wm. Volker Fund (Monterey, CA), and more than 100 other individuals and foundations provided financial support prior to 1981, without which this program would have been impossible. During that period we also received equipment donations from dozens of corporations, including Phillips Electronics, Fairchild Inc., Siguestis Inc., Soldran Inc., Bell Telephone Laboratories Inc., Hughes Aircraft Corp., Sanyo Corp., General Atomic Corp., Thermionics Inc., and TRW Inc. Since 1981, all financial support has been provided by the Dobelle Institute, Inc. and its United States and Swiss affiliates. Financing our R&D on artificial vision entirely by sale of related neurological stimulators was consciously modeled on the Wright Brothers, who developed the airplane with proceeds from their bicycle factory. Like the airplane, the artificial vision project has entailed a high risk of failure, and a long development time, which are incompatible with conventional venture capital horizons. Advice and assistance in this respect has been provided by P. Baldi, P. Conley, † R. Downey, D. Ellis III, C. Giffuni, A. Gutman, E. Heil, I. Lustgarten, † J. McGarrahan, P. G. Pedersen, S. Sawyier, L. Towler, T. Young, and L. Weltman among others.
FOOTNOTES
*From Watson W: An account of Mr. Benjamin Franklin’s treatise, lately published, entitled Experiments and Observations on Electricity, made at Philadelphia in America. Philos Trans R Soc London 47: 202–211.
†The first implant was removed, as planned, after 3 months. The second volunteer agreed to continue participation but his implant was removed due to a blood borne infection that did not originate with the implant.
Cited Here...
†Deceased.
Cited Here...
References
1. Brindley GS, Lewin WS: The sensations produced by electrical stimulation of the visual cortex. J Physiol (Lond) 196:479–493, 1968.
2. Dobelle WH, Mladejovsky MG: Phosphenes produced by electrical stimulation of human occipital cortex, and their application to the development of a prosthesis for the blind. J Physiol (Lond) 243:553–576, 1974.
3. Dobelle WH, Mladejovsky MG, Girvin JP: Artificial vision for the blind: Electrical stimulation of visual cortex offers hope for a functional prosthesis. Science 183:440–444, 1974.
4. Dobelle WH, Mladejovsky MG, Evans JR, Roberts TS, Girvin JP: “Braille” reading by a blind volunteer by visual cortex stimulation. Nature 259:111–112, 1976.
5. Dobelle WH, Quest D, Antunes J, Roberts T, Girvin JP: Artificial vision for the blind by electrical stimulation of the visual cortex. Neurosurgery 5:521–527, 1979.
6. Klomp GF, Womack MVB, Dobelle WH: Fabrication of large arrays of cortical electrodes for use in man. J Biomed Mater Res 11:347–364, 1977.
7. Klomp GF, Womack MVB, Dobelle WH: Percutaneous transmission of electrical energy in humans. Trans ASAIO 25:1–7, 1979.
8. Stenaas SS, Eddington DK, Dobelle WH: The topography and variability of the primary visual cortex in man. J Neurosurg 40:747–754, 1974.
9. Mladejovsky MG, Eddington DK, Evans JR, Dobelle WH: A computer-based brain stimulation system to investigate sensory prostheses for the blind and deaf. IEEE Trans Biomed Eng 23:286–296, 1976.
10. Valvo A: Sight restoration after long-term blindness: The problems and behavior patterns of visual rehabilitation. New York, American Foundation for the Blind, 1971.
11. Henderson DC, Evans JR, Dobelle WH: The relationship between stimulus parameters and phosphene threshold/brightness, during stimulation of human visual cortex. Trans ASAIO 25:367–370, 1979.
12. Dobelle WH: Artificial vision for the blind: The summit may be closer than you think. ASAIO J 40:919–921, 1994.
13. Leonard R: Statistics on vision impairment: A resource manual.
Lighthouse International, February 1999.
15. Watson W: An account of Mr. Benjamin Franklin’s treatise, lately published, entitled Experiments and Observations on Electricity, made at Philadelphia in America. Philos Trans R Soc London 47:202–211, 1751–1752.
16. Sobel I: Camera models and machine perception. AIM–21. Stanford Artificial Intelligence Laboratory, Palo Alto, California, 1970.
Afterword
Our team has continued to develop the hardware and software of this artificial vision system. Five key developments have occurred in the 2 months since submission of this paper for publication in September, 1999.
Development of a New Technique for Phosphene Mapping
Phosphene mapping is complicated by the fact that all phosphenes are produced in a relatively small area, which makes pointing difficult. This is compounded by the fact that phosphenes move with eye movements. In the refined technique, two phosphenes are selected to provide a vertical scale. The volunteer is then asked to estimate the vertical distance between each phosphene and these two references, as well as the distance to the left or right of an imaginary line connecting the reference phosphenes. This approach resulted in some small changes in the map described in Figure 6, but the principal result was to “compress” the map horizontally from 3 inches across to about 2 inches across.
Use of a More Powerful Computer
During the last two decades, many improvements in our hardware and software have developed because of rapid technological advancements in computer technology (“Moore’s Law”). Shortly after submission of this paper, we were able to obtain a new computer in an almost identical small package. This more powerful system uses a 233 MHz processor, 32 MB of RAM, and a 4 GB hard disk. After debugging the software, the extra computing power proved important in two areas: (1) magnification in software and (2) image pre–processing, particularly edge detection.
Electronic Magnification
The pinhole camera we have been using is small, light, and inconspicuous. However, it has a 69° field of view. Conventional optics would be heavy and conspicuous. Moreover, it is difficult to conceive a “zoom” version without using a motor drive. Using the more powerful computer, we were able to implement software magnification algorithms that were not possible with the initial portable system discussed above. The gray value for all pixels (120 × 160) were recorded and then 2, 4, 8, or 16 pixels were combined to create a single pixel for transmission to the patient. Using magnifications of 4 (and sometimes 8) times the patient’s resolution improved to the point where he can now recognize a 2 inch high letter at 5 feet, as opposed to a 6 inch high letter at the same distance. This represents an acuity improvement from roughly 20/1200 to 20/400. Less magnification (e.g., 2×) was insufficient. Due to the patient’s tunnel vision, at 16× the image far overlapped the tunnel, with effects similar to the acuity degradation described for letters larger than 6 inches at 5 feet in Figure 8 above.
Edge Detection
In 1969–1970, our team (M. Mladejovsky and W. Dobelle, unpublished data), at the University of Utah began exploring computer simulations of artificial vision displays using a head mounted display (originally designed by Ivan Sutherland) attached to a “single user” PDP–1 computer. This research was part of a much larger (unclassified) program on computerized image processing sponsored by the Advanced Research Projects Agency of the Department of Defense. Edge detection clearly extracted important information and removed “noise.” However, this computer (which occupied approximately 8,000 square feet) required hours to process a single frame. The 120 MHz system described above was able to process approximately one frame per second, which is too slow for mobility. The new 233 MHz system using Sobel filters 16 for edge detection, can process and transmit images to the volunteer at speeds up to 8 frames/second. A mannequin as pictured by the television camera (Figure 1) is shown in Figure 9A. The same scene is also shown after edge-detection processing in Figure 9B. We believe that such processing will be an integral part of all clinical visual prostheses.
Ultrasonic Rangefinder
Using edge detection, it is particularly helpful for the blind patient to know how far the wall is located behind the mannequin (Figures 9A and B). Ultrasonic rangefinders for the blind have been known for many years, however they have typically translated distance into audio signals that interfere with the ability of blind patients to use their hearing. (Indeed, this writer almost fell down a stairway at the University of Utah while blindfolded and trying to use an ultrasonic-to-audio conversion device. I did not hear the warning of a companion). However, by placing an electrostatic transducer on the left lens of the patient’s eyeglasses (lateral to the camera and below the laser pointer) we have begun exploring the supplementary information that can be provided by modulating brightness, blink rate, and identity of selected phosphenes.
Discussion
The blind volunteer is now able to navigate among a “family” of three mannequins—standing adult male, seated adult female, and standing 3 year old child—randomly placed in a large room, without bumping into any of them. He can then go to the wall and retrieve a cap that has been placed on the wall at a random location. Navigating back in the direction from which he came, he can find any of the three mannequins and place the cap on the head of whichever one we request. As the volunteer gains more experience, and we make further refinements in the system, rapid progress can be expected. Even more rapid advances can be anticipated with larger electrode arrays, more powerful computers, and more sophisticated image pre–processing algorithms.
Wm. H. Dobelle, PhD
December 1, 1999
New York, NY
Copyright © 2000 by the American Society for Artificial Internal Organs
Source
ASAIO Journal46(1):3-9, January-February 2000.
-
-
-
-
Your message has been successfully sent to your colleague.
Some error has occurred while processing your request. Please try after some time.
Your message has been successfully sent to your colleague.
Some error has occurred while processing your request. Please try after some time.
The item(s) has been successfully added to "".