Dobelle, Wm. H.

This new visual prosthesis produces black and white displays of visual cortex “phosphenes” analogous to the images projected on the light bulb arrays of some sports stadium scoreboards. The system was primarily designed to promote independent mobility, not reading. We have also provided a battery powered, electronic interface that is radio frequency (RF) isolated from line currents for safety. This interface can replace the camera, permitting the volunteer to directly watch television and use a computer, including access to the Internet. Because of their potential importance for education, and to help integrate blind people into the workforce, such television, computer, and Internet capabilities may prove even more valuable in the future than independent mobility. In addition, the image from the camera or interface and an overlaid simulated real-time display of the phosphene image seen by the volunteer can be re–broadcast from the system over an RF link to a remote videotape recorder and viewing screen. This allows real-time monitoring, as well as post-trial analysis, by the experimental team.

The television camera, which is built into a pair of sunglasses, is shown in Figure 1; the prosthesis, worn by the blind volunteer, is pictured in Figure 2, and the complete system is described schematically in Figure 3, including both the television/computer/Internet interface and the remote Video Screen/VCR monitor, neither of which are shown in Figure 2.

Figure 1

Figure 2

Figure 3

These efforts were inspired by a seminal paper published by Giles Brindley’s group in 1968. ¹ Our first human experiments in 1970–1972 ² involved cortical stimulation of 37 sighted volunteers who were undergoing surgery on their occipital lobe under local anesthesia to remove tumors and other lesions. In 1972–1973 we then stimulated the visual cortex of three blind volunteers who were temporarily implanted for a few days with electrode arrays passed through a Penrose drain. ³ Our subsequent experiments have involved four blind volunteers implanted with permanent electrode arrays using percutaneous connecting pedestals. Two volunteers were implanted in 1974. ⁴ One array was removed 3 months after surgery and the second after 14 years. ^† The first five volunteers were operated on at the University of Western Ontario in London, Canada. Two additional blind volunteers, including the subject of this article, were implanted in 1978 at the Columbia–Presbyterian Medical Center in New York City. ⁵ They have both retained their implants for more than 20 years without infection or other problems.

The Volunteer and Implant

The 62 year old subject of this article traumatically lost vision in one eye at age 22 and was totally blinded at age 36 by a second trauma. He was continuously employed, before and after losing his sight, as an administrator by the State of New York. He retired in 1997 after 32 years of service. The electrode array was implanted in 1978 when he was 41 years old. Because of discomfort during surgery caused by mechanical impingement of the teflon electrode matrix on the volunteer’s falx and tentorium, his electrode array is posterior to the position of the arrays implanted in our six other blind volunteers. We have been using this implanted pedestal and intracranial electrode array to experimentally stimulate the visual cortex, on the mesial surface of the right occipital lobe, for more than 20 years. However, the fifth generation external electronics package and software are entirely new, taking advantage of cutting edge technology that has only recently become available. An X–ray of the implanted visual cortex electrode array is shown in Figure 4, and the numbered electrode layout is detailed in Figure 5.

Figure 4

Figure 5

A platinum foil ground plane is perforated with a hexagonal array of 5 mm diameter holes on 3 mm centers, and the flat platinum electrodes centered in each hole are 1 mm in diameter. This ground plane keeps all current beneath the dura. This eliminates discomfort due to dural excitation when stimulating some single electrodes (such as number 19) and when other arrays of electrodes are stimulated simultaneously. The ground plane also eliminates most phosphene interactions ³ when multiple electrodes are stimulated simultaneously and provides an additional measure of electrical safety that is not possible when stimulating between cortical electrodes and a ground plane outside the skull. Each electrode is connected by a separate teflon insulated wire to a connector contained in a carbon percutaneous pedestal. Fabrication techniques for these electrodes ⁶ and pedestals ⁷ have been previously described. The original surgery in 1978 was performed under local anesthesia, and implants in future patients can probably be performed on an outpatient basis by most neurosurgeons.

Phosphenes and Their Map in The Visual Field

When stimulated, each electrode produces 1–4 closely spaced phosphenes. Each phosphene in a cluster ranges up to the diameter of a pencil at arms length. Neighboring phosphenes in each cluster are generally too close to the adjacent phosphenes for another phosphene to be located between them. These “multiples” are unlike the phosphenes described by our other blind volunteers, or those reported by Brindley’s volunteers. ¹ They may be due to the use of a ground plane array, although we have used a similar ground plane array in one temporarily implanted blind volunteer without producing multiple phosphenes. Other possible causes for these multiples include the fact that the volunteer lost vision in his two eyes at different times or that we may be stimulating visual association cortex (areas 18 and 19) rather than primary visual cortex (area 17).

All phosphenes flicker at a rate that seems unrelated to the pulse, repetition frequency, or any other parameter of stimulation, or to cardiac pulse, breathing rate, or other physiologic function. Using a variety of computer and manual mapping techniques, we determined that the phosphene map occupies an area roughly 8 inches in height and 3 inches wide, at arms length. The map and the parameters for stimulation both appear to be stable over the last two decades. The map of some of the phosphenes in this volunteer’s visual space is shown in Figure 6, and is more nearly a vertical line than the larger, more two-dimensional maps reported by our earlier volunteers, or by the volunteers of Brindley. ¹ We suspect, but cannot prove, that this unusual map, like the clusters of multiple phosphenes, is due to placement of the electrodes on visual association cortex (areas 18 and 19) rather than primary visual cortex (area 17). In the future, we may implant up to 256 additional surface electrodes, particularly on the left occipital lobe of this volunteer, to increase the resolution of this system. However, trying to place additional electrodes within sulci is impractical, at least at this time. Our anatomic studies in cadavers ⁸ indicate that primary visual cortex (area 17) would permit placement of 256 surface electrodes on 3 mm centers on each lobe in most humans (512 electrodes total). However, stimulating adjacent visual association cortex—as we believe we are doing in this volunteer—would substantially expand the number of possible electrodes in the matrix. The organization of the stimulator is modular, and the system described here is being expanded to allow us to stimulate 256 electrodes on each hemisphere.

Figure 6

The Electronics Package

The 492 × 512 pixel charge coupled devices (CCD) black and white television camera is powered by a 9 V battery and connects via a battery–powered National Television Standards Committee (NTSC) link to a sub–notebook computer in a belt pack. This f 14.5 camera, with a 69° field of view, uses a pinhole aperture, instead of a lens, to minimize size and weight. It also incorporates an electronic “iris” for automatic exposure control.

The sub-notebook computer incorporates a 120 MHz microprocessor with 32 MB of RAM and a 1.5 GB hard drive. It also has an LCD screen and keyboard. It was selected because of its very small size and light weight. The belt pack also contains a second microcontroller and associated electronics to stimulate the brain. This stimulus generator is connected through a percutaneous pedestal to the electrodes implanted on the visual cortex. The computer and electronics package together are approximately the size of a dictionary and weigh approximately 10 pounds, including camera, cables, and rechargeable batteries. The battery pack for the computer will operate for approximately 3 hours, and the battery pack for the other electronics will operate for approximately 6 hours.

This general architecture, in which one computer interfaces with the camera and a second computer controls the stimulating electronics, has been used by us in this, and four other substantially equivalent systems, since 1969. ⁹ The software involves approximately 25,000 lines of code in addition to the sub-notebooks’ operating system. Most of the code is written in C++, whereas some is written in C. The second microcontroller is programmed in assembly language.

Stimulation Parameters

Stimulation delivered to each electrode typically consists of a train of six pulses delivered at 30 Hz to produce each frame of the image. Frames have been produced with 1–50 pulses, and frame rates have been varied from 1 to 20 frames per second. As expected, ⁴ frame rates of 4 per second currently seem best, even with trains containing only a single pulse. Each pulse is symmetric, biphasic (−/+), with a pulse width of 500 μsec per phase (1,000 μsec total). Threshold amplitudes of 10–20 volts (zero–peak) may vary ±20% from day to day; they are higher than the thresholds of similar electrodes without the ground plane, presumably because current shunts across the surface of the pia–arachnoid and encapsulating membrane. The system is calibrated each morning by recomputing the thresholds for each electrode, a simple procedure that takes the volunteer approximately 15 minutes with a numeric keypad.

Performance of the System

We know of no objective method for comparing our “artificial vision” system with a cane, guide dog, or other aid for the blind. For example, there is no standard obstacle course on which such devices, or the performance of volunteers using them, can be rated. Indeed, even the vision test for drivers’ licenses in most jurisdictions uses only static Snellen tests. Furthermore, there are really no analogous low vision patients with parafoveal tunnel vision, plus scattered field defects (due to gaps between phosphenes), no color vision, and no depth perception to provide models for testing.

Initially, the volunteer was unable to recognize letters or numbers. Based on extensive personal experience in the 1960s with corneal transplant patients whose vision had been restored after many years of blindness, I expected that it might take the volunteer more than a year to learn how to use our new artificial vision system. This expectation was reinforced by the work of Valvo ¹⁰ and others. However, within 10 one-day sessions, the patient learned to use the system, and he has continued to practice 3–4 hours per day 2 or 3 days per week.

With scanning, he can now routinely recognize a 6 inch square “tumbling E” at five feet, as well as Snellen letters, HOTV test, Landolt rings, and Lea figures of similar size. These psychophysical tests are summarized in Figure 7. He can also count fingers. With the exception of finger counting, these acuity tests have been conducted using pure black characters on a pure white background at an illumination greater than 1,000 lux. Six inch characters at 5 feet corresponds to a visual acuity of approximately 20/1200. A frequency-of-seeing curve for the “tumbling E” and for Landolt’s ring is shown in Figure 8.

Figure 7

Figure 8

Paradoxically, larger characters are slightly more difficult for this volunteer because they extend well beyond the limits of his visual “tunnel.” The rapid fall–off with characters smaller than 20/1200 is also quite reproducible, but the explanation is uncertain. In the future, more sophisticated psychophysical experiments may compare this volunteer with normal patients, separating effect due to processing at the retina and lateral geniculate from those occurring at cortical levels or beyond.

Similar acuity results have been achieved with the television/computer/Internet interface replacing the camera, although scanning is slower because a keypad is currently used for control, rather than neck movements. The volunteer believes that his performance will continue to improve with additional experience, particularly practice in scanning. The resolution of the system itself is ultimately limited by the analog-to-digital conversion in the NTSC link between the camera or other source and the computer, and thus can be improved by a better link, a different camera, or both.

Of course, visual acuity is normally measured with optimal correction. Adding a lens to the existing camera is one possibility; however, because of size, weight, and cosmetic considerations, we have chosen to accomplish magnification “correction” in software, which proved very difficult to write and is still being debugged. In addition, we are exploring use of image processing techniques, including edge detection. This additional computer processing required for edge detection slows the frame rate to approximately 1 per second, but the volunteer is practicing use of such displays for mobility. In a larger (benchtop) development system, with a different camera, no NTSC link, and a 300 MHz processor of slightly different design, frame rates can be increased up to 7 per second.

We had expected that the patient might have trouble with apparent changes in size or shape of the phosphene image, particularly because the electrodes seem to be on visual association cortex. However, at this point there are no signs of either metamorphopsia or dysmetropsia, and corrective image processing has not been necessary.

As we have reported with earlier volunteers, ² brightness can easily be modulated by changes in pulse amplitude. ¹¹ However, provision of “gray scale” has not proven very valuable so far, probably because of the combination of tunnel vision and limited resolution. The phosphene display is planar, but is of uncertain distance, like the stars in the sky. We, therefore, plan to add an ultrasonic or infra-red “rangefinder”¹² in which the brightness of an easily identifiable phosphene, probably the one produced by electrode number 14 in this volunteer, is a function of distance. This is analogous to the “heads up” displays used by military pilots.

Although stimulation of visual cortex in sighted patients ² frequently produces colored phosphenes, the phosphenes reported by this volunteer (and all previous blind volunteers to the best of our knowledge) are colorless. We speculate that this is the result of postdeprivation deterioration of the cells and/or synaptic connections required for color vision. Consequently, color vision may never be possible in this volunteer or in future patients. However, optical filters could help differentiate colors, and it is also conceivable that chromatic sensations could be produced if future patients are implanted shortly after being blinded, before atrophy of the neural network responsible for color vision.

Contrast is entirely a function of the software, with adjustment by the experimental team depending on the experimental situation. The system also allows “reversals” in which the world looks much like a black and white photographic negative. Reversal is particularly useful when presenting black characters on a white background. These characters are then reversed by the computer so they appear as a matrix of white phosphenes on the patient’s (otherwise dark) visual field.

The phosphene map is not congruent with the center of the volunteer’s visual field. Phosphenes also move with eye movement. However, the volunteer’s ability to fixate with this artificial vision system is a function of aiming the camera using neck muscles, rather than eye muscles. It helps that the camera image is displayed on the remote video screen for monitoring by the experimental team. In addition, we use a laser pointer in the temple-piece of the volunteer’s glasses so the experimental team can tell at any moment where the camera is aimed by looking for the red dot.

Low vision patients often follow lines, including the junction between the wall and the floor, and/or lines of lights on the ceiling, and this volunteer has been practicing this approach. People with very limited vision can also achieve excellent mobility by following people. The volunteer has been practicing use of the system for this purpose as well, and can easily follow an 8 year old child.

The volunteer frequently travels alone in the New York metropolitan area, and to other cities, using public transport. He believes that one of the most dangerous errors in mobility is to mistake the space between subway cars for an open car door. He has been using the artificial vision system to practice this differentiation, while we monitor his performance with the remote VCR and viewing screen.

Discussion

In the United States, there are more than 1.1 million legally blind people, including 220,000 with light perception or less. ¹³ Similar statistics are thought to prevail in other economically developed countries. Unlike some other artificial vision proposals, such as retinal stimulators, cortical stimulators are applicable to virtually all causes of blindness. Our device may also help some legally blind low vision patients because the cortex of sighted people responds to stimulation similarly to the cortex of blind people. We believe that some blind children will be particularly good candidates for this new artificial vision system, because of their ability to quickly learn to use the system. In addition, without visual input, the visual cortex of blind children may not develop and this would prevent their use of artificial vision in the future. For example, the second patient implanted on the same day in 1978 as the volunteer reported here, was blinded in an accident at age 5 and implanted at age 62. Although he has retained his implant for more than 20 years, he has never seen phosphenes. However, our device is contraindicated in the very small number of blind people with severe chronic infections and the even smaller number blinded by stroke or cortical trauma.

None of the seven blind volunteers in our series have ever exhibited epileptic symptoms or other systemic problems related to the implant. Based on our clinical experience during the last 30 years, implanting thousands of patients in more than 40 countries with other types of neurostimulators (to control breathing, pain, and the urogenital system), ¹⁴ we believe the principal risk of our artificial vision device is infection, which might require removal of the implant in addition to antibiotic therapy.

To control costs and ensure easy maintenance, our design uses commercial off-the-shelf (COTS) components. The computer, stimulating electronics, and software are all external, facilitating upgrades and repairs. However, despite ongoing software improvements and use of larger numbers of electrodes in the future, it is unlikely that patients will be able to drive an automobile in the foreseeable future, much less get legal approval to do so.

Development of implanted medical devices such as this artificial vision system progresses in three stages. First there is speculation, ¹⁵ then there is hope, ¹ and finally there is promise.

Given our considerable experience with neurostimulator implantation, we believe that we can promise a 512 electrode system that will be cost–competitive with a guide dog. More important, that cost can be expected to drop dramatically in the future, while performance should continue to improve.

Acknowledgments

I thank W.J. Kolff, the mentor and friend who enabled me to begin this project from 1968 to 1976. I also thank this blind volunteer and his family, as well as the more than 50 other sighted, blind, and deaf volunteers who have been involved as surgical subjects in our sensory prostheses research, as well as the thousands of patients in more than 40 countries who have been implanted with our clinical neurological stimulators since 1969. I also thank (alphabetically) R. Avery, G. Brindley, M. Dobelle, ^† M. D. Dobelle, C. Eyzaguirre, D. Evans, ^† H. K. Hartline, ^† B. Lisan, E. F. McNichol, Jr., W. Partridge, W. Penfield, ^† K. Reemstma, D. Rushton, and T. Stockholm, for reasons best known to each of them. J. Girvin has been our principle neurosurgical collaborator since 1970. He implanted all seven of our blind patients assisted by J. Antunes, D. Fink, M. McDonald, D. Quest, T. Roberts, and T. Stanley among others. D. Dohn, C. Drake, ^† P. Gildenberg, M. G. Yasagil, and many others have also provided neurosurgical advice and assistance. M. Mladejovsky and, more recently, P. Ning have guided our computer engineering efforts during the past 30 years with programming assistance from a group including D. Eddington, J. Evans, A. Halpert, M. O’Keefe, and J. Ochs. More than 300 other scientists, physicians, engineers, and surgeons have been involved in our experiments since 1968, including K. Aron, B. Besser, M. D’Angelo, G. Dulmage, S. Fidone, B. Goetz, R. Goldbaum, J. Hanson, D. Hill, R. Huber, D. Kiefer, G. Klomp, T. Lallier, L. Pape, B. Seelig, K. Smith, L. Stensaas, S. Stensaas, and M. Womack III. ^† J. Andrus and L. Homrighausen (Surdna Foundation, New York, NY), Max Fleischman Foundation (Reno, Nevada), H. Geneen ^† (IT&T Corp., New York, NY), E. Grass (Gross Instruments, Boston, Mass.), Wm. Randolph Heart Foundation (San Francisco, CA), E. Land ^† (Polaroid Corp., Cambridge, Mass.), S. Olsen (Digital Equipment Corp., Maynard, Mass.), D. Rose ^† (New York, NY), M. Shapiro ^† (General Instrument Co., New York, NY), Wm. Volker Fund (Monterey, CA), and more than 100 other individuals and foundations provided financial support prior to 1981, without which this program would have been impossible. During that period we also received equipment donations from dozens of corporations, including Phillips Electronics, Fairchild Inc., Siguestis Inc., Soldran Inc., Bell Telephone Laboratories Inc., Hughes Aircraft Corp., Sanyo Corp., General Atomic Corp., Thermionics Inc., and TRW Inc. Since 1981, all financial support has been provided by the Dobelle Institute, Inc. and its United States and Swiss affiliates. Financing our R&D on artificial vision entirely by sale of related neurological stimulators was consciously modeled on the Wright Brothers, who developed the airplane with proceeds from their bicycle factory. Like the airplane, the artificial vision project has entailed a high risk of failure, and a long development time, which are incompatible with conventional venture capital horizons. Advice and assistance in this respect has been provided by P. Baldi, P. Conley, ^† R. Downey, D. Ellis III, C. Giffuni, A. Gutman, E. Heil, I. Lustgarten, ^† J. McGarrahan, P. G. Pedersen, S. Sawyier, L. Towler, T. Young, and L. Weltman among others.

FOOTNOTES

*From Watson W: An account of Mr. Benjamin Franklin’s treatise, lately published, entitled Experiments and Observations on Electricity, made at Philadelphia in America. Philos Trans R Soc London 47: 202–211.

†The first implant was removed, as planned, after 3 months. The second volunteer agreed to continue participation but his implant was removed due to a blood borne infection that did not originate with the implant.
Cited Here...

†Deceased.
Cited Here...

References

Afterword

Our team has continued to develop the hardware and software of this artificial vision system. Five key developments have occurred in the 2 months since submission of this paper for publication in September, 1999.

Development of a New Technique for Phosphene Mapping

Phosphene mapping is complicated by the fact that all phosphenes are produced in a relatively small area, which makes pointing difficult. This is compounded by the fact that phosphenes move with eye movements. In the refined technique, two phosphenes are selected to provide a vertical scale. The volunteer is then asked to estimate the vertical distance between each phosphene and these two references, as well as the distance to the left or right of an imaginary line connecting the reference phosphenes. This approach resulted in some small changes in the map described in Figure 6, but the principal result was to “compress” the map horizontally from 3 inches across to about 2 inches across.

Use of a More Powerful Computer

During the last two decades, many improvements in our hardware and software have developed because of rapid technological advancements in computer technology (“Moore’s Law”). Shortly after submission of this paper, we were able to obtain a new computer in an almost identical small package. This more powerful system uses a 233 MHz processor, 32 MB of RAM, and a 4 GB hard disk. After debugging the software, the extra computing power proved important in two areas: (1) magnification in software and (2) image pre–processing, particularly edge detection.

Electronic Magnification

The pinhole camera we have been using is small, light, and inconspicuous. However, it has a 69° field of view. Conventional optics would be heavy and conspicuous. Moreover, it is difficult to conceive a “zoom” version without using a motor drive. Using the more powerful computer, we were able to implement software magnification algorithms that were not possible with the initial portable system discussed above. The gray value for all pixels (120 × 160) were recorded and then 2, 4, 8, or 16 pixels were combined to create a single pixel for transmission to the patient. Using magnifications of 4 (and sometimes 8) times the patient’s resolution improved to the point where he can now recognize a 2 inch high letter at 5 feet, as opposed to a 6 inch high letter at the same distance. This represents an acuity improvement from roughly 20/1200 to 20/400. Less magnification (e.g., 2×) was insufficient. Due to the patient’s tunnel vision, at 16× the image far overlapped the tunnel, with effects similar to the acuity degradation described for letters larger than 6 inches at 5 feet in Figure 8 above.

Edge Detection

In 1969–1970, our team (M. Mladejovsky and W. Dobelle, unpublished data), at the University of Utah began exploring computer simulations of artificial vision displays using a head mounted display (originally designed by Ivan Sutherland) attached to a “single user” PDP–1 computer. This research was part of a much larger (unclassified) program on computerized image processing sponsored by the Advanced Research Projects Agency of the Department of Defense. Edge detection clearly extracted important information and removed “noise.” However, this computer (which occupied approximately 8,000 square feet) required hours to process a single frame. The 120 MHz system described above was able to process approximately one frame per second, which is too slow for mobility. The new 233 MHz system using Sobel filters ¹⁶ for edge detection, can process and transmit images to the volunteer at speeds up to 8 frames/second. A mannequin as pictured by the television camera (Figure 1) is shown in Figure 9A. The same scene is also shown after edge-detection processing in Figure 9B. We believe that such processing will be an integral part of all clinical visual prostheses.

Figure 9

Ultrasonic Rangefinder

Using edge detection, it is particularly helpful for the blind patient to know how far the wall is located behind the mannequin (Figures 9A and B). Ultrasonic rangefinders for the blind have been known for many years, however they have typically translated distance into audio signals that interfere with the ability of blind patients to use their hearing. (Indeed, this writer almost fell down a stairway at the University of Utah while blindfolded and trying to use an ultrasonic-to-audio conversion device. I did not hear the warning of a companion). However, by placing an electrostatic transducer on the left lens of the patient’s eyeglasses (lateral to the camera and below the laser pointer) we have begun exploring the supplementary information that can be provided by modulating brightness, blink rate, and identity of selected phosphenes.

Discussion

The blind volunteer is now able to navigate among a “family” of three mannequins—standing adult male, seated adult female, and standing 3 year old child—randomly placed in a large room, without bumping into any of them. He can then go to the wall and retrieve a cap that has been placed on the wall at a random location. Navigating back in the direction from which he came, he can find any of the three mannequins and place the cap on the head of whichever one we request. As the volunteer gains more experience, and we make further refinements in the system, rapid progress can be expected. Even more rapid advances can be anticipated with larger electrode arrays, more powerful computers, and more sophisticated image pre–processing algorithms.

Wm. H. Dobelle, PhD

December 1, 1999

New York, NY