Spatial recognition with the HoloLens is based on simultaneous localization and mapping,20 which indicated that the display stability of holograms with respect to real space is excellent. However, there was no function to detect precisely the shape of a specific part in real space and accurately align the hologram with it.
Although libraries that perform automatic alignment by algorithms for fiducial marker recognition or contour extraction have also been released, they apply loads to the central processing unit of the HoloLens, slightly impairing its display stability. The manual alignment, however, has the advantage that simultaneous localization and mapping of the HoloLens is not impaired. Even if automatic positioning is possible, cases where alignment errors remain should be assumed. In such cases, a mechanism that can be adjusted with a simple manual operation should be prepared.
The difference between this method and that performed with the existing software is that any of the three marking points can arbitrarily be selected as a pivot/axis of rotational movement of the hologram. This difference makes manual alignment easy.
Although the HoloLens is far more advanced than conventional devices, it is still in development. It seems that higher performance devices will be available in the future.9 It is already at a stage, however, where it can be used in the clinical setting with only a little ingenuity. Thus, it is possible to use it clinically to obtain otherwise difficult-to-determine information. The error for alignment in this study was not much different from the various reports of other augmented reality systems in recent years.21–25 Although it is currently impossible to use as a strict means of navigation whereby the positional relationship between the tip of the instrument moving in real time and the target organ can be grasped, it is sufficiently accurate to apply to navigation-like use or intraoperative simulation to gain assistance with objective judgment.
We devised a method to align the hologram to the surgical field precisely within a short time using a simple manual operation when using the mixed realty device HoloLens. By using the method, the clinical usefulness of the HoloLens will be expanded.
1. Azuma RT. A survey of augmented reality. Presence 1997;6:355–385.
2. van Krevelen DWF, Poelman R. A survey of augmented reality technologies, applications and limitations. Int J Virt Reality 2010;9:1–20.
3. Stéphane N, Luc S, Didier M, et al. Augmented reality in laparoscopic surgical oncology. Surg Oncol. 2011;20:189–201.
4. Antonio M, Fabrizio C, Marina C, et al. Augmented reality in neurosurgery: A systematic review. Neurosurg Rev. 2017;40:537–548.
5. Yoon JW, Chen RE, Kim EJ, et al. Augmented reality for the surgeon: Systematic review. Int J Med Robot. 2018;14:e1914.
6. Kim Y, Kim H, Kim YO. Virtual reality and augmented reality in plastic surgery: A review. Arch Plast Surg. 2017;44:179–187.
7. Khor WS, Baker B, Amin K, Chan A, Patel K, Wong J. Augmented and virtual reality in surgery: The digital surgical environment. Applications, limitations and legal pitfalls. Ann Transl Med. 2016;4:454.
8. Burdea GC, Coiffet P. Virtual Reality Technology. 1994.London: Wiley-Interscience.
9. Paul M, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst. 1994;77:1321–1329.
10. Billinghurst M, Kato H. Billinghurst M, Kato H. Collaborative mixed reality. In: Proceedings of the First International Symposium on Mixed Reality. 1999:Berlin: Springer Verlag; 261–284.
11. Tamura H, Yamamoto H, Katayama A. Mixed reality: Future dreams seen at the border between real and virtual worlds. IEEE Comput Graph Appl. 2001;21:64–70.
12. Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B. Recent advances in augmented reality. IEEE Comput Graph Appl. 2001;21:34–47.
14. Lia H, Paulin G, Yeo CT, et al. HoloLens in suturing training. In: Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling. 2018:Bellingham, Wash: International Society for Optics and Photonics; 1057628.
15. Hanna MG, Ahmed I, Nine J, Prajapati S, Pantanowitz L. Augmented reality technology using Microsoft HoloLens in anatomic pathology. Arch Pathol Lab Med. 2018;142:638–644.
16. Kuhlemann I, Kleemann M, Jauer P, Schweikard A, Ernst F. Towards X-ray free endovascular interventions: Using HoloLens for on-line holographic visualisation. Healthc Technol Lett. 2017;4:184–187.
17. Oren MT, Hayeem LR, Aaron L, et al. Mixed reality with HoloLens: Where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg. 2017;140:1066–1070.
18. Mitsuno D, Ueda K, Itamiya T, Nuri T, Otsuki Y. Intraoperative evaluation of body surface improvement by an augmented reality system that a clinician can modify. Plast Reconstr Surg Glob Open 2017;5:e1432.
19. Ueda K, Hirota Y, Mitsuno D, et al. Three-dimensional, computer-assisted, three-layer models of the face. Plast Reconstr Surg. 2018;141:199e–200e.
20. Durrant-Whyte H, Bailey T. Simultaneous localization and mapping: Part I. IEEE Robot Autom Mag. 2006;13:99–110.
21. Badiali G, Ferrari V, Cutolo F, et al. Augmented reality as an aid in maxillofacial surgery: Validation of a wearable system allowing maxillary repositioning. J Craniomaxillofac Surg. 2014;42:1970–1976.
22. Qu M, Hou Y, Xu Y, et al. Precise positioning of an intraoral distractor using augmented reality in patients with hemifacial microsomia. J Craniomaxillofac Surg. 2015;43:106–112.
23. Wang H, Wang F, Leong AP, Xu L, Chen X, Wang Q. Precision insertion of percutaneous sacroiliac screws using a novel augmented reality-based navigation system: A pilot study. Int Orthop. 2016;40:1941–1947.
24. Mahmoud N, Grasa ÓG, Nicolau SA, et al. On-patient see-through augmented reality based on visual SLAM. Int J Comput Assist Radiol Surg. 2017;12:1–11.
25. Kilgus T, Heim E, Haase S, et al. Mobile markerless augmented reality and its application in forensic medicine. Int J Comput Assist Radiol Surg. 2015;10:573–586.