Secondary Logo

Journal Logo

Original Article - Evaluative Study

Mixed Reality in the Reconstruction of Orbital Floor: An Experimental and Clinical Evaluative Study

Rahimov, Chingiz R.,; Aliyev, Daniz U.; Rahimov, Nurmammad R.1; Farzaliyev, Ismayil M.

Author Information
Annals of Maxillofacial Surgery: Jan–Jun 2022 - Volume 12 - Issue 1 - p 46-53
doi: 10.4103/ams.ams_141_21
  • Open



Orbits are one of the most common sites of midface fractures which account for approximately 40%.[1] These fractures are often associated with changes in orbital anatomy[23] that leads to changes in its pattern and volume. As a result, the orbital dimensions could be compromised by shifting the position of intraorbital contents, resulting in diplopia, enophthalmos, vision, and aesthetic disturbances.[24]

Nowadays for the improvement of functional and aesthetic outcomes, medical rapid prototyping (MRP) and intraoperative navigation systems are being introduced to orbital reconstruction.[238] Particularly, the application of MRP technology allows understanding of patient’s anatomy, perform orbital implants, and preplanning surgical procedures.[27] An intraoperative navigation system can use the radiologic and preoperative preplanning data as a map, to help position the reconstruction materials accurately, and decrease the rate of complications.[48]

Recently a new trend of mixed reality (MR) technology was introduced in different fields of surgery as pilot research. As a surgical navigation tool MR-based technology has been used for orthognathic surgery, face contouring, bone tumour resection, and neurosurgery.[131415161718]

The current study aims to demonstrate the possibilities of the application of MR technologies in orbital floor reconstruction.


The current study was performed at the Department of Oral and Maxillofacial Surgery during the period of January 2017 to December 2019. The study design included both experimental and clinical implementation of holographic technology for the improvement of orbital floor reconstruction (all procedures performed in the study were conducted under the ethical standards given in the 1964 Declaration of Helsinki, revised in 2013, as well as the Ethical Committee of Azerbaijan Medical University, Faculty of Dentistry Protocol N 15).

A total number of 10 participants who are residents in the same department and 5 experienced maxillofacial surgeons were added to the experimental part of the study. All data and customised software were well documented and then used in a single clinical case of orbital floor reconstruction. The patient consent for the application of MR technology and publication was obtained before the procedure.

Experiment design

To achieve a “proof of concept” state of the study one proposed preclinical experiment. The random computed tomography (CT) scan data delivered from the database was imported to Mimics Medical 17.0 (Materialise, Belgium) software for further virtual simulation. Inclusion criteria were CT scans of the patients who were scanned for different reasons, while exclusion criteria included any pathology of the facial skeleton. The general aim was to simulate orbital floor reconstruction. Therefore, an artificial defect in the right orbit was created. Within the simulation, a virtual protocol that included the generation of a virtual orbital plate was conducted [Figure 1a-f]. All virtual procedures were done within 3-Matic Medical 9.0 (Materialise, Belgium) software.

Figure 1:
Virtual simulation of orbital reconstruction: (a) importing of patient’s CT scan data; (b) cropping and simplifying of the virtual model; (c) creation of artificial defect in the orbital floor; (d) mirroring; (e) forming perimeter lines; (f) generation of the virtual orbital implant; (g) positioning of the virtual orbital implant-coronary view; (h) sagittal view; (i) 3D positioning; (j) generation of the scene; (k) reference points plate’s positioning assessment. CT: Computed tomography, 3D: Three-dimensional

Then virtual implant was loaded to Mimics Medical 17.0 software and appropriate positioning of the implant was achieved [Figure 1g and h]. The position and relationship to adjacent anatomical structures were double-checked in two-dimensional (2D) and three-dimensional (3D) views [Figure 1i].

As a next step, the obtained 3D data was used both for plastic model fabrication and virtual scene generation. The plastic model fabrication was done by Ultimaker Cura 4.1 software and Prusa i3 3D printer (Layer Height 0.06 mm, Infill 50%). The obtained plastic model was covered by a standard rubber mask. In addition, the double-sided tape was inserted within the region of the left orbital floor of the model to simulate plate fixation.

Generation of the virtual scene was done in Unity3d 2018.3.0 (Unity Technologies) by importing all components achieved from CT and virtual simulation [Figure 1j and k]. The main purpose of the developed software is to ensure the accuracy of the relationship between anatomical and virtual objects during the simulation. Custom C# scripts were used to control the simulation process. The user interface of the program includes two main components: one for turning on/off the virtual elements and the second to navigate virtual objects in 3D space and place them in the correct spot.

To achieve 3D mapping of the virtual object to a plastic model one used the principle of triangulation, which is the process of determining the location of a point by measuring only angles to it from the known points at either end of a fixed baseline, rather than measuring distances to the point directly as in trilateration. For these reasons, three cylinders were added both to the virtual and plastic model. The general idea was the superimposition of these cylinders in 3D space, and the achievement of accurate mapping.

Then all the data was loaded to Microsoft HoloLens MR headset (Microsoft Company, USA) to achieve 3D holograms in real space. The translocation and rotation of the holograms were realised by the usage of a clipper (a component of the HoloLens device) that works as a remote control.


The experiment’s scenario included the placement of a preformed orbital mesh implant in the region of a defected orbital floor similar to a real-time procedure. Ten participants were asked to use Hartman alligator ear forceps and Farabeuf retractor to place the orbital implant into the orbit through a preincised rubber mask. The visualisation of the orbital floor was similar to what one can achieve within real-time surgery. For fixation of orbital plate, one used double-sided tape that was placed in the region of the defect of the orbital floor (being buried into the defect and therefore without production of any additional eminences). The plate was simply stuck to tape; therefore, no additional fixation was needed. The evaluation criteria were the position of the rear plate as compared to preoperative planning data [Figure 2a-d]. The evaluation was performed by experienced maxillofacial surgeons, who were provided with a preoperative CT scan and postoperative plastic model with the orbital implant. In addition, virtual and real measurements of plate position were done by means of a virtual and real caliper. As reference points one used distance from the infraorbital rim, lacrimal crest, and the apex of the orbit, relative to the plate’s terminal corners [Figure 1l]. Statistical analysis was performed by Microsoft Excel 2013 and MedCalc 12.7 software (P < 0.03).

Figure 2:
Mixed reality aided surgical simulation: (a) mapping of the virtual hologram to plastic model; (b) insertion of the implant (side view); (c) insertion of the implant (see-through display view); (d) inserted implant


Experiment’s results

Subjective assessment of plate positioning revealed that the plate was positioned properly in 8 cases. The reason for inaccuracy in the rest of the 2 cases was the inability of participants to concentrate their attention on both holographic and real objects and the fact, that participants were inexperienced in gaming technology and augmented reality.

Objective assessment by the means of comparison of virtual and real measurements data showed inaccuracy in a range of 0.3–0.5 mm [Table 1 and Graph 1].

Table 1:
Summary statistics table
Graph 1:
Hologram of statistical analysis

Clinical case presentation

The 38-year-old male presented with the symptoms of slight enophthalmos and significant immobility of the left eyeball in inferior and medial quadrants [Figure 3]. Subjectively, diplopia in all directions and paraesthesia on the upper lip left side were noted. Upon anamnesis, he had blunt trauma to the left orbit a month ago.

Figure 3:
Clinical evaluation of eyeballs mobility: significant immobility was found in inferior and internal quadrants

CT scan revealed L-side “blowout” fracture [Figure 4a and b]. The inferior rectus muscle and medial wall of the left orbit were intact. 3D evaluation and orbital volume determination revealed its increase of up to 4.7 cc [Figure 4c]. Surgical reconstruction of the left orbital floor was chosen as a treatment option.

Figure 4:
Virtual planning workflow: (a and b) blowout fracture of the left orbit; (c) explanation of orbital volume; (d) tracing of perimeter and guiding lines; (e) virtual plate generation; (f) virtual measurements. (g) Positioning of the virtual orbital implant-coronary view; (h) sagittal view; (i) 3D positioning; (j and k) generation of the scene. 3D: Three-dimensional

According to the clinical protocol accepted in our department, the virtual planning was performed and the patient-specific orbital implant was fabricated and its position to adjacent anatomical structures was checked in 2D and 3D views [Figure 4d-i].

Thereafter patient-specific scene was generated within customised software like the one used for an experiment that was described earlier [Figure 4j and k].

Surgical procedure has been done under GA through the transconjunctival approach. After exposure of the infraorbital margin, an accurate dissection within the orbital floor was carried out. Prolapsed soft tissues were extracted and returned to the orbit. Thus, the orbital floor and defected region were completely visualised. Once it was done, HoloLens holographic headset was placed on the surgeon’s head and the virtual scene was activated. After the mapping procedure, which was realised with the help of a clipper device (which was wrapped in a sterile sheath) the virtual hologram was superimposed on the patient’s head. As a reference, anatomical landmarks of both virtual and real teeth of the patient, as well as infraorbital margin were used [Figure 5].

Figure 5:
The intraoperative wearing of the headset and see-through display view

Once the mapping procedure was completed the surgeon had the opportunity to verify the real orbital implant position through technology. In this step, the main task was the superimposition of the movable real patient-specific implant with a stable holographic implant. In this step, the surgeon was provided with the opportunity to change the intensity of hologram visualisation until the best fitting of real and virtual implants was achieved. Once superimposition was completed final fixation of the implant was done. On this step, the holographic smart glasses were taken off and a naked eye assessment of plate position as well as forced duction test was performed.

The postoperative course was smooth without complications. Soon after the surgical procedure, the patient showed diplopia related to the intra-periorbital oedema and temporary paresis of oculomotor muscles. A month later, the patient showed improvement in ocular mobility and reduction of diplopia. A postoperative CT scan a month after the procedure showed the plate to be placed properly in 2D and 3D sections [Figure 6]. However residual oedema was found in the region of medial and inferior rectal muscles. Therefore, the patient was referred to the Department of Ophthalmology for postoperative conservative treatment and observation.

Figure 6:
Postoperative CT scan of the patient. CT: Computed tomography


Orbital “blowout” fractures are one of the most frequent types of fractures among all midfacial injuries.[19] Typically, clinical symptoms include evident enophthalmos, dystopia, diplopia of more than 2 weeks duration, and a forced duction test. Without adequate treatment, these fractures are showing a tendency for malunion and further changes in the position and integrity of the soft tissues.[20] Orbital reconstruction of its normal anatomy and volume in these cases is mandatory for the further prevention of listed symptoms.[2] However, postoperative complications are not rare and include residual diplopia and enophthalmos, infraorbital nerve dysfunction, which could be associated with misposition of the reconstruction materials.[3]

Nowadays progress of digital technologies allows for a significant increase in the accuracy of orbital reconstruction by application of virtual preplanning and MRP.[2345678] Moreover, the application of navigation systems leads to improvement of intraoperative positioning of implant materials, thus improving accuracy and diminishing mean surgical time.[48]

However, the application of this technology could be related to several technical problems, such as the necessity to switch the surgeon’s attention between the display of the navigating device and the time for system installation and patient positioning.[9]

As opposed to the traditional “head’s up” approach recently visualisation of navigation map could be executed by see-through display technology.[9] The concept of a headset with a superimposed display was introduced by Ivan Sutherland to the military in 1965 and consisted of a head-worn display and an image generation subsystem. However, these devices were cumbersome, heavy, and expensive that limited their application for health-care needs.[1011] Recently introduced less costly headsets such as Microsoft HoloLens could be successfully implemented in real-time surgery.[12]

Several publications described the application of mixed or augmented reality technologies in the field of plastic and reconstructive surgery. One could classify the range of applications of these technologies in 3 main directions: planning, navigation, and training.

In the case of preoperative planning in most of the publications, the general idea was to use the haptic device, immersive workbench, 3D eyewear, or 3D displays to simulate real-time surgery, thus finding the best option for every clinical case. This approach was used for preplanning of orthognathic surgery,[21] facial contouring surgery,[22] mandibular reconstructive surgery,[2324] cleft lip repair surgery, and craniomaxillofacial complex fracture reduction.[2526] Described technology was also used in hip fracture reduction surgery,[27] femur fracture reduction surgery planning,[28] orthopaedic fracture reduction surgery,[29] orthopaedic burring simulation,[30] and spinal fixation simulation.[34] In all cases, preoperative planning was done in real-time mode by virtual interaction of the operator through a haptic device and 3D eyewear with computer software.

As opposed to the presented technologies in our study, the hardware was limited to single 3D eyewear and a plastic model of the patient. Surgical planning was done before simulation and included virtual implant generation, implant template, and plastic model fabrication. In our opinion, this approach could simplify virtual planning workflow and reduce expensive hardware, such as haptic devices.

In cases of surgical navigation in most publications, the general principle included the usage of 3D patient mesh models that were reconstructed from CT or magnetic resonance imaging data and integrated into different AR devices, such as head-mounted displays, interactive portable displays with a camera, or tablet PC with an embedded camera. This technology was implemented for orthognathic surgery,[1314] mandibular angle osteotomy,[15] and tracking of maxillofacial bone, nerves, and vessels by application of markerless AR-based technology.[16] This technology was also implemented in bone tumour resection in the pelvic region.[17] However, authors reported difficulty in switching AR and VR display images that provide the closest distance information from a real surgical tool and a 3D patient mesh model.[1831]

The authors report that on average, participants made errors of up to 5.9 mm in length versus 2.8 mm during the naked eye tasks.[31]

In our study, navigation was provided by the application of the triangulation method that most probably leads to less error, with a maximum magnitude of up to 1.75 mm. Moreover, intraoperative navigation was achieved by multiple points (teeth and infraorbital margin) superimposition that makes the 3D hologram more stable. In addition, the accuracy of surgical procedures could also be increased by the tactile sensitivity of an experienced surgeon which was proved in our case.

Finally, in cases of surgical training, authors used both AR and VR technologies that included the application of the different haptic devices, immersive workbench, 3D eyewear, or 3D displays. These approaches were used for training in orthognathic surgery.[3233] AR/VR surgical training was also used in different surgical specialties such as training in orthopaedic surgery[34, 35] and in the field of neurosurgery.[36]

In our study of 3D eyewear and plastic, the model is similar to what was implemented for surgical simulation and planning. The application of a haptic device was not necessary, due to the experiment scenario that makes such technology less costly and therefore more applicable.

Another challenging problem that is related to the implementation of MR technology in medicine is the realisation of 6 degrees of freedom (6-DOF) which has received increased attention in the past few years.[3738] The main aim of the 6-DOF principle is to provide object tracking of typically small objects, such as patient anatomy or surgical instrumentation.[39] This kind of approach of combining navigation principles with MR will significantly increase the accuracy of surgical procedures and reduce operating time.[40] However, these approaches require the existence of surgical navigation systems, which makes this technology expensive.


Implementation of MR technology in combination with CAD could significantly improve the results of preoperative planning, intraoperative navigation, and surgical training. However, existing technical limitations within described methods are still producing several mistakes and errors in the application of such approaches intraoperatively. Therefore, improvement of MR technology that should be focused on the 6-DOF problem solution could be considered a reasonable way of eliminating of listed limitations.

Declaration of patient consent

The authors certify that they have obtained all appropriate patient consent forms. In the form, the patient(s) has/have given his/her/their consent for his/her/their images and other clinical information to be reported in the journal. The patients understand that their names and initials will not be published and due efforts will be made to conceal their identity, but anonymity cannot be guaranteed.

Financial support and sponsorship

This work was financially supported by the Ministry of Transport, Communications, and High Technologies of the Azerbaijan Republic.

Conflicts of interest

There are no conflicts of interest.


1. Cha JH, Lee YH, Ruy WC, Roe Y, Moon MH, Jung SG Application of rapid prototyping technique and intraoperative navigation system for the repair and reconstruction of orbital wall fractures Arch Craniofac Surg 2016 17 146 53
2. Choi JW, Kim N Clinical application of three-dimensional printing technology in craniofacial plastic surgery Arch Plast Surg 2015 42 267 77
3. Ramponi DR, Astorino T, Bessetti-Barrett CR Orbital floor fractures Adv Emerg Nurs J 2017 39 240 7
4. Susarla SM, Duncan K, Mahoney NR, Merbs SL, Grant MP Virtual surgical planning for orbital reconstruction Middle East Afr J Ophthalmol 2015 22 442 6
5. Mourits DL, Wolff J, Forouzanfar T, Ridwan-Pramana A, Moll AC, de Graaf P, et al. 3D orbital reconstruction in a patient with microphthalmos and a large orbital cyst-A case report Ophthalmic Genet 2016 37 233 7
6. Beliakin SA, Khyshov VB, Khyshov MB, Klimova NA, Saĭfullina SN, Éĭzenbraun OV Reconstruction of posttraumatic skull and facial bones injuries with the use of perforated titanium plates and meshes Voen Med Zh 2012 333 12 7
7. Rahimov CR, Ahmadov SG, Rahimli MC, Farzaliyev IM Three-dimensional diagnosis in orbital reconstructive surgery Ann Maxillofac Surg 2020 10 3 9
8. Kormi E, Männistö V, Lusila N, Naukkarinen H, Suojanen J Accuracy of patient-specific meshes as a reconstruction of orbital floor blow-out fractures J Craniofac Surg 2021 32 e116 9
9. Khor WS, Baker B, Amin K, Chan A, Patel K, Wong J Augmented and virtual reality in surgery-the digital surgical environment:Applications, limitations and legal pitfalls Ann Transl Med 2016 4 454
10. Sutherland IE The ultimate display Proc IFIP Congr 1965 2 506 8
11. Curtis D, Mizell DW, GruenbaumPE, Janin AL Several Devils in the Details:Making an AR Application Work in The Airplane Factory Natick A. K. Peters Ltd 1999
12. Kim Y, Kim H, Kim YO Virtual reality and augmented reality in plastic surgery:A review Arch Plast Surg 2017 44 179 87
13. Badiali G, Ferrari V, Cutolo F, Freschi C, Caramella D, Bianchi A, et al. Augmented reality as an aid in maxillofacial surgery:Validation of a wearable system allowing maxillary repositioning J Craniomaxillofac Surg 2014 42 1970 6
14. Park JH, Lee YB, Kim SY, Kim HJ, Jung YS, Jung HD Accuracy of modified CAD/CAM generated wafer for orthognathic surgery PLoS One 2019 14 e0216945
15. Lin L, Shi Y, Tan A, Bogari M, Zhu M, Xin Y, et al. Mandibular angle split osteotomy based on a novel augmented reality navigation using specialized robot-assisted arms-A feasibility study J Craniomaxillofac Surg 2016 44 215 23
16. Wang J, Suenaga H, Yang L, Kobayashi E, Sakuma I Video see-through augmented reality for oral and maxillofacial surgery Int J Med Robot 2017 13 10.1002/rcs.1754
17. Choi H, Park Y, Cho H, Hong J Augmented Reality-Based Simple Navigation System for Pelvic Tumor Resection Proceedings of the 11th Asian Conference on Computer-Aided Surgery (ACCAS 2015); 2015 Jul 9-11 Singapore 2015
18. Choi H, Cho B, Masamune K, Hashizume M, Hong J An effective visualization technique for depth perception in augmented reality-based surgical navigation Int J Med Robot 2016 12 62 72
19. Garg V, Girardi GB, Roy S Comparison of efficacy of mandible and iliac bone as autogenous bone graft for orbital floor reconstruction J Maxillofac Oral Surg 2015 14 291 8
20. Grob S, Yonkers M, Tao J Orbital fracture repair Semin Plast Surg 2017 31 31 9
21. Fushima K, Kobayashi M Mixed-reality simulation for orthognathic surgery Maxillofac Plast Reconstr Surg 2016 38 13
22. Tsai MD, Liu CS, Liu HY, Hsieh MS, Tsai FC Virtual Reality Facial Contouring Surgery Simulator Based on CT Transversal Slices. Proceedings of the 5th International Conference on Bioinformatics and Biomedical Engineering; 2011 May 10-12 Wuhan, China 2011 1 4
23. Rahimov CR, Farzaliyev IM, Fathi HR, Davudov MM, Aliyev A, Hasanov E The application of virtual planning and navigation devices for mandible reconstruction and immediate dental implantation Craniomaxillofac Trauma Reconstr 2016 9 125 33
24. Olsson P, Nysjö F, Rodríguez-Lorenzo A, Thor A, Hirsch JM, Carlbom IB Haptics-assisted virtual planning of bone, soft tissue, and vessels in fibula osteocutaneous free flaps Plast Reconstr Surg Glob Open 2015 3 e479
25. Olsson P, Nysjö F, Hirsch JM, Carlbom IB A haptics-assisted cranio-maxillofacial surgery planning system for restoring skeletal anatomy in complex trauma cases Int J Comput Assist Radiol Surg 2013 8 887 94
26. Zhang J, Li D, Liu Q, He L, Huang Y, Li P Virtual Surgical System in Reduction of the Maxillary Fracture Proceedings of the 2015 IEEE International Conference on Digital Signal Processing (DSP); 2015 Jul 21-24 Singapore 2015 1102 5
27. Kovler I, Joskowicz L, Weil YA, Khoury A, Kronman A, Mosheiff R, et al. Haptic computer-assisted patient-specific preoperative planning for orthopedic fractures surgery Int J Comput Assist Radiol Surg 2015 10 1535 46
28. Cecil J, Ramanathan P, Rahneshin V, Prakash A, Pirela-Cruz M Collaborative Virtual Environments for Orthopedic Surgery. Proceedings of the 2013 IEEE International Conference on Automation Science and Engineering (CASE); 2013 Aug 17-20 Madison, WI 2013 133 7
29. Shen F, Chen B, Guo Q, Qi Y, Shen Y Augmented reality patient-specific reconstruction plate design for pelvic and acetabular fracture surgery Int J Comput Assist Radiol Surg 2013 8 169 79
30. Chan S, Li P, Locketz G, Salisbury K, Blevins NH High-fidelity haptic and visual rendering for patient-specific simulation of temporal bone surgery Comput Assist Surg (Abingdon) 2016 21 85 101
31. Condino S, Carbone M, Piazza R, Ferrari M, Ferrari V Perceptual limits of optical see-through visors for augmented reality guidance of manual tasks IEEE Trans Biomed Eng 2020 67 411 9
32. Wu F, Chen X, Lin Y, Wang C, Wang X, Shen G, et al. Avirtual training system for maxillofacial surgery using advanced haptic feedback and immersive workbench Int J Med Robot 2014 10 78 87
33. Lin Y, Wang X, Wu F, Chen X, Wang C, Shen G Development and validation of a surgical training simulator with haptic feedback for learning bone-sawing skill J Biomed Inform 2014 48 122 9
34. Seah TE, Barrow A, Baskaradas A, Gupte C, Bello F Avirtual reality system to train image-guided placement of Kirschner-wires for distal radius fractures Bello F, Cotin S Biomedical Simulation: 6th International Symposium, ISBMS 2014, 2014 Oct 16-17 Strasbourg, FR. Cham Springer International Publishing 2014 20 9
35. Thomas GW, Johns BD, Kho JY, Anderson DD The validity and reliability of a hybrid reality simulator for wire navigation in orthopedic surgery IEEE Trans Hum Mach Syst 2014 45 119 25
36. Alaraj A, Luciano CJ, Bailey DP, Elsenousi A, Roitberg BZ, Bernardo A, et al. Virtual reality cerebral aneurysm clipping simulation with real-time haptic feedback Neurosurgery 2015 11 Suppl 2 52 8
37. Kehl W, Tombari F, Ilic S, Navab N Real-Time 3D Model Tracking in Color and Depth on a Single CPU Core In:IEEE Conference on Computer Vision and Pattern Recognition 2017
38. Tan DJ, Navab N, Tombari F Looking beyond the simple scenarios:Combining learners and optimizers in 3D temporal tracking IEEE Trans Vis Comput Graph 2017 23 2399 409
39. Garon M, Lalonde JF Deep 6-DOF tracking IEEE Trans Vis Comput Graph 2017 23 2410 8
40. Liebmann F, Roner S, von Atzigen M, Scaramuzza D, Sutter R, Snedeker J, et al. Pedicle screw navigation using surface digitization on the Microsoft HoloLens Int J Comput Assist Radiol Surg 2019 14 1157 65

Augmented reality; blowout fracture; orbit; reconstructive surgical procedure

Copyright: © 2022 Annals of Maxillofacial Surgery