XR Technologies to Advance Maintenance Training

While extended reality (XR) technologies – the new catch-all designation for VR, AR, MR and other such emerging “realities” – are seeing widespread commercial applications to entertainment, education and manufacturing, the aviation industry has also started to adopt them as well. Chuck Weirauch explores what is available.

Boeing is notably employing AR for some areas of aircraft assembly such as wiring, while Pratt & Whitney and United Technologies have teamed up to invest in VR-based engine maintenance training for airline mechanics. According to MRO Network, Kevin Deal, VP of Aerospace and Defense for aerospace software company IFS, MROs are starting to use AR and VR to “help bridge the aircraft maintenance technician skills shortage gap.” Japan Airlines is employing the Microsoft HoloLens AR headset to train engine mechanics and flight crew members.

To take a further look into the potential, success and future of XR technologies applications for aircraft maintenance training, CAT solicited input from some leading aviation training experts, a university aviation educator and a manufacturer of visualization technologies. The list includes Boeing’s Pete Boeskov, Chief Technologist for Training and Professional Services; Jason Thompson, Business Development Manager for L3 Commercial Training Solutions; Kirk Thorson, Program Manager of Training Services for Rockwell Collins; Cory Frost, Rockwell Collins Principal Marketing Manager; Fernando Petruzziello, Co-Founder of Imagine 4D; and Lori Brown, Associate Professor for Western Michigan University’s College of Aviation.

CAT: Has your company been involved in the development of training tools for aviation maintenance training that employ virtual reality, augmented reality or mixed reality technologies? And if so, how?

“Extended reality (XR) has burst onto the scene in the last few years, offering us amazing visualization possibilities in a myriad of application areas along the aerospace value chain.” Boeskov said. “Boeing is exploring and leveraging these emerging technologies in all of its business units. Applications include delivery of work instructions for assembly, repair and maintenance, training and design.”

According to Brown, Western Michigan University (WMU) has created a suite of interactive virtual aircraft systems tutorials which can be delivered on a PC, tablet or in virtual, augmented, and mixed reality. These tools are used in the basic and advanced aircraft systems laboratory courses at WMU.

Petruzziello’s company has developed Station IX, a round VR-based self-contained virtual classroom system that can project an up to 300-degree field of view, where students are encapsulated in a VR environment. He describes the product as the “world’s largest headset” - one that can be employed for aviation maintenance training.

CAT: Are you currently employing such tools for aviation maintenance training? Again, if so, in what manner?

According to Boeskov, Boeing has experience in creating a variety of XR systems for pilots, maintenance personnel, and crew. Currently the company is collaborating with customers continuously to understand their needs and to define the right XR applications for a given situation.

“We are evaluating complex learning objectives in maintenance training and troubleshooting and developing proof of concept applications that have received positive customer feedback,” Boeskov explained. “We started by taking one of the most complex learning objectives in our maintenance training offering - the landing gear. What makes it so difficult is that when the landing gear is up, a student can’t see exactly how it articulates and how it’s positioned, since the landing gear doors are closed.”

“So, we created the AR experience to show the most common issues that may arise and how they would approach troubleshooting. When we created this experience and shared it at our events with airline customers, they immediately saw the value in how this helps to drive competency. We are developing a full spectrum of capabilities that will enable us to efficiently develop and deploy these new XR applications to our customers and improve competency of maintainers.”

According L3’s Thompson, his company has been exploring development and use of immersive technologies for aviation maintenance and aviation maintenance training for some time now. Through discussions with airlines, training organizations and other aviation maintenance professionals, he reported that the training provider has identified key areas in training, as well as on-the-job uses, that are suited for incorporation of immersive technologies that include virtual and augmented reality training products.

“Virtual reality tools are well positioned to assist with aviation maintenance training,” Thompson said. “Students, particularly the digital native generation that is now entering the aviation maintenance industry, expect training that is engaging and provides exposure to scenarios. We have heard of the challenges faced by airlines and schools in obtaining access to aircraft to permit hands-on, practical experience.”

Thorson reported that Rockwell Collins has conducted “a couple of demonstrations” for VR-based maintenance training, specifically for LRU removal.

“We send instructors all over the world training on Rockwell Collins equipment,” Thorson said. “And there are times when we do not have access to certain assets, or the assets are flying, so technicians are not being trained. The lack of equipment availability is really a sweet spot for virtual, augmented and mixed reality. We want to at least have the student experience the procedure prior to even getting on the aircraft. There is nothing more that makes a technician insecure than not having done it. So, if we can - we will put people in VR inside an airplane. This technology is only going to get better and improved, so such capabilities are going to be added to training applications.”

“We think that maintenance training is absolutely an underserved market,” Frost pointed out. “It’s not something we can do with just specific scenarios. An instructor can help students visually through it. We see a big market for these XR technologies for learning tasks.”

CAT: How do you view such products; as a viable means to enhance learning and increase productivity, or just as the latest technology product that may or may not become a useful addition to the aviation maintenance technician curriculum?

“These products will absolutely be a viable means to enhance learning and increase productivity for the aviation maintenance technician curriculum,” Boeskov said. “We see this as more than just the introduction of a new piece of technology into a training curriculum. It’s a catalyst for change in terms of how we think about training and learning overall. Coupling cross-reality (XR) technologies with mobility initiatives and data analytics will transform how, when and where students learn and how they grow through more personalized learning. XR is one part of that solution.”

Thompson sees the primary usage for immersive technology tools to address training and maintenance on transport category aircraft. The maintenance training tools that L3 has developed include full aircraft interactive 3D models, which are currently deployed in conventional 2D displays and are being leveraged in a 3D environment for research into immersive technologies. L3 has been in contact with several airlines and training providers that have expressed interest in incorporating immersive technologies into their programs and continues to develop capabilities to meet these industry requirements, Thompson added.

Brown stated that the Title 14 Code of Federal Regulations (CFR) part 147, the regulation dictating aviation maintenance education curriculum requirements, hasn’t been significantly updated since 1962.

“Now we have the opportunity to revolutionize the education of aviation technicians,” she pointed out. “The use of 3D modeling can reduce the traditional use of labs, while increasing retention and comprehension of the material. Some of the curriculum could be delivered through online courses, utilizing interactive 3D with less need for hands-on time. Traditionally AMT students complete 2,000 hours of instruction.”

“With interactive augmented reality, students can see internal functions of the components, and have the opportunity to practice outside of the lab, just as medical students do,” Brown related. “Additionally, this allows the training provider to allow the trainee to see technical manuals, checklists and step-by-step instructions on the training screen to decrease heads-down time and loss of situational awareness. Even in remote situations, an instructor can provide virtual feedback to a student by giving virtual pointers to where and how they should be performing a certain task.”

CAT: Which technology do you feel has the greatest potential to become the most valuable as a learning aid for aviation maintenance technicians?

“Based on feedback we have received from industry, we believe that augmented reality will provide the highest level of value to aviation maintenance,” Thompson stated. “Augmented reality has the potential to provide benefits not only as a learning aid, but more than that, we believe that it can be used for continuous learning and as an aid to be utilized throughout a mechanics career, moving beyond training tools that will only be used for initial or recurrent training.”

CAT: Considering the shortage of qualified aviation maintenance technicians, do you feel that the learning and training applications of such technologies will help improve the skill levels of the technicians and/or the throughput for AMT schools?

“Just by having that technology, and having taught AMT schools before, I think that in five years, you are going to see this type of technology entering into these schools,” Thorson predicted. “From an OM perspective, we see it as a huge benefit just from the fact that there are some cases where we cannot get them a hands-on experience. And this will allow students to get familiarized with the equipment in that compartment of that specific aircraft.”

“With this XR technology, the students have been there before,” Thorson continued. “With these AR, VR and MR-based training applications, there is visualization, there is the theory of operation that they normally struggle with because they can’t see the electrons. But this is bringing a visual environment to them and allowing them to experience it in a different way, so that they can get some understanding such as fluid flows and all of the other factors that we used to have mock-up boards for. There are plenty of opportunities - it’s just a matter of what training problem you are trying to resolve, and if there is a gap in your training curriculum.”

CAT: What are the major challenges of 1) developing such products, and 2) gaining their acceptance as practical learning and training tools?

“Because augmented and virtual reality are at the forefront of emerging technologies, there hasn’t yet been any best practices truly defined to handle design,” Thompson pointed out. “You really have to be innovative on how you execute development.”

“There definitely is merit in increasing the immersion of training, I don’t think anyone will argue with that,” he continued. “Some of the challenges in gaining its acceptance come down to physical and technological obstacles. Take resolution, for example. The same goes for text. These abnormalities create a level of discomfort that can disengage the user. Physical interaction, such as tactile feedback, is also a challenge. Of course, as technology improves then so will the ability to develop higher caliber products that capture and hold users. So, while there are some barriers in acceptance at the moment, they will diminish as the frontier expands.”

“The main challenge is the development of proper content and appropriate viewing technology to make the experience of displaying digital images fully immersive,” Petruzziello summed up. “Until an environment is put forward that provides the sensation of full immersion, it will be challenging for these types of products to be readily adopted. What needs to be understood by the community is that tools are being developed, and that it will take time to have a fully immersive environment that will be effective as a learning aid.”

“The next generation of air transportation training requires training which is engaging, cost effective, mobile and reconfigurable with real-world environments and equipment,” Brown concluded. “Aviation maintenance and inspection requires an understanding of several interrelated human and machine components requiring practice and immersion. With the use of virtual and augmented reality, we can provide hands-free, step-by-step instructions and feedback in front of the trainee, whether they're using a mobile device or augmented reality glasses. Virtual environments allow us to create new equipment and experiences the student can interact with. Augmented reality allows the student the opportunity to walk inside of the component such as an aircraft engine. This can enhance the current training.”

Published in CAT issue 4/2018