Embracing Extended Reality for a New Era in Flight Training: Promise and Pitfalls

By: Stephanie Fussell, PhD

Extended Reality (XR), which encompasses Mixed, Virtual, and Augmented Realities, is a hot topic these days, not just for its promise but also for its potential pitfalls.

XR can enhance various aspects of aviation training. It can bring in different sensory cues that traditional simulation lacks to make training more immersive and real, or superimpose and overlay content onto the environment to make information more accessible and understandable. It can enable interaction through voice and gestures to reduce cognitive load.

Within XR you can do all these things. But like any newer technology, XR also raises questions on how it should be used and integrated into existing training frameworks.

When adding XR technology into a simulator, what will need to change? How will it affect various human factors and trainee behavior? Will it actually improve learning and skills development? What does the research show?

Many of these questions and considerations are being addressed by the International Civil Aviation Organization (ICAO) Personnel Training and Licensing Panel, which includes people who want to see XR used in aviation training applications—but used thoughtfully and correctly. These include not only regulators from the different civil aviation authorities, such as the FAA, the EU Aviation Safety Agency, and the International Air Transport Association, but also industry and manufacturers such as Boeing, Airbus and CAE.

At a recent meeting of the Flight Simulation Training Devices & Simulation Training Devices Working Group, a variety of XR use cases and applications were presented, along with some of the different implementation challenges, and the work and research being done to advance XR’s use.

Now that companies are creating new XR devices and XR-enabled training systems that they need to qualify and certify, a key challenge remains: How do we integrate these devices into today’s existing training systems, or use them as stand-alone training technology?

One area the FSTD Working Group is focused on is developing guidance on how to incorporate XR technologies into current training paradigms. This guidance, which will include best practices and considerations to mitigate cybersickness, will address how XR should be added to existing frameworks that flight schools worldwide use for training. Another focus relates to the technical aspects of the XR devices themselves, which impact user experience, such as camera quality, latency, field of view and field of regard. The Working Group is creating a minimum requirements checklist to ensure devices meet certain recommendations, which could help smooth the certification process for products that meet them or require an explanation for those that don’t.

XR’s General Usability and Impact on Fidelity

Many of the conversations around XR center around its usability and ergonomic integration. And not just the physicality of sitting in a simulator with a headset on, but the environment as a whole, and the importance of designing a safe and effective virtual environment.

When adding XR to an existing training system, it must be done in a way that the system is still usable and safe. A pilot wearing a headset in a confined cockpit, for example, shouldn’t hit the physical space around them when leaning forward.

Then there are a host of cognitive considerations that come into play with XR and the different types of fidelity.

One is conceptual fidelity, which is how accurately the simulation reflects the real world. This is obviously important because pilots develop mental models through their training. If a simulated cockpit or checklist isn’t accurate, they can learn procedures that don't transfer to the aircraft. Aviation regulations recognize the discrepancy between learning to ‘fly in a sim’ and learning to ‘fly in the air,’ however, with the addition of XR, some of those standards will need to be re-evaluated to determine how the technology impacts them.

Another consideration is physical fidelity, which is the realism of the simulation’s appearance and feel. Flight training relies not just on visual information but also sense of touch and feel. This is particularly true with proprioception, which is the sensation of the body’s position and movement in space. As an aircraft banks, rolls, and ascends or descends, the vestibular system is stimulated.

This is where the use of motion, or inability to use motion in training can be problematic. Motion platforms can be costly and hard to maintain, and so there’s interest in whether XR can mimic those conditions. While Mixed Reality and haptics have advanced, there are limitations with the technology, and adding XR into existing devices is more complicated than many realize.

A particular problem is simulator sickness and cybersickness. Some of that can be attributed to technical matters such as the field of view and field of regard, but work is being done to better understand the factors that impact cybersickness, and what can be done to mitigate it.

One option being considered is the use of galvanic vestibular stimulation (GVS), which stimulates the vestibular part of the ear to create a sense of motion. Research is looking at whether GVS could be used with XR devices to help manage cybersickness, either to reduce it or alleviate it altogether.

The research is also looking at whether the use of GVS can also enhance learning. For example, if a pilot can now sense motion and experience the roll of a turn, will GVS reduce their response time and improve performance? If they should be in a turn but sense they’re not, will they maneuver more responsively now to correct the flight?

Research is being done in other areas as well. Several studies have shown positive results for the use of XR in flight training, demonstrating effectiveness for basic skill acquisition and procedural training. Some of the more complex behaviors, however, have yet to be tested, and so there are still gaps in the body of knowledge.

Overall, XR offers plenty of promise.

If a training environment that uses XR is well-designed, then immersion, engagement and motivation should increase, which can lead to enhanced learning and skills development. And if mastery occurs faster, perhaps less time will be needed in training and the overall pipeline will decrease with students getting through faster.

With continued research into XR and a focused understanding on how to thoughtfully integrate it into aviation training and current frameworks, the path to these benefits will be more assured.

(Editor’s note: Stephanie G. Fussell, PhD, is a Scientist at Aptima, Inc., aeronautics professional, and a proponent of using extended reality (XR) efficiently and effectively. Halldale Group has followed her previous efforts in the S&T community by way of our companion department MS&T. We are most pleased to gain her contributions for CAT. Her research concentrates on ensuring training competency, enhancing usability, meeting user experience (UX) requirements, and demonstrating transfer of training. At Aptima, she has worked on projects supporting the Air Force Research Laboratory, NASA, Naval Air Warfare Training Systems Division, and Air Education and Training Command.