AR/MR/VR/XR as Training Disruptors

Halldale Group Editor Marty Kauchak surveys the latest A/M/V/XR hardware developments and applications in civil aviation pilot training.

There are few instances when CAT and MS&T writers use the words ‘transformational’ and ‘disruption’ across the broadening simulation and training industry. But transformation and disruption are among the descriptors to convey the developments in augmented, mixed, and virtual reality. In use cases where ‘eXtended realities’ are being brought to bear, the plate tectonics of instructional design are shifting, with evidence of increased student performance and other returns on investment. Beyond traditional classrooms and other learning venues, this technology sector’s ecosystem is further expanding as the number of suppliers of products and services, and their portfolio breadth, increase.

It’s also necessary to manage expectations for this still nascent, emerging technology space. While some ATOs and other end users are beginning to use A/M/V/XR-enabled learning, others have simply not invested in content and systems designed to deliver this mode of instruction. At the same time, content suppliers in this market space are continuing to bend the laws of physics and make other adjustments, to overcome technology challenges and other impediments to support a wider use of A/M/V/XR tools in customers’ programs.

Training Domain Limitations

Current capabilities for instructional design and other learning practices vary, are evolving and in some instances merging in the domains of pilot, maintenance technician, and cabin crew training.

Rick Parker, Co-Founder and CTO of VTR (Visionary Training Resources, Woodlands, Texas) observed that his team has “found that AR still has too many device-physical limitations to make its use practical for airline pilot training, yet there is real potential for this to change in the future. VR and MR have already demonstrated their value in military pilot training. The choice between the two really depends on the training objective.”

Thor Paulli Andersen, CTO at VRpilot (Lystrup, Denmark), said his company’s focus is VR, rather than MR or AR, as both require a physical environment to interact with. “With VR, everything is represented virtually, not requiring, e.g., a physical cockpit, which would reduce flexibility and increase cost of a training system,” he said and continued, “The immersiveness of VR is also much greater than AR, which gives a better training experience. From a hardware standpoint, the AR technology is still in its infancy with VR being mature for professional training use.”

VRpilot's scenarios include multi-crew training for the A320. Image credit: VRpilot.

Democratization-Enabling Tools

While the S&T industry is fielding a cascading number of A/M/V/XR products and enabling technologies, there is nothing less than a technical transformation occurring in HMDs and other content with huge outcomes. In one case, unit prices for these articles are decreasing to permit a “democratization”, or increased affordability, of this content for more organizations, and the suppliers of these products are expanding technology envelopes. As HMDs and other articles become less ‘klunky’ and easier to use, industry is improving other aspects of the user experience – reducing eyestrain and other impediments restricting or limiting equipment usage.

The following is a snapshot, certainly not a definitive list, of products available to pilot training enterprises. As this is a dynamic market, end users often indicate they remain agnostic in the content they are integrating into training programs, resulting in different brands populating a training organization.

Varjo VR-3, XR-3 and Aero. John Burwell, Global Head of Simulation and Training at Varjo (Helsinki, Finland), said his company’s VR-3 HMD is a preferred choice for VR training as it provides human-eye resolution so pilots can see cockpit displays and objectives as they can in the real world. He added, “Mixed-reality training is enabled using Varjo’s XR-3, which combines the human-eye resolution of the VR-3 with high-resolution digital cameras that seamlessly blend video of the outside world with computer-generated content. This enables tactile feedback so pilots can see their hands and feel the stick and throttle and buttons/switches that populate the crew station.”

Varjo announced in October the release of Varjo Aero – worn by the Halldale Media team for the first time at 2021 I/ITSEC. Burwell offered that, where Varjo has traditionally focused on demanding, enterprise-class customers that require the highest image quality available, the Aero is “our first device offered for more value-conscious customers who still want the high quality, but at a lower price point.” Accordingly, “The Varjo Aero fits into this ‘prosumer’ class offering the highest image quality and field of view at a price point under $2,000. This is an ideal device to support much of the basic flight and maintenance training that must be accomplished across the industry.”

Pico Neo 2 Eye/Neo 3 Pro Eye. Pico deferred CAT queries about its offerings to the commercial pilot training market to VTR’s Parker. The executive explained VTR’s FlightDeckToGo 737-800 product is currently in use with VA Airline Training in the UK for APS MCC [Airline Pilot Standard Multi-Crew Cooperation] students, using the Pico Neo 2 Eye. More significant, “We are in active discussions with numerous airlines and training providers in the EU and United States. We are currently transitioning all our content to the Pico Neo 3 Pro Eye” – and with good reasons, as VTR has found “the Pico Neo 3 Pro Eye currently offers the most effective training platform for the early-stage airline pilot curriculum due to its fully mobile capabilities, allowing the pilot to train anytime and almost anywhere.” The Pico website notes, in part, beyond Tobii eye tracking, the headset is “the world's first XR platform that unites 5G and AI.”

Microsoft HoloLens. Camille Elmenhurst, Microsoft Media Relations Team Member, noted in one use case, “Using Microsoft HoloLens, Japan Airlines (JAL) has developed two solutions to provide supplemental training for engine mechanics, and for flight crew trainees who want to be promoted to co-pilot status. With HoloLens, they have a detailed hologram in front of their eyes that will display cockpit devices and switches that they can operate themselves, with visual and voice guidance provided through HoloLens.”

HP Omnicept platform & HP Reverb G2 Omnicept Edition VR headset. These devices have the potential to further expand the VR technology envelope. Conor Driscoll, HP spokesperson, noted the company’s offering provides VR with biometric insights and a cost-effective approach to immersive flight simulation, which may be paired with platform inertia mechanics or not, depending on level of immersion and realism being sought. He added, “The most significant benefit will be after-action review. Being able to understand cognitive load in context of important actions and decisions that must be made in the moment during a simulation exercise. Being able to review afterwards and remember the stress or complexity of influence and how it relates to mental load provides opportunity for critical insights and learning. This combines with eye-tracking information, so attention and focus are documented along with level of mental exertion.” The HP Store website noted the VR headset cost an attention-getting $499.

JVCKenwood HMD-VS1W. The HMD-VS1W head-mounted display reportedly allows users to view virtual images while operating actual equipment through MR, which combines the real and virtual worlds. This MR-capable HMD is further said to enable applications including simulators. Two of the HMD’s attributes to bring state-of-the-art technology to learning scenarios is its 5K total horizontal resolution and FOV (field of view) 120 degrees horizontal and 45 degrees vertical.

No one supplier is able to claim it has cornered the airline training market’s evolving and numerous requirements. For instance, while VRpilot develops its software themselves, in the past the Danish firm relied exclusively on the Oculus hardware platform. Andersen pointed out, “We have expanded our hardware support to include all commonly available standalone VR headsets, such as the HTC Vive Focus 3 and Pico Neo 3 Pro. We currently also have a retail agreement for the HTC headset.”

VRpilot presented another vital point on the business side. This relatively new firm has a founding date of February 2019. The company’s main product is VRflow, the collaborative cockpit procedure training platform for VR. Andersen added, “We create aircraft- and operator-specific training scenarios for ab initio training, MCC courses, type ratings and recurring training. VRflow is license-based to enable continuous customer support. We deliver turnkey solutions to our customers, including hardware retail, software customization and support.”

JVCKenwood’s new HMD-VS1W capabilities include 5K total horizontal resolution and 120 x 45 field of view. Image credit: JVCKenwood.

Tools Beyond Headsets

Articles beyond headsets are helping to more fully immerse training audiences in scenarios.

Amanda Bentley, Senior Director of Sales and Business Development, Tobii Techhnology, noted A/M/V/XR-enabled pilot training has seen fast-growing interest and called attention to the adoption of eye tracking. “A good training experience depends on an insightful evaluation, a powerful VR/AR/XR headset, and an immersive training experience,” she noted and offered two examples of leveraging Tobii eye tracking in pilot training are Cineon Training and VTR.

The technology firm provides developers with support and tools like Tobii Ocumen, an analytics toolkit to observe human behavior in VR and a framework to record, organize and analyze data.

Tobii is allowing eye tracking to be used through a training scenario’s life cycle. In one instance, Tobii eye tracking allows developers to verify effectiveness before content development, and focus on content that drives results. Trainers are empowered with analytics and insights from eye tracking to evaluate and enhance the training and performance of trainees. “For trainees, they can improve their performance through an immersive training experience enabled by Tobii eye tracking,” the executive added.

Significantly, Tobii is casting a wide net for hardware and other partners to expand its business ecosystem. Bentley noted Tobii eye tracking allows for a high-performing headset without compromise in rich content or higher frame rate requirement through foveated rendering, and added, “It also helps standalone devices to run longer sessions with better battery life. The mainstream enterprise VR devices in the market, like HP Reverb G2 Omnicept Edition, Pico Neo 3 Pro Eye and Neo 2 Pro Eye, and Vive Pro Eye, are all integrated with Tobii eye tracking.”

Tobii eye tracking is used in Training Aircrew Competency (TACET) using Eye Tracking by UK company Cineon. Image credit: Tobii.

Best Practices: Flight Training

Training organizations and providers are able to articulate where VR is integrated in their curricula and provide current and evolving outcomes.

VRpilot strongly believes that VR is a powerful tool to increase the quality and flexibility of pilot training. “It also holds the potential of big cost savings, both on simulator and instructor resources and travel expenses as remote training is possible,” Andersen emphasized. “EASA is supportive of the emergence of immersive technologies in flight training, and various working groups at EASA are working on how to credit training tools like these in the future.”

VRpilot’s Andersen provided a number of other outcomes to justify his positive assessment of this sector’s future, pointing out his customers use VR training from ab initio through MCC courses, to type ratings and recurring training. The corporate executive pointed out this is a powerful training technology that enables faster learning and better knowledge retention. He added, “Using VR training will prepare the pilots better for simulator passes or real flying, as many learning objectives may be trained in a flexible and cost-efficient manner with VR, increasing the quality of the time spent in FSTDs or the real aircraft. Self-training VR scenarios have proven to save instructor resources as students are able to learn procedures, cockpit layouts, callouts, etc. by themselves.”

VRpilot’s current customers include military, ab initio schools and airlines. He noted, “We are expanding our customer portfolio to include training centers, such as BAA Training, with whom we signed a contract during EATS.”

They also signed Sunclass Airlines at Halldale/CAT’s EATS 2021 in Berlin. Sunclass will be using VRpilot’s virtual reality-based procedure trainer, VRflow, to support its A320s. The Copenhagen-headquartered carrier’s chief pilot and trainer, Nicolai Bondo Rasmussen, spoke to the use case and corresponding best practices for the newly introduced capability, initially explaining his airline will use Pico VR goggles and VRflow. “The setup will be used in a competency-based approach, where we want to improve the application of procedures, knowledge and leadership and teamwork, to allow for a cognitive spare capacity when the crew enter the full-flight simulator.” When CAT engaged with Rasmussen, the airline was expecting to soon recall a group of pilots, who have been absent from flying for almost two years – a case all too familiar. “VR will allow the pilots to train and rehearse in the virtual flightdeck. We believe this will prepare the pilots for a smooth transition back into the flightdeck and eventually result in better usage of the simulator time. We expect to see less need for extra training with this preparation.”

The airline training leader pointed out VR can support and instruct the pilots performing their procedures and flows. “We have an embedded instruction that will highlight the correct sequence in procedures and flows, tailored to both PF [pilot flying] and PM [pilot monitoring]. The pilots can enter the same virtual flightdeck from their respective homes, and thereby train together in both roles. We will assign an instructor to some of the sessions to support them and also to assess the level of competence before and after the use VR.”

An oft-noted observation CAT has heard is that A/M/V/XR will, for the foreseeable future, complement, and not replace, full-flight simulators. In one instance, Rasmussen emphasized, “The backbone of our high safety standard is the use of the full-flight simulator; however, I see a huge potential in the use of VR in a competency-based training approach,” and noted, “I believe we should have a talk about the right tool for the right task, e.g., cockpit preparation in a full-flight simulator occupies a substantial amount of time and forms a costly training environment. I further believe this could be performed in VR, allowing the pilots to make better use of the full-flight simulator. Low-visibility training could also be performed or some special airport training, where the focus is on local knowledge and awareness.”

VRpilot observed that despite the enormous training potential of A/M/V/XR, it doesn’t believe that it will replace full-motion Level D simulators. The company’s Andersen noted they expect immersive technologies to fulfill a significant part of the learning objectives of a type rating, with authorities (FAA, EASA) recognizing and crediting hours spent using these devices. “As the AR technology matures, we expect this technology to be a supplement in the actual simulator where visual aids and training guidance may be overlaid on the physical cockpit.”

VTR’s Parker elaborated that, as with most airline pilot training platforms, the most compelling value-add of VR, and for its purposes, the Pico Neo 3 Pro Eye, is in the early stages of new hire pilot and new aircraft transition training. He pointed out this stage involves aircraft systems and routine task training to bring pilots up to a minimum knowledge and proficiency level necessary to function and learn more advanced skills in an FSTD with a flight instructor. Parker added, “We can now train a pilot in actual systems operations, routine normal and non-normal procedures and validate their procedures proficiency in the FlightDeck before they arrive at their first simulator event. All this training is delivered anytime and anywhere on a fully mobile, untethered, no computer or internet required, virtual reality device.”

Parker emphasized the value-add is, “We can deliver an entire month of this new training capability, in most cases, for less than the cost of a single 4-hour full-flight simulator period and collect almost any KPI (Key Performance Indicator) data that airlines need.”

Training Device Provider Perspective

Randy Gawenda, Business Development Manager at Frasca International (now part of FlightSafety International), provided another key stakeholder’s insights on this technology space, first noting this Urbana, Illinois-based company thinks there is some growing demand in this sector, which is somewhat paced by the advancement in technology of various HMDs. “Some of that demand also depends on the size of the operator and the throughput, or quantity of student pilots needing to be trained at any given organization in conjunction with the rest of the training system. As the civil/commercial training experts on Halldale’s XR webinar pointed out, this realm is approaching these types of devices as supplements or enhancements to the more traditional training tools, not as substitutes or replacements. If lower-level tasks of orientation and familiarization can be done via a cheaper, yet effective method, then there is certainly value there. Those organizations can help streamline and become more efficient, potentially, by a thought-out, deliberate application of these newer training aids.”

Frasca reports it has delivered 10 “pure VR devices” that it modified to become more type-specific to the customer’s aircraft. “This included adding a high-fidelity flight model, main panel instrumentation relevant to the aircraft type, and added visual elements for the training area for the customer. We are close to finalizing a contract for another group of devices, this time with a mixed-reality solution.” More significant, Gawenda said, “By early 2022, we will have experience with both VR and MR types of training devices.” Beyond that forecast, Frasca reportedly has several, related on-going R&D projects in this domain to stay current with the evolving technology.

Challenges and Limitations

While the commercial aviation training community’s demand signal trends upward for more A/M/V/XR-enabled learning solutions, and business development opportunities emerge for companies as content and service suppliers, challenges and limitations remain. The good news is many stakeholders are focused on fixing these shortfalls.

Varjo has four technology thrusts in progress to address current shortfalls in the learner’s HMD experience.

Burwell first noted his company has invested “a tremendous amount into building HMDs that reduce or eliminate cyber sickness,” pointing out recent studies have shown that an improperly adjusted interpupillary distance (IPD) is the cause of 40% of all cyber sickness, so Varjo headsets come standard with a built-in eye tracker that automatically adjusts the IPD every time a user puts on the device.

A second human performance issue involves eye strain, which can be caused by forcing the visual system to integrate low-resolution scenes, especially when the content is displayed close to the eye. “Varjo devices minimize the effects of eye strain by displaying scenes at human-eye resolution,” the executive added.

Elsewhere, mixed-reality visual systems that rely on video captured by cameras can create visual anomalies, including double imaging and mis-matched scenes with virtual content. Burwell noted the Varjo video-pass-through capture system provides synchronized video to the GPU (graphics processing unit) in less than 20 msecs, enabling the seamless integration of real and virtual content. In addition, Varjo devices support time-warp capabilities to compensate for graphics systems that generate new frames less than 60Hz, reducing the effects of lag and increased transport delay.

Finally, Varjo headsets are said to all come with a built-in fan that circulates air away from the eyes to eliminate fogging. “All these features combine to create a training experience without simulator sickness, which is an industry first,” he claimed.

From its OEM vantage point, Frasca still sees a limitation in resolution in certain HMDs and headsets, as well as a limited FOV, in some cases, both direct FOV as well as peripheral. Gawenda continued, “However, I think the training community is getting much better in understanding these limitations and applying the tool to only the training tasks that the tool best supports. In pure VR, the tactile operation of knobs and switches is either lacking or requires special gloves and haptic sensors to help simulate operation of the knobs and switches. There are also limitations in terms of avionics functionality, especially glass flight decks with integrated PFDs [primary flight displays], MFDs [multi-function displays], FMSs [flight management systems], and AFCSs [automatic flight control systems].” The OEM has observed when they move to MR applications, this mitigates some of these issues. However, the quality of the HMD can still present some limitations.

On the Fast-Growth Track

Tobii’s Bentley provided another ‘insider’s’ affirmation that the A/M/V/XR market “is experiencing fast growth,” and called attention that training in enterprises, “including pilot training, is one of the fastest-growing use cases we have seen, and we expect it to continue this growth over the next few years.”

Tobii is working on a next-generation eye-tracking platform that addresses the challenges of future hardware form factors. Bentley concluded, with the integration of Tobii eye tracking into more A/M/V/XR devices, developers and trainers will have access to a variety of hardware supporting their needs. “We are also working on OpenXR, which is an open, royalty-free standard for access to VR and AR platforms and devices.”

The A/M/V/XR sector appears to be on the cusp of even more growth due to the rapid emergence of the burgeoning eVTOL aircraft sector.

One indicator of the potential this new market offers the S&T sector was noted when Volocopter teamed with CAE to develop an eVTOL pilot training program in preparation for Volocopter's first aircraft entry-into-service.

This trend has not been lost on others in this industry. Varjo’s Burwell noted that with demand from emerging eVTOL manufacturers like Beta Corporation, “Varjo sees a growing need for more cost-effective, low-footprint training systems across the civil training landscape.”

Not Only for Aircrews

The commercial aviation learning enterprise is stepping up its pace to integrate VR and AR training products into its aircrews’ curricula. The technology envelopes for these products are expanding, providing an opportunity for more onboard tasks to be offloaded from the supported aircraft and even FFS onto lower-level training devices. As costs for these articles continue to decrease, and they become more capable with AI and other enablers, more ATOs and other organizations will invest in these products. At the same time this market and other adjacent S&T markets are due to rapidly expand with the stand up of eVTOL carriers and their supporting training organizations.

A/M/V/XR training capabilities are not, of course, the exclusive purview of the commercial pilot training community – with the products being integrated into other sector training enterprises – for maintainers and cabin crew members.

Definitions: AR, MR, VR, XR

Adapted from the Unity Technologies Glossary

Augmented Reality (AR) is the overlaying of digitally created content on top of the real world. AR allows the user to interact with both the real world and digital elements or augmentations.

Mixed Reality (MR) blends the user’s real-world environment and digitally created content, where both environments can coexist and interact with each other.

Virtual Reality (VR) computer-generated stereo visuals surround the user, entirely replacing the real-world environment. Real-time user interaction within the virtual environment is possible, whether through detailed interactions, or simply being able to look around within the experience.

XR (eXtended Reality) is generally used as an umbrella term, a casual shorthand to group technologies such as VR, AR, and MR together. Most definitions of XR encompass platforms and content where the user can take digital objects into reality, or, conversely, see physical objects as present in a digital scene. “X” represents an undefined or variable quality/quantity.

To represent the spectrum of “realities,” CAT and our sister publication MS&T will use the shorthand ‘A/M/V/XR’.