The debut of the Apple Vision Pro (AVP) mixed-reality (MR) headset has re-ignited interest in using A/M/V/XR technology for aviation training applications.

But beware the hype. Consider the limitations.

“Apple Vision Pro is another step toward an immersive future. But it is a long way from being Pro,” commented technology reviewer Benjamin Dannenberg. “It is currently primarily another platform for displaying 2D content, i.e., flat applications that are partially enriched with 3D or depth functionality.”

Which may be quite adequate for some of the cabin crew, maintenance training, and even flight procedures trainers flowing into the market.

But AVP 1.0, despite the $3499 price tag, has significant technical weaknesses, especially when it comes to professional flight training:

• The headset is front-heavy, weighing 650 grams, and puts pressure on the cheekbones. One early purchaser said the device gave her two black eyes after wearing it for “about an hour.”
• Glasses do not fit under the Vision Pro. Outside the US, you will need to wear contact lenses. In the US, magnetic prescription lenses may be available.
• Hand-tracking suffers from lag during fast movements.

Passthrough latency of MR headsets may be an issue for aviation training applications involving quick motion, such as a flight simulator. According to Optofidelity, photon-to-photon latency (the time it takes for an optical change to reach the retina of the user – camera capture, algorithm reconstruction, screen display), is a minimum 11 milliseconds for AVP. Meta Quest Pro ($999) and HTC Vive XR Elite ($1099) measured minimum latencies of 35 to 40 milliseconds.

Potentially more serious are the findings of a study by Stanford University, led by Professor Jeremy Bailenson, founding director of the Virtual Human Interaction Lab (VHIL).

The Stanford testers, despite being “headset veterans,” all suffered motion or ‘simulator sickness’ from extended use – a phenomenon well known to early developers of full-flight simulators from the 1960s (and a term many thought had been long ago retired).

The study’s authors urged caution against prolonged use of passthrough MR headsets.

Instead of mixed-realty headsets, which use front-facing cameras to capture the real world and blend it with synthetic visuals, virtual-reality applications use computer-generated visuals exclusively.

Meta $999 headset users in the Stanford field test often underestimated distances to objects. Fabi Riesen, CEO of Loft Dynamics, said the disconnect is because the cameras “are not where the eyes are. Which means what the camera sees is not exactly what my eyes would see. And this is the geometric distortion.”

Professor Bailenson warned “such overcompensation could linger after prolonged headset usage. What are the aftereffects and how long do they last? A plausible scenario could be walking down a flight of stairs and you miss a step, or driving a car and you misjudge distances.”

Riesen challenged, “Before anyone continues to use current MR, or even thinks about qualifying current MR for use in pilot training, they must refute the study’s findings. The reported impacts could lead to negative training. It completely destroys the immersiveness if you're suddenly reminded that this isn't actually reality.”

“The study is actually proving that we are doing the right thing” (in opting for VR instead of MR), he added. “The whole mixed reality thing is something which most likely will fail if you want to apply it to a regulatory framework like EASA or FAA.”

For the aviation industry, “a lot has been done with mixed reality because it's far simpler to implement it,” Riesen noted. “For example, in maintenance training where you're not worried about flying an aircraft in real time, you're using it more to inspect things.”

“The results are as expected,” John Burwell told me, “when using low-cost, low-fidelity, commodity MR headsets that don’t support adequate video see-through and image resolution, reduced latency, and lens projections that are needed to support enterprise-class applications.” Burwell is Global Head of Simulation and Training for Varjo and a 30-year veteran of flight simulation visual technology.

“Users of Varjo headsets (such as the $3990 XR-4) don’t see the same issues because they are specifically designed for enterprise users rather than gamers.”

Consumer-grade helmet-mounted displays (HMDs), he explained, typically are “all-in-one” devices where the computer and associated battery are mounted inside the HMD with the displays and optics. “This limits the graphics performance of consumer HMDs to what the simulation industry used in the 1990s – and for limited amounts of time due to battery life. Where this might be fine for some training applications, the industry has moved well beyond these limitations, and I don’t see them going back.”

With regard to the Apple Vision Pro, Burwell said software is a big issue -- only supporting Apple’s new spatial iOS and downloads from the Apple store. “This infrastructure will not likely ever support the industry’s image-generation systems like Lockheed Prepare3D, FlightSafety Vital, CAE Prodigy, Aechelon PC-Nova, etc. It will support limited Unreal Engine and Unity applications, but these are far from the large-scale, sophisticated IG systems used by the flight simulation industry.”

Security is also a significant issue. “Devices like Meta are not allowed to be used in any sensitive government applications because the devices require accounts and logins and sharing of data,” Burwell noted. “This is likely the same with Apple, so these devices are simply non-starters for any secure options.”

He continued: “The resolution of the displays, and particularly the resolution of the video see-through lenses used for mixed reality are a fraction of what’s needed to be able to read text required for most training applications. For example the FOV of the Apple Vision Pro is reported to be somewhere in the 90-95 degree range verses the 120 degree range of the Varjo XR-4. Where looking through a straw is good enough for some training applications, we moved to wider FOVs 5 years ago because of industry demands.”

Apple CEO Tim Cook lauded the AVP during the long-anticipated product launch as “the dawning of the era of spatial computing – You’ve never seen anything like this before.” But that’s not quite true; the term spatial computing dates from the 1980s, and has long been a cornerstone of the geographic information system (GIS) technology that is foundational to flight simulation.

For that matter, headsets for flight simulation are not a recent innovation either. In 1981, CAE received contracts from the Canadian and American governments for the rather bulky but effective tethered Fiber Optic Helmet Mounted Display (FOHMD).

NOTE: Loft Dynamics and Varjo will be exhibiting at WATS, 30 April to 2 May, in Orlando (Booth 411) – featuring the world’s only regulatory-approved VR-based flight simulator.