Emerging X-Realities

VR, AR and MR technologies – collectively known as XR (Extended Reality) – are delivering dramatic advances in capability at increasingly affordable prices. Atul Chandra explores examples of how XR solutions are becoming rapidly adopted by world defence training organisations.

“When it comes to XR, we generally observe that end users and aircraft manufacturers adopt a technology-driven approach. Many seem to embrace XR technologies and their apparent financial benefits before even connecting them to operational use. Instead, we have adopted a more careful and strictly use-case-oriented approach. This is why, for instance, we are assessing which of MR or VR better suits a specific training task in relation with the level of fidelity of cockpit representation,” Bernard Del Ghingaro, VP Marketing & Sales, Sogitec told MS&T.

French training solutions specialist Sogitec’s work in the XR arena is substantially advanced, having developed and presented several XR simulation demonstrators over the past four years, including at the Paris Air Show. These include fighter and helicopter piloted simulation with high-end headsets, simulated helmet-mounted displays, tactical situation viewer with smartglasses and a multi-user virtual reality maintenance simulator.

Another necessary assessment, Del Ghingaro said, is to cater for the overall physiological effect (e.g. motion sickness) of wearing XR devices during lengthy training sessions. “Our vision is broader in order to assess how applicable XR is to more advanced training phases.”

Sogitec is also evaluating the relevance of haptic feedback to pilots and trying to adjust fidelity, depending on the type of training which is performed. “Looking further into the future, we are already supporting our parent company Dassault Aviation for the S&T aspects of the FCAS programme (European Future Combat Air System under development by Airbus, Thales Group, Indra Sistemas and Dassault Aviation). Other fields of interests include drone simulation, LVC training, and embedded simulation,” Del Ghingaro said.

Reduced Footprint for Forward Deployment

“The primary use cases we see are for pilot and rear-crew training, although we see direct application to land, naval and maintenance training,” Phil Perey, Head of Technology for CAE Defence & Security, told MS&T.  “In all cases, the significant reduction in physical size can make these solutions attractive for forward-deployed training and mission rehearsal. With new commercial-off-the-shelf capabilities introduced in the past months, the adoption trend is likely to accelerate.”

While XR is a recent phenomenon, CAE was working with XR-type systems over 30 years ago and delivered a custom-built Fiber-Optic Helmet Mounted Display to the US Army’s Simulator Complexity Test Bed (SCTB) in the early 90s.

Until 2019, Perey noted, the largest constraint for widespread adoption of XR was the combination of resolution and large field-of-view for headsets, adding, “With the significant increases in these areas through 2020, this is becoming much less of a constraint.”

CAE is also working on XR systems for future pilot training which will use either optical see-through helmet displays or video merging headsets. While pros and cons exist for these capabilities, Perey said that “the rapid progression in image resolution, field-of-view and update rates make these helmets a compelling alternative to large display screens.” Update rates on these systems are all at 90Hz or more, which is sufficient to get a solid image with little ‘swimming’ or lag. “A few systems are now wireless, which is important for solutions where you need to move around. Wireless solutions are still limited in overall performance compared to wired (or tethered) displays,” Perey said.

CAE USA was awarded a contract from the Defense Innovation Unit (DIU) in August 2020 to support the US Air Force Air Education and Training Command (AETC) with its Undergraduate Pilot Training (UPT 2.5) initiative. While CAE is not delivering the entire solution, UPT 2.5 remains a significant contract for the company, which has the responsibility of delivering the Learning Management System – which some call a Knowledge Management System or Adaptive Learning System. CAE will deliver this for the T-6 (which will likely expand to include the T-38). Limited user testing will commence at Randolph AFB, Texas during the first half of 2021; rollout to multiple air bases is slated for the second half of the year.

CAE’s Defence & Security business unit also launched its Mission Augmented Virtual Reality/Rear Crew (MAVRC) Trainer in 2020, which has already been delivered to a European customer. The MAVRC trainer delivers MR via a head-mounted display (HMD), which projects what the crewmembers see as the out-the-window virtual environment and also see and interact with everything inside the fuselage. It allows rear-crew members to work together as a team in tasks such as gunning, winching and confined-area landing, as well as search-and-rescue operations. The trainer makes use of a fully functional and reconfigurable aircraft fuselage and is equipped with stimulated and simulated aircraft panels including multi-function displays and communications.

Gamer or High-Performance Headsets?

Cost of headsets remains a factor, particularly so for armed forces with smaller budgets, as the highest performance devices are still priced at US$5-10,000 a set, which can limit widespread adoption. According to David Kanahele, Product Manager (Mission Systems), Collins Aerospace, while there have been some exciting improvements with gamer and enterprise level head-worn displays in key metrics such as resolution, latency and ergonomics, he cautions that further technological advances in those areas are still needed for some applications and “there remain fundamental gaps in current product offerings towards providing consistent focus and vergence cues to match a natural ‘real world’ visual experience.”

XR solutions will continue to deliver increases in performance even as competition drives prices down and at the present moment are at an inflection point for wider industry adoption. “Beyond the display hardware, technological advances that sense and integrate the real and synthetic environments, such as environmental mapping and AI, will be critical for advancing XR. Likewise, for the underlying networks and platforms, where 5G communications, edge computing and ‘cloud XR’ will be key enabling technologies,” Kanahele told MS&T.

As a provider of near-eye XR display solutions that cover a wide spectrum of applications, Collins Aerospace has delivered well over 100 HMDs on the Army Aviation Collective Training (AVCATT-A) program utilizing an optical see-through design for mixed-reality training. Now part of Raytheon Technologies, they are combining capabilities and working together in delivering VR-based training systems. In development for the Army’s Synthetic Training Environment (STE), is a new immersive military training product, the Soldier Virtual Trainer (SVT), which uses virtual and mixed reality to train squads of soldiers in multiple scenarios while using real and virtual weapons.

Collins Aerospace also offers a VR head-up display (HUD) trainer which uses COTS VR headsets to deliver faster crew training on the Collins Enhanced Vision System (EVS) at a lower cost. The VR HUD trainers are currently in use for human factors research by the US Federal Aviation Administration and are also under evaluation by various airlines that utilise the Collins EVS HUD. Also being offered are Virtual Reality Avionics Trainers, which fully immerse the user in the training environment and utilise the company’s rehosted avionics software to maximise the training fidelity.

Collins Aerospace’s ‘Coalescence’ mixed-reality system enables a trainee to interact with real objects and people while still immersed in the synthetic environment. Image credit: Collins Aerospace.

Digital Twins for Aircraft Design

Italian defence conglomerate Leonardo has also been working on XR technologies for a few years and their introduction for training of military operators, both pilots and maintainers. “Integration of XR in pilot training allows the pilot to fly in a fully virtual environment, synchronized with the simulation framework, introducing innovative additional data visualizations. The integration of the real displays/panels with virtual ones allows to move training faster towards the future where the cockpit will be not hardcoded but customized based on mission needs,” a Leonardo spokesperson conveyed to MS&T.

The firm is also working on integration of XR in maintenance training with a replica of the aircraft in a virtual hangar, allowing the maintainer to practice maintenance procedures while interacting using tools and ground support equipment to perform troubleshooting and remove/install operations.

XR cockpit simulators are effectively digital twins of the real cockpit, allowing training with a very low ratio of cost/effectiveness and are now also used for fast prototyping during the aircraft preliminary design phase. The mix of the physical objects with the digital manipulation allows innovative use of training scenarios dealing with complex data visualization without impacting the cockpit hardware. The XR cockpit is a kind of “services” machine providing data visualization without impact on the hardware side. The main technological challenge, according to Leonardo, is to allow people to interact with virtual displays/panels in the most natural way, breaking the barrier of the devices used for virtual data manipulation. The endeavour being to allow the pilot to interact with the digital twin in the same way as they interact with the physical hardware, to allow a swap from XR training and full-flight training without any impact on the pilot behaviour.

Australian XR Strides

The Australian Defence Force (ADF) is significantly invested in XR and leads other militaries in the Asia Pacific region in this regard. “The Army is already using VR for several individual and small team training outcomes. The Army is also exploring XR technologies for technical trade training and in support of wider technology-enabled learning, and is studying the potential benefits of augmented reality in support of both training and operational requirements and Free Roam Virtual Reality technology for dismounted combat training and combat resilience,” a Department of Defence spokesperson informed MS&T.

The Australian Army’s Land Simulation Core 2.0 project aims to provide a persistent and efficient land simulation system for training outcomes. Initial operating capability for Land Simulation Core 2.0 is slated in 2023, with final operating capability in 2024. The project aims to deliver synthetic training tools and a supporting network to enable remote/virtual training within the Army from the individual level to the battle group level. Several small-scale trials with a range of VR and AR technologies are already underway to assist the Army in defining its broader requirements for training and learning support.

Studies into the impacts and effectiveness of immersive training are also being undertaken by the Army and Defence Science and Technology Group. “In 2020, the Army evaluated the Free Roam Virtual Reality technology in Townsville for dismounted small team training and combat resilience. More than 200 participants experienced various urban combat scenarios,” the spokesperson said, adding that under the home station training concept the Army is exploring the value of integrating existing and future live and synthetic training tools to maximize the value of collective training without leaving the barracks environment.

XR ‘proof of concepts’, VR and/or AR technologies for use in training are being experimented with by some training and operational units of the Royal Australian Air Force (RAAF). The RAAF Training Group Headquarters established a learning technologies and integration capability in 2019 to better enable the use of various learning technologies and the Air Force Air Warfare Centre (AWC) has an established innovation hub which is experimenting with XR. The RAAF continues to engage with the wider defence partners and industry to identify and examine new and emerging technologies that could be used to enhance military simulation and training capabilities. The Chief Information Officer Group (CIOG) innovation team, Telstra, and Microsoft are working with Jericho Disruptive Innovation to develop an AR Operations Room to support battlespace awareness. The system uses Microsoft’s HoloLens 2 to project a hologram of the battlespace in front of the user, enabling them to quickly understand where all the various pieces such as aircraft, drones, ground units, etc. are located to support training in operational decision making.

The Royal Australian Navy (RAN) is using XR as one means to deliver training, both ashore and at sea, from initial employment training through to complex multi-role scenarios. XR used by the RAN for training includes simulation models of ships, allowing a sailor to navigate through a vessel prior to joining, through to aviation flying training in full-immersion simulators. The RAN has found in the inclusion of VR training that it is more of a challenge in the at-sea environment, primarily due to the necessity to embark and disembark the equipment. The RAN is of the view that simulation will play an increasingly dominant role in at-sea training and that developments in connectivity across multiple ships will improve simulated training across multiple ships and joint synthetic exercises.