AI Makes the Invisible More Visible in Simulator Training

AI Makes the Invisible More Visible in Simulator Training

Pilot instructors are accustomed to spotting the moment a trainee becomes overloaded, startled or begins to struggle. But what if technology could help them identify those moments more precisely, and then take the instructor and pilot straight back to the evidence?

That is the thinking behind Simvidence Powered by Simnest, an AI-assisted simulator training-review platform being developed through a collaboration between AviatePro Services, a developer of AI-supported human factors and debriefing technology for aviation training, and Simnest Group, which combines flight simulator manufacturer Simnest Aviation and training organisation Simnest Pilot Academy.

The platform combines synchronised simulator video, multichannel audio, speech and session events with observable human-performance signals. Rather than attempting to assess the pilot, it identifies moments that may warrant closer examination during debriefing.

For AviatePro founder and CEO Krisztian Makai, that distinction is fundamental.

“We don't want to provide mind reading,” he told CAT at APATS 2026. “We’re just reading the signs.”

From observation to evidence

Those signs can come from multiple sources. The system analyses facial movements, voice tone, head orientation, gaze and the speed of head movements, looking for changes in behaviour and deviations from an individual pilot's baseline.

AviatePro has spent more than 18 months developing what Makai describes as an “interpretation layer” that turns raw AI signals into understandable behavioural patterns.

Crucially, the technology does not conclude that a pilot is stressed, overloaded or lacking resilience. Instead, it can identify a change in behaviour and direct the instructor to the relevant moment. “The system highlights a change from the pilot’s own baseline,” Makai explained.

Different signals can also reinforce one another. A startle response, for example, might be visible through facial signals, voice tone or changes in scanning behaviour. A pilot might develop tunnel vision or, conversely, begin rapidly scanning instruments and controls.

But Makai is particularly interested in what happens next. “How they recover from the startle – that matters,” he said.

The expected recovery time will depend on the pilot's stage of training. Following performance across successive sessions could therefore allow instructors to see whether recovery becomes faster as experience develops.

Giving instructors somewhere to look

Simvidence combines synchronised recordings with behavioural analysis to help instructors identify and review relevant moments in a simulator session. If it detects an event, warning or significant behavioural deviation, instructors can return directly to that point afterwards rather than searching through an entire recording.

Instructors can also flag moments themselves using a small button device, perhaps identifying an example of strong teamwork or leadership, or something they want to revisit during the debrief.

The result is a collection of reviewable moments rather than an automated verdict. “It’s like a starting point for a facilitated debriefing discussion,” Makai said. “Here’s the tool. It highlighted these five moments. Let’s discuss it, guys.”

Instead of simply telling a trainee they appeared overloaded, for example, the instructor and pilot can revisit what happened and discuss the pilot's own interpretation. The system also provides role-separated transcripts alongside the recordings.

That evidence may be particularly valuable as younger pilots enter training, Makai believes. He said cadets increasingly expect detailed feedback and the ability to review their own performance rather than simply being told where they need to improve.

Keeping AI away from the grade

As AI becomes more capable, an obvious question is whether technology like this could eventually move from supporting an instructor to assessing the trainee.

Makai is emphatic that this is not what Simvidence is designed to do. He emphasises that evidence captured during a simulator session forms only part of the wider context that informs an instructor’s assessment.  “The instructor starts building the bigger picture during the briefing, before the simulator session even begins, and the debriefing can add further context to what was observed,” he said. “Even a multimodal fusion AI system does not have access to everything that informs that professional judgement. Simvidence provides insights and traceable evidence from the session, but the instructor brings those findings together with observations and discussions beyond the system’s view. Our role is to enrich that picture, not to turn individual moments into competency grades.” The distinction is particularly important in Competency-Based Training and Assessment (CBTA), where observations during the simulator form only part of the training and assessment process.

Makai therefore rejects the idea that Simvidence should automatically determine competency grades. “AI should support the instructor’s judgement, not replace it or determine competency grades,” he said. “We decided to show highlights: please review this moment.”

The technology could nevertheless support instructor standardisation. Because moments are timestamped and shareable, instructors and standardisation teams can examine the same evidence and discuss why they interpreted it differently.

But Makai argues that the industry itself needs much greater agreement about how human performance should be interpreted before asking AI to make those decisions.

One million hours

That need for evidence sits behind an ambitious AviatePro objective: collecting one million hours of simulator training data.

Makai calls it the “Wunderlich Project”, inspired by 19th-century physician Carl Wunderlich's large-scale work measuring human body temperature. The principle is that reliable knowledge requires large amounts of evidence.

“Our long-term ambition is to collect one million hours of simulator training data to support the validation and refinement of the system’s findings.”

The collaboration combines AviatePro’s AI-supported human-performance technology with Simnest’s simulator expertise and real-world training environment. Simvidence is being developed and validated using Simnest Aviation simulators at Simnest Pilot Academy, where approximately 800 cadets train each year.  “The academy’s instructors are highly experienced pilots, whose operational experience and instructional expertise strengthen both the quality of training and the rigour of the software validation process,” said Makai. “Simnest’s forward-thinking approach as a simulator manufacturer is equally valuable, both in bringing Simvidence into the training environment and in exploring how simulator telemetry can enrich our multimodal behavioural insights. That combination of instructional expertise and engineering collaboration is helping shape the next stage of Simvidence’s development.”

“The real value of Simvidence is in giving instructors and cadets more evidence to work with during debriefing. By developing it together with AviatePro in a real training environment, we can create a tool that supports more evidence-based feedback while keeping the instructor at the centre of the process,” said Simnest Group CEO Gyula Kühtreiber.

According to Simnest, use is voluntary, but instructors are already reviewing the system's insights and using selected material during debriefing, while cadets are returning to review their own performances.

As part of the collaboration, the partners are also exploring integration of simulator telemetry, aircraft-state and event data to provide additional technical context.

Deployment is intended to remain non-intrusive. Makai said the system can be installed using a computer, four small cameras and an instructor interface, with further installations at Simnest planned as the partners increase the amount of training data available.

For Makai, however, accumulating more data does not change the underlying principle: AI should help instructors identify and examine relevant patterns, while leaving their significance for training and assessment to the instructor.

“AI can help us find and examine the evidence. The instructor remains responsible for interpreting it in context and assessing competence.”

For Simnest and AviatePro, that principle sits at the heart of Simvidence Powered by Simnest: using AI not to replace instructor judgement, but to make more of the evidence behind that judgement visible.