EATS Data: One-Third of Instructors Miss Training Needs

Key Takeaways

• One-third of instructors miss training needs: Research shows only 67.7% of standardized instructors identify training needs that operators define as critical
• Unidentified needs create safety risks: When instructors fail to spot deficiencies, pilots continue training or return to line operations without addressing performance gaps
• Over-grading wastes resources: 28.2% of instructors identify training needs in pilots the operator standard deems proficient, leading to unnecessary training costs
• Concordance isn't EBT-exclusive: Despite regulatory requirements only in Evidence-Based Training, instructor concordance issues affect all aviation training programs
• Instructors are the only sensors: As the sole evaluators observing pilot performance directly, instructor accuracy is critical for both safety and efficiency

A Universal Challenge

Frank Steiner, Managing Director of conavitra GmbH, opened his EATS 2025 presentation with a provocative question: "Is instructor concordance only for the EBT world?" His answer: "I wouldn't be here if the answer would be no."

What followed was a groundbreaking research study analyzing 2,662 instructor ratings that revealed troubling gaps in how aviation instructors assess pilot performance—gaps that extend far beyond Evidence-Based Training (EBT) programs to affect ab initio training, type ratings, and all forms of aviation instruction.

The findings have profound implications for both safety and cost management across the industry.

Understanding Instructor Concordance

Before diving into the research, Steiner provided essential context on what instructor concordance means and why it matters.

Why Concordance Matters

Steiner identified five critical reasons for ensuring instructor concordance:

Fairness and objectivity: "All pilots have the right to get a fair and objective grading and fair objective assessment of the instructors they are dealing with," Steiner stated. "That's the right of our trainees."

Data-driven programs: Modern training increasingly relies on performance data. "Only if the instructors have the ability to grade correctly, to record the correct things, then we can adopt the training efficiently to the really the needs of the pilots."

Alignment with competency standards: Every operator and ATO establishes standards they expect pilots to achieve. Concordance ensures these standards remain consistent regardless of which instructor evaluates performance.

Instructor feedback: "The instructors are some kind of lonely riders," Steiner observed. "They are instructing, and the only feedback they get is maybe from their students." Most students, eager to progress, will say "everything was fine" rather than provide critical feedback. Concordance programs give instructors meaningful data about their assessment patterns.

Learning objectives: When instructors accurately identify pilot needs, they can adapt training effectively to address those needs.

Reliability vs. Validity: The Technical Foundation

Steiner explained the scientific concepts underlying concordance using clear examples:

Reliability means consistency among instructors. "An inter-rater reliability of 100% would mean that every pilot gets the same grading of every instructor for a given performance," he explained. While perfect reliability is unrealistic, instructors should show reasonable agreement when evaluating the same performance.

Validity adds a second dimension: matching the operator's standard of reference. "Our instructors are all agreeing on the same, and this agreed grading matches with the operator standard of reference," Steiner defined. "The operator said that's exactly what we expected to be graded in such a situation."

Using visual examples, Steiner showed how instructors might agree with each other (high reliability) but still miss the operator's standard (poor validity), or disagree entirely (poor reliability), making validity impossible to achieve.

"Only if agreement is high, then we can think of alignment," he emphasized. "Then we can think of getting a good alignment."

EASA and IATA have recognized concordance's importance by incorporating Instructor Concordance Assurance Programs into EBT regulations (AMC1 ORO.FC.231). These programs require annual instructor standardization, including concordance assessment.

However, Steiner's research demonstrates that concordance challenges exist wherever instructors evaluate pilot performance—not just in EBT programs.

The Research: Analyzing 2,662 Instructor Ratings

Steiner's team conducted a comprehensive study using the ICAP (Instructor Concordance Assurance Program) system, where instructors evaluate standardized video materials that have predetermined operator standards of reference.

Study One: The Safety Gap—Missing Training Needs

The first analysis examined whether standardized instructors could identify when pilots needed additional training.

Methodology: Researchers identified scenarios where the operator standard of reference included at least one competency graded 1 or 2 on the five-point EBT scale—indicating a training need. They then checked whether instructors identified any training need (grade 1 or 2 in any competency), even if they misidentified which specific competency needed work.

"We're not looking at the competencies," Steiner clarified. "It's the highest level of performance. It's the easiest way of performance. Do the instructors identify a training need or not?"

Results: "Only 67.7% of the instructors found the training need, or any training need, for the pilots they saw," Steiner revealed.

This means nearly one-third of standardized instructors failed to identify that a pilot needed additional training—even when the performance clearly warranted it by operator standards.

Safety implications: "There we have a safety issue," Steiner warned. "If we have an unidentified training need, this pilot will get thumbs up, goes back to the line, or continues the training, and then will maybe be producing some safety issues."

Study Two: The Cost Factor—Unnecessary Training

The second analysis flipped the question: Do instructors identify training needs where none exist?

Methodology: Researchers examined scenarios where the operator standard of reference showed good performance (grades 3, 4, and 5 only—no training needs). They checked how many instructors still assigned grades of 1 or 2, indicating they saw deficiencies requiring additional training.

Results: "More than 28% of the instructors found a training need for pilots that, defined by operator standard of reference, had no training need."

Cost implications: "This is an efficiency thing," Steiner explained. "Only with consistent assessments can we reduce unnecessary training capacities and unnecessary trainings and save resources."

Over-identifying training needs wastes simulator time, instructor hours, and operational resources on pilots who don't require additional instruction.

Instructors as Sensors: Why Accuracy Matters

Steiner emphasized a crucial point: "Our instructors are our sensors. The instructors are the only sensors that we have. They are the ones who are sitting in the simulator or in the aircraft with the pilots, with the trainees, and we can only rely on their expertise, on their evaluation."

This reality makes instructor concordance critical. Unlike automated systems that can cross-check data, pilot evaluation depends entirely on instructor judgment. When that judgment lacks consistency or alignment with standards, both safety and efficiency suffer.

The Solution: Concordance Training

"We highly recommend concordance training, which enhances the safety," Steiner concluded. "With concordance training, we can reduce the number of the 67.7%—the instructors who didn't see the training need—on the one hand, and on the other hand, we can also reduce training costs, because the instructors really can identify where do I have a training need with a pilot or not."

The message is clear: concordance training isn't a regulatory checkbox for EBT programs. It's a fundamental need across all aviation training to ensure instructors can accurately identify when pilots need additional training and when they're ready to progress.

Conclusion: A Universal Imperative

Steiner's research provides compelling evidence that instructor concordance challenges transcend training methodologies. Whether conducting ab initio training, type ratings, recurrent training, or EBT, organizations face the same fundamental problem: instructors don't consistently identify training needs accurately.

With 32.3% of instructors missing critical deficiencies and 28.2% over-identifying problems, the aviation training industry has substantial room for improvement. The safety implications of missed training needs and the financial waste of unnecessary training make concordance training a priority for every ATO and operator—regardless of whether they operate under EBT regulations.

Want to stay at the forefront of aviation training innovation? Join us at WATS 2026 in Orlando, Florida, May 5-7, where industry leaders will explore the latest developments in pilot training and ATO operations.