Can You Trust Your Instructors to Grade Correctly?

Based on the presentation, Importance of Instructor Concordance Program (ICAP), by Captain Graham McNally, CBTA and Instructor Training Lead, Learning Design, Boeing

Pilot Stream, WATS 2026

Aviation training has invested heavily in data. Competency-based training and assessment frameworks, evidence-based training programs, sophisticated simulator environments — all of them generate performance data that organizations rely on to make decisions about crew readiness, training effectiveness, and safety risk. But there is a foundational question that often goes unasked: how reliable is that data in the first place?

Captain Graham McNally, CBTA and Instructor Training Lead in Boeing's Learning Design team, has spent considerable time working on exactly that question. His answer, presented at the recent WATS conference, is that data quality in aviation training is inseparable from instructor standardization — and that instructor standardization requires something most organizations have not yet systematically built: a formal concordance assurance process.

The program Boeing has developed is called ICAP — the Instructor Concordance Assurance Program. McNally's eight-year-old grandson, when asked what ICAP might stand for, offered a definition that turned out to be remarkably accurate: instructors can assess properly. That, in essence, is what ICAP is designed to measure and ensure.

Why Instructor Standardization Is a Data Problem

The case for instructor standardization is not new. Most training organizations understand intuitively that different instructors should assess the same performance consistently. What is less well understood is the downstream consequence when they do not.

If two instructors observe the same crew performing the same maneuver and award meaningfully different competency grades, the problem is not just unfairness to the pilots being assessed. The problem is that every data point generated by those instructors is now of uncertain reliability. Aggregate that across a fleet, a training center, or a global operation, and the data that management is using for risk oversight, training program decisions, and regulatory reporting may be systematically distorted.

McNally frames this as a risk management issue as much as a training quality issue. Data collection is only valuable if the data is reliable. Without instructor standardization, it is not.

There is also a regulatory dimension. In the EASA regulatory environment, transitioning from mixed-implementation evidence-based training to a full baseline EBT program requires two years of ICAP data before the transition can be approved. Concordance assurance is not merely good practice in this context — it is a formal prerequisite for program advancement.

Building the ICAP Process

Boeing's ICAP program operates across six structured steps, and its design addresses one of the central practical challenges of instructor standardization at scale: getting instructors in the same room at the same time is often not feasible across a global operation. The program needed to function both in-person and remotely.

The foundation of the process is a library of purpose-built assessment videos, filmed in simulator environments with pilots performing at deliberately varied competency levels across the nine CBTA competencies. Separate videos were produced for instructor briefing and debriefing scenarios, recognizing that instructor competency in facilitating learning is as important to measure as instructor competency in assessment. Additional training videos, distinct from the assessment videos, allow organizations to use the same material for targeted upskilling.

For the assessment itself, instructors observe each video once and once only — mirroring the conditions of an actual simulator session, where there is no opportunity to pause or rewind. They then submit competency grades through an electronic assessment system that feeds directly into Boeing's performance analysis tool, Performance and Training Advisor.

This single-viewing rule is deliberate. It prevents instructors from over-analyzing footage in a way that would not reflect their real assessment behavior, and it ensures that the data captured reflects genuine, in-the-moment judgement rather than a considered retrospective opinion.

Setting the Standard of Reference

For concordance assessment to be meaningful, there must be an agreed benchmark against which instructor grades are measured. Boeing terms this the standard of reference — sometimes referred to in the broader industry as a gold standard. It is established through a structured process involving a team of subject matter experts who independently complete the same e-assessment before meeting to align on expected grades for each competency in each video.

Critically, the subject matter experts are themselves treated as instructors in this process. They complete the e-assessment under the same single-viewing conditions as any other participant, and their individual grades are captured before the group alignment meeting takes place. This prevents the standard of reference from being shaped purely by discussion rather than independent judgement.

The output is a set of reference grades against which every participating instructor's responses can be compared. The analysis then looks for two things: agreement among instructors — meaning consistency with each other — and alignment with the standard of reference. Both matter. A group of instructors can agree with one another while being collectively misaligned with the established standard, which is a different and potentially more serious problem.

From Assessment to Action

The value of the ICAP process lies not in identifying underperforming instructors but in generating actionable intelligence about where the assessment community as a whole needs development. McNally is explicit that the program is not designed to catch instructors out. It mirrors what good instructors do for pilots: identify specific areas for improvement and provide structured support to address them.

Where individual instructors show significant deviation from the standard of reference on particular competencies, customized development can be provided — working through the relevant assessment videos with an instructor trainer or standards captain, reviewing grading rationale, and recalibrating. Where a collective pattern emerges across a group of instructors — for example, a tendency to systematically over or under-grade a specific competency — group training can be designed to address it at the program level.

Boeing's current implementation plan runs two assessment cycles per year, approximately six months apart, giving the process enough frequency to be a meaningful feedback loop without creating an unmanageable administrative burden.

One important note of context: McNally positions ICAP accurately within the broader landscape of instructor standardization. It is not a complete solution in itself. Simulator observations, direct instructor evaluation, and other components of a comprehensive standards program all remain necessary. ICAP addresses one specific and measurable element — whether instructors are assessing consistently and in alignment with an established benchmark — and it addresses that element rigorously.

In a training environment increasingly dependent on data-driven decisions about crew competency, program design, and regulatory compliance, that reliability is not a secondary consideration. It is the foundation on which everything else rests. As McNally's grandson put it with accidental precision: instructors can assess properly. ICAP exists to make sure they do.

Want more insights on Pilot Training? Join us at EATS, 3–5 November at the Estoril Congress Centre in Cascais, Portugal, and explore the Pilot Conference Stream. Register below.