Performance Based Oversight and Regulation

Jules Kneepkens asks if Europe has Performance based Regulation and Performance based Oversight in place in relation to Approved Training Organizations?

The simple answer to this question is “no”. We therefore need to ask ourselves, “Why not?” There are likely to be a number of answers to this question, including the fact that the concept is, for many, difficult to grasp. This is understandable because the concept is very different from that of ‘compliance’ which has been the basis for regulations and oversight for many decades.In addition, there is the matter of whether performance based regulation and oversight are of real benefit to the industry. This is especially visible to organizations located outside the EU with advanced SMS in place. Such organizations may get ‘credit points’ from their own Authority, such as the FAA, which reduces the regulatory burden. They do not as yet get any equivalent benefit from the European Oversight system.

Within the European Union, and also outside the EU, there is a need to establish a common understanding on exactly what a Regulator means by “performance based regulation (PBR)” and subsequently how to set up and implement an effective performance based oversight methodology. Many people, consultants and training providers have individual opinions and views on PBO and PBR, but a common view and understanding doesn’t yet exist.

Common UnderstandingThe initiative to create a common understanding and clarification must come from the Regulators. In order to create “commonality” it is preferable that regulators such as the European EASA, American FAA and Canadian Transport Canada, take the lead in a joint initiative. Such co-operation between those regulators in other aspects of aviation regulation has already proven to be highly successful. These regulators have previously worked effectively together, and with ICAO, to create a common understanding on Safety Management and Safety Management Systems. Supported by some other States they have been able to develop a ‘common SMS-view‘ which, via the ICAO decision-making process with support of the ICAO Council, cumulated in the new ICAO Annex 19, Safety Management.

Elements from Annex 19 relating to Safety Management Systems will be helpful in creating a common understanding of PBR, the use of PBR, and to ensure effective PBO. For example, in defining appropriate performance indicators, and in setting targets for these indicators within the existing aviation organizations. It will be essential that this happens soon to ensure that the regulator implements its own regulations correctly. Already SMS provide the opportunity to improve the quality, effectiveness, efficiency and capacity of the regulator, and the industry that it oversees. PBO has the capability of taking these improvements to the next level.

A few companies in the training industry have already adapted their way of working, and already meet SMS requirements, but are still nevertheless overseen in the ‘old fashioned way (compliance)’ because the regulator’s oversight staff has not adapted the basic approach and working methods to align with the new regulations. PBO offers immediate advantages in situations today where, for example, ATOs have implemented SMS to the full extent. It is now up to the overseeing authority to adapt its way of working to ‘do more with less’, to increase ‘efficiency’ and reduce costs and regulatory burden. It will however require strong leadership from the regulators and a firm change management process.

In August 2014 EASA published on its website a paper on “A Harmonised European Approach to a Performance Based Environment”. This paper was a result of discussions between EASA staff and the EASA Rulemaking Advisory bodies. The discussion highlighted that the various aviation domains had different views on the meaning and interpretation of the terms PBO and PBR.

The production of the EASA paper was a first step along the path to create a common understanding on what PBR and PBO are in Europe. Not being pessimistic but realistic, the estimation is that less than 0.1 % of people in the European Aviation overseeing system is aware of the “August 2014 EASA document” and much less ‘perform their work‘ according to the principles/ideas described in the EASA paper. Although this could be done today at national and European level, the opportunity has yet to be exploited.

InitiativesIt is clear that mechanisms have to be found to trigger a wide distribution and understanding of the PBR and PBO approach and, importantly, the benefits that this concept offers. This could be initiated via pilot projects that demonstrate the advantages of PBO in terms of effectiveness and efficiency. The initiatives should include methods used to define performance objectives/targets, and to train oversight staff in the oversight of organizations with a well functioning SMS. This should be supported by the initiation of a process to create and define a common approach between EU/US. This is essential to further improve safety. In fact PBR and PBO invite people/organizations to measure data, analyse information and work pro-actively in taking timely, informed intervention action. Another way of looking at this is that organizations need to take full responsibility for safety: they can no longer rely on “being safe” simply because their Authority's audit has found them to “be in compliance”. This is a major paradigm shift that may need input from social behaviorists and organizational experts with expertise in how to implement such significant changes. In view of the magnitude of the task ahead it will require a ‘top-down’ and ‘bottom-up’ approach at the same time. In addition, strong cooperation (and please no-competition who will be first) will be needed between the major aviation regulation bodies like ICAO, EU, FAA, TC, CAAC, the NAAs from the Middle East, India, Brazil etc, as well as the industry and its representative bodies.

Safety ManagementICAO has introduced in the new Annex 19 the requirements and guidance for Safety Management and Safety Management Systems. Safety management recognises the existing interdependencies between domains in the aviation system and encourages, in fact, requires cooperation among them. Those interdependencies and their importance were not fully recognised before. In order to control how the different aviation domains influence each other there is a clear need to monitor aviation as a system and not ‘per domain’ or business activity. As it is correctly stated in the EASA paper, “Safety performance measurement and risk assessment need to be developed at a total system level to correctly ascertain the interfaces and interdependencies between all the domains of aviation. Safety performance and risk management applies not just to any one organization, State or regional agency, but also to all domains. This is also appropriate to any process related to certification and oversight by addressing all the interactions and streamlining the collection and analysis of data.”

Fundamental to the concept of a total system approach is an understanding of how the different parts in systems interact. Each player in the system has to think how they can contribute to the total system, how their actions can impact upon others, and how they in turn could be affected by others. Consequently, each player must be able to take and have full responsibility for their part in the system. The recent development of the regulations, which introduced SMS, reflects this principle, although this has yet to be fully implemented in practice. The ‘system thinking approach’ should, and will have an effect on oversight.

Experience shows that inspection of compliance often interferes with the management of the organizations inspected. Some inspectors see themselves as a part of the correction process, and act in that way. Interfering in the management of compliance makes it impossible for the management to take full responsibility for the safety performance of the organization. Oversight should instead concentrate on how safety and compliance are (pro-actively) managed and assured by the organization.

Implementation of the ICAO requirements for SSPs and SMSs will focus on how to pro-actively identify and minimise risk, with less emphasis on correcting the past (whilst not of course neglecting lessons learned). Analysing data, identifying risks and the taking of mitigation measures to address issues at the right level requires a pro-active approach by all in the system; by the regulators, and the industry. Such a change is not easy to realise but it must be facilitated. Examples of such ‘facilitating elements’ are the (FAA)-ASIAS-system (owned by MITRE), fed with information from different sources (industry and authority), and used by the FAA to identify risks and measures, and the European safety analysis systems that are designed to be compatible. In this way the funder of this EU-data-system, the EU Tax payer, gets real value for money.

The publication of ICAO Annex 19 recognizes safety management as the way forward in dealing with aviation safety risks. Any performance-based approach to oversight must consider the safety performance of regulated entities both at an individual and aggregate level (state), in respect of:- effective safety management,- the identification and management of safety risks by setting and achieving relevant and appropriate targets, and- safe operations.

The above changes are not easy to realise but must be the goal.

Are We Ready to Start PBO?In order to be effective, the PBO methodology must enable the assessment of whether the required safety performance or objectives established in the regulations have been met in a manner that is appropriate to the organization. To that end, the Competent Authority must understand the safety objectives, and be able to distinguish different levels of achievement, recognising good and poor performance. But enabling the regulators to adopt PBO will be of no benefit if the industry itself does not seize this opportunity and embraces the concept.

Competence of inspectors and oversight methods will consequently need to evolve to include more interaction, monitoring, negotiation of targets and objective judgment. This would apply equally to regulated entities and to the Authorities’ staff.

The role of Authority inspectors will need to change, working in closer collaboration with the organization, engaging in a broader dialogue on safety assurance and safety objectives rather than just checking compliance with prescriptive regulatory requirements. The focus will be more on how risks are identified and mitigated, and on assessing the effectiveness of the mitigation process. Finally, as oversight will be mainly based on performance, the ability to measure safety performance must also become part of the inspectors’ competence. This means a basic understanding of safety analysis techniques and an understanding on how to work with SPIs.

ChallengesThe effective introduction of PBR and PBO will require a fundamental change in the safety regulatory mind - a shift of responsibilities will take place as the regulated entity will be able to choose the means that will fulfill the performance goals. Oversight of performance based rules will require a different and more advanced approach as it will trigger discussions on the basis of SPIs and targets and on whether the applied method is best suited to meet the performance goal. An exchange of views, rather than one-way communication from the competent authority to the regulated entity, will need to take place. Hence, as the overseeing authority will have the last say, this different kind of oversight becomes a challenge in terms of appropriately measuring performance. The same type of dialogue will be necessary during the design and maintenance of PBR.

The improvement of safety requires one to focus on the management of risks. The increasing globalisation, the different ‘regional temporary shortages in pilots’, the evolution in training of Pilots (EBT etc.) and the expected expansion of aviation activity over the next years require;

a. A common understanding between the main players on the difference between prescriptive and Performance Based Regulation, and what this means in terms of the Training requirements (agreed standards for, and indicators of safety performance for training organizations are needed, taking into account the interdependencies with other aviation domains.) There is an opportunity here for industry and its representative bodies to seize the initiative in defining these standards and indicators.b. More differentiation in these Oversight methods by EASA and NAAs of the ATOs.c. Training of EASA and NAA-staff (management and inspectors) for the new approach (focus on safety objectives and performance).d. The ATOs to embrace the introduction and use of SMS in their daily business.e. Recognition that the transition is ‘not a free ride’ but requires investment by all of the involved parties.

About the AuthorJules Kneepkens has been the Rulemaking Director of EASA (2008-2014). Before joining EASA Jules was Director General of Civil Aviation in Belgium and before Director Civil Aviation in the Netherlands. Since the beginning of 2015 Jules has established with Ray Elgy, former Head of Licensing and Training Standards at the UK CAA, a new company with the name TEASAS. TEASAS takes a total system approach to aviation safety management to help ensure that safety risks are properly identified and to enable organisations to work together effectively in mitigating risks and enhancing overall aviation safety performance. (www.teasas.com)