Risk analysis: myths, confusion and common sense

Over the past decade, methods of risk analysis have been actively propagated for pharmaceutical manufacturing, and companies are strongly encouraged to apply these methods. However, do such risk analyses have any real value?

Broadly speaking, a risk is the possibility that a dangerous or unwanted event may occur. It could be financial, traffic-related or to do with equipment faults, safety (e.g. with nuclear power stations). People spend time every day estimating or analysing risks to protect themselves against unwanted events. The purpose of risk analysis is to understand the ‘Reasons–Consequences’ chain with a view to finding the best protection. But any new method should be supported by demonstrations of its effectiveness and practicality. The medicinal products sector, however, has a special feature in that no risk is acceptable.

EU GMP and ICH Q9

In 2010, the ICH Q9 Quality Risk Management document was included in Part III of the EU GMP Guideline. The introduction to EU GMP says: ‘The aim of Part III is to clarify regulatory expectations and it should be viewed as a source of information on current best practices’. ICH Q9 says: ‘Quality risk management supports a scientific and practical approach to decision-making’. These are promising statements, but are they correct?

Methods of risk management can be separated into general and ‘other’. General methods include flow charts, check sheets, cause and effect diagram (e.g. the Fishbone or Ishikawa diagram) and others. Other methods include Failure Mode Effect Analysis (FMEA), Failure Mode, Effects and Criticality Analysis (FMECA) and Hazard Analysis and Critical Control Point (HACCP), etc.

The following highlights a widely disseminated example of an FMEA method. It is stated that FMEA provides a ‘Quantity estimation of risk’ by carrying out the following analysis:

1. All risks shall be grouped based on the following evaluation criteria: Severity/Impact (I), Occurrence or estimation of probability of event (O), and Detectability (D).
2. Each criterion has a numerical value, e.g. from 1 to 5, where 1 represents the lowest risk and 5 the highest risk.
3. The Risk Priority Number (RPN) is calculated by multiplying the evaluation criteria. The RPN range is from 1 (when all criteria are 1) to 125 (when all criteria are 5). Thus the RPN grows in value with increasing risk.
4. The acceptance level shall be specified in advance. If the RPN < Acceptance level, then risk is low and no further action needs to be implemented but if RPN > Acceptance, action is required.

I would argue that this approach has three fundamental problems: a) The evaluation criteria are selected subjectively; b) values of a different type (or sense) are multiplied, which empirically is not allowed; and c) this arbitrary estimation is put forward as a basis for responsible decision-making when no risk should be acceptable.

The FMEA method attempts to compare the incomparable and thus its basic principle is not correct. As a short illustration, take the following two examples shown in Tables 1 and 2.

Example 1 (Table 1): The delay of a plane’s arrival is not dangerous (Severity is equal to 1), but can occur rather often (Occurrence is 5); a plane crash is dangerous (Severity is 5), but very rare (Occurrence is 1).

According to the FMEA method the detectability for both events is numerically 5 (both events would be noticed). Multiplying the evaluation criteria gives an RPN of 25 for both cases, so according to this method they are equivalent in respect of risk analysis. But, in reality, no-one would consider the ‘delay’ and the ‘crash’ of a plane as equivalent.

Example 2 (Table 2): An application of the FMEA method for a tablet press, published in GMP Report, No.3 p.144¹ provides a typical example provided to illustrate the method. The results of the example analysis are given in Table 2, which was constructed and published as an example to follow.

The acceptance level is specified as 27 and is not exceeded in any line of the table. The calculation for line 5 (patient death) gave a RPN of 10, which is less than 27. So the decision would be no action needs to be taken because the risk is lower than the Acceptance level. This so-called scientific approach also suggests that in terms of risk analysis a mix-up of products which could cause patient death (RPN of 10) is a lower risk level than having the wrong weight of medicine, which carries an RPN of 15.

This is, of course, nonsense; death and potential harm cannot be compared using any numerical manipulations. Common sense suggests that any situation where a danger of human death exists should be excluded without the need for any evaluation or manipulation involving numbers. It is not possible to compare the severity of an event and the probability of it happening. Dangerous events should be excluded and not averaged. An arbitrary chosen number (why 27 and not 7, or 80?) should not be used in responsible decision-making. It is better to make such decisions in an honest way, without hiding behind pseudoscientific methods. The manufacture of medicinal products should be entrusted only to professionals who understand the responsibility of their job.

Inspections and action

ICH Q9 (Part III of EU GMP) claims that the drawing of process flow charts to find critical hazard points, and of HVAC, water treatment and other schemes, helps manufacturers and the inspectors. ICH Q9 also suggests that manufacturers should arrange routine testing/control and document both the results and methodology. But does this really help, as these should already be available in the initial plant design drawings and the testing and control should already be in place as part of GMP.

This analysis may be useful to understand one’s own processes better, but is it necessary to request such additional methods rather that execute the GMP norms?

It is not possible to compare the severity of an event and the probability of it happening

Does risk analysis really help the inspector? One inspector argued in his article² that an inspector has not enough time to read, and cannot see, all the details during an inspection, and that papers on risk analysis prepared by the manufacturer make the inspector’s task of understanding the workings of the plant easier. As such, the inspector does not observe the primary documents (water treatment schemes, batch records, laboratory records, etc.), but secondary ones – papers that only partially reflect the primary sources.

There is a fundamental danger hidden in this approach, which is that the inspector makes a conclusion on compliance with GMP requirements based not on primary documents, but rather he observes their interpretation. It would be interesting to consider how a tax inspector would check the company on interpretations of financial documents made by people who are being inspected, and not on the actual documents.

When a customer buys a medicinal product at a drug store it should comply with primary documents and not exercises. If three days are not enough for the inspector to evaluate the plant, let him spend two weeks on a proper study. Consumers need only high quality inspector’s reports, not substitute fiction.

Rational risk analysis

So is there any rational purpose to risk analysis? Yes, but it needs to be professional and useful. For example, the build of a new facility should follow various steps:

1. The design stage – this is the fundamental step and many risks can be excluded at this stage simply by following GMP Guidance.
2. Construction of the facility according to the design, followed by commissioning and testing.
3. The development of documentation and operation following GMP requirements.
4. Selection of suppliers so as to exclude risk of any defect because of materials.
5. Training of personnel to decrease or exclude risk.
6. Carrying out OOS analysis to check the work, etc.

The real purpose of risk analysis should be to show how the facility design protects against:

• Cross contamination (layouts, airflows, pressure differences, materials, personnel flows, etc.)
• Mixing of materials and products
• Mixing of sterile and non-sterile products
• Non-sterility in aseptic processes
• Contamination (particles, viables…)
• Surface contamination
• Other key GMP requirements

Formalised risk analysis will not help this but may merely mask poor work. Instead the following actions should be undertaken:

1. Estimations of manufacturing compliance with GMP requirements.
2. Reviews of documentation to ensure compliance with GMP and to ensure that all arrangements/changes are executed according to GMP.
3. To organise a distinct analysis system for OOS deviations. It is the most important element of risk analysis for existing facilities. It allows the prevention of many dangerous situations, complaints and product recalls.
4. To arrange analysis of complaints, returns and product recalls.
5. To develop and undertake necessary measures based on the results of OOS analysis and to define whether the actions were effective.
6. To continue process analysis (OOS), analysis of reliability (failures) and removal of disadvantages.
7. To do all these jobs continuously and to arrange regular self-inspections.
8. To carry out trend analysis, to construct a database for deviations and failures, systematically, sorting out the reasons. To make an annual report on all findings and actions undertaken. To develop a programme of improvements for the following year.

This work should be properly organised; the experience of the company Nutricia, in the Netherlands, is a good example of clever and effective risk analysis. This company is well known as a manufacturer of nutrients for children. In 1993, a batch of product contained residues of disinfectants and was recalled from the market. This accident pressed the company to implement a risk analysis system. Rather than a formalised approach, the company applied an intellectual, ‘thinking’ approach focused on the need to achieve effective results.

This system had the following elements: 36 risk analysis groups were arranged at the head factory; 20% of the factory’s personnel were members of these groups. The mean number of persons in the group was five and each group met weekly for short working meetings. Internal audits (self-inspections) were arranged and an analysis of customer complaints was organised. Inspections of suppliers started to be conducted systematically and a factory database on contamination was arranged and was continually supported and revised.

Soon problematic areas were revealed involving personnel, contamination, raw materials defects, and out-of-standards deviations. These are not dissimilar to the problems faced by pharmaceutical factories. The scheme of work for a risk analysis was simple:

• to create a register of out-of-standards deviations (complaints, etc.);
• to find pinch points (problems in critical control points);
• to eliminate weak spots;
• to analyse the factory’s workflow;
• to find pinch points again, etc.

Typical risks for pharmaceutical manufacturing are:

• the wrong content in product
• contamination
• accidental mixing of products
• the wrong labels, etc.

It is time to reflect on the wide acceptance of the kind of tabulated or numerical methods of risk analysis described above. We all belong to the species homo sapiens which means ‘wise man’, so, why do we accept exercises such as the FMEA method and what is the driving force behind them? The answer can be perhaps found in the fact that the global turnover of validation companies amounts to billions of dollars. Many of them have a pseudo-sense of GMP (known more fittingly as Great Mounds of Paper) and provide new ways of making money in the name of ‘risk analysis’.

Everybody talks about manufacturers, inspectors and consultants, but what about the customers – who should be the main concern

More importantly, what first arrived as an ICH Q9 guide (i.e. a recommended document), was then transferred to the EU GMP as Annex 20 (again only as an informational document), then was changed to form Part III. The idea behind these moves is clear: to increase the role of risk analysis from that of ‘information only’ to a more or less mandatory level, bearing in mind that recommendations become mandatory for practitioners if an inspector expects that they are to be followed.

Everybody talks about manufacturers, inspectors and consultants, but what about the customers – who should be the main concern. They are not aware of how the quality of medicinal products is regulated, so what would their reaction be to the methods described?

This article argues that the ICH Q9 methods are wrong and misleading for users and give trivial results (i.e. result that can be obtained in a simpler way or which are obvious) and therefore are not suitable for use.

Risk analysis can be useful but it is not a panacea and has a limited area of application that should not be overrated

Risk analysis can be useful but it is not a panacea and has a limited area of application that should not be overrated. There is a particular danger that enforcing such methods can lead to the acceptance of unacceptable events. These methods, moving from the office of consultant to manufacturer, can be used by some to justify poor work.

It is necessary to promote a wider discussion on risk analysis methods, with all the pro and contra arguments, to formulate public opinion. GMP rules appeared 50 years ago because the requirements of how to arrange manufacturing facilities were needed. At first they were clear and transparent. Further editions have added a lot of confusing texts, clouding rather than improving quality.

Risk analysis could be considered the last straw as people realise that modern GMP is moving too far from its primary purpose. It is time to clean up GMP guidance and create a high quality normative document.

References

1. Dr Antje Knoll. GMP Report, No 3 Risk Management in the Pharmaceutical Industry, 2008, pp139–152.

2. Rudolf Völler. GMP Report, No 3 Risk management in the Pharmaceutical Industry 2008, pp11–21.