Risky Business

By Dr. Scott Rudge

“Risk Analysis” is a big topic in pharmaceutical and biotech product development. The International Conference on Harmonization (ICH) has even issued a guidance document on Risk Analysis (Q9). Despite the documentation available and the regulatory emphasis, these tools remain poorly understood. They are used to justify limiting the extent of fundamental understanding that can be gained on a process, while simultaneously used as a cure-all for management challenges that we face in pharmaceutical process development.

Risk analysis focuses only on failure modes. Failure mode effect and analysis (FMEA) was developed by the military, first published as MIL-P-1629, “Procedures for Performing a Failure Mode, Effects and Criticality Analysis” in November 1949. The procedure was established to discern the effects of system failure on mission success and personnel safety. Since then, we have found a much broader range of applications for this type of analysis. The methodology is used in the manufacture of airplanes, automobiles, software and elsewhere. People have devised “design” FMEAs and “process” FMEAs (DFMEA and PFMEA). These are all great tools, and help us to design and build better, safer products with better, safer processes.

Where Risk Analysis Falls Down

The FMEA is such a great hammer, it can make everything look like a nail. And when regulatory authorities are encouraging companies to use Risk Analysis for product design and process validation, the temptation to apply it further can be overwhelming. In particular, risk analysis is used in three inappropriate ways, in my estimation:

 

1. Decision analysis
2. Project management
3. Work avoidance

Quite often, risk analysis tools are used to guide decisions. Here, the pros and cons of selecting a particular path are weighed, using the three criteria of risk analysis (occurrence, detectability and severity) as guides. However, not every decision path leads to a failure mode or an outcome that could be measured as a consequence. And detectability and occurrence may not be the only or most appropriate factors by which to weight a consequence. There are excellent decision tools that are designed specifically for weighting and evaluating preference criteria. A very simple tool is the Kepner Tregoe (KT) decision matrix. Decision analysis uses very detailed descriptions of the decision to model the potential outcomes. The KT decision matrix is sufficient for many decisions, large and small. But if you really want to study the anatomy of a decision, decision analysis is the most satisfactory method.

 

FMEAs are sometimes inappropriately applied in project management, to assign prioritization and order in which tasks should be completed. This may be handy in some instances, but is somewhat misleading or inappropriate in others. The riskiness of an objective should not be the sole determinant in its prioritization. Quite often, the fundamentals or building blocks need to be in place in order to best address the riskiest proposition in a project. Prematurely addressing the pieces of a project that present the greatest risk of failure may lead to that failure. On the other hand, being “fast to fail” and eliminate projects that might not bear fruit with the least amount of resources spent, is critical to overall company or project success. Project management requires consideration of failure modes, but also resource programming and timeline management. FMEAs can be an element of that formula, but should not be the focus.

 

Finally, and perhaps most painfully, FMEAs are used to justify avoiding work. Too often, risk analysis is applied to a problem, not to identify the elements that are most deserving of attention, but to justify neglecting areas that do not rank sufficiently high in the risk matrix. Sometimes, the smallest risks are the easiest to address, and in addressing them, variability can be removed from the process. Variability is the “elephant in the room” when it comes to pharmaceutical quality, as has been concisely pointed out by Johnston and Zhang.

 

The FMEA is a stellar tool, and it is applicable to problems in design, process and strategy across many industries. Its quantitative feel makes practitioners feel as though they are actually measuring something, and can make fine distinctions between risks that they were unable to articulate before. However, the FMEA can be applied rather too widely, or sometimes unscrupulously, yielding bad data and bad decisions.

Informing the FMEA

By Dr. Scott Rudge
Risk reduction tools are all the rage in pharmaceutical, biotech and medical device process/product development and manufacturing. The International Conference on Harmonization has enshrined some of the techniques common in risk management in their Q9 guidance, “Quality Risk Management”. The Failure Modes and Effects Analysis, or FMEA, is one of the most useful and popular tools described. The FMEA stems from a military procedure, published in 1949 as MIL-P-1629, and has been applied in many different ways. The most used method in the health care involves making a list of potential ways in which a process can “fail”, or produce out of specification results. After this list has been generated, each failure mode in the list is assessed for its proclivity to “Occur”, “Be Severe” and “Be Detected”. Typically, these are scored from 1 to 10, with 10 being the worst case for each category, and 1 being the best case. The scores are multiplied together, and the product (mathematically speaking) is called the “Risk Priority Number”, or RPN. Then, typically, development work is directed towards the failure modes with the highest RPN.

The problem is, it’s very hard to assign a ranking from 1 to 10 for each of these categories in a scientific manner. More often, a diverse group of experts from process and product development, quality, manufacturing, regulatory, analytical and other stake holding departments, convene a meeting and assign rankings based on their experience. This is done once in the product life-cycle, and never revisited as actual manufacturing data start to accumulate. And, while large companies with mature products have become more sophisticated, and can pull data from other similar or “platform” products, small companies and development companies can really only rely on the opinion of experts, either from internal or external sources. The same considerations apply to small market or orphan drugs.

Each of these categories can probably be informed by data, but by far the easiest to assign a numerical value to is the “Occurrence” ranking. A typical Occurrence ranking chart might look something like this:

These rankings come from “piece” manufacturing, where thousands to millions of widgets might be manufactured in a short period of time. This kind of manufacturing rarely applies in the health care industry. However, this evaluation fits very nicely with the Capability Index analysis.

The Capability Index is calculated by dividing the variability of a process into its allowable variable range. Or, said less obtusely, dividing the specification range by the standard deviation of the process performance. The capability index is directly related to the probability that a process will operate out of range or out of specification. This table, found on Wikipedia (my source for truth), gives an example of the correlation between the calculated capability index to the probability of failure:

As a reminder, the capability index is the upper specification limit minus the lower specification limit divided by six times the standard deviation of the process. The two tables can be combined to be approximately:

How many process data points are required to calculate a capability index? Of course, the larger the number of points, the better the estimate of average and standard deviation, but technically, two or three data points will get you started. Is it better than guessing?

Do You Use Risk Assessments in Auditing?

Audits are a critical component of quality systems, but are they guided by formal assessments of risk to your products? In this world of ICH Q9, can you offer even a semi-quantitative justification for your audit priorities? We have spoken to many people in the industry, and almost all mention a risk assessment being undertaken prior to an audit. But we have not found many people that formalize that risk assessment, or keep it updated from audit to audit. Even fewer communicate their scoring of risk to either their internal clients or the vendor that has been audited.

A new trend in auditing is to use a form of risk assessment both before and after the audit. A popular form is the Failure Modes and Effects Assessment, or FMEA (see, for example, http://www.sre.org/pubs/Mil-Std-1629A.pdf). In a traditional FMEA, risks of failure are identified in a detailed fashion, and scored in three categories related to the failure’s probability, detectability and severity. Scoring is done on a semi-quantitative or relative basis using an arbitrary scale such as 1-10. For an audit, you might use the same categories as they relate to a particular vendor's (or department's) ability to deliver a product or service, failure free. You could organize your FMEA according to the critical quality attributes of the product or service being delivered or according to a list of requirements from a guideline or the CFR's. Your FMEA should receive input from affected departments, and should be used for prioritization of audit items. You should have the FMEA in mind as you conduct your audit, and remember why various items received high prioritization. You may change ratings for probability or detectability based on what you observe. If instead, you confirm your evaluation, you should probe remediations that decrease your firm's primary concern. A remediation that addresses detectability, when the issue was probability, likely won’t mitigate the risk of failure.
When you return from your audit, rescore the FMEA with assessments based on your observations and data that you collected. Make sure that you share your analysis with the stakeholders. And monitor the performance of the vendor until the next audit; the data will help inform your next FMEA.

Do you already use FMEA's in audit preparation and reporting? Let us know your practices.