Metrics, Key Performance Indicators & Risk Management
Now KPIs are very interesting to us, as they represent the “holy grail” of what us “quants” are looking for. And while I agree with Geer, Jaquith and Soo Hoo when they tell us that the future belongs to the quants I think that “quals” too, look for KPI’s of their own. After all at the heart of the “checklist vs. risk” debate is the simple question - “What really is a useful Key Performance Indicator for Security/Survivability/Risk Management?”
Now in this debate, it isn’t that the risk geek claims that “compliance” to a checklist provides no value. No, we acknowledge that such a statement is a useful piece of prior information. We just don’t believe it is a Key Performance Indicator (and yet there a still those that claim that this is the only KPI we should ever use). But it *is* something, right? It is certainly evidence that we can use (for FAIR, it is one piece of prior information that we can use to develop Control Strength).
A KEY PERFORMANCE INDICATOR IS JUST THAT, “KEY”
This distinction of usefulness is important. Indicators (or metrics, if you’d like - I’m tending to use the terms interchangeably, and perhaps I shouldn’t) have different meanings to different folks. They can be priors, posteriors, and often both. Chandler talks about this distinction in his blog post:
For a non-security example, consider Gross Domestic Product. This is total value of goods and services produced by an economy, typically a country. Economists along with government and financial leaders use this as The Number for measuring how well or poorly an economy is doing.
There’s only one problem with it: it’s largely useless to the Average Joe who’s getting up and going to work every day to actually produce the Product.
He’s more likely interested in a KPI like “dollars earned per hour/week/year” or paycheck size, or something otherwise downward-focused and inconsistent across the economy.
What we should note about a particular KPI is its ultimate importance to the particular stakeholder - it means most to those to whom it should mean the most (duh). In Chandler’s example, GDP trends (and interpretation of trends) do have some relevance to the everyman as it may eventually effect his/her KPI - paycheck amount, and vice versa (the trends for one individual paycheck, even if it is one of 100,000,000 - means *something*).
FROM EXAMPLE TO OUR LITTLE WORLD OF RISK
So here’s why “Risk Management” and “Risk Analysis” shouldn’t be confused in my opinion. The risk for a discreet issue (say, some percentage of systems are unpatched) is different than a nebulous aggregate amount of risk facing the organization. So when we talk about failures in current approaches to risk management - Curphey is right. To decision makers, discreet risk issues can be seen to be simply a lot of lettuce. They know that having a lot of “red” is bad, but how bad? How can we develop a KPI from hundreds or thousands of stoplight ratings (or even quantitative analysis)? It’s akin to the impact of a Chandler’s everyman’s paycheck to GDP. Useful prior information but we just can’t aggregate them all together and say “Aha! The risk to us is X”.
This is, unfortunately, another mistake I see with ISO efforts - the aggregation of risk issues within the context of your ISMS, or even the most absurdly detailed “Enterprise Risk Assessment” cannot be said to be representative of the ability of an organization to manage its risk. Over time such efforts can provide you with metrics that are indicators of your ability manage risk, but they, in and of themselves, cannot do that much more for you than prioritize discreet tactical efforts. They are “paychecks” in Chandler’s example, but their aggregate is no GDP. And that’s what we need.
This concept of a Risk Management KPI is what our measurements should be pointing us to. They are why we should be measuring. It and it’s component metrics are what we should be using to make decisions. And tracking this KPI and it’s component metrics (they themselves could be little KPIs to various levels of management) should be the focus of how/what we manage. Not a discreet risk issue.
Tags: financial statement, Key Indicators, monitor your progress
