Security vendors have long been criticized for making grandiose claims about the efficacy of their wares. They have taken to presenting return on investment (ROI) arguments to justify the sales of their gear.
In his article , Bruce Schneier makes this point and further argues, quite effectively, that calculating a specific product or solution's potential ROI "is mostly bunk in practice."
I thoroughly agree. Further, I have seen vendors make a fast U-turn when a customer asks for a "guarantee" or "warranty" on the performance of the specific product in question. In my mind, any sort of ROI should be associated with a performance assurance. Or, one could equally ask the vendor, "OK, I see your ROI assumptions.
Now, since you won't give me a written warranty, how much should I discount the value of your ROI proposition?" The two cannot be separated.
Bruce also makes well-reasoned cases for the use of ALE , or Annualized Loss Expectancy, a risk view of security budgeting.
My only fear is that readers might, as I did at first, come to the conclusion that security is not measurable. Far from it -- there are many additional specific data points that can contribute to the evaluation of risk and security. Some of them are indeed even derivable directly from a wide range of security products and integrated solutions.
First, I don't believe security professionals should get dragged into the endless and futile debate over the value of information. That is not our job. The value of information should be determined at the highest levels of the organization. As Bruce says, in the event of a security incident, many of those values are intangible -- customer reactions, market perception, cost to re-brand along with the more traditional event mitigation and direct consequential costs.
But we can help. A few basic concepts should help understand that some aspects of security are measurable--and meaningful--if we find a common metric. Without getting into all of the math and formulas, that metric is time. From a risk and security standpoint, time is a common theme where we can bridge other information. A few mantras:
-- A security event should be detected rapidly.
-- A detected security event should trigger an alert as quickly as possible.
-- The cyber-first-responder should react to the alarm in as close to zero-time as required by policy and the detected and evaluated severity of the threat.
-- Threats should be thoroughly matrixed according to class, vector, severity, and time to infect, propagate and so on.
-- The greater the time it takes to detect and react to the event, the greater the risk.
Since we cannot measure, or be assured of the efficacy of primary security protection devices, we have to look at the problem backwards.
-- Anyone can measure the speed of their detection system; this is not just IDS, but password errors, login time outs, A/V and products of that ilk.
-- Anyone can measure the amount of time it takes to trigger an alarm and alert the first responder.
-- Anyone can measure and know (at least the range) of how long it take the responder to appropriately react to and cut off the primary threat vector.
Assuming that the protective products (procedures are more difficult to assess) fail or are breached, adding these three numbers gives us a worst case quantitative security and risk value: E' (Exposure) is the total of the amount of time a security event begins to the time it is mitigated technically. This, of course, does not take into account any of the downstream collateral damages, but gives us a handle on how to scale security spending.
My book Time Based Security outlines the principles in detail. A highly attractive fallout from this approach is embedding quantitative security metrics inside applications at the design stage. Employing the use of process control feedback is another means to inculcate time directly into the security process. This method provides additional high-speed security detection points to trigger lower-level alarms in an effort to stifle events before they become significant.
An advanced metrics application of Time Based Security is to analyze network administration topologies, develop Trust Factors for key operational staff, and use these data points to enhance the architecture's impact on security.
Security metrics also come into play when we realize that a vast component of enterprise security is the staff itself. By training staff to become security aware, and using their 'better-than-a-machine' skills to act as human firewalls and detection systems, the security posture necessarily increases. Poor security behavior by staff is arguably worse than many technical problems we face today.
With cyber insurance becoming more common, demonstrating that a company's security aware staff is a security asset can potentially hold down insurance and bank rates. In addition, the enterprise will be directly addressing the concerns of auditors and governance organizations, all the while making the likelihood of insider-triggered traumatic security events lower.
And this too can be measured. SCIPP International, a non-profit Security Awareness Certification company I founded with ITPG a couple of years ago, does just that. Using the SCIPP GAP (Generally Accepted Principles), organizations become awareness certified by passing pre-assessment and post-assessment gauges, vetted by pyschometricians, and completing annualized on-line training.
Consistently, for the last 10 years, security awareness has been measurable, and the effects are quite noticeable; security reports (detection and reaction) go up while security breaches go down. In this case, I could argue that security awareness provides a very high ROI, but the ROI measurement in this case cannot be, in good faith, tallied against a specific security data point. What can be measured in many cases, though, are the lowered rates, fewer help desk and operational problem, and less time-consuming intrusion by compliance groups.
Keep these kinds of security metric concepts in mind when looking at your networks. All the while, mind Bruce's warnings about self-interested claims on the product side. There is no perfect way to measure security's ROI, and claims to that effect should be met with skepticism. But there are tools that can help refine, and to a certain extent, define risk, mitigation and cost versus value.
A respected keynote speaker, seminar leader and author on numerous topics concerning information security and electronic privacy, Winn Schwartau has used his expertise in the field to create and build corporate security awareness programs, as well as provide consulting services to multinational organizations and governments. In 2006, Schwartau founded SCIPP International with the vision of raising and validating the security awareness of enterprise end-users on a worldwide scale.