More firms want proof their recovery plans and backup systems work when a disaster hits. Many are staging mock disasters that summon not only IT staff, but also call for quick reactions of others throughout the organization. That’s the view of Ralph Dunham, senior vice-president and national business continuity practice leader with Marsh Canada, Toronto.
At one time organizations and auditors were satisfied if a plan was in a binder and tested annually, says Dunham. Now executives and directors are challenging the plan and want evidence it works.
Some organizations have plans in place for five to ten years. In some cases, those plans worked fine for years, but suddenly they didn’t work anymore because companies increased their technology dependence, changed their IT environments, adopted new processes, or lost the people who knew how to make the plans work.
Now many companies are conducting full-blown exercises at least annually that amount to widespread disaster recovery fire drills, notes Dunham. They build more realistic scenarios that draw in not only the computer room but also users, suppliers and external auditors. Not only does the testing prove whether or not you can do it, it also strengthens organizational reactions.
“People can actually adapt to situations as changes are introduced throughout the course of the exercise. They learn from it, improve and get better,” Dunham says.
One client fired a disaster recovery consultant who wanted to run an exercise based on an airplane crashing into the building. The client thought the idea was preposterous. Soon after, 9/11 occurred.
Dunham knows of another company that stages disaster recovery practice exercises 11 times a year. Each one has a different twist. It’s not a company with a more critical recovery need than others, he says. The firm has just built a culture that understands that the whole enterprise depends upon proper disaster recovery execution.
Five components of effective recovery
Marsh has consulted in business continuity and disaster recovery for scores of CIOs in companies across the country. He maintains there are five components to effective recovery.
• You need to store data in an offsite location.
• You need a document of procedures others can follow in case of loss of staff.
• You need money to make it happen.
• You need a place to go with access to electricity and water.
• You need ongoing executive support and commitment.
The last is often the most difficult to secure, he says. Many executives spend 30 years or more making decisions based on business cases that yield return on investment. But it’s virtually impossible to build a disaster recovery business case based on ROI.
“If you’ve spent 30 years learning how to improve the bottom line, this doesn’t help. It only saves the bottom line,” says Dunham.
Outsourcing data centre backup
Disaster recovery is becoming more sophisticated and complex, and as a result, many companies are now outsourcing the function in whole or in part. A lot of medium-size enterprises in particular can’t afford the skills in-house to keep their plans current. A common practice is to outsource offsite data centres to organizations that provide those facilities, such as SunGard, IBM Canada, and Fusepoint Management Services.
George Kerns, president and CEO Fusepoint, confirms the trend to outsourcing. He says most firms can’t match the economies of scale his company can obtain by supporting the IT and data processing needs of many clients. This includes purchases in hardware and software and all the training and skills of technical support staff.
Typically clients get better performance at the same cost, or equal performance at less cost, from outsourcing. He says the former is more prevalent.
Fusepoint runs a round-the-clock operation. It is just as prepared to deal with an event that happens at 2:00 a.m. as 2:00 p.m. As a rule, most clients aren’t as prepared to cope with threats and disruptions in non-business hours.
Kerns says an outsourcing relationship often starts with this company supplying the offsite backup services. It frequently evolves into Fusepoint acting as the primary production site with the client’s facility turning into the disaster recovery site.
Alternatively, Fusepoint will suggest to clients that its site perform part of the production. If a problem develops in one site, the production load is transferred to the unaffected facility.
“You are not bringing up a cold site or a warm site,” says Kerns. “It is hot from the get-go because it is part of your production. And you have geographically dispersed it so that if there is ever an issue in one facility or the other, the load is transferred to the secure site.”
Executives in the firing line
Natural disasters and terrorist attacks have made C-suite executives more acutely aware of disaster recovery, says Chris Toushan, country manager for SunGard Availability Services Canada. Compliance issues are further heightening that awareness.
“C-level executives are being held more responsible for making sure the infrastructure is available so companies can support their customers and protect their shareholders,” Toushan notes. “If they don’t do proper due diligence to make sure a game plan is in place should the unexpected happen — and consequently the company loses market share, and its customers and shareholders suffer — they are going to be held personally accountable.”
More stringent executive accountability has already come to pass in the U.S. through Sarbanes Oxley, he says. Similar measures are coming in Canada through Bill 198, which is expected to provide a workable regulatory framework by next June.
A lot of modelling can be done to put a tangible value around the risk companies face, says Toushan. That financial exposure can then be balanced with the investments necessary to make the infrastructure secure. It involves examining business functions within organizations.
“A call centre might generate $50 million in sales every day. If it were out of action for one day, it amounts to a $50 million revenue hit, two days $100 million, and three days $150 million. You then balance that with what it costs to restore the centre. Recovery within one day may cost $100,000 a month or $1.2 million annually. Immediate recovery may cost $200,000 a month or $2.4 million. You then map out the revenue losses and recovery costs to figure out the sweet spot.”
No matter how you look at it, disaster recovery is becoming more than an insurance policy — it’s becoming an economic imperative.
QuickLink: 056598
–Ron Glen is a freelance writer and editor based in Toronto. He is one of Canada’s longest serving technology journalists.
