The statistics are disturbing. An IDC Canada study released in August found only one quarter of Canadian organizations have a tested business continuity plan.
Another third have a plan but don’t test it regularly, says David Senf, IDC Canada’s director of security and software research, and 30 per cent have an “ad-hoc plan,” which in many cases is “not really a plan at all,” Senf says. The rest have nothing.
Dave Dobbin, president of Toronto Hydro Telecom Inc., a Toronto utility telco and data services provider that recently launched backup and archiving services, says he has seen businesses where continuity planning consisted of an IT staffer storing tape backups of critical data in his car.
“If shareholders knew what was going on,” Dobbin maintains, “there’d be lynchings.” Large companies — those that must answer to shareholders and securities regulators – are generally in better shape than smaller ones. Senf says 32 per cent of large organizations have a tested plan and only 11 per cent have no plan at all, but “that’s still a lot of organizations that don’t have a plan.”
“The big banks and most of big businesses are very well prepared and they test and exercise their plans often,” says Ann Wyganowski, Toronto chapter president of the Disaster Recovery Information Exchange (DRIE). “In small to medium-sized business, there hasn’t been the level of awareness.”
The problem is not that nobody thinks business continuity matters. Most businesses IDC surveyed thought they should spend more on the issue. So why don’t they? The top reasons, IDC found, are budget constraints, lack of management buy-in and “not a business priority.” They know it matters, but aren’t worried enough to do something about it.
“Until a small-d or big-d disaster happens which actually causes a negative business impact, a firm is less willing to shell out real dollars,” Senf says.
Business continuity planning can be complex, Wyganowski says. The first step is to analyze what could happen, how it would affect your business and how quickly the business must recover. A realistic plan recognizes what is essential and sets priorities. Some automated systems are non-essential — it may be possible to return to manual processes temporarily or wait until the system is restored. Others are vital.
Recover fast
When GHY International Ltd. began reassessing its IT recovery plan after consolidating all applications on a virtualized IBM System i server, the original objective was to be able to recover everything in four hours, says Nigel Fortlage, vice-president of information technology. But after looking more closely at the impact of a longer outage and the cost of quick recovery, GHY extended its recovery time to 24 to 36 hours.
On the other hand, says George Kerns, chief executive of Toronto-based Fusepoint Managed Services Inc., for an online trading system five minutes might be too long.
Businesses with applications hosted by third parties can rely on those providers to address many IT recovery issues.
Insurance Pay Canada Inc., a Toronto-based firm that provides insurance financing, has key applications hosted at Fusepoint’s data centre in Mississauga, Ont., with redundancy at Fusepoint’s Vancouver facility. Fusepoint can move a customer’s processing quickly in an emergency, says Kerns.
A walk-through
That leaves customers like Insurance Pay responsible for issues like staffing. “If we have to evacuate our building,” says Stuart Bruce, Insurance Pay’s founder and chief executive, “we can move our people down the block or into another location very quickly.”
Testing the plan without disrupting your business may seem daunting, but in fact, says Robert Boyce, manager of the Canadian security practice at consulting firm Accenture, “you don’t have to actually go through the process of actually shutting the system down.”
Nortel Networks Corp. regularly conducts business continuity simulation exercises, explains Maribeth Tessier, BCP consulting manager at the Brampton, Ont.-based telecom equipment manufacturer. The company gives a small team a disaster scenario and they walk through the recovery process.
“We don’t turn anything off. We don’t order anything.” If the scenario requires obtaining replacements for destroyed equipment, for instance, the team calls suppliers to ensure it is available, but wouldn’t have it shipped.
Some systems have scheduled downtime during which it’s fairly easy to run tests of things like switching to backup servers, Wyganowski says. Other tests can be run by bringing up backup systems without actually shutting down production systems.
Tessier says Nortel conducts failover tests on some key applications early every morning. Others are tested quarterly. The company periodically tests systems that are recoverable within 24 hours by shipping tapes between data centres. That sometimes uncovers glitches that haven’t been considered. In one weekend test, the team found tapes couldn’t be picked up until the next day. Finding another shipping service resolved the problem.
“It’s ensuring that you have alternate means of reaching your end,” says Tessier. “Plan A and B and C and D are very, very useful.”
Steve Taylor, vice-president of sales at AT&T Global Services Canada, says AT&T conducts regular “business continuity days” at facilities around the world. The company has trailers filled with telephone switching equipment to replace failed or destroyed facilities. Its tests involve dispatching these to different locations and switching operations over.
Exact state
It’s important not only that individual pieces work but that everything works together and that systems can recover to the exact state where they were at a specific time, says Yvette Ray, business unit executive for business continuity and resiliency services at IBM Canada. And no business continuity plan is complete that doesn’t consider suppliers and business partners’ ability to recover, another reason for planning matters, notes David Denault, general manager of AT&T Global Services Canada. “If you have an important customer come to you and ask for a business continuity plan, you may not have a customer any more if you don’t have one.”
IBM Canada has a business continuity and resiliency services (BCRS) centre where clients can test their disaster recovery capabilities. A customer takes a snapshot of its IT environment and brings it to the Markham, Ont., centre, Ray explains. “They bring a team of professionals to our centre and declare — in quotes, if you will — a disaster.”
IBM also provides contingency planning assessment services, helping customers determine if plans and testing are adequate. AT&T also conducts business continuity and security assessments for customers, Taylor says, and will develop continuity plans and even build recovery facilities.
The frequency of testing depends partly on how critical the systems are. “At minimum, every plan should be exercised once a year,” Wyganowski says, “but it depends on how often your environment changes.”
Plans should be reviewed quarterly, Boyce suggests, with a complete walk-through once or twice a year, depending on criticality of the system and the feasibility of a full-scale simulation.
GHY International is still ironing out details of its plan, which depends primarily on hosted service providers. Key applications for U.S. operations are already hosted, so GHY’s main concern is to ensure the hosting provider follows its official test plan. Applications supporting Canadian operations run on GHY’s System i, Fortlage says, but the software provider can host the software in an emergency. He foresees running tests twice a year.
Insurance Pay reviews its continuity plan quarterly, Bruce says, and regularly tests individual pieces. “We don’t go so far as to actually shut down the pipe to Fusepoint and see what happens,” he says.
SIDEBAR
Make the risks seem real
Once a disaster occurs, C-level executives will understand the importance of being prepared — but by then it may be too late, at least for the CIO.
“They know it’s a threat out there,” observes Steve Taylor, vice-president of sales at AT&T Global Services Canada, “but everyone has fiscal constraints.”
So how do you convince top management of the need to invest in business continuity planning? It’s a question of making the risks real, says George Kerns, chief executive officer of Fusepoint Managed Services.
He advises IT executives to pick a critical system vital to the company and ask what would happen if it were out of service for five minutes. Or what about an hour? “Keep on working up until you get to the point where somebody says, ‘Ouch!’,” Kerns says. Then ask what needs to be done to ensure that outage doesn’t happen.
It’s very important to tie the impact of any risk back to people and dollars, says Maribeth Tessier, BCP consulting manager at Nortel Networks. “That’s how you get the executives’ attention.”
If the CEO and CFO understand what the financial impact of a system failure would be, investing in preventing that failure will make a lot more sense — and understanding the potential impact also helps determine how much is appropriate to invest in prevention.
CSA offers guidelines for continuity planning
If you’re convinced of the need for a business continuity plan, but not sure what it should include, here’s good news. The Canadian Standards Association is developing a Standard on Emergency Management and Business Continuity Programs.
CSA Z1600 is a broad outline of what such plans should cover. “It’s really a high-level framework standard,” says Ron Meyers, project manager at CSA in Toronto. “It tells you what you need to have.”
CSA Z1600 is largely based on an existing U.S. standard called NFPA 1600, but unlike its American counterpart it includes a section on recovery planning. The U.S. standard will probably get a similar section in its next revision, Meyers says, but for now Canada is a bit ahead. Apart from this, differences between the two standards are mainly in things like the terminology used in definitions and in references to other documents.
Public comment on Z1600 closed in mid-September and the standard committee was scheduled to meet in late September to review and respond to all the comments received - and there were a fair number, Meyers says. The next step will be to prepare a final draft, which the committee must vote on. The hope is to publish a final standard in February or March.
CSA standards are voluntary, though legislators and regulators can require compliance. Government agencies or private companies could also insist that those they do business with comply. But for anyone seeking guidelines to developing a comprehensive business continuity plan, the new standard could be a good place to start.