By Richard Bray
Are there important lessons for IT security managers from the Big Blackout of ’03? Maybe not. In government offices across Ontario, most essential computers kept running through the outage; people returning from their unexpected holidays found they could log on and resume work more or less where they left off.
IT survived because it was ready. Most IT departments took advantage of the Y2K review to replace hardware and software, revise procedures and develop contingency plans. In the years before and after Y2K, a flood of virus, worm, exploit and “social engineering” attacks tested their defences and ability to recover. And, to their credit, senior management recognized the essential nature of IT in their departments and allocated the resources needed to ensure continuity.
On the Monday after the Thursday blackout, things in Toronto were quiet enough for James Ridge, Toronto’s chief information officer, to brief a journalist on how his department managed the crisis.
“Nothing can be less expected than what happened Thursday afternoon,” said Ridge, who has 14,000 networked clients at 460 sites. “. . . Our city data centre does have battery backup and the appropriate fail-safes as well as generator support, and all of that worked quite flawlessly for us. We have a well-ordered process and I think we executed it perfectly. We were quite stabilized through the weekend.”
All the major production systems stayed online, Ridge said, and people were able to return to work with few difficulties.
In Ottawa, almost a week after the outage, Greg Geddes, the city’s director of information technology services, reported that “we are trying to help out and keep the power consumption down, so a lot of the municipal services are offline right now.
“We have quite a number of our facilities running on generator power as well, to try and reduce the load on the power grid. All our major data centres are running on backup power as well, so that takes a fair amount of the power requirement off.”
Because the blackout came at the end of the day on Thursday, Geddes said, Ottawa staff were able to work towards Friday morning. “For the areas where we did have problems, we had most things back up and running the following morning for the buildings that were going to be opened,” he said. “We had the 1998 ice storm as an early rehearsal, followed by the Y2K planning, so there wasn’t much left to miss.”
Governments provided the same uninterrupted or quickly restored service in their outward-looking, citizen-facing operations. Most governments host their Web servers “off-site.” In Ottawa, for example, Magma Communications rents space to various governments, including some in the United States.
From the outside, Magma is a cement and steel cube with biometrically protected access points. On the inside, it is a system of control rooms, backup batteries and diesel generators designed to keep clients’ servers online through practically any emergency.
According to Ron Ethier, the Magma centre’s manager, “we were actually unaffected, other than that we were running on diesel backup power for the duration of the outage. It started around 4 p.m. on Thursday and I believe we were back on street power again by midday on Friday.”
The only other indication that something was amiss was a sudden drop in volume in the call centre, because with no power to their desktop computers, users could not connect anyway.
“There were some calls from large customers with Web sites hosted at the data centre, including some large government customers,” Ethier said, “but they just wanted to know if everything at the data centre was running, and we said, ‘yes, we’re running with diesel generator power and we don’t anticipate being offline at all during the duration of the power outage.'”
With a few minor exceptions, essential government data was available right through the blackout, either on internal networks or over the Internet. Ironically, many computers in government offices were functioning with nobody there to use them; while Internet service was available across the province, home computers were without power.
In the days and weeks after the blackout, there was a determined effort to assign at least some blame to IT. It may be more useful for other essential services to look past the few computers that didn’t work and study an entire sector, public and private, that did.
Richard Bray ( [email protected]) is an Ottawa-based writer specializing in high technology issues.
