Did IT learn nothing from the blackout of 2003?

Five years after a major blackout cut power to most of Ontario, an IT analyst warns many firms are still unprepared for a disaster of this magnitude.

“On the power infrastructure side of things, there were some lessons learned but unfortunately they haven’t been resolved yet,” said Ross Armstrong, senior research analyst with London, Ont.-based Info-Tech Research Group. “For example, if you’re in a wide-scale power outage and you have your diesel generators that you bought five years ago after the last outage, if power goes out to the entire city or county or wherever you are, you’re not going to be the only guy looking for diesel to fill those generators.”

Shortly after 4:00 p.m. on Aug. 14, 2003, a failure of the Niagara-Mohawk power grid resulted in a loss of power for most of Ontario, New York, Ohio, Pennsylvania and surrounding areas. Power was restored to most of the Toronto area at 6:00 the following morning, but companies, managed service providers and government agencies were forced to implement what disaster recovery plans they had.

However, some were caught with their pants down.

“For organizations who had been growing more reliant upon electronic information over the last six or seven years, it became very apparent to many of these organizations that they did not have adequate disaster recovery plans in place,” Armstrong said. “It also woke people up to the fact that widespread disasters can and do occur here in comfy, cozy North America. 50 million people were affected by that.”

As a result, Armstrong said, many companies started spending more money on disaster recovery planning and paying attention to methods used to measure preparedness, such as recovery time objectives and recovery point objectives.

Indeed, the water in people’s ice cube trays had barely re-frozen when storage vendors like Veritas (now part of Symantec Corp.) used the outage to promote data recovery technologies.

At the same time, Armstrong said, companies started looking at technologies they could use to mirror their critical systems and stream data to off-site backup centres.

Though many firms lack adequate power back, the shortcomings in preparation are usually not the fault of IT staff, who find themselves trying to convince executives to spend the money.

“A real stumbling block is getting CFO approval for disaster recovery and continuity expenditures,” he said. “This is really an insurance policy that we’re talking about here and of course people begrudge spending money on insurance because they will probably never have to use it but if they do they’ll be glad they had it.”

And those who do pony up the cash often don’t provide enough to get them through one day of a blackout, Armstrong said.

“People who have bought diesel generators want to make sure they have large enough fuel reservoirs on those things,” he said. “I’ve spoken to a number of clients whose CFOs cheaped out at the last second and got the lesser quality model of diesel generator that had the smaller fuel reservoirs, and it turns out they only provided enough fuel to power it for two to three hours.”

After the 2003 blackout, Toronto had double the normal number police officers on hand the evening of Aug. 14, 2003. The power went out just before shift change, so officers who were originally supposed to go home at the supper hour stayed on.

And like the great blackout, more recent disasters have necessitated a callout of emergency responders.

For example, last Sunday, shortly before 4:00 a.m., an explosion at Sunrise Propane Energy Group Inc. in north Toronto resulted in the evacuation of thousands of nearby residents. Police deployed extra officers to the area to herd people away from the area and keep gawkers out. Even after the fire subsided, police continued preventing some residents from returning after officials found asbestos, a carcinogen, near the site.

The explosion and fire, which at press time was reportedly under investigation by the province of Ontario’s technical standard and safety authority, occurred immediately south of the Downsview airfield, east of Keele St. and north of Wilson Ave.

The fire kept Toronto Fire Services busy for days, and to notify key officials, the department uses a combination of text messaging, paging and phones.

“We still really still rely on telephone communications, in the event of an emergency,” said Capt Adrian Ratushniak, public information officer for Toronto Fire Services. “We do rely on phone and paging systems.”

He added they have a system that sends automated text alerts to notify people of incidents.

“Whenever we have a second alarm fire or higher, it’s a text message that comes through and describes the incident number, the day, time, location, the closest area the address is cross-referenced to,” he said. “That gives you a reference to search generally via a computer what the incident is about.”

Toronto Fire Services has no specific plans to upgrade the system.

“It’s basically working for us now so let’s not go any further than we need to go,” Ratushniak said. “That’s not to say we’re not going to advance to an automated system in the future but right now we’re using what we have.”

Related Download
Can we save the open web? Sponsor: Acquia
Can we save the open web?
Join the creator of Drupal, Dries Buytaert, in a discussion about the web’s evolution, how we can put the power of the internet back into the hands of the people, and how you can prepare your organization.
Register Now