Running through a disaster recovery drill

It’s 5:30 a.m., and the top executives and their staff of United Services Automobile Association (USAA) are summoned to company headquarters. In a room flanked by computers, 650 of USAA’s executives and staff have gathered to hear a terse announcement by the company’s CIO, Steve Yates. “A major U.S. bank has just reported a bomb at its headquarters,” he says.

Outside, employees are starting a normal workday, slowly trickling in to USAA’s sprawling 286-acre San Antonio campus. To them, it’s just another hot, muggy day, the kind that one expects at the end of July in south Texas. But deep below the company’s headquarters – in a concrete bunker built to withstand a Force 3 tornado or a direct hit from a 727 – things are anything but normal.

The day before, the corporate situation management team (CSMT), composed of business unit representatives and many of the company’s executive counsel – along with staff members from finance, human resources, e-business, general counsel and corporate communications – were told that the U.S. Federal Bureau of Investigation (FBI) was warning of possible terrorist activity at financial institutions planned for July 24. In addition, a hurricane was reported to be forming off the Virginia coast, posing a potential threat to USAA’s Norfolk office.

Yates’s report hushes the room. “We have a live event,” he tells them, setting off a flurry of activity as the team shifts into response mode. In truth, however, there was no live event. This was only part of a drill, an elaborate business continuity exercise USAA had devised to teach its executives and employees to deal with disasters – everything from anthrax and bombs to an outbreak of severe food poisoning. Before this daylong exercise, the company’s employees knew they would be participating in a drill. What they didn’t know, however, was the exact nature of the “emergencies” they would face throughout the day.

While USAA’s approach to continuity planning is extreme, so are the stakes. This Fortune 200 company manages US$65 billion in assets. Its Texas campus houses 16,000 to 20,000 people on any given day – roughly the same number of people who work in downtown San Antonio. With 5 million square feet of office space, it is one of the largest horizontal office buildings in the world (second only to the Pentagon). So if a major security event ever hit the facility, significant casualties are a possibility. USAA provides property, casualty and life insurance and banking, brokerage and investment management services, primarily to members of the military and their families. So, as an insurer, USAA is also in the risk management business and has significant experience dealing with unanticipated disasters.

While few CSOs could afford to run exercises this elaborate, even those executives swimming in the shallow end of the risk pool can learn some lessons from observing these well-planned war games. This story looks at how USAA developed a contingency plan that suited its risk model and how other CSOs can determine where they and their company belong on that continuum.

It’s 7:15 a.m. A bomb is found at the headquarters of a major East Coast bank. Reporting to the corporate situation management team (SMT) are individual business unit SMTs that relay to the top executives in the command centre what’s happening and keep the business units functioning when an emergency hits. Each SMT is composed of three smaller teams – red, white and blue – that alternate shifts.

As each new event is thrown into the scenario, the SMTs face the challenge of trying to understand its implications for their business unit, not only from a human perspective but also from a customer support perspective. Yates points out that the company is often dealing in “live-money” transactions, where members (USAA’s term for customers) want to sell stock, transfer money or get cash right away. In an emergency, in particular, people want access to their money, and in those situations USAA can’t afford to be unavailable.

It’s 9:00 a.m. A loud explosion is heard in the building. Several casualties are reported. USAA has learned to embrace Murphy’s Law. “In combat, anything that can go wrong will,” says Yates. “So you need to be working on instinct and training rather than emotion and fear.” Many of the events that were injected into the exercise were done so precisely to test that training.

In an emergency, company leaders won’t always be available. This is a principal tenet in USAA’s approach to continuity planning. The CSMT executives held a lottery first thing in the morning to simulate this loss of leadership, and they removed three executives from the exercise. Other individuals had to unexpectedly take over, testing their ability to suddenly lead without relying on the executive staff for guidance.

Moreover, business unit SMTs working inside the “bombed” building also had to simulate that team members were lost. The surviving members of each group had to figure out how to carry on without those coworkers. On the IT department’s team, for example, the entire group responsible for relocating workstations during the exercise was declared dead. In other cases, evacuations forced the situation management team members out onto the lawn where they had to try to keep their business unit functioning and their employees organized via cell phone.

The series of evacuations actually produced one of the event’s most interesting lessons. The employees of USAA’s life insurance unit were evacuated from their building and were supposed to be relocated to another area where IT was setting up computers and phones for them. But the process would take almost two hours. During that time, employees would be standing on the lawn in the hot Texas sun. An executive in the CSMT questioned leaving them out there. Was there a safer place to put those employees in the interim? How should USAA determine if or when employees could be allowed back in the building? How would thousands of people access their vehicle if their car key was still sitting on their desk? And was there an alternate transportation plan if the company needed to send employees home? Just imagine trying to quickly evacuate a football stadium full of people, and you can see the challenges.

It’s 10:30 a.m. The CEO has been confirmed dead. “We did that to let his succession plan unfold,” says John Blaha, USAA’s assistant vice president of business continuation. The property and casualty SMT ran through some of the steps it would have to take in such an event – like notifying the state’s insurance commissioner.

It’s 11:30 a.m. The hurricane has reached Category 4 and is projected to make landfall on the Virginia coast at 4:30 p.m. Even though weather and on-campus chaos were intentionally thrown into the exercise, reality also presented its own challenges.

A bona fide emergency call actually came in during the July 24 drill when a suspicious substance was found in the company’s cash processing centre. The USAA staff reacted appropriately. To avoid confusion, employees had been instructed to say, “This is the exercise,” before exchanging information about the different scenarios so that everyone would understand what was real and what was simulated. As a result, no one was confused when the real call came in.

“Within minutes, we had guys suited up, the security team in force, and the area cordoned off,” says Wayne Peacock, the senior vice president of corporate real estate. Although ultimately it turned out to be a false alarm, Peacock was impressed with the employees’ ability to quickly take what they were practicing and apply it in real life.

It’s 12:45 p.m. Employees begin to evacuate the campus. Part of continuity planning involves preparing for the unknown. The goal at USAA was to begin challenging people’s emotions so that they could learn how they would react and then plan for that reaction. Beyond faking deaths, the scheme called for simulated injuries. Courtesy of some artfully applied makeup, a dozen employees were gashed, caked with blood and placed on the lawn where other employees were being evacuated. To further the role-playing, the employees had been coached as to how people with those wounds would react. Obviously, the evacuated employees knew that the situation was fake, but the moaning and pleas for help from their injured coworkers added a dose of reality, and also gave USAA’s safety and environmental affairs group a chance to practice maintaining employee calm.

To add to the realism, employees exposed in the mock anthrax attack had to go through a decontamination shower set up for possible hazardous-material exposure (employees were forewarned to wear bathing suits). They were then escorted by specially trained USAA employees who guided them into a HazMat tent for further decontamination. After leaving the tent, injured employees were escorted to tarps where USAA’s medical personnel were on hand to patch up the wounded. While staffers rehearsed the evacuation and decontamination process, small groups of USAA employees acted as observers, making notes about possible improvements.

Far from resenting these elaborate machinations and the time away from their job, USAA employees are enthusiastic about these exercises. “Before 9/11, if you conducted a fire drill, people ignored it,” says Wendi Strong, senior vice president of corporate communications. “But now they don’t see it as an inconvenience; it’s a valued exercise – something that their employer is doing to protect them.”

It’s 1:30 p.m. and local news stations are onsite requesting a human interest story. In a crisis, communication is often the first part of the corporate machinery to break down. Recognizing this, USAA has put a great deal of time and money into building as many avenues and techniques for emergency communication as possible.

“We’re highly dependent on our internal communications network – video, e-mail, telephones and intercom,” says Yates. “If we had something really bad happen, all wires might be cut and we could have thousands of people wondering what to do.” During the exercise, the SMTs experimented with using both cell phones and walkie-talkies to communicate with each other. In addition, the company bought 18 satellite phones – at US$1,250 each – and dispersed them among senior staff in the event that the whole phone network goes down.

The company also has what Blaha, a former NASA astronaut, refers to as the No-Comm (no communication) plan. The senior staff and SMT members have laminated white cards with directions written on them to point executives to a location where they can go to meet up with the rest of their team in the event that something massive in scale occurs and the phones are jammed.

Not only is it important to know how to communicate in crisis, it’s also critical to know what to communicate. Strong wanted to test and find out how quickly her corporate communications team could draft a message as well as what kind of language they would use under pressure. So during the exercise, Strong and her group practiced what they would say to the company’s employees and customers. They wrote memos and press releases and communicated updates to employees over a limited number of public announcement systems.

Strong had the CFO go to an on-campus studio, where USAA has its own closed-circuit television system, so that he could record a message to the employee population. This exercise gave Strong a chance to test these messages with a number of employees to see how they, in turn, would react to certain kinds of language.

It’s 3:00 p.m. The exercise ends. How did they fare? The July 24 exercise was the largest the company had ever done and the first time it had included the San Antonio fire department and EMTs in a broadscale drill. During the exercise, the fire department helped evacuate employees and get them to medical assistance, and it also had the opportunity to interact with USAA’s own emergency personnel.

Now, if a real emergency occurs, the partnership USAA has built with the city will benefit both groups.

At the end of the day, all the teams talked through the major lessons learned, highlighting areas where improvements should be built into the company’s continuity plan. The next morning another meeting was held to analyze the exercise at a deeper level, and each situation management team presented the top three problems it had encountered along with a plan to fix them. The company documented all of those findings and actions, and set a turnaround time of one month to implement the fixes.

Obviously, continuity plans cannot exist only on paper. Regularly putting them into practice lets the company see how it would function in a real situation. USAA plans to continue running full-scale exercises at least once a year, with smaller exercises every few months.

“There are so many interdependencies today. It’s not just a physical security issue, it’s not just a technology issue, it’s not just a line of business issue, and it’s not just a corporate issue,” says Peacock. “They’re all going on at the same time. On paper, you can guess at how they fit together and how they interrelate, but until you’ve actually gone through the exercise, you don’t see how it might unfold. The more times you do it, the better prepared you’ll be.”

Contingency planning checklist

The elaborate machinations that USAA goes through in developing and testing its contingency plans might strike the average CSO as a bit over the top. Like much of security, the issue of continuity planning comes down to basic risk management: How much risk can your company tolerate, and how can that risk can be effectively mitigated? USAA has found that testing your plan is an inexpensive and important step. Here’s a contingency planning toolkit:

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Featured Articles

Empowering the hybrid workforce: how technology can build a better employee experience

Across the country, employees from organizations of all sizes expect flexibility...

What’s behind the best customer experience: How to make it real for your business

The best customer experience – the kind that builds businesses and...

Overcoming the obstacles to optimized operations

Network-driven optimization is a top priority for many Canadian business leaders...

Thriving amid Canada’s tech talent shortage

With today’s tight labour market, rising customer demands, fast-evolving cyber threats...

Staying protected and compliant in an evolving IT landscape

Canadian businesses have changed remarkably and quickly over the last few...

Related Tech News

Tech Jobs

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Tech Companies Hiring Right Now