IT dept should have disaster recovery plan in place, experts say


A lot has been written about what companies should do to keep things running when a Hurricane Katrina-size disaster strikes. But what about those who come in to clean up the mess? For Anthony Cicco Jr., then CIO for the U.S. Government Accountability Office and now a director at SRA International Inc., this meant immediately building a new portal for staffers to coordinate their actions.

“The GAO managing director overseeing the recovery asked if we could put something together that would allow the teams to coordinate their work, share documents and keep up to speed on what other people were doing,” says Cicco. “The key goals were to make it easier for them to plan their work and to ensure efficiency by avoiding overlap.”

Congress had assigned the GAO the task of reviewing the hurricane recovery efforts. The job entailed the coordinated efforts of 13 mission teams specializing in areas as diverse as banking, public health and flood control — as well as the participation of staff from the U.S. Department of Homeland Security, the Federal Emergency Management Agency (FEMA) and state and local agencies. Cicco presented a plan within four days, and a new portal — Hurricane Central — was online in two and a half weeks.

The companies that do best in an emergency are those that have their disaster recovery plans firmly in place. In the GAO’s case, the work started well before the disaster. About a year before Katrina struck, IT staffers were volunteering their time to explore the issue in a skunk works project, says Cicco.

“We were forward- thinking enough that we were in the right place at the right time,” he says. “To the extent that you have a standard infrastructure with standard Web-based tools and a staff who knows how to use them, you can turn something around like this a lot easier.”

The challenge the GAO faced was that it didn’t normally have multiple teams working on a large-scale project. Katrina operations, however, required a dozen separate teams working on as many as 30 simultaneous missions. Many of them needed access to similar information and documents.

“We had no way of sharing such documents and information in an easily searchable, readily accessible way,” says Bill Jenkins, the GAO’s director of homeland security and justice issues. “Generally, prior to Hurricane Central, staff had to search for such information across a number of engagements by job code, which assumed one knew the job code for each of the 30 engagements.”

There was also the matter of making redundant information requests of FEMA and other agencies, which distracted them from their vital recovery missions.

Cicco’s Hurricane Central portal proposal addressed each of those issues. It contained all the key information, such as national plans, reports and congressional testimony, as well as state and federal contacts. Users could access support services for travel and videoconferncing, a calendar listing the GAO’s Katrina activities, and a discussion forum. Team members could post their current work and access the GAO’s document management system.

“Four things make collaboration work: You have to know who’s involved, what they are doing, how they are going to be connected and what the time constraints are,” says Jessica Lipnack, co-author of Virtual Teams (Wiley, 2000). “It looks like they hit all four of these.”

Another key factor was balancing speed and usability, since federal workers couldn’t spend a year designing and testing the system. From a technology standpoint, this meant using existing resources.

“The most important lesson is that they built it in a hurry using Apache and HTML rather than formal enterprise portal software,” says Tony Byrne, founder of CMS Watch in Silver Spring, Md. “When you are in a hurry, simpler is better.”

The portal technology was the easy part. The harder part was making sure it met the users’ needs exactly. “My biggest fear was that we would build something real quick that had no value to anybody,” says Cicco. His team worked side by side with a customer advisory group that could give quick feedback on portal iterations so the system could be continually refined.

“It was also a good model of how the IT folks and analysts could work together to create IT tools that enhanced the GAO’s ability to do its work effectively and efficiently,” says Jenkins. “Throughout the process, IT was constantly asking questions about how they could make the site both more user-friendly and useful.”


The site was up within three weeks, enabling faster access to information and tighter coordination between the teams.

“It would have been considerably more difficult to coordinate all of the Katrina-related engagements across the GAO and would have resulted in greater duplication of data requests from the DHS to other agencies,” says Jenkins. “It would also have taken more time and effort to answer congressional inquiries about our Katrina-related work.”

Hurricane Central no longer needs to support the number of Katrina projects it once did, but its underlying technologies are being put to new uses at the GAO.

“Although our Katrina- related work is winding down, the portal has proved so useful for ongoing emergency preparedness and response work that we continue to keep it updated,” says Jenkins. “The portal has been the model for creating a similar portal for our work on a potential influenza pandemic, which involves a number of GAO teams.”

Byrne says that keeping things simple made the site a success — and that’s something other organizations can replicate.

“See what value you can bring by building a ‘small p’ portal with existing tools and technologies already resident in your enterprise, before investing in new tools that will — by the time you purchase, install, customize, pilot, debug and roll them out — push your project back at least a year,” he says.

Lipnack agrees that it’s not necessary to go overboard building a portal. “You can throw all the technology in the world at these things, but unless there are good work processes, good people processes and a lot of trust, it just isn’t going to happen,” she says.

Cicco did this by establishing a tight working relationship with users. He says the GAO’s portal project includes a “rigorous process simplification” approach, so data is entered once and the system builds the documentation needed for an audit.

Although Hurricane Central was a success, Cicco says if he had it to do over again, he would break down the deployment of new features to make it easier for users to absorb the data. The main lesson is that organizations can get far more out of existing resources than they realize, so they don’t have to look for something new.

“Organizations probably use 30 percent to 40 percent of the functions of their existing technology,” Cicco says. “Just by giving people time to research and get under the hood of what you already own boosts productivity.

“Nobody could foresee the events of Katrina,” he adds, “but if CIOs are constantly looking at how to make jobs easier, they will be in a better position to respond.”


Related Download
3 reasons why Hyperconverged is the cost-efficient, simplified infrastructure for the modern data center Sponsor: Lenovo
3 reasons why Hyperconverged is the cost-efficient, simplified infrastructure for the modern data center
Find out how Hyperconverged systems can help you meet the challenges of the modern IT department. Click here to find out more.
Register Now