“When written in Chinese,” John F. Kennedy once said, “the word ‘crisis’ is composed of two characters. One represents danger, and the other represents opportunity.”
Crisis in this sense hit Canaccord Capital Corp. last summer. The company, Canada’s largest independent investment firm with 26 offices and 1,200 employees, was infected with “Blaster,” a worm.
Blaster’s claim to fame: it exploited a months-old vulnerability in Microsoft Corp.’s operating systems. Its effect on IT systems around the world suggested that perhaps many companies were not as up on patch management as they should be.
“It may have come in with a laptop,” says Scott Collins, Canaccord’s manager, technology and infrastructure. Most of the 1,700-odd computers were OK, and “luckily our most critical system…is Unix-based. It wasn’t affected. However, the traffic storm that was created on the network actually slowed down that Unix server, hit the interface. We had a tough time getting connected, because the network was just on fire.
“I would say it was a half-day to three-quarters of a day of serious downtime, where end users were affected enough to say, ‘Hey, what’s going on?’”
Canaccord’s users are not the sit-back-and-relax types. Time is money. When Blaster lit up the company’s network, the IT department felt the heat.
“We realized that we needed to have a proper patch-management policy in place,” says Collins, pointing out that patch rollouts were taking nearly 20 days to complete — not good enough come Blaster. But out of this threat came an opportunity. Canaccord reviewed its patch-management process, found it lacking and undertook a project to improve the situation.
The company installed some new technology, subscribing to Microsoft’s Software Update Service (SUS) for software service packs and patches. Canaccord also purchased Microsoft’s Operations Management system to provide an overarching view of the enterprise IT infrastructure. And the firm started to follow Microsoft’s Operations Framework, a set of best practices meant to keep companies as updated as possible.
Most importantly, Canaccord changed its mindset. “The technology behind it is very important, because it reduces the amount of time it takes to deploy patches,” Collins says. “But communication is one of the biggest components: communication with the vendor; communication with the end user; communication with the stakeholders in each of the departments.”
In amending the communication flow between IT and other parts of the business, Canaccord rearranged the test-and-rollout procedure for service packs and patches. For instance, whereas pre-Blaster the company asked end users to test newly patched systems, these days IT does that job. Collins says the new method makes more sense. IT no longer bothers end users for testing, so his department can impose its own procedures on the process, ensuring that patches work the way they should. Not to mention, testing now takes substantially less time than it did when users took care of it.
As well, today Canaccord employs one person to watch over patch management. This person is tuned in to software vendors and security services firms, ever mindful that a minor program change could affect the whole corporation.
When that patch-minded person discovers a vulnerability or gets a new patch, a senior IT team decides what the next step is. The team consists of Collins and other senior IT folks at Canaccord. They review the patch and develop a course of action, which the team submits for executive sign-off. When that’s done, Collins et al. go through the rollout process. The crew always tells end users what’s happening.
“This happens really quick, usually over the period of a day,” Collins says. Of note is the high degree of executive buy-in that Canaccord was able to foster for its new patch management process. According to Collins, “if you get executive buy-in, it allows the IT department to formulate a plan and deploy patches as needed.”
Collins says the new patch-management plan is saving his company $86,000 per year. Most of the savings come from productivity improvements courtesy of the patch monitor, that employee who watches for incoming programs or problems. Now that he’s there, senior IT managers can spend less time worrying about patches and more time on strategic thinking.
Patches take three to five days to complete, he says. It’s a far sight better than the 20 days patching used to take.
David Senf, an IT analyst at IDC Canada Ltd. in Toronto, says there are certain best practices that companies can undertake to succeed with patch management: create a business case that compares and contrasts the cost of patch management with the cost of a security breach; garner executive buy-in; and train IT staff, who “should understand the patching process and the pitfalls that they are likely to encounter.”
Collins says Canaccord plans to extend patch management beyond the office walls. People working remotely would meet a quarantine area before being granted access to info served on Canaccord’s infrastructure. A program would check the remote user’s computer and verify that it’s running the most up-to-date software. If the computer isn’t properly patched, the program would send the requisite software to the remote machine. Once everything’s patched, the user would be allowed to enter the network.
Here’s to a worm-free future.
