In June 2002, as Washington Mutual Inc. moved toward finalizing contracts for a huge Multi-protocol Label Switching-based (MPLS) network, CIO Jerry Gross was getting nervous.
The United States’ seventh-largest bank was poised to commit millions to the project, a gamble of sorts on a technology used more by big carriers than corporations. In fact, WaMu wanted MPLS for the same reasons a Tier 1 ISP does — it scales readily; segments easily, with connections and teardowns handled automatically; and handily accommodates big surges in demand, says Nabil Badr, the company’s resident protocol genius, who architected the network.
Confident as Gross might have been about his backbone choice, he was worried about the stability and viability of the primary carrier with whom he was negotiating — MCI, then known as WorldCom Inc. With word on the street putting MCI in deeper financial trouble than its executives were letting on, Gross flew to the carrier’s New York headquarters for assurances. Over dinner, then-CFO Scott Sullivan convinced Gross that the rumours surrounding the carrier were baseless and that everything would be fine.
So Gross was probably more shocked than most when, less than a week later while watching CNBC in his Seattle office, he saw law enforcement officials leading Sullivan away in handcuffs. “I’ve been involved in some white-knucklers in all my years in business, but this one took the cake,” Gross recalls.
With MCI in such dire straits, Gross needed to get even more pragmatic and contingency-minded than he had been regarding this massive network overhaul. Gross and his network staff still believed that the carrier was the best in the business to provide MPLS. ISP UUNET, acquired by MCI, pioneered the technology in the 1990s. It uses MPLS to run its own massive IP backbone, which comprises much of the public Internet. So rather than ditch MCI in favour of AT&T Corp. or another carrier as the primary service provider, WaMu structured its MPLS contracts assuming MCI would go broke and cease operations. That is, it spelled out terms for shunting all WaMu traffic on MCI’s backbone to back-up carrier AT&T. The MCI and AT&T contracts also contain such unique terms and conditions as the ability to change the mix of services without penalty at any time, Gross says.
The financial services company had plenty of incentive to get this MPLS project right and minimize risk. WaMu’s business spans consumer banking, mortgage lending and insurance in the U.S., with assets totalling more than US$286 billion. The MPLS network, connecting 3,000 locations and the 60,000 people who work at them, would help fuel unprecedented business opportunities. WaMu, for example, relied on the MPLS network in opening 32 bank branches in Chicago in one day in August — a figure of which the company is justifiably proud. “Our network takes shocks like adding that many branches, and still the needle stays on ‘normal,’” Badr says. “MPLS can take it.”
Conservative cost estimates for such a large-scale MPLS deployment range in the tens, if not hundreds, of millions of dollars. While WaMu officials won’t divulge budget information, they did verify that first-year ROI was a 25 per cent reduction in IT operational costs, a figure they forecast will remain constant over the next three years.
While few companies can warrant such a massive network overhaul, WaMu was at a crossroads. Gross describes the situation he encountered his first week at WaMu, in August 2001: intranet pages took three minutes to load; the network went down a lot, but no one could explain why; and remote connectivity was so bad that he opted to use Hotmail when working at home rather than try to access the corporate e-mail server.
Then consider the thousands of bank tellers trying to get account information, or the multitudes trying to close on the best interest rate, or the financial services company’s inability to handle its own internal business processes. Applications had been developed and then shelved because the data infrastructure couldn’t support them. Long outsourced to IBM Corp., that data infrastructure had been intended to support nothing more than general office applications such as file serving, e-mail and accounts payable.
Part of the problem was that the old network couldn’t scale enough to meet WaMu’s rapid expansion — an acquisition binge started in the late ‘90s wrapped up with purchases of three financial institutions in the past year. In the past six years, the number of WaMu employees more than doubled from 27,000 to 60,000. The number of locations also more than doubled, to 3,000 nationwide.
“It was a burning platform,” Gross says. “There was a basic need for a scalable, stable, hardened network that could support WaMu’s growth.” And although WaMu scaled back acquisition plans, continued growth there would be. The company planned to open as many as 300 financial centres through 2004. WaMu was willing to prioritize IT spending on a companywide data network that would span local, metropolitan and wide areas — and eventually voice and wireless data traffic. Plug and play of new sites was an imperative, given the company’s growth projections.
“MPLS was the choice for flexibility and scalability, which makes for a better total cost of ownership,” says Badr, noting the company also considered fully meshed hub-and-spoke and ATM-based backbones.
And so WaMu created a six-tier architecture using the MCI and AT&T services and network gear from Cisco Systems Inc. and Juniper Networks Inc.
The new LAN architecture provides 75,000 Ethernet ports (a mix of 10Mbps, 100Mbps and 1Gbps links) at branch banks, financial centres, home loan sites, corporate office buildings and for automatic teller machines. Each WaMu site connects to one of multiple metropolitan-area networks (MAN) for connectivity to the WAN. A WaMu-developed node, called a supercore, further consolidates MAN traffic across any of three different SONET OC-3 lines, which operate at 155Mbps. From the supercore, traffic passes onto any one of nine MPLS nodes, which in turn collapse into a three-node backbone. Back office, system-to-system and dial back-up layers round out the six-tiered WaMu network.
Design and testing began in the fall of 2002, with implementation following through 2003.
WaMu holds network specifics close to the vest, not willing to share too much information for competitive reasons. But Mike Spalter, team leader for WaMu’s Infrastructure Services Group, did explain how the company ensured such a smooth rollout. WaMu partitioned the supercores from each other and from the rest of the network until they were certain each was stable, he says.
Also critical was teamwork among WaMu’s IT staff and the MPLS players. Spalter describes 60-person meetings at which all the vendors were represented so blame couldn’t get shifted and everyone could agree on cooperative action. Offloading a lot of responsibility to its vendors helped WaMu achieve a “remarkably” smooth and quick implementation, the IT folks say.
To ensure the network continues operating seamlessly, WaMu holds monthly quality improvement meetings during which vendors present their network statistics in front of each other. “It gives the frontline vendors the tools they need to manage the back office,” Spalter says.
WaMu and its carrier partners manage the MPLS network from three network operations centres (NOC) — one each owned and operated by WaMu, MCI and AT&T. All the major vendors and service providers have full-time staff working in WaMu’s NOC.
Gross and the network team are under no small pressure to leverage the MPLS backbone. And while it’s not a front-burner action item, WaMu is looking at the MPLS network as a means to handle VoIP traffic across the corporation and to be able to create virtual call centres rather than invest in physical infrastructure for customer service. Gross also sees the MPLS backbone as a means to tie local wireless LANs to the MAN and WAN and even broadcast interest rate changes to bank customers or inform them of other market news.
But for now, Gross and his team have their hands full. The good news is, what looked like a huge risk in June 2002 already has begun paying off for WaMu by late 2003.
“Downtime is all but gone,” says Gross, citing a network uptime statistic of 99.997 percent, up from 97 per cent when he joined the company. “And we can deploy new capabilities much faster. Our operational excellence wouldn’t be possible without IP applications. How do you calculate the cost of not having an intranet — millions of dollars? MPLS has transformed how we grow and how productive we are.”
