Coffers hit by outages

If you think network downtime negatively affects your company’s bottom line, a recent report shows just how right you are.

Infonetics Research Inc. in San Jose scrutinized six Fortune 100 enterprises to see how network availability – or lack thereof – impacted each firm’s revenues. The analysis found that these guinea pigs were losing up to US$96,632 per hour thanks to network outages.

“We calculate two costs,” said Jeff Wilson, Infonetics’ executive director. “One is the cost of lost revenue because of network outages or service degradations for people who use the network to generate revenue, such as order entry. The second cost is loss of employee productivity because of network downtime.”

Revenue drain is the bigger concern for most companies, which makes sense, Wilson said. When a revenue-generating employee can’t access the network, his productivity goes down along with revenue generation. Productivity loss is relatively benign by comparison, although Wilson said most companies have a harder time understanding the impact of productivity loss.

In the study, companies lost up to one per cent of their revenue to network problems, which is a lot of money for some firms. “One of the companies we interviewed was a $40-billion company,” Wilson said. “One per cent of your revenue is significant at this level.”

Infonetics found that screwy servers are the primary cause of network downtime. Wilson said this is a function of critical mass. “There are lots and lots of servers and applications, and they’re hard to control.”

Network elements were also blamed for lost connectivity. Router, switch and Network Interface Card (NIC) failures accounted for one quarter of the problems that companies faced.

What’s an enterprise to do? First off, mind the network architecture, Wilson said.

“There are two theories: you can centralize your infrastructure and focus on making those points bullet-proof, or you can go with a highly distributed infrastructure and make sure you have lots of failover and backup, so if one server is down, people are re-routed somewhere else.

“Overall, it looks like the people building the distributed, highly diverse networks are having a better time minimizing outages. They might have more downtime events, but the events are less costly. The total cost is minimized.”

Nonetheless, redundancy, advanced failover techniques and other characteristics of distributed architecture can be more expensive to implement than centralized schemes. Wilson said equipment vendors recognize an opportunity here, and stand ready to swoop in.

“[Distributed network architecture is] good for the product managers. Of course, they’re not going to raise that issue very much.”

A look into network resiliency plans among Canadian organizations suggests that there are many ways to enhance uptime. At Vantis Credit Union in Winnipeg, for example, centralization is the name of the game.

“Everything is centralized here,” said Brian Hansen, director of IT. “We have load balancing to keep things up and running, should one piece of equipment fail. We have multiple network connections to the servers, so should one network card go down, you don’t lose the server completely.”

Hansen said network downtime is important at Vantis. Employee productivity rises and falls with network availability – no network, no productivity – and can impact the firm’s payroll, as well as its ability to serve clients.

“When the network goes down, we lose access to the banking system. Tellers have to stay late in order to post things and get people’s transactions up to snuff.”

At McGill University in Montreal, network availability also dictates client satisfaction, although in this case the clients are the school’s students, staff and faculty members.

Quan Nguyen, associate director, system engineering in the university’s network and communications services unit, said it’s difficult to assess the bottom line impact of network outages at educational institutions. Nonetheless, availability is instrumental in at least one, important respect.

“In a university environment, we have a lot of researchers. They’re usually working in the evening or at night. Whenever they have a bright idea, they want to do research. The network has to be available.…This is the only way we can be fired from our jobs, if we’re hindering research and education.”

Nguyen and his team stave off unemployment by constantly monitoring the network – 24-7 – and developing automated programs that alert the IT crew to problems before users run into trouble.

Nguyen said common sense plays a major role in maintaining network availability. For example, “Have a good inventory of your network. You need maps to troubleshoot a complex network. When you’re troubleshooting, that’s not the time to draw the map.”

As well, “make sure things are tested before you put them into production,” he said, explaining that although it sounds logical, you’d be surprised how many skip this integral step and mess up the network as a result.

Wilson said no matter the cost, smart architecture and tools can help companies save money otherwise lost to the void of network downtime.

In the Infonetics study, “these are people who have invested a lot already in infrastructure to reduce downtime,” he said. “But even one per cent of revenue when you’re a $40-billion company is worth recovering if you make an investment of, say $25,000, to identify the causes and fix them. You can recoup a lot of those costs.”