You have a great product, so you hired a hot Web designer to give your e-commerce site just the right look, poured a fortune into a state-of-the-art warehouse and a customer service support system, and now you’re all set to watch the dollars flood in.
Unless, of course, you forgot to involve the networking team in the planning and building of a high-availability infrastructure to support this new on-line operation. Infrastructure managers may not know much about showy graphics and colourful prose, but they do know how much bandwidth is required to keep up with 100,000 hits a day and what equipment is required to keep a network humming 24 hours a day, seven days a week. The bottom line is it doesn’t matter how much glitz and glamour an e-commerce operation has if it’s not available when customers want to use it.
The uptime four-step
There are four fundamental attributes to keep in mind when designing a high-availability network for e-commerce, according to Alistair Croll, CEO of Networkshop, an Ottawa-based network consultancy specializing in e-commerce.
These principles are: performance, delivering a completed page within the expectations of a customer; availability, ensuring a page is always available from a site, no matter what problems might be affecting the site; security, securing transmissions through SSL and restricting site users to areas they’re supposed to access; and scalability, making sure a site can grow by 100 per cent without breaking down.
From a technology perspective, there are several valuable tools no self-respecting e-commerce operation should be without, Croll said.
Perhaps the most important of these is load balancers, which allow infrastructure managers to share traffic equally among a group of servers. Croll noted load balancers also allow managers to better handle server failures or upgrades.
“With a load balancer you can make operational changes to your network without taking the network down,” he said. While a system manager works on one server, the load balancer will continue directing traffic to still-functioning servers, ensuring a Web site stays up.
In the area of security, SSL accelerators are useful, Croll said, as they take the processor-intensive security function off of the server’s back. A firewall with good traffic-shaping abilities is also handy, he added. And if a business is co-locating with an ISP, a VPN between the enterprise and ISP will probably be necessary so network staff can make changes to the e-commerce site securely.
Web caches can also be used to take more of the workload off of an e-commerce operation’s servers. The caches can be used to feed static information to users, while the servers handle requests for dynamic content only.
“Your servers are precious resources and you really only need them for dynamic traffic,” Croll explained.
When selecting equipment to power an e-commerce site, Croll said the needs of the site must always outweigh the name of the vendor on the equipment. This is because each vendor’s gear operates differently and is better-suited to particular operations.
“You should start with your requirements and then select a vendor – not the other way around,” Croll said. “There is a lot of complexity in vendor equipment and you shouldn’t let your applications change because of that.”
For enterprises that decide to co-locate their e-commerce operations at a service provider, the selection of the provider is very important, Croll noted.
“You need a good, robust, first-tier ISP,” he said. “Most companies can’t afford all sorts of backup. With a first tier ISP you get all that.”
Rick Segal, president of Toronto-based on-line retailer Chapters Online, subscribes to this provider theory. Chapters uses AT&T Canada to host its on-line operations.
“I tend to go with a telco like AT&T, because they’re used to keeping dialtone,” Segal said. “You really appreciate the grey hairs of the telecom guys.”
Paying for redundancy
As for how much redundancy is necessary for an e-commerce operation, Croll said it’s basically a question of how much downtime a business can afford. Once the acceptable downtime is determined, a firm can decide how much redundancy it needs to meet that downtime objective.
For outfits that never want to be down, Croll said Networkshop will co-locate its clients’ e-business operations at two ISPs. Each ISP will have a connection to the other and the Networkshop clients will have completely redundant equipment, including DNS servers, at each ISP. That way if the main site goes down, the backup site will pick up right where the primary site left off.
Chapters Online’s Segal is a big advocate of redundancy. All of Chapters’ network gear has a hot spare – the servers have hot standbys, its Cisco LocalDirector load balancers have redundant partners and its switches all have standbys.
“It’s simple for the people doing my hardware books,” Segal said. “Every time they see an order they multiply it by two.”
Although Intria-HP of Mississauga, Ont., doesn’t use its network to sell goods over the Web, it does use the network to handle most of the electronic delivery systems of Toronto-based bank CIBC. This includes CIBC’s PC banking and approximately 4,500 automated teller machine, 1,400 branch operations and 120,000 point-of-sale terminals. Given the amount of business-critical information Intria-HP supports, the firm’s vice-president of technology planning and technical services, Mike Somerville, knows a thing or two about high-availability networks.
One of the main pillars of Inria-HP’s high availability is complete redundancy. The firm has two operations centres – one in the east end of Toronto and one in the west. Somerville said the two are far enough apart that they can function as disaster recovery centres for one another.
Each site has double the capacity it actually needs so it can handle the traffic of the other site in the event of a failure.
“They’d have the capacity to be able to handle [critical applications] in a hot standby mode,” Somerville said.
Test live systems
Intria-HP has made sure its network can handle any problems by performing extensive destructive testing. Destructive testing involves actually simulating events on a live network to see if the network can handle those events.
“If you don’t test like that during the buildout of the network, you run into a number of real-life situations that cause the backup not to kick-in the way you want it to,” Somerville said.
For CIBC, backup options include a failover dial-up system, Somerville said.
“If the main circuit drops, or if any of the equipment in that configuration drops, the dial backup automatically dials in and users (in this case the branch) doesn’t even know they’re on dial backup – only the network control centre knows.”
But, Somerville said, situations could arise where the configuration on the branch’s end doesn’t realize the circuit connection is down and doesn’t switch to the dial backup. For example, a circuit card could be going in and out intermittently and that may not be enough to trigger the backup. The only way to find out if the backup will work properly is to test the situation on the network.
Intria-HP’s testing has included situations where the business-critical workload for one operations centre has been switched over live to the other operations centre.
Somerville said there is some risk involved in testing on a live network, but he believes it is the only way to find out if backup systems will work the way they’re designed to work.
“If you’re not actually exercising your business recovery plan, you can’t be sure you can count on it when you’re going to need it,” he said.
Making sure an enterprise’s own network is fully redundant isn’t the only step in building high availability, Somerville said.
“If you’re using public services, you need a fundamental understanding of how that infrastructure’s being built out, what the recovery capabilities are, even if it’s something you’re buying from someone else,” he noted. For its own public facilities, Intria-HP uses both AT&T Canada and Bell Nexxia to ensure redundancy.
Network managers also need to think about redundancy on the border between the public and private network. Somerville said Intria-HP has diversity on fibre routes running into its data centres and buildings, so if one route is accidentally cut by a backhoe, the other route will kick in and keep traffic flowing.
Building for the long run
Toronto-based hardware and software distributor Ingram Micro Inc. did not build its network to handle e-commerce, but they did build it to be extensible.
Dave Falconer, manager of Ingram Micro’s systems development group, said the network can support e-commerce operations when required to do so.
“For things like hosting our resellers’ Web sites…we wanted to be flexible enough with our systems to do that,” he said. Ingram Micro hosts its own Web site at the Toronto office.
Ingram’s network supports between 700 and 800 devices housed in the firm’s new office space and distribution centre. Because Ingram runs fibre to the desktop, the company has the capability to push 1,000Mbps out to end stations. Falconer said that capacity is necessary because the company plans to implement high-end customer relationship management applications and possibly streaming video on some desktops in the future.
Redundancy is key to Ingram’s network operations. The firm has data lines coming in from redundant central offices, so if one central office goes down the other one will be able to take over. For its own equipment, Ingram uses UPS systems and a diesel generator to protect from power outages.
“We’re capable of running up to 24 hours under our own power if we’re knocked off the grid,” Falconer said.
All of Ingram’s end stations are connected back into a group of 3Com CoreBuilder 9000s. Ingram has seven 9000s – five for its operations centre, one for the distribution centre and one in standby mode as a hot spare. Each CoreBuilder also has its own redundancy in the form of a spare concentrator blade in each unit.
Fibre to the desktop might seem like an expensive extravagance, but in the long run Falconer believes the fibre will prove cheaper than traditional Category 5 cabling. He explained Cat5 would likely have met Ingram’s networking needs only for the next five to seven years. Cat6 cabling was also considered, but the Cat6 standard hasn’t been ratified, so Ingram’s networking team settled on fibre as the best option.
Because electronic signals travel further along fibre than over Cat5 cabling, Falconer said Ingram was able to eliminate the need for wiring closets full of repeaters and hubs. This helped reduce the overall cost of the fibre implementation, he said.
With fibre to the desktop and extensive redundancy, Falconer believes Ingram’s network is set to handle any future challenges.
“We wanted to make sure our people were on-line and active in supporting our resellers as much as they possibly could,” he said. “And if we do move to e-commerce, we wanted to make sure we had enough bandwidth to host our resellers’ sites if it came to that.”
SIDEBAR
Dollars inside or outside
It can be an expensive process for a business to hammer together a network conforming to all the principles of high availability.
“My biased advice is to outsource [the e-commerce infrastructure] to someone who knows this stuff,” said Alistair Croll, CEO of Ottawa-based Networkshop. “We’ve all heard Bill Gates say you can plug Windows 2000 in and make a million selling handbags on the Internet, but it’s not that simple.”
To set up an e-commerce infrastructure in-house, Croll estimated a firm would need to spend between $30,000 and $40,000 in hardware, $2,000 per month to co-locate with a top tier ISP and between $120,000 and $160,000 per year to hire two technical staffers to maintain and manage the site.
With an outsourced managed service, a firm would pay only a fraction of that, because the costs would be shared amongst other businesses. Also, Croll noted, by outsourcing, a firm avoids the biggest pitfall in maintaining an e-commerce operation – retaining talent.
