As companies become increasingly dependent upon electronic transactions as a primary means of customer interaction, many have found that it’s no longer adequate to simply monitor overall performance. For businesses such as Starwood Hotels & Resorts Worldwide Inc. in White Plains, N.Y., monitoring each user’s experience has become critical.
Starwood, which owns or manages 700 Westin, Sheraton and other hotels and resorts in 80 countries, relied on its Web site to produce more than US$1 billion in booking revenue last year. If customers can’t instantly get the service they need, however, they take their business to the competition.
“Unless you are in a unique industry, Internet users probably have hundreds of options, and they are all a mouse click away,” says Keith Kelly, Starwood’s vice-president of Web technology. “Every guest interaction must be handled quickly and flawlessly.”
To gain better insight into their service to customers and employees, Starwood and other companies are turning to a new generation of appliances and software tools that let them monitor the level of service for each transaction.
Starwood opted for the RealiTea end-user monitoring system from TeaLeaf Technology Inc., installing the San Francisco-based vendor’s tools on passive capture appliances and servers.
The data it obtained helped Starwood spot defects earlier, resulting in better performance and availability metrics, fewer help desk calls and less chatter about Web site problems in Internet discussion forums. Other companies use the tools to measure the effectiveness of technology rollouts or to reverse-engineer reported bugs.
“If your Web site is or will be an important part of your business, this type of product is no longer an option. It’s a necessity,” says Kelly.
A tale of two architectures
Traditional system- and network-monitoring tools do a good job of looking at the overall health of IT systems but don’t provide visibility into the experiences of individuals, even though it’s the individual’s experience that matters. For example, diners care less about a restaurant’s rating in the Zagat guide than they do about whether they have to wait an hour to be served.
“Slow response times can cause impatience, frustration and errors as users press keys multiple times and potentially duplicate transactions,” says Teresa Jones, a senior analyst at Butler Group in Hull, England, in an e-mail. “Most of the existing tools can monitor only one aspect (e.g., the network) and not the end-to-end service.”
This is where the new tools come into play. These products, which provide data on individual transactions, are offered by niche vendors like TeaLeaf and Coradiant Inc., as well as by vendors of systems management products, such as Mercury Interactive Corp., Quest Software Inc. and Compuware Corp.
“There is a huge variety of tools provided by all the major systems management vendors and niche players,” says Tony Lock, an analyst at Bloor Research in Towcester, England. “Their sophistication and their applicability to different services has gotten a lot better in the last few years.”
There are several approaches these tools take. One is to use agents, either on a company’s own desktops or as a hosted service at remote locations, that periodically run a script to execute a transaction on the firm’s servers. These provide some insight, but not about what actual users are doing.
Other tools sit as agents on workstations to monitor response times or on the network to capture data. They provide data on actual user experiences and are necessary for public Web applications where companies can’t install user agents.
For example, AutoTrader.com LLC, whose Web site lists more than 3 million vehicles from 40,000 dealers and 250,000 private owners, chose Compuware’s ClientVantage devices, because they don’t require the use of agents.
“It is not feasible to have collection agents installed on an auto shopper’s PC,” says Paul Leadingham, director of technical operations at Atlanta-based AutoTrader.com. “By ‘sniffing’ end-user network timings, ClientVantage can draw a complete picture of the auto shopper’s experience from their PC at home through AutoTrader.com’s application infrastructure, including the elusive ‘last mile’ metrics.”
The drawback is that these types of tools work only when someone is sitting at a computer and generating transactions. And although they work for heavy traffic sites like AutoTrader, which attracts millions of visitors per month, they won’t work if you plan on installing a single agent on an employee’s desktop as a way to monitor performance of a workgroup. You lose visibility every time he takes a coffee break.
“You can’t monitor a transaction from a machine if the platform is not running at all, so you usually end up monitoring a large population of users,” says Lock.
“By using a combination of both architectures, you can gather a lot of valuable information.”
That’s the approach Starwood takes. It runs synthetic transaction monitoring, which provides general periodic information on the performance and availability of its site, but supplements that by using TeaLeaf to look at the individual user service levels.
“No longer are we estimating how the site performs for each guest,” says Kelly. “We know.”
Inside jobs
Such tools, however, are not limited to monitoring Web site traffic. “The monitoring devices have even been used on many internal and back-office applications to understand AutoTrader.com’s employee experience,” says Leadingham.
Patrick Clark, technical operations manager at Rexel Inc., a Dallas-based electrical parts distributor, is using Quest’s Foglight software to monitor user performance as it embarks upon a multiyear project to implement Oracle Corp.’s E-Business Suite.
“We were spending all our time chasing down performance problems, most of which were unrelated to our applications.
However, we did not have the monitoring and reporting tools to conclusively prove that the applications were not the problem,” says Clark. That changed, once Foglight was set up, a process that took three weeks working with Quest.
Side bar
A true overhaul in Halifax
Don Allen, senior director of IT operations at Drugstore.com Inc. in Bellevue, Wash., and Kamal Jain, director of ASP operations at BrassRing LLC in Waltham, Mass., both use TrueSight appliances from Coradiant. BrassRing provides recruiting services to approximately 15,000 enterprise users and more than 150,000 job applicants on a daily basis.
Jain says that it took his staff less than 30 minutes to do the initial setup of the TrueSight appliances. The company now uses TrueSight to report to its clients regarding contractual service-level agreements.
“Users of Web-based applications aren’t as willing to accept poor performance and errors as they were in the mid-’90s through 2002,” says Jain. “They don’t care what your operational or technical challenges are, because there are almost always alternatives to your offering.”
Drugstore.com uses its TrueSight boxes to monitor not only its Web site, but also its internal applications. For example, the company is currently upgrading the network at its call centre in Halifax and rolling out new tools for customer service representatives there.
Allen used TrueSight to establish a baseline level of performance before the rollout and then used it again to demonstrate the success of the project. He also uses it to resolve help desk calls. The tool doesn’t meet all of his needs; he still uses uptime- and network-monitoring tools, and he runs test scripts to check the Web server and query the database. But TrueSight does provide a way to solve problems faster.
“Rather than calling the user and asking them ‘What was the error? What did you click on?’ we can actually see the user’s actions click by click and reverse-engineer what the problem was,” Allen says.
Although they may not be complete solutions, monitoring tools do provide a different view of what’s happening, and they improve the problem-solving process when used in conjunction with other monitoring and debugging software.
“You need to know the limitations,” says Allen. “It is not a panacea, but it does give you what the user sees.”
Side bar
Monitoring to improve best practices
While real-time user-monitoring tools are generally used to monitor response times for quality of service, companies can also use the tools to improve business practices, says Tony Lock, an analyst at Bloor Research in England. “You should start with doing the basics, but then you should also look into the collateral benefits you can get from this,” he says. “You can improve the business by finding out different levels of efficiency and look to exploit the information in a business context.”
When analyzing the data, patterns start to emerge. For example, users may be processing a greater number of transactions during certain times of the day, a particular office may be running more transactions per employee than other branches, or an office may be using an application for different processes than other offices. One can look into whether these patterns are the result of differences in employee training or the way different locations have implemented business processes.
“You can get real business information out of these systems,” says Lock. “It could be that one office found a new way of running business processes so they can get more transactions out. If so, spread the word.” But he also cautions that collecting data on usage may run afoul of employee privacy regulations. Look into that before using the data to monitor productivity, he says.
QuickLink 063585