Virtual server technologies used in preproduction environments promise cost, time and labour savings, yet the same tools left unchecked can result in complex configurations, wasted resources and management nightmares for IT staff.
Virtualization removes the physical server constraints of test environments and enables sharing of resources among IT staff to make test work easier, but its use must be carefully controlled, industry analysts and IT professionals say.
"One of the pitfalls of using virtualization in test environments is the proliferation of images, especially when testing multiple configurations across different operating systems," says Carey Schwaber, a senior analyst at Forrester Research. "There has to be a real effort around controlling this environment with policies to prevent the environment from growing too much or becoming unused resources."
Avoiding test-server sprawl
Tim Antonowicz, systems engineer at Bowdoin College in Brunswick, Maine, says virtualization helps his team test software without requiring the build of a new operating system or cluttering a developer's workstation with another piece of software. He has 55 test stage, or sandbox, virtual machines (VM) running.
"Sandboxes are basic VMs where we test and evaluate various software offerings without expectation. If we want to try out something new, run a beta version or just play with a new idea, we roll out a sandbox VM," Antonowicz says.
Using virtualization in such a way -- as a tactical tool for testing -- is common. But most IT organizations haven't standardized their use of virtualization for testing across the enterprise. Different IT groups wind up operating their own pockets of virtual servers that aren't always properly managed or decommissioned. Industry watchers argue the benefits of using virtualization in test labs have yet to be fully realized because of these inconsistencies.
"It is important to have consistency when testing, and IT needs a comprehensive management approach to ensure proper co-ordination between physical machines and virtual resources," says Melinda Ballou, a principal analyst at IDC.
To help IT managers gain control of their testing resources, virtual test lab management vendors have been coming out with new tools.
Vendors such as Akimbi (acquired by VMware), CollabNet, VMLogix and Surgient have emerged in the past two years with products aimed squarely at those enterprise companies using virtual server tools to quickly build up and tear down testing environments. The products include automated features that track virtual machines and capture configuration data to be stored in libraries for future use.
For instance, Akimbi's Slingshot product, now VMware's Lab Manager, lets IT managers build a software test infrastructure to automate the setup and teardown of multiple VM environments. Surgient's Virtual QA/Test Lab Management System speeds the test process for enterprise IT managers by consolidating test infrastructure and making it possible to automate the setup and teardown of complex test configurations on demand.
IT staff at Sisters of Mercy Healthcare in St. Louis, Mo., turned to VMware and Surgient when they realized the prospect of upgrading 24,000 desktops for a workstation refresh would drain staff resources without delivering the desired results.
"We had a desktop refresh cycle that involves all the computers in the enterprise being upgraded to the same operating system and the same lockdown strategy. We had multiple environments we had to bring up to speed," says Brian Boresi, manager of client engineering. "Doing that across 24,000 workstations, to say the least, is labour- and time-intensive, too much for us because we have to follow a very rapid deployment schedule."
While the IT team realized virtualization was the only realistic option for such a large desktop rollout, Boresi says he knew they needed help managing the test lab as well. Rather than have an IT staff member physically meet with each desktop owner to determine application requirements, Boresi says Surgient enables his team to automate the process of creating multiple configurations in the test lab and change those configurations based on the user workstation environment.
"We currently support 600 applications, have a short turnaround time and aggressive rollout schedule. There is no way we could do this without an automated way to test and deploy these applications," Boresi says.
Virtual lab limitations
Still, virtual test lab management tools won't be enough to stop the environments from going awry, some say. IT organizations must define what can be tested, approach it with best practices and ensure anything tested on the VMs also is run in tests on physical machines before hitting production.
With that in mind, Sisters of Mercy Healthcare uses a virtual environment for one of three stages of tests and always completes a test run on physical machines before going live.
"We deploy an application package to production workstations in a pilot stage prior to going live. This way we make sure the software meets all the requirements and doesn't hit any snags specific to a physical machine and performs as expected," Neubauer says.
Edward Christensen, director of technical operations at Cars.com in Chicago, say he refrains from load or performance testing in the virtual test environment. "We limit our virtualization to functional and integration testing only. And unless your production environment is also virtualized, it shouldn't be used in performance testing," he says.
Others agree that performance testing, such as that for application load and availability, is not suited for virtual test labs.
"You can't ramp up the number of concurrent users to see how well the application will perform under of a load of say 10,000 concurrent users. Virtual machines do share some resources with the physical machine, no matter how few, that would cause those types of performance tests to be in accurate," Forrester's Schwaber says.
Gary Chen, senior analyst at Yankee Group, says he encourages clients to adopt virtualization for test environments because "if they do, their lives will get much easier and they will get more testing for less money." But he also warns IT professionals not to fall victim to the promise of virtualization without heeding some simple advice.
"No one should rely fully on a virtual environment for testing. Physical testing is still a must," Chen says.