“But experts cautioned that the test lacked some real-world conditions….”

Testing being at the core of what we do at The Tolly Group, a newspaper story with these words was certain to attract my attention. It might seem to some that any test is an incomplete test, and by extension, without value — but I don’t agree.

Interestingly, the quote above is not from a trade publication but from a recent front-page story in The Washington Post titled “Target Intercepted in Anti-Missile Test.” In this case, experts bemoaned that the test attack was not a surprise attack, did not involve multiple incoming missiles and did not involve an enemy trying to thwart the tracking system. The implication was that the Pentagon test was a pointless waste of time (and, to be sure, it cost a lot of money).

So, who is right? From reading about all of the flaws in the test, one might concur with the experts. Again, I would disagree.

When you think about it, a test that tries to do too much often accomplishes nothing. Imagine if the missile test had all of the extra real-world conditions listed above and it failed. The first thing one would want to know is why. With so many variables, the likely answer would be: Who knows — we tried to do too much.

The essence of testing, whether in IT or elsewhere, is to isolate certain elements to establish a baseline of performance or functionality. Subsequent tests can build upon the base knowledge and be used to exercise more sophisticated features.

We need to build our testing — as the Pentagon did — by testing core functions and then increasing complexity. It doesn’t make sense to conduct, say, a test trying to establish the maximum throughput of a wireless LAN (WLAN) in an environment that you know is loaded with interference and physical obstructions. What would be the point?

Just as we let athletes optimize their performance by wearing performance clothing at track and swim meets, it makes sense to determine the best-case performance of a technology before adding other elements. As important as other real-world elements are, they are often meaningless without some baseline numbers for comparison.

In the case of WLANs, for example, it didn’t take us long to realize that the best throughput, even under optimal conditions, was about half of the rated speed (we learned to expect no more than about 20Mbps to 22Mbps out of 802.11g 54Mbps LANs).

If we hadn’t established this in ideal, laboratory conditions, one might have thought interference was driving down throughput dramatically. (As it is, there are architectural reasons for this number.)

So, yes, let’s do WLAN tests in environments with obstructions and interference, but remember the essential of having controlled results with which to understand this. Let’s recognize that a test piling on complexity is not inherently better than a straightforward test of a single aspect of a product done under controlled conditions.

A test must be repeatable to carry much meaning. Given the nature of some technologies, reproducing a test environment isn’t always possible. Still, similar results should be expected for similar conditions.

Finally, and most importantly, we need to remember that numbers without analysis often tell us nothing. Look beyond the numbers; look for meaning.

QuickLink: 064291

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Featured Articles

Empowering the hybrid workforce: how technology can build a better employee experience

Across the country, employees from organizations of all sizes expect flexibility...

What’s behind the best customer experience: How to make it real for your business

The best customer experience – the kind that builds businesses and...

Overcoming the obstacles to optimized operations

Network-driven optimization is a top priority for many Canadian business leaders...

Thriving amid Canada’s tech talent shortage

With today’s tight labour market, rising customer demands, fast-evolving cyber threats...

Staying protected and compliant in an evolving IT landscape

Canadian businesses have changed remarkably and quickly over the last few...

Related Tech News

Tech Jobs

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Tech Companies Hiring Right Now