The Storage Performance Council (SPC) on Wednesday rolled out some of the first vendor results from its SPC Benchmark 1 (SPC-1), a newly defined metric for determining the performance of storage sub-systems.
SPC-1 ratings for storage products from IBM Corp., LSI Logic Corp., and Sun Microsystems Inc. were released.
The purpose of SPC-1 is to help storage customers gauge the performance of individual storage products and thus create a short list of vendors to be considered when deploying a storage infrastructure, said SPC representative Walter Baker.
The benchmarks represent “the first of several benchmarks to cover the spectrum of storage performance. This first benchmark is primarily focused towards random I/O operation that you might find in multi-user environment,” said Baker, who added that SPC-1 was created “because the existing industry standard benchmarks were insufficient for producing results that would differentiate based upon storage requirements.”
Roger Reich, a senior technical director for Mountain View, Calif.-based Veritas Software Corp. and an SPC representative, said that the SPC and SPC-1 offer a top-to-bottom way for storage vendors to accurately benchmark their products. SPC-1 includes the benchmarking specification, a test kit, and a performance analyzer. The SPC provides auditors and technical assistance, he said.
“We have built a complete environment that for the first time allows users to accurately compare the performance of storage products,” said Reich.
Industry observers such as Chuck Standerfer, a principal analyst with the Evaluator Group in Greewood Village, Colo., agreed that SPC-1 will improve the manner in which storage performance benchmarking is done.
“We believe this is a major milestone in bringing clarity to various storage performance issues,” Standerfer said.
But SPC representatives and the participating storage vendors at Wednesday’s announcement came under fire by launch attendees who charged that the absences of storage switch products from companies such as Brocade and McData made the SPC-1 benchmarks something less than real-world performance numbers.
This suggestion re-kindled the issue surrounding the departure of EMC from the SPC approximately a year and a half ago. At the time, EMC representatives charged that the SPC benchmarks were not based on “real-world” storage networking scenarios.
But representatives from IBM and LSI Logic defended the SPC-1 benchmarks, saying that switches were not a factor in internal sub-system performance.
Of Big Blue’s SPC-1 benchmark, Brian Smith, an official for IBM, based in Armonk, N.Y., said “this particular configuration has no switch in it. The switches are sort of a performance-neutral thing in these configurations. They don’t add, and certainly don’t subtract from, the performance capabilities; [a switch is] just an additional component. I don’t see it as an essential element in characterizing the performance of an I/O sub-system.”
Milpitas, Calif.-based LSI Logic’s Steve Gartner turned EMC’s past criticism of the SPC back toward EMC itself.
“I would comment that if you don’t like the benchmark, then why don’t you stay in the organization and make it so where it is representative [of real-world performance] instead of taking your ball and going home,” Gartner said.
IBM, LSI, and Sun were only the first vendors to release SPC-1 benchmark numbers. Additional vendor results using the SPC-1 benchmark will begin arriving over the next few weeks, Baker said.
The SPC is a Redwood City, Calif.-based consortium founded to define, standardize, and promote storage subsystem benchmarks.
