Mathematics of the SPC-1 Benchmark

Hi folks,

When the Storage Performance Council SPC-1 benchmark was introduced several years ago, and the first few results hit the streets, I noticed something astounding: the SPC-1 IOPS scaled linearly with the number of drives in the tested storage array, independent of vendor, array, drive size or drive speed!

It was not astounding as a technical result - I would expect a cache hostile benchmark like the SPC-1 to be loosely dependent on spindle count. What was astounding was that no one ever called any attention to it!

Vendor Olympics:

I saw vendor after vendor claiming technical superiority based on the magnitude of the SPC-1 IOPS measurement they had made. I saw challenges to EMC and Netapp to participate in this.  So I decided to look at the most recent results to see if the benchmark or vendor arrays had changes so significantly in the past few years.

I took "high-end" systems - with a lot of scalability, and plotted their SPC-1 IOPS against the number of drives they had. The results:

The equation is a simple straight line fit done with Excel. The R^2 value is the quality of the fit, 1 is considered the best, the 0.996 is pretty darned good.

What did this mean?

Well... I could have probably saved HDS a lot of money and testing! Just knowing the previous results with the DS8300 Turbo, SVC 3.1 and SVC 4.2, I could have predicted the SPC-1 IOPS number for the 1024 drive HDS USP V almost exactly! Pretty good for arithmetic!

The HDS USP V has a tested configuration (ASU Storage) of 26 TB, but had 150+ of raw storage in it. In fact, even discounting for the RAID-1 protection for the storage, the benchmark only used 34% of the storage in the array. But spread out over all 1024 spindles. The USP V had 146 GB 15K Drives. The press release says none of this! You have to read the Full Disclosure report to find this out. What gets the headlines is just the fact that they have 200,000+ SPC-1 IOPS for 26 TB!

Even more surprising, the SVC matches the performance almost exactly as well - a completely different architecture! And it had sub-arrays that used 73 GB 15K drives.

So all HDS has to do to beat the standing SVC record is to load up the USP V with 1536 drives.

Ooops - can't do that.

Or for the DS 8300 to match the USP V record - go from 512 to 1024 drives.

Regardless, the SPC-1 IOPS has no discriminatory power for any of these 4 systems. The benchmark results are completely determined by one thing: the number of physical spindles in the configuration.

Lets look at it all!

What if I included not just highly-scalable systems, but mid-sized systems too? Well... you be the judge.

Here we see two classes of storage - the scalable arrays, and the mid-range ones.The lines have the same slope, and the correlation is pretty linear in both the blue and red bands.  For almost every system tested, the results can be predicted to within a few percent with just knowledge of the number of spindles. [Note: I have excluded some older results at the low end (red band) in the interest of time (mine!). This is an exercise I encourage everyone to do at home - the data is public at the SPC Home Page]

The real useful metric from the SPC-1 is $/SPC-1 IOPS - one that has unfortunately faded from prominence. The over configuration of the USP V makes the SVC give the best bang for the buck.

So I'd love to see what these platforms can do without the spindle count advantage. How about a test with 26 TB where there is 52 TB in a RAID-1 configuration in the array? I can predict the result: 1/3 of the current SPC-1 IOPS for the USP V with 1024 drives.

Why I am upset:

This is no magic folks - I cannot imagine that this is not a well known fact for vendors and the SPC - that the benchmark dumbs down ALL vendors to the level close to JBOD.  If SPC-1 performance is predetermined by drive count, customer decisions on storage investments are purely an exercise in pricing - all vendors are the same from a performance standpoint.

If this was known - not drawing customer attention to it is, mildly put, disingenuous.

If this wasn't known - this is not rocket science, people. Thats hard to believe.

Lets work instead to see if there is a real way to benchmark these systems - which is actually useful to our customers. This is exactly why EMC pulled out of the SPC years ago.

I know this is not easy, but am willing to work on it.

Cheers,

.Connector

Comments

[Open Systems Guy]

What's the problem? Random, non sequential reads are what most transactional and database workloads do most of the time, and the performance these application will get is pretty much directly proportional to the spindle count. Cache can only really help you write or do predictable reads (less than 30%, in my experience)

While I don't agree that the SPC is a silver bullet for comparing disk systems, and I certainly don't think it even remotely mimics the real life environment of any company's server room, I do think these results are right in line with what I would expect from a good benchmark. More spindles should yield higher "behind cache" performance. The benefit of benchmarking is that it allows buyers who care about performance to find out if this is true for the system they're thinking of acquiring- I'll bet that on an old disk system, the performance measured would quickly plateau off as drives continued to be added. I would also expect this of low power controllers that have more intensive features (like the write anywhere file system in a Netapps box).

Of course, performance is a distant third in my book behind features and reliability, and every corporate IT person is going to have to prioritize their needs.

[.Connector]

No issue with what you said, Open Systems Guy. The results are not at all unexpected for a benchmark like SPC-1 - like I stated in the beginning of my post. As I said, the result is not astounding, and neither is this rocket science.

But given the complete predictability of the results, why bother having a benchmark? What is a vendor trying to prove or disprove by publishing a result that uses much more raw storage than the adressable storage? Why should one cook the books in this fashion?

Maybe lower end systems will reveal choke points before disk IO saturates, but for all the systems benchmarked to date, the spindle count is it.

I agree that functionality and availability should come before, especially for high end arrays.

[Open Systems Guy]

In my experience, even some mid range systems level off before disk IO saturation. This may not be an issue- if I use a Netapps filer, chances are I am paying for its unique snapshot algorithm, but if I didn't know that it could choke on IO if I hit a certain level of performance, then I might be unpleasantly surprised.

All I'm saying is that some sort of benchmarking should be made available. Just because most systems will perform the same per spindle does not negate the need for a benchmark to support vendor data.

[Barry Whyte]

Some additional analysis over on my blog - would have posted here, but needed to include the complete pictures :)

http://www.ibm.com/developerworks/blogs/page/storagevirtualization?entry=detailed_mathematics_of_spc_1