When the Storage Performance Council SPC-1 benchmark was introduced several years ago, and the first few results hit the streets, I noticed something astounding: the SPC-1 IOPS scaled linearly with the number of drives in the tested storage array, independent of vendor, array, drive size or drive speed!
It was not astounding as a technical result - I would expect a cache hostile benchmark like the SPC-1 to be loosely dependent on spindle count. What was astounding was that no one ever called any attention to it!
I saw vendor after vendor claiming technical superiority based on the magnitude of the SPC-1 IOPS measurement they had made. I saw challenges to EMC and Netapp to participate in this. So I decided to look at the most recent results to see if the benchmark or vendor arrays had changes so significantly in the past few years.
I took "high-end" systems - with a lot of scalability, and plotted their SPC-1 IOPS against the number of drives they had. The results:
The equation is a simple straight line fit done with Excel. The R^2 value is the quality of the fit, 1 is considered the best, the 0.996 is pretty darned good.
What did this mean?
Well... I could have probably saved HDS a lot of money and testing! Just knowing the previous results with the DS8300 Turbo, SVC 3.1 and SVC 4.2, I could have predicted the SPC-1 IOPS number for the 1024 drive HDS USP V almost exactly! Pretty good for arithmetic!
The HDS USP V has a tested configuration (ASU Storage) of 26 TB, but had 150+ of raw storage in it. In fact, even discounting for the RAID-1 protection for the storage, the benchmark only used 34% of the storage in the array. But spread out over all 1024 spindles. The USP V had 146 GB 15K Drives. The press release says none of this! You have to read the Full Disclosure report to find this out. What gets the headlines is just the fact that they have 200,000+ SPC-1 IOPS for 26 TB!
Even more surprising, the SVC matches the performance almost exactly as well - a completely different architecture! And it had sub-arrays that used 73 GB 15K drives.
So all HDS has to do to beat the standing SVC record is to load up the USP V with 1536 drives.
Ooops - can't do that.
Or for the DS 8300 to match the USP V record - go from 512 to 1024 drives.
Regardless, the SPC-1 IOPS has no discriminatory power for any of these 4 systems. The benchmark results are completely determined by one thing: the number of physical spindles in the configuration.
Lets look at it all!
What if I included not just highly-scalable systems, but mid-sized systems too? Well... you be the judge.
Here we see two classes of storage - the scalable arrays, and the mid-range ones.The lines have the same slope, and the correlation is pretty linear in both the blue and red bands. For almost every system tested, the results can be predicted to within a few percent with just knowledge of the number of spindles. [Note: I have excluded some older results at the low end (red band) in the interest of time (mine!). This is an exercise I encourage everyone to do at home - the data is public at the SPC Home Page]
The real useful metric from the SPC-1 is $/SPC-1 IOPS - one that has unfortunately faded from prominence. The over configuration of the USP V makes the SVC give the best bang for the buck.
So I'd love to see what these platforms can do without the spindle count advantage. How about a test with 26 TB where there is 52 TB in a RAID-1 configuration in the array? I can predict the result: 1/3 of the current SPC-1 IOPS for the USP V with 1024 drives.
Why I am upset:
This is no magic folks - I cannot imagine that this is not a well known fact for vendors and the SPC - that the benchmark dumbs down ALL vendors to the level close to JBOD. If SPC-1 performance is predetermined by drive count, customer decisions on storage investments are purely an exercise in pricing - all vendors are the same from a performance standpoint.
If this was known - not drawing customer attention to it is, mildly put, disingenuous.
If this wasn't known - this is not rocket science, people. Thats hard to believe.
Lets work instead to see if there is a real way to benchmark these systems - which is actually useful to our customers. This is exactly why EMC pulled out of the SPC years ago.
I know this is not easy, but am willing to work on it.