Lots of good stuff in response to my last post! Good comments from Open Systems Storage Guy; Barry Burke's tounge-firmly-in-cheek DMX-4 SPC-1 IOPS calculation was very entertaining; and an interesting Barry Whyte post in response to the observations I made.
Out with the old...
My interest in what the SPC-1 benchmark has been used for - lately! Like in the last 3 years or so. Apparently BarryW has a significant amount of academic interest in a LOT of older results, from an era when cache was at a cost premium, and modular 2-controller systems were the new rage, and the benchmark results highlighted that. Many of the systems have been EOL'd, or superseded.
If anyone wants to harp on how the SPC-1 IOPS results can help an end user to decide whether to buy Fujitsu Eternus 3000 Model 300 or the Dell PERQ/3 QC today - have a blast! I just don't see the relevance.
And in with the new...
Welcome to 2007! My customers are likely to be using much more modern stuff, and thats MY interest. So I restricted my analysis (and stated so in my post that I am excluding older results) to roughly submssion dates post October 2004.
I missed a few SUN/STK systems in the cutoff (my bad! Add them in...won't change a thing), and I did include the SVC 1.1.1 from June 2004 ( I was curious about that!). Others are from another era - not relavant for the points I am making.
My observation is simple: for the results in the last 3 years, all the results reflect direct proportionality between the SPC-1 IOPS benchmark and spindle count. This is a significantly stronger statement than stating that there is a monotonic increase of SPC-1 IOPS with spindle count (which is intutive). The data speaks for itself.
However, thanks, BarryW, for reminding me to discuss the DS8300 family SPC-1 IOPS. During my data collection phase for this series of posts, I too noticed this very enigmatic puzzle.
Here were two systems, identical in HW configuration (same number of disks, same testing configuration, same amount of cache, processors, 32 channels) driven by exactly the same server, a P5 p595 Model 9119 32-way processor with 32 channels running AIX 5.3.
Yes, the adressable storage was a bit different (6.6TB for the DS8300 and 8.9 TB for the Turbo), but in true form like other vendors, the data only occupied about 32-36% of the total storage in the system, ostensibly to increase spindle count and thereby inflate the SPC-1 IOPS result (just curious: so what is the end customer supposed to do with the rest of the storage?). So that couldn't be the reason for the mystery below....
So it was intriguing to see the Turbo post 123,033 SPC-1 IOPS and the DS8300 post 101,102 SPC-1 IOPS! 22% better!
Could it be... that the microcode on the DS8300 Turbo was better, and the SPC-1 actually caught that? Wow! I mean, Gee Whiz! Holy Cow! Or...
Could it be... Satan?
The devil in the details
Let me ask this question:
Why is the driving host configuration different for the Turbo benchmark compared to the vanilla 8300?
Specifically, why were the queue_depth and max_transfer parameters changed from their defaults (20,256K) to higher values (64, 1024K) for the DS8300 Turbo benchmark? This is buried in the Full Disclosure report, in the link above.
The queue_depth parameter increase lets more IOs queue up at the host - a good thing for large arrays, where many disks make appear aggregated as one logical volume. This makes sure that the disks are not twiddling their thumbs, while the operating is working under the wrong assumption that the volume will be overwhelmed if more IOs are pushed. Set it too low, and the disk array seems to underperform, as not enough work is going its way. I have seen many instances when increasing operating system queue_depth gives significant gains in IO throughput, especially with internal striping in the array (like the RAID-10 the DS8300 uses).
The max_transfer parameter, makes sure that large IO's are broken up into bite sized chunks. Set it too small, and the operating system does a lot of work for nothing.
Could it be.... that the DS8300 test was actually throttled by the inability of the host to drive enough IOs? So actually the DS8300 and the Turbo might have posted the same result - except, the driving P595 was not queuing enough IOs for the older array.
Or was it really the SPC-1 suddenly becoming sensitive to something other than spindle count, just for the DS8300 family? Somehow, I don't think so...
I could be wrong... but on the other hand....
I know I am speculating - but right now, the SPC-1 IOPS FUll Disclosure report for both is no help in clarifying what reality is. Short of a new measurement with the new parameters for the vanilla DS8300, I don't see how one can argue that the storage performance under SPC-1 actually improved for the DS8300 Turbo compared to the DS8300. I would submit that this could equally be an artifact of a restricted IO driver. If two things changed, which one caused the difference in the measurement?
But, oh, didn't I hear somewhere that the DS8300 just got EOL'd?
So it is an interesting and anomalous discrepancy, but not one that can be resolved conclusively with the data at hand. I'm not buying it, BarryW.
Is the SVC really high-end?
Hmmm... the fact that the USP V and the SVC seem to perform the same means one of two things:
a) The SVC is truly on par with a USP V from an array capability point of view or
b) the SPC-1 benchmark has done great disservice to the USP V, and forced it down to the least common denominator - spindle count.
The USP V SPC-1 result did show one thing - that a full configuration shows no other component saturation effects. It is purely spindle bound - with no discernable choke points outside of that. My personal belief (not EMCs!) is that HDS sold their technology short by participating in this benchmark. Like the DMX, their microcode engineers have spent many hundreds if not thousands of man-years optimizing their systems for MF, multiple-host workloads, etc. The benchmark lets none of that shine through. I think they can do a lot better with a real life workload than the SVC.
But, as I said, thats one mans opinion..... the same man who is also positive that the DMX will do better that both the SVC and USP V with real customer multi-host multi-function workloads, with a boatload more functionality to boot.
Thanks for the welcome to blogosphere, BarryW! I am sure we will agree and disagree on many other topics over the coming years- and I hope to continue to learn from that as we go on.