OSSG shared some interesting ideas about who does the benchmarking, and how to manage these results. The core question here is: are vendors the only ones who can benchmark? Why not allow customers who own the equipment to benchmark and publish their own results?
Perhaps the right approach is to create a support structure for this. A Web 2.0 collaborative portal with a wiki such as the one from Wikibon can give guidance on what constitutes a benchmark, how to run it, and what the pitfalls are, as well as what the utility of the result is in making a storage platform decision. The the results can be commented on and sliced and diced by the world at large, and in true internet anarchy style, converge on the “truth” and “vaidity” of each result.
I have misgivings about this, although the approach is very intriguing. First, most customers I know don't benchmark storage, they benchmark an entire stack. For example, a customer of mine recently re-platformed (Unix variant change) one of their critical OLTP applications running on a Progress database - their benchmark was a particular piece of batch code wall clock timings, which we used as a basis for tuning storage performance. Can this be used as a general purpose result? Likely not.
Another fundamental problem here is constucting a benchmark for storage that can bring an array to it's knees, without the IO driver infrastructure saturating. Most customers don't have the capability of using say 12 Linux systems with a distributed IO driving piece of code to really test such large arrays, and run the risk of saturating the driving server before the array chokes.
But, there is something appealing about anarchy...... and regardless of the whether customer benchmarks should be collated, interpreted and governed, a central body of guidelines and definitions would be very helpful.
Thoughts and comments?
Marc Farley asked why the SPC couldn't be modified to satisfy EMC's objections, and whether EMC's objections were religious or technical.
Disclaimer: These are my opinions only, and may not reflect the opinions of EMC as a company.
Well, I don't believe EMC will (re)join the SPC because SPC-1 set a direction that has proven to be unproductive. The reasons are technical, and well described in earlier posts in this blog. In a nutshell, the SPC-1's cache hostile workload profile reduces its utility to counting the number of spindles in an array, and therefore a) provides no new useable information and b) deliberately "dumbs" down storage arrays and the intelligence inherent in them. It has no discriminating power between offerings from different vendors. EMC left the SPC for this reason, and I suspect (now, this is only my personal opinion) EMC wont join again. So by all means we can discuss modifying the SPC - I'm not holding my breath that it will be the standard benchmark from EMCs point of view. This whole effort is to discuss the creation of a new set of gudelines on how to characterize performance outside the SPC.