dotConnector's Blog

Should end-users do the benchmarking?

OSSG shared some interesting ideas about who does the benchmarking, and how to manage these results. The core question here is: are vendors the only ones who can benchmark? Why not allow customers who own the equipment to benchmark and publish their own results?

Perhaps the right approach is to create a support structure for this. A Web 2.0 collaborative portal with a wiki such as the one from Wikibon can give guidance on what constitutes a benchmark, how to run it, and what the pitfalls are, as well as what the utility of the result is in making a storage platform decision. The the results can be commented on and sliced and diced by the world at large, and in true internet anarchy style, converge on the “truth” and “vaidity” of each result.

I have misgivings about this, although the approach is very intriguing. First, most customers I know don't benchmark storage, they benchmark an entire stack. For example, a customer of mine recently re-platformed (Unix variant change) one of their critical OLTP applications running on a Progress database - their benchmark was a particular piece of batch code wall clock timings, which we used as a basis for tuning storage performance. Can this be used as a general purpose result? Likely not. 

Another fundamental problem here is constucting a benchmark for storage that can bring an array to it's knees, without the IO driver infrastructure saturating. Most customers don't have the capability of using say 12 Linux systems with a distributed IO driving piece of code to really test such large arrays, and run the risk of saturating the driving server before the array chokes.

But, there is something appealing about anarchy...... and regardless of the whether customer benchmarks should be collated, interpreted and governed, a central body of guidelines and definitions would be very helpful.

Thoughts and comments?

Sidebar

Marc Farley asked why the SPC couldn't be modified to satisfy EMC's objections, and whether EMC's objections were religious or technical.

Disclaimer: These are my opinions only, and may not reflect the opinions of EMC as a company.

Well, I don't believe EMC will (re)join the SPC because SPC-1 set a direction  that has proven to be unproductive. The reasons are technical, and well described in earlier posts in this blog. In a nutshell, the SPC-1's cache hostile workload profile reduces its utility to counting the number of spindles in an array, and therefore a) provides no new useable information and b) deliberately "dumbs" down storage arrays and the intelligence inherent in them. It has no discriminating power between offerings from different vendors. EMC left the SPC for this reason, and I suspect (now, this is only my personal opinion) EMC wont join again. So by all means we can discuss modifying the SPC - I'm not holding my breath that it will be the standard benchmark from EMCs point of view. This whole effort is to discuss the creation of a new set of gudelines on how to characterize performance outside the SPC.

Cheers,

.Connector.

The Quest continues....

Hullo there Barry W.!

Thanks for the comment  on my old post - woke me up out of a year-long hibernation. You succeded where others have tried and failed....;^)

Yeah.. lets look at realistic benchmarks again - and I agree, its going to take more than the two of us. And don't hold your breath on EMC joining the SPC anytime soon.....

Here's one way of looking at it

Lets me grant that the intent behind the SPC was noble - to have a benchmark that customers could look to as a guideline for their performance needs from storage. There are then two major aspects that I find objectionable about it today, namely:

1. The benchmark itself is very narrow - cache hostile, basically counts spindles, not representative of real life workloads.

2. Governance - as the specific HW configurations are unconstrained, many of the tested systems are highly optimized, short-stroked, and not representative of what customers would buy. Apples-to-apples comparisons are very difficult. Even though the cost of the configuration is a weak measure of how efficiently assets were used, lets face it, no one pays attention to that. Every press release focuses on IOPS, nopt the cost, and that is where customer attention is drawn to.

So instead of designing the uber-benchmark from first principles, perhaps addressing these deficiencies for the SPC is one way of converging quickly. So for example, include more measurements for a range of workloads - ones that will let the underlying array architecture show its mettle. And demand high asset utilization (say 70%+) for ports, spindles, capacity etc. to discourage jury-rigging configurations for the test.

This way, the work done by SPC can be leveraged, and the meaningfullness of the results can be enhanced. A more complete benchmark with mixed concurrent workloads, backend processes like array replication etc are desirable, but will take a lot longer to craft.

Thoughts?

Cheers,

.Connector