Sorry folks! My day job and family effectively chewed up all my spare time for the past week- hence the delay in responding to the excellent suggestions in the comments to the last one.
Inch observed correctly that Open Source workload generators may end up writing to buffer cache instead. True. Reminds me of an old story...
Many years ago, when I was still doing REAL work, a customer called me and said they were running Oracle RAC on an Symmetrix 8830. We had told them they should get about 240 MB/s large block throughput from a well partitioned workload across the two node cluster.
The customer measured about 160 MB/s and was understandably concerned. So I went over and looked at the configuration - 2-node Solaris 8 VCS DBED/AC with Veritas' cluster file system. Enough HBA's, no architectural bottlenecks on the SAN. So my interest was piqued. I tuned up max_phys in /etc/system to 1 MB and that improved stuff a bit - 200 MB/s. Still short.
I used one of our internal workload generators, and measured the throughput for the vxfs filesystems. Sure enough, I got 200 MB/s too. So I went to raw devices, and viola, 240 MB/s. Turned out that going through vxfs and CFS, even with Direct IO turned on, I still saw a lot of overhead from operating system meta-structures like file systems.
Moral: Be very careful how the operating system is configured. Many unintended effects may rear their head if not done carefully. Raw devices are the best to get a true measure of storage performance. (I still maintain that the DS8300 would have posted better results on SPC-1 if only the queue_depth had been increased - one mans opinion!)
So if one is careful, maybe open source workload generators could be used. By stipulating only raw devices as targets. Its the multi-threading and building correlated IO threads that is daunting to me - any ideas? And then moving to multi-host benchmarks after that.....
Overwhelmingly, the sentiment echoed by people seemed to be - if someone wants to short-stroke, let 'em!
OSSG and TechEnki said just disclose it clearly, for all you know, for a fixed price, that may be the best configuration to achieve a certain metric. The open used community will decide if it is relavant or useful. The Anarchist suggested a minimum limit on utilization - spindles, ports, etc.
Here's my suggestion. I like Anarchist's idea of setting a limit, but tend to agree with what OSSG and TechEnki suggest as well. So why not have a default benchmark report with a utilization limit (such as 70% for component utilization), which I hope will be the majority, and have the tester explicitly call out any result which had been optimized for a specific metric like response time or back-end throughput?
The issue is one of constraints, IMHO. One can optimize a configuration for cost, throughput, power consumption (as Anarchist suggested as well) or other metrics. This should be highlighted in the results. So I may have Copan giving me results that suck from an IOPS perspective, but which rock from a power consumption standpoint because of the MAID technology it incorporates. Inquiring minds should want to know!
So should we have a loose framework of constraints, and have classes of results optimizing for the maximum bang for that constraint? Or have the constraints (spindles, total capacity, cache, $$, power, etc.) for a given configuration declared prominently?
This way, if someone wants to short-stroke, or have a a much greater number of spindles that needed for the storage, so be it. But its optimized for response time and throughput, and we know ot explicitly because it's posted in that class. BarryW echoed this sentiment as well - constraining # spindles may not make sense as spindle sizes grow.
BarryW also called for requirements on replication "overlay" tests. I agree - I'm only wondering if we should dive into this now, or hash out the pure performance benchmark first, and then move to overlays.
So my modified Postulate #2 could be:
Postulate #2: Dont overconfigure! (unless you have to)
Postulate #3: If you DO overconfigure, publish all constraints.
On a tangent...
I was tickled by the financial analyst reaction to Oracle VM - suddenly a bunch of them were going gaga over how this was going to compete with VMWare! Huh? Didn't anyone see that it was Xen based? Xen is nice, but its still got a ways to go.... even funnier is how both VMWare and Citrix tanked! Priceless!