« The Quest continues.... | Main

November 13, 2008

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

open systems storage guy

SPC is unlikely to change. More to the point, it's run by vendors. Users are in a better position to create benchmarks because aside from technical caveats that would be discovered, it's unlikely that they would try to deliberately show a high or low number (by short stroking, for example).

Wikis are a good first step, but what we need is an organization that will collect and publish benchmark submissions that conform to a certain standard of documentation and repeatability.

I imagine it would work like this:

-Client performs benchmark using real world workload, documents all variables (config, data used, etc), and then submits it for review.

-The vendors and bloggers chew over it to point out any caveats that make the data less useful or general.

-Other people replicate the benchmark to see if they get similar results.

Once all these things have happened, and everyone agrees that under these conditions, this performance is reasonable, the benchmark goes from "submitted for review" to "published", and can be used for comparison.

marc farley

I don't agree on the appeal of anarchy. It would look more like a presidential campaign than a benchmark.

marc farley

What if SPC developed a new mixed workload benchmark, say SPC-3 or 4 as Barry Whyte has suggested? I assume religion is a larger factor in this scenario?

.Connector

My thoughts on anarchy - Wikipedia is anarchy - it as no centralized leader or power center, but it works. So its not all bad. The regulating force is community self-policing. So I ain't knocking OSSGs idea yet.

Its meaningless for me to comment on a hypothetical SPC-4 and EMCs possible reaction to it. Lets see it and we'll see how to respond to it. I too think SPC is unlikely to change.

Cheers,
.Connector

marc farley

Analogies to Linux, Wikipedia or any other open collaborative effort aren't very good here as the context for a benchmark is by nature much more competitive. We ought to acknowledge this in our assumptions and expectations.

The "best result" would be for this effort to take on a life of its own - independent of vendor steering. But that is also probably not realistic. I guess that's why I like the university approach suggested by inch in the previous discussion. Not that influence couldn't be applied unevenly there, but there might be ways to make it more transparent.

The comments to this entry are closed.