« The quest for a better benchmark | Main | Moving right along.... »

November 05, 2007

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Open Systems Guy

I like the idea of having an open standard benchmark that anyone can perform. Since there are so many variables that will need to be measured and there will probably end up being a plethora of benchmarks, having an open standard is the best way to make sure all ideas get considered.

dotconnector

Hi OSSG,
I say let the community decide on usability. This is an experiment, but I would say that if enough people participate, it will converge. Look at Wikipedia - one would naively think that an edit-by-anyone forum would be pure anarchy (no offence - BarryB!), but au contraire!

A large enough social community seems to reach stable equilibrium pretty well.

Will an open source open usage benchmark suite fare differently? Maybe - maybe not.

I see this discussion leading to a framework for a good benchmark - not necessarily a code base. Where the community takes that remains to be seen.

Lets see if participation and support for this discussion is passionate enough to develop its own inertia. We may end up concluding that the task is too complex and too daunting.

Thanks for the input!
Cheers, .Connector.

inch

G'day,

Firstly, I would like to say that I'm so glad to finally see a storage blog with no petty "He said she said my array goes faster not your array" crap.

Other bloggers whom remain nameless that work for various organisations don't do themselves or the company's they work for any justice with the constant slamming of each others products.

Anyway, to the topic at hand....

I think it is about time that a conversation has started about tools and methodologies for storage performance testing.

I have a bit of experience evaluating various storage technologies and generally have to concoct my own tools on the fly depending on the product.

Most of these tools are a mash of various open source products which are not meant for testing out disk arrays, nas equipment and switch they are generally made for filesystem or protocol testing.

I'm sure each vendor would have their own tools for burning in disk arrays and load generation, why not open source the lot... :-)

I will also raise my hand along with David and OSSG to assist where I can

Cheers

the storage anarchist

No offense taken. As you note, the Wikipedia proves that anarchy can work. And importantly, anarchy isn't about not having any rules, it's about not having a RULER. This was actually the foundation of my blog's title - I'm an activist trying to ensure that in the absence of a ruler in the storage blogosphere (by definition), we all still play by common rules of fairness and judgement.

And I TOTALLY support the community approach to this effort...but as you say, it really depends upon a commitment to participate from a broad enough community. What the effort really needs soon are:

1) participants from other storage vendors (HP, Sun, Hitachi, Dell, 3par, etc.). We even need more from EMC and IBM, given the breadth of our respective product lines.

2) participation from the storage network infrastructure vendors. HBAs, switches and the like. Whether or not we decide these play a significant role, we need to consider the impact of these components on the benchmark configurations and results.

3) Server vendors...and this one will undoubtedly be controversial. But the power of the server driving a workload frequently plays a role in the results. In order to be repeatable, the benchmark is going to have to account for the differences in server platforms. Involving the server vendors may (or may not) help address this aspect more efficiently.

4) Most importantly, we need CUSTOMER participation - and this is critical. In fact, I'd like to see the CUSTOMER definition of what's needed before we go much further. As technologists, we can imagine a plethora of dimensions and angles - but it's possible that many or even most of those are irrelevant to the customer. So getting the voice of the customer involved should of highest priority.

I'll be recruiting every opportunity I get, but let me know if there's anything else I can do...

open systems storage guy

Here's a start: http://www.wikibon.org/Benchmark_storage

I have not finished laying out the format, and have to do some real work for a bit ;). I'll take another crack at it tonight, but in the meantime if someone wants to go and start adding content (or copy-pasting .Connector's post if we have permission), then please do :)

You may need to sign up for a wikibon acct to edit, but that's probably a good thing- keeps people accountable, more or less. Also, if you get a "server not responding" error, just try again- it seems to be an intermittent problem. Make sure you copy the text of your edits to notepad or something before clicking submit though...

dotConnector

Inch,
Thanks for your kind words - and I really appreciate your willingness to help.

I second Anarchist's call for more participation. This effort is pointless unless we have strong customer and vendor participation. I too will recruit tirelessly.

Gotta run now - more later.
Cheers, .Connector.

The comments to this entry are closed.