turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written.
If you have found other sources where the Optimizely site publicly describes their statistical methodology, please share them. I think several people following this discussion would be interested.
So ask them; they aren't stupid, they wouldn't be building a business based on A/B testing without using valid methods of testing and displaying results. That you call the results crap because you don't have the perfect details of everything is simply absurd.
I didn't call the results "crap". I am simply pointing out that they are meaningless without knowing the methodology behind them. (And we aren't just missing the "perfect details of everything" here. As far as I can see, we have no rigorous details whatsoever.)
I would remind you that you were the person who was attacking another poster's position based on your interpretation of those currently meaningless numbers. It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.
> I am simply pointing out that they are meaningless without knowing the methodology behind them.
Only if you assume incompetence or malice on the part of Optimizely, neither of which you have any valid reason to do. It's perfectly reasonable to assume they aren't stupid and the results are valid.
> It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.
Um, my claim is don't assume they're idiots; that doesn't require me to back anything up.
The poster I replied to wasn't attacking them, he was attacking statistic in general, which is what I was replying to.
Your response was to imply that Optimizely doesn't know what they're doing and therefore their results are invalid until you see how they're crunching the data; that's simply absurd.
