Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I think the consensus on why this happens boils down to regression to the mean, selective experimentation (exciting things over boring things like attempting to duplicate results), and selective publishing (e.g. positive results are much more interesting to journals than hypotheses being rejected).

Did I miss any reasons?



Plus if you have a million researches in your area of research a 1 per million chance your results are wrong is still no assurance. Should statistically happen to one of the researchers in your area.

Very good article by Warren Buffet that touches on the same issue: http://www.tilsonfunds.com/superinvestors.html


I'd add bad math. Fisher significance testing is on shaky grounds[0] both theoretically (significance testing violates the likelihood principle) and practically (just by nature of having a small sample size, a lot of null hypotheses can be rejected, while with a larger sample size hardly none are rejected). Hypothesis testing is little better. The "Bayesian revolution" has barely begun and has yet to influence a majority of researchers.

[0] http://uncertainty.stat.cmu.edu/ Chapter 12.


I don't think this is quite right. If the null hypothesis is true and you conduct the same study an infinite number of times, you will reject the null hypothesis 5% of the time at alpha = 0.05 regardless of sample size, unless there is something wrong with your sampling procedure or hypothesis test. This is the point of null hypothesis significance testing.

In practice, if you fix p=0.05 and increase your n, the probability that you will find a statistically significant result often increases because your power increases, and in many situations, the probability that the null hypothesis is true is close to zero. (Andrew Gelman uses the example of asking whether there are significant differences between voting patterns of men and women.)

On the other hand, effect size estimates become more accurate as the sample size increases. This mitigates the above issue, provided you actually report your effect size. It also means that small sample studies that report statistically significant results are more likely to overestimate their effect size, which is especially problematic if you are applying null hypothesis significance testing when you know the null hypothesis is false.


Unknowable chance events and hidden variables (John Crabbe's experiments towards the end of the article).


Just one. Financial incentive to obtain positive results. Especially in the drug market.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: