Prestigious but wrong?

In his book Expert Political Judgment, Philip Tetlock examined the accuracy of forecasts made by political and economic experts. Among his many interesting findings, he found that experts with a track record of media appearances were more overconfident in their judgments than their office-bound colleagues; that is, the probabilities that they attached to possible future events of interest tended to be too extreme.

Now it is claimed, by Neal Young and John Ioannidis, that academic journals are affected by a similar bias. The pressure to publish, combined with the selectivity of top journals, means that the top journals are likely to publish dramatic research that will more often turn out to be wrong. An earlier study of 49 papers published in leading journals found that nearly a third of them had been refuted by other studies within just a few years, although it should be noted that they did not look to see whether the same was true for less prestigious journals.

A short account of the Young and Ioannidis paper can also be found at the Economist online.