Leaving the Land of Woo

Until recently, we could trust that academic scientific journals published papers from respected scientists which demonstrated a high standard of evidence and methodology.  As a result, the conclusions of these papers, often stated in cautious terms with appropriate caveats, represented reasonable judgements based on the presented evidence.  Before the papers were published, they would have been reviewed by other leading scientists of the field and perhaps the authors would have had to respond to pertinent questions and made corrections.

This peer review process is designed to achieve a high standard of published research, to ensure that unjustified claims were not made in print.  The problem is that once a paper makes it into the journal, it will be referenced by others who will quote the results and conclusions as reliable.  If they are not reliable, then nevertheless they will still be quoted as such.  Poor science when published will still be propagated widely.

We mentioned on this blog a recent case highlighted by PZ Myers on the Pharyngula blog in which the Southern Medical Journal was to publish a very poor paper on faith healing, in which no valid conclusions could be drawn at all.  I am still waiting for a response to an email I sent asking how such a lapse of publishing standards could occur and what their view was of the likely effect on their reputation.

Now, the wonderful Orac has found another instance, this time in the public access hournal PLoS ONE.  These guys are really committed to putting into the public domain medical research for ordinary folks who can't afford thousands of dollars in medical journal subscriptions, and who don't happen to have a public medical library on thier doorstep.  It's a laudable aim.  But now it seems they too have been caught out.

Orac's recent post looks at a paper claiming to have investigated impedance in intermuscular collagen bands, the idea surely being to look for evidence of the acupuncturist meridians - since there is no evidence for the existence of these meridians.  The study takes as an assumption that those meridians exist, and they are then going hunting for the evidence.  Their initial attempts failed, so they tried somewhere else and came up with - nothing.

Not to be put off, they tried to derive some comfort from statistical manipulation of the data.  They found a small correlation with a proposed site of the large intestine meridian, but here's the crucial point.  Instead of assuming that this variation correlated with their claimed meridian, they should take the presence of this anomalous reading and test a hypothesis derived from it, not the other way round.

In this case, there were very many reasons why any of their readings might have been anomalous, not least that the trial wasn't blinded.   They did at least realise that tissue thickness and varying echogenicity (the ability of tissue to bounce an echo) easily explained it.  So the paper showed nothing.

They went fishing to see if they could find any evidence consistent with their predetermined theory, and didn't.  In any case, you start with the data and derive your theory from that.  First the data, then the theory.  These guys start with their fanciful theory, that Orac calls fairy dust, and then go fishing for anything that they can quote to support it.

But the unfortunate fact is that once they get this sort of poor study published, all the other fairy dust collectors will quote it, in their blogs, in their own papers, in their press releases, in their advertising.  Unless journals like PLoS ONE keep to their high standards and reject this sort of low standard paper, the reputations of the journals themselves will become coopted into the marketing effort of Woo merchants.