Journals of Alternative Medicine: insufficient scepticism = Cargo Cult Science

Or: Plus ça change, plus c’est la meme chose

Starring the “Mellow rats in pyramids”, with walk-ons for the late Jacques Benveniste (boo), Richard Feynman (science guru Nobel prize winner) and James Randi (magic man and debunker). Join Dr Aust on a wander through the wacky world of Journals that look like scientific ones, but aren’t.

 

 

One of the things about “Alternative Medicine” – AltMed for short – that can surprise mainstream scientists is the sheer number of Journals of Alternative Worldview that are listed on Medline / PubMed. PubMed is the listing and indexing service run by the US National Library of Medicine, and being listed on PubMed is pretty much a must for any journal in medicine, or the biosciences, that wants to be taken even slightly seriously.

 

Once you start looking, there are loads and loads of alternative journals, many published by the same large publishing conglomerates that publish mainstream scientific and medical journals, like Elsevier and Blackwells. There’s money in AltMed. There is even one AltMed journal (Evidence-based Complementary and Alternative Medicine, aka eCAM) published by the normally thoroughly sensible, and impeccably academic, Oxford University Press.

 

Now, some alternative types would have you believe that they are ruthlessly and unfairly excluded from the mainstream, and that that is why they need their own journals. Not so. It is perfectly possible to publish well-constructed research into complementary and alternative therapies in mainstream medical journals. Studies of things like herbal medicines, acupuncture, or even homeopathy, appear regularly in places like the British Medical Journal, The Lancet, the Annals of Internal Medicine, and so on.

 

 

So the dedicated AltMed journals are just taking the “overspill”, right?

 

Well, not quite. There is a noticeable difference between the AltMed journals, and the mainstream journals that publish studies in alternative medicine.

 

Essentially, many of the “dedicated” AltMed journals, and notably their peer reviewers, display an astounding lack of scientific scepticism.

 

In a nutshell, they don’t have any.

 

Reading the AltMed journals one always has a weird sense of entering The Twilight Zone, or some other kind of parallel reality – “It’s science, Jim, but not as we know it”. The reason for this is that the papers are set out exactly like mainstream scientific papers – they look like the real thing.

 

But looks can be deceptive. Because despite the appearance, a critical ingredient of science is lacking – scientific scepticism.. No-one here seems to ever ask; “Am I fooling myself? Is it really clear that this effect occurs by some mysterious energy-force? Have I eliminated all the possible less exciting explanations?”

 

The authors of the paper don’t ask themselves this. And the peer-reviewers, who are probably authors of similar studies themselves, don’t ask the authors either. And the editors – well, see peer reviewers, or authors.

 

A hypothetical example of what most scientists would say ought to be the thought-process would run something like this.

 

“Hmm, before I publish this paper that says that Reiki Healers can make peoples’ painful knees feel better, which would mean the Healers can beam healing invisible energy out of their hands, I should check out exactly how reliable the patients’ own estimates of how much their knees hurt are, especially how much the reported symptoms vary day-to-day…. And I should check how often a painful knee would just start to feel better of its own accord… and perhaps I should see how much “healing effect” the patients report they get if I pay an actor to pretend to be a Reiki Healer.”

 

In other words, before you blithely publish something which requires the wholesale dumping of the known laws of physics, chemistry and biology, you should make absolutely sure that there is not an obvious explanation for what you thought you saw.

 

One of my scientific heroes, Richard Feynman, nailed this more than thirty years ago in a famous speech about what he called “Cargo Cult Science”, given as the Commencement address at Caltech in 1974, and reproduced in the brilliant book “Surely you’re joking, Mr. Feynman!”.

 

 

Talking about “science that isn’t science”, Feynman said:

 

“I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential…”

“Now it behooves me, of course, to tell you what they’re missing. …there is one feature I notice that is generally missing in cargo cult science. That is… a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty – a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid – not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked – to make sure the other fellow can tell they have been eliminated” [1]

 

Let’s apply this kind of logic to a real example from an AltMed journal .

Here is the “Mellow rats in pyramids” study, published earlier this year in eCAM. [2].

 

Housing in Pyramid Counteracts Neuroendocrine and Oxidative Stress Caused by Chronic Restraint in Rats

M.S. Bhat, G. Rao, K.D. Murthy & P.G. Bhat

eCAM 2007 4(1):35-42.

 

The first sentence sets the scene:

 

“Pyramid research to date reveals some evidence that the space within the great pyramid and its smaller replicas enhances, intensifies and/or generates energy of the electromagnetic spectrum and other forms or degrees of the so-called universal energy.”

 

Hmmm. Incidentally, the reference cited for this “pyramid research to date” is not a scientific study. It is a thirty-year old book extolling the healing powers of pyramids [3].

 

A quick summary of the paper. The authors took rats and placed them in very small cages for 6 hrs a day for 14 days to cause stress. At the end of the 14 days they measured several parameters related to neuroendocrine and oxidative (free radical) stress, including blood levels of the stress hormone cortisol. Placing the rats in the small cages raised their cortisol levels. But the cortisol level did not rise if the small cages with the rats in were placed inside a large wooden pyramid (picture here). In contrast, placing the small cages inside a large square wooden box gave values similar to when the “stress cages” were left on a lab bench.

 

The authors’ conclusion: being in a wooden pyramid made the rats less stressed.

 

 

So what is wrong with the paper?

 

The problem is that there is not enough information to rule out a whole load of possibly “confounding” explanations.

 

One major possibility is differences in animal handling. Handling lab animals, if they are not accustomed to it, is a major stressor. Being picked up and moved between cages is stressful. Thus any systematic difference in how animals are handled can show up in the results – for instance if the animals going into the pyramid were not handled identically to the others.

 

Time of day is another possibility. Stress hormones like cortisol have diurnal rhythms. The blood samples were all taken at the same time of day (within an hour), but the time of day the stress was applied could make a difference.

 

Next, the surroundings could make a difference. How shielded were all the housings (especially the wooden pyramid and box) from sources of noise?

 

Or – how crowded were the cages are in which the rats were housed when not in the stress cages? Another thing that would make a difference to stress responses. Were all the rats housed in identical-sized groups?

 

And so on.

 

None of the above, NB, implies deliberate “fixing”. It can easily be just a readiness to believe the improbable, and thus to accept said improbable at face value instead of looking for an explanation in terms of the already known. There are lots of ways unconscious biases can creep in. For instance, suppose an animal technician does the rat handling. Suppose one day he notices his bosses are happier with their experimental results when the rats that went into the pyramid were the ones he happened to have handled a bit more than the others, perhaps because these rats had been in his animal house slightly longer. The technician wants his bosses to be happy, so he keeps doing it.

 

To reiterate, the point is that one can think of many other explanations for the “pyramid stress reduction” effect, explanations that are not ruled out by the information given in the paper and do not require one to accept that a wooden pyramid produces a soothing effect that a similarly-sized square-sided wooden box does not.

 

So what should the authors have done? Well, I would argue that they should have told us, in detail, exactly how they made utterly sure there was no possibility of these kinds of effect occurring and “contaminating” the results. See Feynman above.

 

However, no such information appears in the paper. There is no discussion at all of the inherent biological implausibility of the result. There is no explicit discussion of how confounding effects were avoided.

 

Instead the authors quote a number of “studies” reporting on effects of sitting in pyramids in humans (again, not one of these is a study published in an actual scientific journal, even a complementary one) and then comment:

“[Our work] suggests that the shape of the housing has its effects [on stress] and pyramid shape appears to have beneficial effects… Thus, sitting inside a pyramid-shaped structure can be used as an effective technique for stress management and for non-invasive treatment of diseases in which the role of free radicals and ROS has been implicated.”

 

 

Music, I predict, to the ears of pyramid sellers everywhere.

 

Unfortunately, this intrinsic suspension of disbelief is rife among people looking into alternative therapies. They just don’t get it. Instead, what we get is – Cargo Cult City.

 

 

 

Enter M. le Docteur Benveniste

Perhaps the ur-example of recent years of suspending disbelief is the celebrated 1988 Nature paper by the late Jacques Benveniste and his co-workers [4]. This paper purported to provide evidence that homeopathic dilutions of antibodies (i.e. solutions so repeatedly diluted that there was not a single molecule of antibody left in them) could cause activation of mast cells (cells that release histamine and give you allergic reactions). The mast cells are activated by antibody. Except that here there wasn’t any.

 

The measurements in the paper involved an observer counting reddish-coloured cells in among a big mish-mash of cells under a microscope. About one in a hundred cells or less was expected to be red.

 

Applying our rule of scepticism, to make sure you were not fooling yourself in this experiment, you would want to rule out a whole range of possibilities: inconsistent counting, bias in counting (person counting knows in advance what answer they want to get), selective “editing” of results (only keeping results of counts on days when the results went the preferred way) etc etc.

 

As the subsequent report of Nature’s investigators (Nature editor John Maddox and two others) makes clear [5], the Benveniste lab took no precautions to avoid any of these problems. None. In other words, they totally failed to ask themselves what the ways were that their truly startling “result” (apparent immune reaction-generating activity of solutions with no antibodies left in them) could have been generated by observer bias. And then construct an experiment that systematically eliminated the chance for such bias to shape the results.

 

When the investigators did exactly that, the “startling” result vanished completely.

 

Of course, Benveniste’s paper had in the meantime been seized upon by homeopaths as proof that the water in infinitely-diluted (homeopathic) solutions could “remember” the structure of something that had once been in it. This idea, of course, flies in the face of all of physics and chemistry as we know it. The homeopaths refused to accept Nature’s explanation of why the experiment was totally unreliable, preferring to see conspiracy theory and suppression.

 

Scientists just saw a bunch of credulous and technically naive workers who had mislaid their scepticism – a classic piece of Cargo Cult Science. And when it further emerged that two of the people in the lab had been having their salaries paid by a company – Boiron – that manufactured homeopathic remedies, Benveniste was sunk.

 

Benveniste claimed to his dying day that he had been wronged, and that his failure to get research money from the French Govt afterwards showed that he had been “silenced”.

A less hysterical view would be that once he had been very publically shown to have mislaid his appreciation of basic statistics, or any recognition of the need to eliminate conscious and unconscious biases from his labs’ work, people began to wonder whether you could believe the work they produced on other topics, and effectively figured that putting more money into his lab was a waste of time. After all, if you can’t do an experiment properly, the data you generate cannot be relied on by anyone else. And if your data can’t be relied on, they are useless.

 

 

But…. The Alternativists have a counter-argument.

“That’s NOT FAIR”, they say. “You don’t make the mainstream scientists do all this extra work eliminating every possible source of bias. Why should we do it, when they don’t have to?”

 

This sounds good, but the main answer is actually an obvious one. The wilder and less plausible the statement, or the supposed experimental result, the better quality the evidence for it has to be.

 

And this finally, is where James Randi comes in. Randi was one of the people Nature sent to investigate Benveniste’s lab.

 

At the end of Nature’s report on the affair, the Nature trio describe trying to explain to Benveniste and his team, both why the data were seriously flawed, and also why the data needed to be watertight (which they clearly weren’t) if their “effect of no molecules” were to be believed.

 

““If I told you that I keep a goat in the backyard of my house” Randi said “…and if you happened to have a man nearby, you might ask him to look over my garden fence, when he’d say “That man keeps a goat”. But what would you do if I said “I keep a unicorn in my backyard?”” [5]

 

In essence, would you accept the same level of proof for the second statement as for the first? Of course you wouldn’t. You would want photographs, experts’ reports, forensic horn analysis, x-rays and DNA genetic testing before you would accept that the animal was what it was said to be. And quite right too.

 

Unfortunately, too often AltMed people are like Fox Mulder. They simply “want to believe”.

 

The counter-argument is put most pithily, as usual, by Feynman.

 

“The first principle [of scientific integrity] is that you must not fool yourself – and you are the easiest person to fool.” [1].

 

Words to live by.

If you stick to that, you are a scientist. If you don’t, you are into Cargo Cult territory. And all the AltMed journals in the world won’t change that.

 

 

1. Feynman RP (1974) reprinted in “Surely you’re joking, Mr Feynman” 1985.

2. Bhat MS, Rao G, Murthy KD, Bhat PG. (2007). Housing in pyramid counteracts neuroendocrine and oxidative stress caused by chronic restraint in rats.Evid Bas Comp Alt Med 4(1):35-42.

3. Schul, B, Pettit, E. (1975). The Secret Power of Pyramids. New York: Fawcett Gold Medal.

4. Davenas EF et al. (1988) “Human basophil degranulation triggered by very dilute antiserum against IgE”, Nature, 333(6176):816-18.

5 Maddox, James Randi and Walter W. Stewart (1988). ““High dilution” experiments a delusion.” Nature 334: 287-290.  Full text online here

Tags: Alternative medicine, bad science, Benveniste, Homeopathy, pyramids