At the risk of repeating myself: more on AltMed journals

The day after I put up the rambling post on AltMed journals, Badscience Guru Ben Goldacre’s definitive five-star demolition of homeopathy ran over several pages of The Guardian and on the Badscience blog.

The resulting comments thread at Badscience.net spawned the occasional defender of homeopathy, but what caught my eye was the one, calling him/herself Budicius, who wrote as follows:

“I get bored to tears with the same old rigmarole from sceptics. To say that Homoeopathy is nothing but placebo is an uneducated and ludicrous comment… …Look at http://www.liebertonline.com at the article – “Efficacy of the potentized Drug, Carcinosin 200 fed Alone and in combination with another drug – Chelidonium 200, in Amelioration of p-Dimethylaminoazobenzene- induced Hepatocarcinogenisis in Mice”. These Homoeopathic drugs have been diluted two hundred fold and are successful in the treatment of and inhibition of Hepatic carcinogens.”

- The abstract of the paper, which appeared in the Journal of Alternative and Complementary Medicine (henceforth “JACM”) is on Pubmed here.

And a little later:

“I found another one with favourable results. “Amelioration of Carcinogenesis Induced Toxicity in Mice by Administration of a potentized Homoeopathic drug, Natrum Sulphuricum 200″ This one is at ecam.oxfordjournals.org”

- this study, which is from the same authors as the other one s/he cites, is in Evidence-based Alternative and Complementary Medicine and can be found, with link to the free full-text version, here.

*Sigh*. Just in case anyone found the last post on AltMed journals too verbose, let me re-state it in simple language. In my experience, studies published in such journals of “Alternative Medicine” (or complementary, or integrative, or whatever other studiedly neutral term they are camouflaging themselves with this year) usually turn out to be so flawed as to be scientifically valueless. IMHO this is because the standards of “expert peer review” at AltMed journals like OUP’s eCAM, and Liebert’s JACM, are, to put it very mildly, debatable. Or more trenchantly, a joke.

The Modus Operandi in these journals, as I was trying to convey with my analysis of the Mellow Rats in Pyramids paper, seems typically to be as follows:

(i) Carry out poorly-controlled experiment into “alternative modality”;

(ii) Resolutely ignore all negative findings (always assuming your publication bias means that you bother reporting them in the first place);

(iii) Resolutely ignore any and all possible confounding factors that could affect the results- instead:

(iv) Attribute any “positive” results to mystic effect of homeopathic remedy / pyramid / Reiki energy field;

(v) Publish in AltMed journal where “expert peer reviewers” ignore lack of controls in (i), repeat one-eyed (ii) and (iii) and unquestioningly accept daft interpretation in (iv), because they share your belief in mystic nonsense.

Pure Cargo Cult Science
. It clothes itself in the appearance of science, but utterly lacks the critical ingredient of necessary scepticism. What you get instead is a collective act of suspension of critical thinking, and indeed of implicit belief in magic.

Clearly I am generalising, but every time I read a paper in one of the AltMed journals that claims to provide “scientific evidence for homeopathy” (or similar), this is what I see.

BTW, the editor in chief of the JACM is Professor Kim Jobst (brief bio here). Kim Jobst is a medical homeopath, was a founding Council Member of the Prince of Wales’s Foundation for Integrated Health, and was for several years (see post below from David Colquhoun) a Visiting Professor of “Healthcare and Integrated Medicine” (that I-word again) at Oxford Brookes University. Apart from editing the JACM, he is best known in the Badscience world for his endorsement of the laughable qLink “anti-EMF” pendant, a medical marvel (not) thoroughly debunked by Ben Goldacre here.

To re-iterate the point. Here is a journal whose Editor-in-Chief seems to believe a device containing no active electrical components can nonetheless have health effects by producing a mysterious “Sympathetic Resonance” effect hitherto unknown to biology.

Given this, how is one to believe the said journal and its “expert reviewers”use any meaningful scientific standards in deciding what to publish?

PS In case Shpalman or any other proper physicist is reading, and would like to engage with the qLink and how it claims to work, the manufacturer offers an explanation here

About these ads

23 Responses to “At the risk of repeating myself: more on AltMed journals”

  1. Mojo Says:

    Huntsman, a poster on the JREF forum used to have a signature reading “Science is the process of crash testing ideas: you ram the idea head-on into a brick wall at 60mph, and knowledge is gained by examining the pieces. If the theory is solid, the pieces are from the wall. Then we build a bigger wall.”

    The AltMed journals don’t try hard enough to break the ideas.

  2. misterjohn Says:

    Thanks for the reposting with correct links. I looked at the explanation of qLink, and despite having studied some quantum mechanics in my University days, remain unconvinced that anyone can produce a gadget that focuses “subtle” forces which “cannot be observed or measured by any known instrumentation.”
    My simple point is that if you can’t sense them in any way how the heck can you do anything with them?
    I know we don’t really know what gravity is, but at least we see its effects.

  3. Andrew Taylor Says:

    Hold on, the drugs “have been diluted two hundred fold”? That’s not homeopathic — people dilute conventional medicines more than that. A lot of them’d be lethal if you didn’t.

  4. Ulrich Berger Says:

    When I came across the qLink pendant on Badscience several months ago, I found to my embarrasment that four of the “scientific studies” the manufacturer listed not only originated from my home country (Austria), but two of them even from the university I studied at (U Vienna). I used these – amongst others – as examples of the involvement of even university based researchers in financially promising pseudosciences in an article in a widely read Austrian newspaper. http://derstandard.at/?url=/?id=2975156

    Only one of the four study authors bothered to reply. It seems several years ago he undertook some efforts, even legal ones, not to let his name be associated with the qLink pendant. Only partly effective, however. The other one did a statistical evaluation of some obvious quack data, and while the statistics are fine, I am sure he knew what was really going on.

    When looking up the names associated with the pseudoscientific gadgets I was amazed to find how many of those where homeopaths or more general CAM practitioners…

  5. Scooby Says:

    Andrew – the “two hundred fold” comment was from Budicius, not Dr Aust.

    I presume it means 200 C homeopathic dilutions i.e about 1 * 10 -400 (can’t figure out superscript).

  6. David Colquhoun Says:

    Concerning Kim Jobst. I recently submitted an FoI request to Oxford Brookes, hoping to see the documents relating to his appointment as a visiting professor.

    It seems he was a visiting professor from 1999 to 2004. I was told on Sept 10 2007

    I can confirm that Dr Jobst no longer has any connection with the University.
    Most of the people involved in the appointment in 1999 no longer work for the University so the committee papers from the time are all that we could find.

    So it seems that Oxford Brookes are doing a bit better than the University of Teesside.

  7. draust Says:

    Thanks for dropping by, Ulrich.

    Haben Sie vielleicht gedacht, ihren Q-link Artikel in Englisch uberzusetzen?

    – I don’t know that I have seen any info in English on the “research” cited by qLink, and it sounds quite interesting.

    By the way, Frau Dr Aust, die aus tiefsten Oberbayern stammt, says she thinks the South German and Austrian mindset is often quite susceptible to mystic mumbo-jumbo.

    Herzlichen Glückwunsche zu ihre Dr Oec Habil

  8. draust Says:

    Thanks for the info on “Prof” Kim Jobst, David.

    He has an interesting history since from 1988-1996, when he was presumably still vaguely in the mainstream, he was part of an ongoing research project on memory and ageing in Oxford called OPTIMA. The project is still headed by founder Prof David Smith.

  9. EEJ Says:

    Dr Aust,

    I just wanted to comment and congratulate you on the blog. I’ve read your comments (and enjoyed them!) on other blogs.

    Hoping you continue the effort on the blog, and looking forward to much more interesting reading.

    Thanks!

  10. emily Says:

    p.s. I slid a reference to the pyramid paper into my current book chapter. Thanks for bring it to my attention. I would offer a finder fee but, um, as is so often the case I am not being paid. If you come across more rat woo I hope you will post it :)

  11. draust Says:

    No fee- ’twas ever thus, Emily.

    It always makes me laugh when I see the Alties going on about scientists being BOUGHT AND PAID FOR with huge amounts of PharmaCo cash. In my twenty-odd years’ experience most academics will spill their guts for a sandwich and two pints of beer. Or maybe just one pint. It may be that Movers and Shakers in Medicine are more pricey.

    I will keep my eyes peeled for more Alt.Rattery. Talking of animal behaviourism, do you remember the part of Feynman’s Cargo Cult Science speech where he talks about “Mr Young” and the rats-in-mazes experiments from the 30s (almost at the end of the speech)? Do you know if the original work is discussed in detail on the net anywhere? It is a good example of “doing proper control experiments”, so I would be curious to see the original papers.

    The other rat behavioural study I always meant to look up some time (but so far haven’t) was Bruce Alexander’s Rat Park work.

  12. emily Says:

    I don’t know the Young experiment but I am writing another chapter in my book that rants, um, discusses this issue of false standardisation rather than looking at how the animal actually does what it does. Seems like a few commentators keep saying this but it just isn’t getting through. We end up with results we can explain, but the explanation is in fact just wrong. Very depressing. One theory I have is that many rat scientists are insufficiently interested in the ratness of rats and the ratness of how rats do things (i.e. everything a rat does is a rat thing, and best inderstood in that context–not one’s favorite model).

  13. draust Says:

    Emily wrote: One theory I have is that many rat scientists are insufficiently interested in the ratness of rats and the ratness of how rats do things.

    Yes, nicely put. Of course, the big money is in using rats and mice as models for people, so in that context the anthropomorphism is pretty unsurprising when coming from the bioscience / medicine types (like me!).

    I am surprised if the animal behaviourists are like that too. I vaguely thought that, post Temple Grandin’s work and all the coverage of it, they would have been wiser to the perils of overly anthropomorphic interpretation.

  14. Life’s 4 Living, The Energy Clinic, Claire Sutton and Sarah McCrum « Holford Watch: Patrick Holford, nutritionism and bad science Says:

    [...] to have an advisory board that is made up of eminent physicians although none is named except for Professor Kim Jobst (he of Q-link). Holfordwatch hopes that some appropriately-qualified people have oversight of the medical, [...]

  15. Joe Says:

    @Andrew Taylor,

    Concerning the 200-fold (0.5%) dilution of a homeopathic product. This has been discussed elsewhere, with respect to even more concentrated products. I suggested that the first dilution and succussion makes something homeopathic. However, I asked a colleague who knows far more about the subject “what makes something homeopathic?” She replied “How long is a piece of string?” Several weeks later, having measured many pieces of string, I am still uncertain.

  16. Surekha Bhat Says:

    all i can do is respond to this saying before publicly condemning a piece of work, go visit the animal houses and the labs in which the works are carried out and study the way they do it, tightly regulated to avoid bias. I am the first author of the pyramid paper and have been involved with mainstream medicine research as well. A journal published in a scientific journal cannot cover the a-z of modus operandi, things like animal houses were constructed according to the international norms specified (constant internal environment, minimal noises, etc), that stress was induced during the same time of the day for all groups of animals, that even the weather conditions were taken into consideration, etc, etc . Some things are just understood by the scientific world. A suggestion here, a genuine interest to know facts drives a seeker to seek direct clarification with the researcher, and not condemn the work by playing with words.

  17. draust Says:

    Hi Surekha

    Thanks for coming by to comment.

    I can see why you might think this post is a bit dismissive about your paper. It is true that coverage on blogs is generally more “blunt” than in the pages of learned journals, and even at conferences, where everyone is being terribly polite.

    In fact, the main discussion of your paper is in a previous post, where we discussed possible things other than “pyramid power” that could explain the effects you reported. I suggest you read that post first, as it gives the background and rationale to the more sarcastic short “caricature” in the current one.

    Anyway, the following response refers to the previous post and the comments there as well as to what you have written.

    First, I am pleased to hear that you and your co-workers conducted the study with great care, but “poorly controlled” (as I wrote in this post) does not necessarily mean the same as “not done with care”.

    I think you need to accept first that implausible claims (like pyramids exerting mysterious “biomagnetic resonance” effects) need far more convincing evidence to be accepted than “everyday” claims – this is the “prior probability” or “prior plausibility” argument (and see also here). In particular, I would argue that there must be extra and stringent effort to rule out “prosaic” (or “routine”) explanations (i.e. explanations that relate to known effects, like observer or expectation bias). These effects are intrinsically more likely (plausible) than an effect of pyramids via some uncharacterised energy. So you have to rule the “routine” effects out rigorously. It really is not enough to just say “science can’t explain everything”. You have to eliminate all the possible scientific explanations rigorously before you can postulate an explanation via an effect wholly unknown to science. Richard Feynman’s famous lecture on “Cargo Cult Science”, as discussed in the earlier post, gives several examples of this.

    Expectation bias (convincing yourself you see what you want to see) of various kinds is well recognised in scientific and “fringe scientific” studies – see e.g. the Jacques Benveniste homeopathy study – so again, before you claim:

    “a pyramid shape has a mysterious unknown effect which caused the results”

    – you need to go the extra mile to convince people that it absolutely could not possibly be something confounding like:

    “the animal handlers knew what result the investigators wanted to see”.

    Remember, in the Benveniste homeopathy study the effect disappeared when the protocol was modified (by Nature’s investigators) to make it impossible for the researchers to bias the results. There was no particularly suggestion that Benveniste’s team biased the results deliberately – indeed, it is far more likely they did it without realising what they were doing.

    Coming to the bit I wrote in this post about “disregarding other possible explanations”: As various commentators on the previous post pointed out, there are things that would be plausible explanations for the effect you observed that are not considered as explanations in your paper. The “higher ceiling” (less cover directly above the animal cage) in the square box (relative to the pyramid box, where the box sides are much closer at cage level ) is one such explanation, as suggested by both Emily (who is an animal behaviourist and specifically a rat behaviour expert) and by Tsu Dho Nimh. This explanation seems the most obvious one as the effect of “cover” on rats is, they both tell us, well known.

    It is also, I note, potentially a testable explanation – e.g. by having a partial “false ceiling” in the square box, say made out of dark paper or cardboard, mounted above the rat cage in the square box so that it that mimics the “contours” of the pyramid box roof, or even just simply provides a closer covering above the cage (i.e. cover near the cage but in no particular shape) . The prediction would be that this would “mimic” the pyramid box effect if it was the level of close cover of the cage that was the important parameter. Another similar suggestion, made by a commenter (“Muscleman“) who is also an experienced biological scientist, was a cone (circular tapering) shape; this is not a pyramid, but (on the above explanation) would be expected to have the same effect of making the rats more secure (less stressed).

    Now, I would say that in a mainstream journal, referees would have pointed out some of these problems and asked that they be addressed or corrected. An animal behaviourist would certainly have drawn attention to the likely cover/enclosure effect on the animals. Anyone of a sceptic mindset familiar with expectation and observer bias would have asked for details of how you made sure these kind of biases and/or handler effects could not be contaminating your results.

    Finally, and not referring to your work specifically; I have read a lot of research in journals of complementary medicine and I am, I am afraid, not terribly impressed with the standard of thinking and rigour on display. Hence my ruder remarks in (v) in this post, which were directed at the journals, their editors, and their editorial boards. Authors improve their experiments and experimental designs, in part, because journal referees make them. At least, that is how it ought to happen.

  18. Surekha Bhat Says:

    Thank you Dr.Aust. My thesis has a covered a lot of information that you have sought here and I am afraid I couldn’t cover the same in journals. In any case, let me respond to your points one by one:

    “One major possibility is differences in animal handling. Handling lab animals, if they are not accustomed to it, is a major stressor. Being picked up and moved between cages is stressful. Thus any systematic difference in how animals are handled can show up in the results – for instance if the animals going into the pyramid were not handled identically to the others”

    As far as both me and my coworker have observed, handling of rats was done in the way expected by us and mishandling as well as differences in handling were minimal, I wouldn say there were never any differences in handling, but these were minimal.

    “Time of day is another possibility. Stress hormones like cortisol have diurnal rhythms. The blood samples were all taken at the same time of day (within an hour), but the time of day the stress was applied could make a difference”

    All animals were stressed beginning from 10 am (plus or minus 15 minutes) and ending at 4 pm (again plus or minus 15 minutes)

    “Next, the surroundings could make a difference. How shielded were all the housings (especially the wooden pyramid and box) from sources of noise? Or – how crowded were the cages are in which the rats were housed when not in the stress cages? Another thing that would make a difference to stress responses. Were all the rats housed in identical-sized groups?”

    We have a set of norms to follow here while working in the animal house, specified areas where rats are stored, specified areas for exposure, etc. Your concern about sources of noise is a concern to our institutional animal house as well and that has been taken care even during construction of the animal house. Three to four rats were housed in each polypropylene cage of dimensions 30x22x14 cm. Paddy husk was used as bedding material, which was changed on alternate days. All the rats were fed ad libitum with synthetic feed containing crude protein (21.2%), crude oil (4.68%), crude fibre (2.95%), ash (6.2%), sand silica (0.95%), moisture (9.25%) and energy (3600 kcal/kg) obtained from Amruth laboratory animal feeds, Pranav Agro Industries ltd., Sangli (Maharashtra). Water was available to the animals at all times. Proper ventilation was provided. Standard hygienic conditions were maintained in the animal house and the rats were exposed to alternate light and dark cycle of 12 hours each. The experimental protocol was approved by the Institutional Animal Ethics Committee of the Manipal Academy of Higher Education (a deemed university) and met the guidelines of Committee for the Purpose of Control and Supervision of Experiments on Animals, Ministry of Social Justice and Empowerment, Government of India. (Registration number of the animal house – 94/1999/CPCSEA)

    Thank you once again for raising these issues and I will try my best to cover these explanations in my forthcoming papers.

  19. DMcILROY Says:

    Come on Surekha, why on earth do you waste your time doing such pointless work?

    No, I don’t want to read your answer. I just want you to think about the question.

    DMc

  20. Garret Moddel Says:

    DMcILROY wrote “Come on Surekha, why on earth do you waste your time doing such pointless work? No, I don’t want to read your answer.” That is an unconscionable comment!

    I have no idea how effects of pyramids could work, but as a scientist I must be willing to look at the data. If repeated data show an effect, I must take it seriously. Knowing the truth in advance and not caring what the data show is unpardonable.

    Clearly Bhat has carried out a decent control experiment. Now let’s see if it can be replicated with possible confounding factors corrected. If not, then apparently there were factors that she didn’t control for. If so, we have something fascinating indeed!

  21. draust Says:

    Hi Garret.

    Thanks for commenting, and sorry for the delay in replying – blame University Spring Semester time here in the UK.

    The two issues that your comment doesn’t address are:

    (i) the control was actually significantly flawed, notably for the reason Emily and Tsu Dho Nimh outlined about overhead cover for the rats and its effect on animal behaviour and stress – see my extended reply to Surekha Bhat for the links, or look back up the comments thread.

    (ii) there were a number of other undescribed (or inadequately described) potential confounding effects, all familiar to bioscientists, that could have influenced the original experimental result. Again, I outlined some of these in the post and again in my reply to Surekha. Her reply deals with some (though not all) of these, but it remains true that essentially no information addressing these things appeared in the paper as published.

    Continuing this general theme. it is surely unsurprising that people want to have the prosaic explanations, or well-recognised potential confounders, ruled out before they accept wacky (or ‘highly implausible’, if you prefer) explanations. The Benveniste homeopathy affair (again discussed above) offers an obvious example of how people can become wedded to their improbable hypothesis/explanation of choice, with the result that it blinds them to the far more obvious way their results actually arise. There are many other examples in other fields, like the Pons & Fleischmann cold fusion saga, and indeed whole books have been written on the topic of scientists fooling themselves. Personally I stand by the old line that:

    ‘Extraordinary claims require extraordinary evidence’

    I do agree with you, though, that the next thing Surekha should do is carry out the more appropriate control that people suggested.

    As to DMcILROY’s comment, I think you are missing that it was an off-the-cuff exasperated remark. It is hard not to get exasperated by the amount of poorly-controlled, and often egregiously over-interpreted, stuff that routinely appears in journals of alternative medicine. I am not referring to Surekha’s paper in particular here. Her study appears painstakingly done, but would have benefitted from better refereeing which should have made – but clearly didn’t – some of the suggestions detailed by commenters here. Sadly, though, this is entirely typical of CAM journals.

    If the response to this Tsunami of author and referee credulity, and scant scepticism, is sometimes a bit of reflex snarking, then that is just human nature. There is also the question of whether all the effort that clearly went into doing the study wouldn’t have been better employed looking into something less inherently implausible.

  22. University of Buckingham does the right thing. The Faculty of Integrated Medicine has been fired. Says:

    [...] Jobst, homoepath and endorser of the obviously fraudulent Q-link pendant.  His Plaxo profile [...]

  23. Chiropractic For Autism « Stuff And Nonsense Says:

    [...] Aust has also been quite outspoken on the trouble with CAM journals: here, and in a follow-up post. The follow-up post contains a summary of what Dr Aust perceives as being the modus operandi [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 1,991 other followers

%d bloggers like this: