Archive for May 8th, 2008

Experts: hired lackeys – & moon made of cream cheese.

May 8, 2008

It is a bit of a tough gig being an “expert” these days.

Not so much in what I do, of course. It is not as if there is all that much controversy – or interest – surrounding the Delta-like cells of the crayfish hepato-pancreas, or whatever it is people like me spend their time fiddling about with. Even if I were to put on my work website that I was a “world-renowned expert in cell physiology”, or whatever, it would raise few hackles, although the thirty people worldwide that actually read my papers would have a quiet snigger at my delusions of grandeur.

The wider world, I predict, would be unconcerned.

– until I start commenting about alternative medicine, or CoQ10, or homeopathy, at which point I will get accused by people I have never met of being in the pay of “Big Pharma”, or, as our nearly four yr old, Junior Aust, says, ”Evil Farmer”.

For the record, the biggest “payday” I have ever had from Big Pharma was a free lunch (once) in AstraZeneca’s nice staff cafeteria at Alderley Park. Other than that, nothing – not even a small consultancy fee. *sigh*

But it is quite different for experts in things like medicine… the stakes are high, and some people nowadays seem to think that all “Experts” are corrupt tools of Shady Corporate Interests, or of “the Medical Establishment”.

So what really makes “an expert”?

It is a truism of science that the more you know about something, the more “expert” you are. For this reason, the average person who reviews scientific papers for a serious scientific journal – an “expert reviewer” – will be someone who has a degree, a PhD, several years’ worth of postdoctoral research experience, and preferably does, has done, and often supervises other people doing, work very similar to the stuff they are reviewing.

The point is that you have to know the subject, and the type of work, and how it is done, inside-out to see the errors, if any. And also to see where the authors may have finessed things slightly by comparing things that are not strictly comparable, and so on.

The result is “expert peer review” – a fully-informed expert opinion on a particular piece of research, or on the latest meta-analysis.

You know you are an expert in this sense because other experts decide that you are. For instance, proper high-end science journals (and this does not mean all the thousands of journals you can find listed on Pubmed) are picky about their expert reviewers, as it is on these peoples’ rigor that a journal’s standards depend. It is for this reason that scientists sometimes list, on their professional CVs, the journals they review work for.

Conversely, I think we can say it a safe bet that the venerable Journal of Physiology is not going to be asking Sir Cliff Richard to review the next paper about whether antioxidants help older athletes get less post-exercise muscle pain.

Sir Cliff is not, whatever the food supplement industry appear to think, an expert.

However, not all public statements about research, especially medical research, or its interpretation, come from the experts.

Indeed, on many issues expert opinion is vastly outweighed by comments from what one might politely call “advocacy groups”. These include, but are not limited to, patient groups.

This presents interesting problems. One in particular is: can one trust the messages emerging from such groups?

And if we say no, are we saying that only real experts should be allowed to have a view?

Context is critical (again)

Well, let’s stay with the profile of an expert, and their ability to interpret the evidence. This ability to interpret research is a question of degrees of relevant knowledge. But it also depends critically on being able to put the research in the proper context – that is, having the background to know things relevant to the research that are not explicitly written down in the paper.

This is something scientists, and doctors, are taught in varying ways. But they are taught it. For instance, take a PhD viva (the oral examination that follows the submission of a PhD Thesis and constitutes – hopefully – the last stage of getting a doctorate).

In the viva, you do not get asked about just the experiments in your PhD thesis.

You also get asked about other experiments in the literature that bear on whatever question or questions you are seeking to answer. You get asked about the broader background area, and where your work fits it. You get asked about the key theories in the area that underpin and lead up to your work, and how strongly the experimental evidence supports them. And you get asked why Dr Y’s experiment got a different answer to yours, particularly in terms of analyzing the differences between your experimental design and Dr Y’s. Was the answer different because the patient blood samples in your experiments came from women aged 16-20 who volunteered for a specific study, while Dr Y’s were from women aged 18-30 who had been sampled for a study into something else? What are the potential ways that that could make your study populations different and thereby explain the different answers you got? Is there other evidence to support any of these potential explanations?

…and so on, and so on.

And apart from being taught it, this “contextual knowledge” is also acquired bit-by-bit over years in research, or medicine. It forms one useful kind of accumulated experience.

The reason I am harping on about this is that many advocacy groups include, or even employ, people who have read a lot of medical and scientific studies, and sometimes consider themselves experts. But their expertise often lacks a critical dimension: they commonly do not understand all the background. And I mean “background” here in a very broad way; I am including not only “the broader science in this field” and “earlier studies like this”, but also “the underlying assumptions of the work”, “the ways in which studies like this can be biased” and – very critically – “the ways in which statistics and statistical testing work in relation to this kind of experiment”

From the above, you can see that there are a lot of different levels of understanding that one can have in relation to a scientific topic.

Which is NOT to say that you cannot have an opinion unless you are a full-on “academic ninja” expert. I mean rather that you should try to recognize the limits of your knowledge, and understand that other people will inevitably know more (or less) about it than you.

[This goes, incidentally, for doctors and scientists, obviously including me. I know more than some people, and less than others, and this is subject-dependent. I know, for instance, more about acid-base balance than most physiologists. But I know less than a hospital doctor, who in turn knows less than a real “acid-base ninja”, like an anaesthetist or intensive care specialist]

Of course, in the public arena, many or even most people seem to not be too good at this “grading” of their level of knowledge.

To take an example…

As an illustration, take the controversy over home birth. Dr Crippen over at NHS Blog Doctor is currently taking a shot at some midwives in Kent (these midwives are independent of the NHS and charge for their services). They ran a home delivery for a mother with a long string of “risk factors” that would have lead any obstetrician, and probably most NHS midwives, to conclude that the mother would have to be bonkers to want a home birth. Dr Crippen is angry because the independent midwives seem to be unaware of the dangers, continually referring to the caution of the mainstream doctors and midwives as “shroud waving”.

The case described by Dr C is pretty extreme, judging by what my medical advisor (Mrs Dr Aust) tells me. But home birth is a contentious issue all around.

And in homebirth, a battle is being fought over precisely what the statistics tell you about whether home birth is more dangerous than giving birth in hospital, and by how much.

The doctors I have asked are unanimous; home birth is more risky for both mother and baby, they say, mainly for the obvious reason that if something does go seriously pear-shaped, you want to be in a hospital.

This extra risk, the doctors concede, may be slight and acceptable for ultra-low-risk normal births under ideal conditions, that is, where the overall (absolute) risk of something going pear-shaped is very small. However, the more “non low-risk” the pregnancy (i.e. the more factors there are suggesting an increased chance of complications) the relatively riskier home birth becomes. And if conditions are not “ideal” (for instance less experienced midwife, long drive from nearest maternity hospital in case of pear-shaped-ness, etc), then that all adds to the risk that is being taken.

For what it’s worth, among all the female doctors with children Mrs Dr Aust and I know, we have never met a single one, in any specialty, who opted for a home delivery.

But the home birth groups dispute the above view, and also the idea that the doctors, and even the hospital midwives, are speaking from expertise rather than ”medical paternalism”. The home-birthers feel they should have the right to choose home birth. They also point to published analyses that say that for low-risk deliveries, home birth is no more dangerous than hospital birth. Typically, a fierce and detailed argument then ensues about exact which figures do (or don’t) mean what different people say they mean. If you want to see this in action, try the discussion thread following Dr Crippen’s post

So – who to believe?

I won’t give my own opinion, for the following reason: I am not an expert in the relevant areas.

However, I think I can say who I would regard as “an expert” on the question.

It is, as ever, a question of “degrees of expert knowledge”. Who are the people most likely to know all the studies, all the background, all the factors influencing the trustworthiness or otherwise of individual studies, and all the hidden catches in comparing different patient groups and studies?

Well, for me, it’s no contest – the answer is that I would put Professors of Obstetrics & Gynaecology (O&G), or experienced hospital O&G consultants, at the top of the expert scale for this particular issue, and Joe and Joanna Public at the bottom.

The advocacy group people, even if they have scientific or medical backgrounds, would rate well below the Professors and consultants, at least for me. This is both because they have a clear agenda going in, and also because they mostly lack the full background necessary to interpret the evidence properly. I would thus be suspicious about whether any message they were putting out was strictly accurate. And my personal bete noire, journalists with no training in science or statistics, would rate pretty damn low on the expert scale, though we hear a lot from them on this issue (for a particularly misleading recent article see last week’s Guardian here).

Personally I think there is a lot of danger in folk who cannot see that there are people who know more than them (or even concede that such people probably exist) when looking at this and other issues to do with interpreting research. To repeat it again, you have to be aware of how much you know, and how much you don’t, if that doesn’t sound too Donald Rumsfeld.

Where to find the good stuff…

Plus, once you have established you don’t know enough, you also have to know where to get the information that you are missing, and from whom.

To give one example: let’s take a family doctor who has heard something about all the homebirth discussion, but hasn’t read all the recent studies, and is confronted by someone who insists that they have read that the statistics say home births are as safe as hospital births.

The doctor knows he or she needs reliable information. Luckily they know that, say, a Cochrane review written by an O&G Professor would be the place to go for the goods. And they would almost certainly rate this higher in their “evidence hierarchy” than a review in a lesser scientific journal penned by the chief advisor of a charity that promotes home births. They would certainly rate published journals studies higher than figures quoted on websites that have never been through peer review.

That is not all there is to it, of course. Our hypothetical doctor might read the two reviews side-by-side, comparing what they said about individual studies, and thus improving their understanding of how research in the area is interpreted.

Where to put your trust?

Now, this is fine if you have relevant expertise and know what to do, and where to go for information.

But: how does this discussion about the nature of experts help Joe Public when he or she is confronted by the need to make a decision? Or in the case of prospective parents and the home birth issue specifically, who should they get their messages from?

As I see it this reduces to – who can you trust to have an expert view on this, and communicate it to you without a hidden agenda, whether ideological or financial?

My answer for childbirth, having been through it as a prospective parent, would be the obstetricians and hospital midwives. Because it is their business to know the facts, and also their professional responsibility to try and get the best outcome for you and your unborn child.

In contrast, I would not expect to get anything like a dispassionate view from a homebirth advocacy group, even when they say they are “quoting the studies”.

The trouble is that there is so much propaganda about, much of it masquerading as journalism – see e.g. the Guardian article linked to above – that it must be incredibly difficult for people to know who to believe.

The danger is that they will make the decision first, based on irrational fears or on preconceptions that may be mistaken. And they will then get this uninformed decision reinforced because there will be some information, somewhere, that agrees with and applauds their choice, even if their choice is objectively ill-advised.

To me this parallels other “reality meets preconceptions” problems in health like the MMR vaccine saga. Once people have bought into the idea that “the doctors / experts / government aren’t telling me everything” – or that the doctors are pushing a particular agenda , “medicalising birth” or “pushing drugs and vaccines for Big Pharma” – then the patients / parents are cast adrift amid a sea of misinformation. They – or perhaps I should say we, since I am a patient too – are at the mercy of lots of advocacy agendas, all too commonly armed with prejudice, misguided certainty, and masses of misused stats.

Experts under fire

So just how did we end up doing this to professionals and experts? Have we, as a society, decided that their opinions are just another optional paid-for add-on?

Experts and professionals certainly are sometimes mistaken. They sometimes do not explain enough. They certainly do not get enough help from professional explainers explaining science and medicine. They are, rather too often for my taste, hung out to dry by Governments who pick them up and dump them as a matter of political expediency. And finally, experts are at times guilty of digging their own holes to fall into – e.g. by doctors trousering large consultancy fees from pharmaceutical companies.

But is that enough to dump expert views entirely? Do we really think all mainstream experts are incompetent and arrogant? Or that self-interest is their only motivation?

Personally I am happy to let the pilots fly me, the anaesthetist anaesthetise me, and the lawyer tell me what is, or isn’t, legal. And I am also aware that you can’t believe everything you read – though on the whole I believe it more if the writer is a real expert, and is not trying to sell me something.

But it sometimes seems, these days, that I am in a minority on both counts.

It is a worry.

PS Sorry for the long and boringly unfunny ramble. Hopefully there will be more satirical snarking next time.

Quote of the day: “I can live with doubt and uncertainty and not knowing. I have approximate answers and possible beliefs and different degrees of certainty about different things… It doesn’t frighten me.” – Richard Feynman