White coats, white lies? Or black marks?

In which Dr Aust ponders what one should do about students who are ‘economical with the actualité’.

——————————————————————————-

One of my scientific internet friends, Steve Caplan, was blogging a week to two back about a student in a lab he worked in who was, shall we say, less than truthful about their actions.

Steve says at one point, recounting what he said to the lab boss about the student:

“Imagine if s/he lied about little things like salt buffers [experimental reagents], what kind of data s/he might have fabricated later on,”

Which caught my attention – partly because research misconduct, and what should be done about it, is in the news at the moment.

But also partly because it deals with a regular part of University life – that problematic character, the student whose habits give cause for concern.

Much of the debate about student conduct recently has tended to focus on plagiarism, but here I am not talking about that, or even what one might call “academic misconduct’. I am talking about the more subtle things to do with how people behave, and how they interact with their colleagues.

Apart from in the lab, another place this turns up  – and perhaps a slightly special setting – is medical school. There is a long-standing debate about how sub-standard conduct should be dealt with, and what behaviours students should be ‘pulled up’ for – especially for students in the pre-clinical years where we have historically tended to take a more relaxed view of how students should behave than our clinical colleagues.

[The main point was that in the past students did not see patients until their third year of medical school. Nowadays in many courses they are around hospitals and GP clinics from the very start of their student career. So they are under the microscope rather more than was formerly the case.]

One argument for what one might call a “Zero tolerance” policy is studies like this one, in the New England Journal of Medicine a few years back.

Of course, the counter-argument, which also has force, is the ‘youthful high jinks’ one – that is, that everyone does daft, or even idiotic, things, and behaves like a plonker, when young, and mostly people grow out of it. For some medical examples you could try many a medical memoir – or for a recent political example you could try Louise Mensch MP’s robust riposte to the Daily Mail a couple of weeks back. There has even been a discussion in the medical literature of whether previous criminal convictions – which one could call an extreme example of idiotic behaviour – should be an absolute bar to entering medical school.

On the whole, when we hear about misbehaviour by students, we tend to try and distinguish between “stupid stuff” (they really don’t know);  and “worrying stuff” (they know, but do it anyway) – of which the most worrying is usually “dishonest stuff”.

A couple of examples of the latter.

In our lab classes for medical students we teach them stuff that they are later tested on, like using peak flow meters and spirometers, and measuring blood pressure. The test they get on this later is in the form of a what is called an OSCE, a kind of practical exam in which students pass through a series of “stations’ where they have typically 5 or 6 minutes to do a task – like measure a volunteer subject’s blood pressure.

As well as teaching students these skills in scheduled classes, we run revision classes a couple of weeks before the OSCE takes place.

Now, the people who teach these revision class are adamant that these classes are for revision – NOT for teaching the skills from scratch to students who couldn’t be arsed to turn up to the regular scheduled sessions earlier in the semester. So the rule for students is that, if you have missed more than 20% of the semester’s scheduled lab classes without explanation, you cannot come into the revision session,

Though this is well publicised, it sometimes seems to come as a shock to less, errm, organised students.

A good few years ago now, a couple of my colleagues were checking the students in at the door on such a revision class when a student appeared who the records showed had exceeded the allowed number of absences. They pointed this out to him and said he would not be allowed in.

“Oh no” said the student “I gave in notes explaining that absence and that absence” (this would get him into the revision class, as ‘excused for something you couldn’t help’, like a doctor’s appointment or illness, or some other stuff, usually doesn’t count as ‘absent without reason’).

Now, one of my colleagues was a touch suspicious, trotted off to the Faculty Office that ‘logs’ such notes, and checked what the student had said.

The student had handed in nothing for the dates in question. No notes. No reasons for the absences.

The student was told to leave the class, and was, I am pretty sure, reported to the Course Director for what in the old days would have been called ‘a right bollocking’.

It was not the being absent from the earlier classes that was the most worrying thing, but rather that the student had flatly and directly fibbed (about the notes for his absences) to a member of the academic teaching staff.

I am curious to know what my readers, especially the medical ones, make of this.

Now, Mrs Dr Aust, the family’s medical expert, has little (read: “no”) sympathy for the student in this story. She is suspicious as to whether leopards change their spots, for one thing. And she says that people who will lie, or even just bend the truth a bit, to get themselves off the hook are a flat-out liability in medicine, full stop. The standard scenario she tends to give is one like this:

In an urgent diagnostic discussion about patient Mr X, gets-by-on-charm-but-a bit-of-an-idler junior doctor Dr Y is asked by a more senior colleague about the result of test Z that he, Dr Y, was supposed to have ordered yesterday. Dr Y had forgotten to order the test. However, fearing a public kicking, he does not admit this but says instead ‘ Errm… the result’s not back yet’.

Of course, you can argue that that latter scenario is unrealistic. But Mrs Dr Aust says that this happened sufficiently frequently to her over the years that eventually she took to phoning the lab herself when she heard this line – to be told, on more than one occasion: “Nope, there’s no sample”.  She also says that there were junior doctors who, when confronted about this, would respond with anger rather than holding their hands up. She says one even called her ‘A Fascist’.

You can also argue that this kind of behaviour is a long way from the skiving student-in-the-lab-class example that I gave above.  But, and this is the crux of the matter,  there is always a suspicion that people who are prepared to lie about one thing are more likely to be prepared to lie about other stuff too.

And there is also a sense that, the more they get away with it, the more they will carry on doing it.

So where should we draw the line? 

———————————————————————————-

The day after I wrote the first version of the above, I was reminded by a discussion on Twitter about something else in a similar vein (artery?).

In our practical tests of measuring blood pressure, which is one of the skills we teach to students, the examiner almost always listens to the same Korotkoff sounds that the student hears via a dual stethoscope.

One of the things you regularly experience as an examiner for this is the student who, when no sounds whatsoever have been audible – typically because the stethoscope bell is in the wrong place – announces:

“One twenty over eighty”

(i.e. a blood pressure of 120/80 mm Hg, which is normal and thus a good random guess for an adult – though it is usually a little high for a fit younger adult, like most of our volunteer subjects tend to be. Of course, we take their BP at the start of the day and periodically thereafter, so we know what their BP is really likely to be).

We were talking about this today, and the thought occurred to me: is offering that answer, when you could not possibly have got a reading of blood pressure, actually – or at least arguably – an attempt to deceive?

And – it would be a pretty draconian way to do it, but should we automatically fail anyone who says it?  

——————————————————————————–

Postscript:

Just to expand on this a bit:

There is always a kind of tension in professional degrees between the need of students repeatedly to pass exams to progress on the course, and the ultimate need for them to learn to do certain things (like measuring blood pressure). One is regularly told (and it is a reasonable axiom) that ‘assessment drives learning’. So if you have a test where there is no downside to guessing (apart from not getting the mark you would get for getting the right answer), students will guess.  [Many University multiple choice exams use what is called ‘negative marking’ specifically to give students a downside to guessing at random when they don’t know the answer].

But in the setting I have described, there are several problems with this.

One is the problem described above – you could say we are allowing the students to assume it is OK to make up an answer, when really it shouldn’t be.

Two is that it is of course important to be able to measure blood pressure accurately; the students should, in turn, be able to see that it is important that they learn to do it properly. If you are being really earnest, you might say they should be able to see that ‘fronting it’ (doing a third rate version but trying to look super-confident) is not really on.

Three is that you could say this represents a lost opportunity – perhaps the students should be pulled up straight away and asked to expand on why they had trotted out a number when there were no sounds.

The point there being to get them to think about why there might be problems with what they had just done that might make it more than just the equivalent of a random guess in a multiple choice exam.

After all, if behaviours that are not acceptable are not identified and challenged, how do the students actually learn…. that the behaviours are not acceptable?

Or – am I just being a hopelessly crusty old git? Highly possible, after all. I am undeniably middle-aged. And famously curmudgeonly.

Anyway – opinions, anyone?  I know it’s a bit cheeky asking for views when it’s been so long since the last post, but I would be interested in what the readers, young and old(er), think.

 

PS – I should also say that, though I’ve been around medical schools a long time, I don’t pretend to claim any great expertise in assessment. Far greater minds than mine etc etc.

Advertisement

20 Responses to “White coats, white lies? Or black marks?”

  1. stevecaplan Says:

    Nice post, Dr. Aust.

    I have to say that I tend to agree with the other Dr. Aust, and I tend to have little sympathy with the “foileshticking” (Yiddish for fooling around) you describe for the med students.

    I don’t think there’s any reason to give a student the benefit of the doubt and equate guessing on a multiple choice exam with making up hearing audible sounds while taking blood pressures. I think that a student who can’t see why these two instances are different will fail to discern between other moral issues and questions.

    What would be fair is to convene all the students at the start of their med school studies, and perhaps even once every few months and remind them how critical their behavior is and how it comprises an essential part of their training. In other words, they have been warned. But I agree in general that people rarely change. Sometimes they will make more of an effort not to get caught. Occasionally, in rare instances people can change, but it usually takes a life-altering experience. For the most part WYSIWYG (what you see is what you get).

  2. Stephen Curry (@Stephen_Curry) Says:

    I would always pick up on this sort of behaviour with students. Given that the medical profession is so keen on puffing up its professionalism (though ironically that is itself a bit of a front), I’d say their students should be held to exacting standards right from the start.

    For a first offence I wouldn’t necessarily impose any sanction but would certainly challenge the behaviour and ask the student to account for it. For some, that pause for thought should be enough to set them back on the straight and narrow. I’d also make it clear that any repeat offence would attract a stiff penalty — as you say, if they will lie about little things, where will it end.

    The movie Before the Devil Knows You’re Dead is an excellent (if slightly extreme) morality tale of how minor misdemeanours can become major ones. ;-)

  3. xtaldave Says:

    As a postdoc who has not yet risen to the higher echelons of command – IMHO any sort of propensity to falsify anything should be beaten out of students (undergrad and postgrad) by tutors and PIs ruthlessly.

    I have just spent a goodly amount of my current post (read: years) trying to discern good data from bad, from a PhD student who has passed their viva, buggered off back to whence they came and now refuse to answer any e-mails.

    I have discovered photoshopped gels that made it into their thesis (photoshopped in powerpoint – and left so that one and ‘ungroup’ the gel to reveal the deception) and irreproducible results along the way. IMHO, the gradstudent in question should be stripped of their PhD – but that is not my decision to make. I have essentially repeated the most important parts of their PhD in order to re-generate the data in a satisfactory manner in order to finish off a paper. Student shall be removed from author list.

    The upshot of this deception has meant that my productivity has suffered whilst the original perpetrator has got away scott free.

    So yes – any and all white lies should be dealt with – maybe not with formal sanctions to begin with, but if left unchecked (as in this case) – this sort of thing can snowball.

  4. Dave Says:

    One approach is to require students to give with every answer an estimate of their confidence in the answer. Someone who honestly doesn’t know won’t be penalised (just helped), but someone who is overconfident looses marks. Something like Brier’s proper scoring rule, as used in quiz.worldofuncertainty.org, should do.

  5. Dr Zorro Says:

    In my experience those doctors who display mendacity, dishonesty, dishonourable and self serving behaviour seem to do rather well, and disproportionately occupy high positions in the profession.

  6. hamlets_ghost Says:

    I feel very old and crusty. I’m currently doing an MSc, and have observed people doing the course for the qualification, rather than to learn the subject. There seem to be two separate issues ahich will encourage mendacity:

    i. the belief that you can do something because someone has given you the qualification (it has nothing to do with whether you believe you can do it)

    ii. a culture which rewards blagging, cheating and lying (cf phone-hacking etc)

    The general prevalence of social Darwinism (if a behaviour gains you status within a group, it is beneficial) against the rigours of eternal punishment for bearing false witness will encourage this sort of behaviour, and since most of the time lying is a successful strategy, we will probably see more of it. I think the “estimate of confidence” sounds wonderful, but regret that many studies show that it is the confidence with which someone is presented, rather than the data itself, that causes it to be believed.

  7. Michael Says:

    An interesting article: I’d like to focus on the specific question asked at the end from a statistical point of view.

    The question was: “And – it would be a pretty draconian way to do it, but should we automatically fail anyone who says [the bog standard, “normal” answer]?”

    My answer is an emphatic no, for two reasons. First, there’s the issue of specificity. Here we consider “giving an answer of ‘120/80′” as a risk factor for “cheating”. How many false positives would that throw up. I’d pity the poor sod who did everything by the book and was unlucky enough to have a genuine reading of “120/80”.

    The second reason boils down to considering things “in the long run”. This is a measurement of behaviour, and behaviour (unlike concentrations measured under controlled conditions in a lab) changes over time, in response to external factors. If word got out that “120/80” was an automatic fail, what would a cheater do in response to this? Pick the next-most plausible value, right?

  8. micktagg Says:

    Dr Aust,

    ah … ethics, damned ethics and damning ethics.

    (i) It’s blooming difficult to fail a student on academic grounds and this might become impossible once they become fully fledged customers. Given the level of pre-selection (ahem) for medical students this is a bit of a surprise – they are thought to be academically excellent from the outset and if it turns out not to be so then why keep them? Unless you factor in something from left field. Oh, I don’t know, maybe the beancounters not wishing to lose some £’s from their spreadsheet column. This will feed in to any similar analysis of what to do with students on the take – having a sliding scale of severity of deviousness if you like. Which is probably fair (especially to identify a panicking student and help them out) but it will mean that it has to be a REALLY serious matter before HR (or whichever administrating body) will sanction a serious punishment such as “there’s the door and here’s your hat”.

    (2) There might be an element of ‘do as your leader does’ i.e. fly by the seat of your pants, keep yourself a moving target etc… Which can also lead to an interesting counter-offensive to be known as ….

    (3) Revenge of the student. To see how (2) and (3) can play out follow:

    http://www.globaltvedmonton.com/events/any/4937399/story.html

    and

    http://www.montrealgazette.com/news/canada/University+dean+medicine+resigns+after+plagiarism+accusations/4964360/story.html

  9. draust Says:

    Dear All

    Thanks for the
    many excellent comments. Currently without internet except via insanely expensive smartphone connection so you may have to wait for a considered reply..!

  10. Stuart Says:

    We’ve had some odd things up North recently. A friend of mine (Senoir post doc) who was supervising a lab project with medical students explained that experiments had to be done properly because you “can’t just publish rubbish”.

    The response (from the group of 4 students) was “why not?”.

    At our University medical students get points for published work, presentations to conferences etc.

    If they publish rubbish they still get the points, there are no extra points for doing a good project that doesn’t end in a publication. All they want is their name and a paper and the points towards their degree.

    To be honest I think it’s a very important question. A significant fraction of most work published in peer reviewed journals could be described as rubbish…..and publishing a huge amout of garbage is, to be perfectly honest, a good career strategy compared to publishing a modest amount of solid work.

  11. drphilyerboots Says:

    Stuart,

    Indeed publishing rubbish often improves the citation index as others disprove the published rubbish. These can be high scoring papers!

    Dr Aust,

    I am faced with similar issues when doing ARCP (annual review of competency progression) on our postgraduate trainees. I am alarmed at some of the evidence presented at times, and have seen some very fishy logbooks.

    It is hard to get an open and shut case that would allow serious consequences. Further surveillance without hard evidence has led accusations of bullying in the past. Due to the rotating nature of placements there is a tendency to get away with it.

    Your student who lied about the attendance slips should be reported to the faculty office. The one who lied about the blood pressure, even when the examiner could hear the sounds were not there is potentially dangerous. There should be a mandatory fail for the exam, even if they passed the other OSCE.

    Negative marking on exams used to be standard, but has recently been dropped as apparently it discriminates agaist females. I am not sure of the evidence for this.

    I worry a great deal more about these issues than about the trainee who is simply not very good. As a trainer that is a straightforward problem. What we need is a supportive training environment, so that trainees can be open about their competence (or lack thereof), and an absolute intolerance of lying and deceit. As long as the risk profile of lying is less than the risk of being found ignorant or feckless, people will lie. Often these are correlated with the ignorant and feckless lying to cover their tracks.

    I am working on a post on some related issues on my blog.

    Thanks for a very interesting post

    Dr phil

  12. draust Says:

    After a lengthy wrestle with ‘dial up internet hell’, a reply to most of the earlier comments. It’s a bit unfinished, but I shall post it now while the bit rate is vaguely workable.

    Will try and respond to Dr Phil and any others I’ve missed in a day or two.

    @Stephen Curry

    Yes, the road to Hell,and all that. My favourite film along these lines is Sam Raimi’s nicely pared-down A Simple Plan
    http://en.wikipedia.org/wiki/A_Simple_Plan_%28film%29

    @Steve Caplan and Stephen Curry

    It is certainly true that you have to be very clear upfront about what is expected of the students if you want to apply any kind of sanctions for unacceptable behaviour. Everything in Univs (as everywhere else) has become very legalistic these last 10-20 years, a trend which is only likely to accelerate in the UK as fees climb ever higher. You do regularly hear people say stuff like

    ‘Well, we can’t do much about [students doing XYZ], it’ll just end up in court if we try and say anything’.

    – Indeed, the need for exceptionally elaborate written rules is one of the things that I think puts Univs off trying to police student behaviour, even when the behaviour is a concern. I am, though, with Scurry that one should take the opportunity to challenge dubious behaviour straight off – and also that by NOT doing something you are missing a good chance to nip the behaviour in the bud. That was Mrs Dr Aust’s main point too – she says that, if you don’t challenge unacceptable behaviour in the medical students the first time it occurs, then you can’t apply the correction, and that by the time the students reach the 3rd year, or even the wards, it may be too late.

    @xtaldave

    That’s a depressing story. There is certainly a widespread feeling amongst scientists that those who finagle early (in their career) will also finagle later, and there are certainly many cases around that seem to back that up (e.g. this one, which has been widely reported in the UK).

    http://retractionwatch.wordpress.com/2011/07/14/jatinder-ahluwalia-out-at-university-of-east-london-report/#wpl-likebox

    Anyway,hope you didn’t waste too much time on it all, and that you at least get a paper at the end of it.

    Like most large Univs, the Univ of Gloomingham runs various workshops for graduate students around topics like research ethics, fraud and authorship – including with the occasional lectures given by yours truly. We do hammer away at the messsage that cheating is unacceptable. But policing it ‘at the grass roots’ is a lot harder – without hard evidence, there is always a fear you will end up in the law courts, see above. It is absolutely imperative that all PIs give out the message that data fiddling is NEVER acceptable – the best time and place to set the message is surely in the labs where people begin their research careers.

    @Dr Zorro

    Hmm. Mrs Dr Aust says she’s met such people in medicine too. Sadly, they do well across all walks of life. There is a running gag about Asperger-y scientists which runs:

    – if you prefer your microscope to people, and can’t do empathy, you may be a scientist.

    – if you prefer your microscope to people, and can’t do empathy, but can fake it with total sincerity, then you are probably a full Professor.

    @Hamlet’s Ghost

    Re i) There is a lot of it about these days, though perhaps it isn’t quite as new a phenomenon as one can sometimes find oneself thinking. Fake medical and dental degrees were all the rage in the late 19th and early 20th century. I believe the real Dr Crippen had a medical dfregree from a homeopathic medical school, for instance.

    On your second point, one does have a strong sense that the privileged have an ever grander sense of entitlement – see e.g. the German Minister (now ex) Herr zu Googleberg with his heavily lifted PhD, or the bankers looting the profits and then expecting the taxpayers to bail them out. And to see our own Eton-and-the-Bullingdon-Club political lot on TV after the recent riots stuck in my craw somewhat. I am certainly with Peter Oborne that the rot starts at the top, really.

    @Michael

    120/80 is only a particular example of a wrong/right answer, so I’m not sure what your objection is. The point is really that without making an actual successful MEASUREMENT, a student should not be taking a random punt – that is, pretending that they have an idea what the blood pressure is in the individual test subject when they necessarily have no idea at all. See Dr Phil’s recent comments.

    In the scenario given, the measurement of blood pressure comes from listening to what are called the Korotkoff sounds (the sounds made by turbulent blood flow in the brachial artery, heard through a stethoscope placed over the artery near the crook of the elbow) as the blood pressure cuff round the upper arm is gradually deflated. If there are audible sounds, but the student then gives the wrong answer, then they obviously need more practise at measuring blood pressure, but they haven’t actually (giving them the benefit of the doubt) simply made up a number. To take an example, if sounds are just audible at 110 mm Hg pressure, and then become inaudible at 70 mm Hg, the BP is 110/70. If the student says ‘100 over 80’, then they are showing they aren’t good enough at the skill, and wouldn’t get whatever marks are available specifically for getting a reasonably accurate value. Depending on how well they actually carry out the measurement, though (where they put the cuff, putting the stethoscope in the right place, letting the pressure down at a sensible rate), they might still get enough marks to pass.

    [The precise details of how the test is marked differ between medical schools, and also often between years of the course – for instance, in the final year you might have to get an accurate reading to pass at all, while in year one it might be enough to be close-ish].

    What I am saying is that that scenario I’ve just given is distinct from a student saying something if there were NO sounds audible at all. In the latter case, they could not possibly have the slightest idea what the subject’s BP is, and any number they then give is just something they have made up. It is this making it up that is unacceptable, and should arguably be penalised more than simply by not getting the marks available for a correct answer. What they should do is to say ‘I couldn’t get a reading, Ill try it again’.

    @Mick Tagg

    The feeling I have is that nowadays, once students are IN medical school in the UK, it is pretty difficult for them to get pitched out, at least for failing exams. People of Dr Grumble’s vintage tell me that, back in the 70s, it was widely understood that a third of those entering medical school would not make it to graduation, but I would be very surprised if the loss rate was that high now.

    Although most UK medical schools run a ‘one re-sit chance only’ system, one hears a lot of stories from around the medical schools about students having their fourth attempt at year one, or similar. One reason for this seems to be the mitigating circumstances system, where any sort of ‘special circumstances’relating to a failure gets you an extra go. Of course, often that is appropriate, but it seems to be being called into action more and more. Again, I suspect fear of litigation plays a part. Students tend to have powerful reasons for being in medical school; most are desperate not to get chucked out and will take any available steps to avoid it.

    @Stuart

    I heard someone refer to this rather snidely as the ‘pets win prizes’ problem – if you don’t get the sticker, then it didn’t happen, and so on. Of course, as crusty old scientists we probably think the point of getting medical students to do a bit of research is to see how research is done, understand that it underpins the knowledge base medicine depends on, and to see that it isn’t easy to do well. Publishing a paper is not the point, mainly because luck plays such a part in whether experiments work.

    Of course, this kind of ‘I must-get-a-gold-star-for-my-CV’business also turns up in science in the form of the modern ‘guest authorship’phenomenon. I’m reminded of David Colquhoun’s story about actually asking people being interviewed for lectureships about the papers on their ‘Five most important publications’ list, and how often they would then say ‘Err… well, I only did the immunostaining for that, I don’t know about the other stuff in the paper’.

  13. draust Says:

    And a response to Dr Phil and to Dave:

    @Dr Phil:

    Thanks for that. I think I recognise a lot of common ground, specifically in how difficult it is to pin down suspected finagling ‘beyond a legal doubt’ (as it now tends to have to be).

    I would agree that at some point in medical school the ‘gives BP answer with no reading’ should be an automatic fail – certainly in a final or nearly final year OSCE. I suppose the question would be whether it should be an instant fail in year 1 or 2, which may be before the students have seen a real patient. One can easily imagine marks schemes getting written (well meaning ones, actually) where you could pass the ‘task’ (station) without getting an accurate reading, e.g. by doing the procedure basically right – see my comment to Michael above. ‘Doing the procedure basically right’ could also include stuff like introducing yourself appropriately to the subject, getting consent, palpating the pulse at the wrist to get an approx systolic pressure, and so on. But I think it is clear that there ought to be some kind of add-on sanction for the ‘inventing’, even if (as Stephen Curry said) it is simply that the behavious is challenged and that the student is told firmly that is not OK, and why.

    PS I’m sure the student in the real ‘invented notes’ incident got reported to the Faculty, BTW – Whether it just stopped at the proverbial bollocking from an authority figure, or whether they might have ended up in front of a Medical School Committee, I can’t be sure. But it definitely triggered some kind of action, which I think everyone agreed was what was needed.

    I think that, at least where I work, the academic teaching staff (like me) would always report something like that, and that when we do it does get taken seriously – the more so since the advent of integrated courses, with students reaching the surgeries and hospitals early in their University career.

    One thing I have found sometimes does fall through the cracks is behaviour in lab classes – the technical staff in the practical labs are sometimes surprisingly reluctant to say anything when they are on the wrong end of unacceptable student behaviour. I can certainly recall a couple of incidents when I and others on the staff have had to persuade them to report what had happened – again on the basis that students (especially in the professional degrees) who think it is OK to mouth off at the practical class technicians need an attitude adjustment.

    On the more general issue of medical training, Mrs Dr A has sometimes been pretty scathing about competency-based assessment, which was just taking over when she was doing her last stint as a hospital speciality trainee. This was mostly on the grounds that it was so difficult to formally certify people as NOT competent that, in the end, everyone tended to be box-ticked as competent – including the trainees a lot of people had misgivings about.

    Mrs Dr A tends to the view that the older-style ‘recompete, recompete, and then recompete again’ system worked better as a method of weeding out those who either weren’t up to it or were somehow dodgy. This was mainly just because there were many more competitive hurdles to jump (and much less formal feedback), so many more chances ‘for the word to have gone out’ on the dangerous or unscrupulous.

    Of course, the flipside was that the old system clearly allowed people to be blackballed or blocked unfairly on grounds other than dodginess or lack of competence – notably not having the kind of ‘face that fits’. This would include the social patina that we hear so much about these days WRT public school and Oxbridge grads. But seeking to eliminate the opportunities for old-school-tie-ery (and other even less savoury forms of prejudice) does seem to have come at a price of it now being perhaps more difficult to do much about the problem cases. And of course one also hears people muttering that early form-and-computer-based selection, and run-through training, exacerbate the problem.

    @Dave:

    Yes, I’ve heard about these kind of ‘confidence-based assessment’ ideas. I actually know one academic that spent a lot of time researching it and pushing for it, and especially advocating it especially for medical student MCQ exams. However, it did’t seem to catch on even at his own institution.

    I think one perceived drawback might be the need for students to fill in two things per question = the A-E answer (or whatever) AND the ‘How sure are you?’statistic. Since you usually reckon on one MCQ question per minute in a medical student exam, there would probably be a long and involved debate (!) about whether you would have to have a lot less questions in the exam with confidence-based assessment.

    Negative marking of MCQs actually does introduce a little bit of a confidence-based aspect, since, if you are not sure of the answer, then you have to decide whether you have enough ‘hunch’ to make it worth taking an educated guess. I usually tell students that if they can definitely rule out TWO options (in a one from five MCQ) then it is certainly worth picking the best guess you can make from the other three in the negative marking scenario. In fact, the research is supposed to say that it is worth a guess even if you are only sure one of the answer options is wrong. The simple statistics there would tell you it is pretty equivocal, but the results are supposed to suggest that, if they students THINK can eliminate even one answer option, then they probably have enough ‘unknown knowledge’ to make a better than random guess – if you see what I mean.

    Of course, that is subtly different from a pure ‘how confident are you of your answer?’ setting, as the consequences of a wrong guess in an MCQ are not too dire even with negative marking. If it was something a bit more serious – like:, in the medical school context

    ‘How sure are you that this patient you are about to infuse with fluids is fluid-deprived and NOT fluid-overloaded?’

    – then the question of how sure you were would assume a much greater significance.

  14. drphilyerboots Says:

    Dr Aust,

    I have major doubts about testing competency withworkplaced based assessments. I think the old style system of repeated hurdles and recompetition did have several major flaws, and the changes of MMC were not entirely unessecary. The old system did have high specificity but low sensitivity, in that people who progressed were highly likely to be competent, but plenty of competent people did not progress. As well as the injustice, this was arguably inefficient of resources.

    The presumption of much medical education theory at both undergraduate and postgraduate level is that most are compent, usually at the 95% level. This underlies OSCEs, WBAs, ARCPs and MMC, and the remainder of the alphabet soup of medical education. This has shifted the burden of proof substantially, with the need for the assessors to prove incompetence rather than the candidate to prove competence. In the legalistic world of modern life this has to be bullet proof for the university/deanery to take action. In practice many doubtful candidates progress. We have moved to a system of high sensitivity and poor specificity. That is strong candidates are certain to progress, but plenty of weak candidates get through. I suspect that Mrs Aust would have progressed easily in the current system if she could get through the lottery of selection.

    I would argue that competence means a lot more than peak performance. To take the analogy of BP, we are not just interested in systolic, but also in mean pulse pressure and diastolic. The Australian Cricketer Shane Warne once stated “it’s not how you play on a good day that makes you a good player, it’s how you perform on a bad day”.

    At the Stafford hospital that recently was in scandal many would have been cured a timely fashion, even during their darkest days. With WBAs the candidate picks both their time and place of assessment, and usually the assessor. The analogy with Stafford is that they could choose to demonstrate competence by picking to be assessed on cherry picked results.

    This is drifting somewhat away from the integrity issue, and rapidly becoming an essay. It is an issue close to my heart and why I remain active in medical education politics, these things do need to be pointed out to the authorities.

    Dr Phil

  15. Stuart Says:

    Dr Aust may have spent much time and effort convincing the world that he is a grumpy old man, but I am currently at a piano summer school and one of my fellow students is, in real life, a medical student in Dr Aust’s University.

    It may come as a surprise that she has described Dr Aust (and here I quote) as ….”cute”.

  16. Dr Aust Says:

    @Stuart.

    Crikey… pretty sure it must be the first time someone has used that word about me in approximately four decades

    But anyway… I am happy to be described as pretty much anything other than ‘old’ (vanity … * Sigh *)… or than Jr Aust’s preferred adjectives for me, which are ‘smelly’ and ‘horrid’.

  17. eveningperson Says:

    “there is always a suspicion that people who are prepared to lie about one thing are more likely to be prepared to lie about other stuff too.”

    Based on some experience with known liars/fantasists, I now suspect that people who are successful initially in lying their way into or out of situations come to be unable to distinguish their lies from the truth. I wonder if there is any firm evidence of this?

  18. draust Says:

    @Eveningperson

    In response to your call for evidence one is tempted to say:

    “Pick any politician of your choice”

    – but I guess one should ask, and I imagine you meant, whether there is proper published evidence. It’s not really my field, but perhaps any psychologists reading could advise?

  19. Stuart Says:

    This issue is addressed in this excellent little book

    http://press.princeton.edu/titles/7929.html

  20. draust Says:

    Ah yes, Harry Frankfurt’s book On Bullshit, which I’ve read about (and watched a video of him talking about), but not read myself. Must put a copy on my Christmas list.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: