Nonblinded studies generate observer bias
The effect of a new drug or method is assessed too positively if the doctors who assess the patients’ course know in advance who received the new experimental treatment and who received the control treatment.
When the effect of a new surgical procedure or a new drug needs to be assessed, it’s done in clinical trials.
Here patients are divided into groups where some are treated with the new experimental method and others with the old control treatment. This enables the scientists to determine which of the two methods achieves the best results and the fewest side effects.
There are various ways of assessing the outcome. And there has long been broad agreement that the most reliable assessment comes from so-called blinded assessments.
In this context, ‘blinded’ means that the assessors don’t know whether they’re looking at data from a patient who has received the new or the old treatment.
For many years it has been assumed that if the assessor knew whether the patient had received new or old treatment, it would have an effect on the assessment. However, it has so far not been possible to find clear evidence of this observer bias. So up to now there has been a difference in what people believed to be the case and what they could prove.
Asbjørn Hróbjartsson
“For many years it has been assumed that if the assessor knew whether the patient had received new or old treatment, it would have an effect on the assessment. However, it has so far not been possible to find clear evidence of this observer bias. So up to now there has been a difference in what people believed to be the case and what they could prove,” says Asbjørn Hróbjartsson, a senior researcher at the Nordic Cochrane Centre in Copenhagen, Denmark.
Greatly biased assessments
He and his colleagues are the first scientists to prove that the assessment of a new treatment is significantly more positive if the study is nonblinded.
”We observed a general tendency for nonblinded assessors to provide more optimistic estimates of the effect. They were also more positive when assessing a new treatment against the old one. So their assumptions are being confirmed,” says the researcher.
“What’s new about this is that we observed a pretty strong effect. And it also turns out that it varies quite considerably. In some studies there is no bias, while others are greatly biased – so much so that even after calculating a weighted average, there is still a considerable bias.”
We observed a general tendency for nonblinded assessors to provide more optimistic estimates of the effect. They were also more positive when assessing a new treatment against the old one. So their assumptions are being confirmed.
Asbjørn Hróbjartsson
The researchers do not, however, have an explanation of the great variation in the bias.
“We did not observe any patterns, but that doesn’t mean there are no patterns to be found. We have reason to believe that for instance doctors who wish to assess the effect of their own treatment or their own surgical skills will find it more difficult to be neutral – but we cannot prove that in our data.”
A clearer experimental design
The study from the Cochrane researchers is a meta-analysis – a systematic review in which they looked at 16 randomised clinical trials and compared them using advanced statistics.
They did not compare the exact differences in old and new surgical procedures in these studies; rather, they looked at differences in the way the interventions were assessed by blinded and nonblinded assessors, respectively.
The researchers found bias is studies with nonblinded assessments – something that other similar studies have failed to do – since they picked out their studies for their meta-analysis very carefully.
There is a tendency to compare apples and oranges is such studies, and that makes it hard to determine with any certainty whether or not the bias is caused by nonblinded assessments.
”The problem with the previous studies is that they have compared study A, which was blinded and may have been conducted in Denmark, with study B, which was nonblinded and may have been conducted in the US. Meanwhile patients, practitioners and the assessors may well all have been blinded, and this is all blended together in the previous studies,” says Hróbjartsson.
“We have instead looked at studies that contain a blinded and a nonblinded assessment. This means that we can make clearer comparisons because the studies themselves, which undergo both blinded and nonblinded assessments, are identical. This enables us to rule out any alternative explanations.”
Blinded studies are the way to go
There is only one obvious way of avoiding biased assessments: by using blinded assessments.
”When you set out to conduct a study, you find yourself in a reality where you need funding, and that can make it tempting to cut corners and use nonblinded assessments because that’s the cheaper option,” says the researcher.
“Our study is useful in this context because we now have evidence that nonblinded assessments generate biased results. This highlights the importance of not making compromises in the experimental design.”
Ultimate truths are rare in science, so we cannot say on the basis of this new meta-analysis that all studies with nonblinded assessments will be biased. In fact, the analysis shows that some studies are not biased at all – but that a vast majority of them do.
“I would say that our findings are well-founded, based on many studies and they cover many different areas.”
------------------------------
Read the Danish version of this article at videnskab.dk
Translated by: Dann Vinther
