Wednesday, 23 June 2010

Is mental health the only field that suffers from publication bias?

The anwser to that is "no" in my opinion. I think many sciences may
suffer from the problem of publication bias, i.e. where negative results
or results that don't support the underlying point of the research. This
happens for many reasons and is a trait I've noticed in a lot of people
and a failing of mine too. The reasons this happens in research and in
organisations is as complex and yet as simple as the human problem. The
simplicity is it has the same reasons as why people lie. And it is as
equally complex.

Mental health has the saving grace of acknowledging it's existence. The
funnel plot is a clever technique where data from lots of studies is
mapped on a graph with sample size on one axis and effect size (or
whatever variable is being measured) is the other. The shape of the plot
should look like a funnel shape and any biases caused by witholding of
data shows up as a blank area in the plot usually found near the wide
end of the funnel. It works on a principle which might relate to
regression to the mean. Studies with a small sample size have larger
errors and these will fall either side of the mean. Larger studies will
be closer to the mean. So the smaller sized studies should have positive
and negative results either side of the average (represented by the
middle of the funnel or the sample size axis). If there's no publication
bias then the spread would be symmetrical either side of the mean. The
smaller studies can have results far away from the mean while the larger
studies will vary less and this creates the funnel shape.

A recent funnel plot from a meta-analysis of around 1000 psycholgoical
therapies studies that was published earlier this year in the British
Journal of Psychiatry showed that the effect size for these studies was
reduced by about a third because of the publication bias. A metanalysis
by Kirsch in 2007 publishing in PloS Medicine that included unpublished
data showed SSRI antidepressants to be as effective as placebo for all
but severe depression. There are other examples in psychiatric research.

Psychotherapy research doesn't have the same motivations for publication
bias, i.e. they are not motivated directly by profit. Professional
pride, enthusiasm for a new treatment, subjective validation skewing
perception of results and other reasons can influence researchers to not
publish results. These qualities are found in many other fields. I
wonder how often funnel plots and other techniques used to identify
other forms of bias are used in other disciplines?

No comments:

Post a Comment

Blog Archive

About Me

We It comes in part from an appreciation that no one can truly sign their own work. Everything is many influences coming together to the one moment where a work exists. The other is a begrudging acceptance that my work was never my own. There is another consciousness or non-corporeal entity that helps and harms me in everything I do. I am not I because of this force or entity. I am "we"