Clinical researchers and epidemiologists are very much concerned about publication bias. That means all factors causing less than the totality of performed experiments to be reported. A well-known bias, for example, is that positive results are much more likely to get published than negative results. If someone then tries to assess all the evidence for and against the benefit of a certain treatment, for example, the picture will be distorted. Therefore, it is important to publish results even when they are not spectacular.
In the laborative biosciences, the problem is far more pervasive than in clinical research. Therefore, I find it a bit surprising that people talk so little about it.
Sins of omission
When we write papers, the objective is to tell a good story. Usually, multiple types of laboratory methods and experimental systems are used in support of each other. Far are we from the studies of yore that investigated a single hypothesis with jut one method. In general, this is a positive development, of course, driven by ever-better methods to generate data easily.
Stories require sacrifices of the narrative. Sidetracks must be removed, darlings killed, and details fitted into the grand plot. Inevitably, data that are not interesting will be thrown out the window.
This is not to say that people deliberately withhold contradictory data, at least not very often. But nearly every experiment starts with a small pilot run, and if it turns out contradictory or confusing, it is so very easy to simply prioritise something else and then never publish it on the grounds that it was never conclusive. The final paper may report just half of the hypotheses that were addressed in the course of the study.
And in this manner, most papers are navigated through a sea of uncertainty, leaving dead hypotheses as corpses under the surface, invisible and unknown to all but the scientists who left them there. And that can be pruriginously annoying. Because then I have to do the experiments myself to find out, even if it is obvious that someone ought to have tried it before. Some experiments, such as optimisation protocols, are never reported simply because it’s boring for everyone except the ten people (sometimes including me) that have to struggle for days or weeks to get the procedure to work in our own laboratories.
Who is to blame?
The current system incentivises everybody quite heavily to publish very selected subsets of their data. Scientific journals want papers with strong evidence that points directly in one direction. Scientists’ grants and reputations are pegged to their success in publishing in the same journals.
In my last project, I did a small experiment which, if positive, would have warranted a bigger experiment to validate the positive finding. But it was negative. I did it three times, and it came out the same way, to I decided to throw it in the paper anyway. Then the next person who might want to do it doesn’t have to, and I haven’t hidden any data. I stuck it in the supplementary section: files that won’t even be printed but are accessible at the publisher’s website.
When we got the review comments they were generally positive, but one of the reviewers lamented, inevitably, that that particular experiment was too weak to prove anything. We considered taking it out. But I decided to argue the point instead, this time, and keep it.
I can’t shake the feeling that there has to be a better way for me as a scientist to make my data available to people!