Bias in the Peer-reviewed Literature, and Crossing the Science/Policy Divide
Reason for rejection
Description
Methods/quality
Unsound methodology, poor write-up
Interest
Insufficient interest to the journal, because results were too specific (not generalisable) or not novel enough
Ideological bias
Paper rejected because reviewers/editor thought findings would affect the MR cause
Personal bias
Paper rejected because of reviewer’s/editor’s personal agenda against the author or competing research programme
Although scientists were chosen for this questionnaire survey because of their publication count, it was not possible to know whether the views of this sample of scientists were representative of the whole scientific community that has studied MRs, so we cannot quantify the extent of any ‘bias’ occurring in the MR scientific literature.
4.2.2 Results and Discussion
One hundred and four scientists out of 200 responded to the survey; the largest number (42 %) came from the USA, with the remainder from Spain (9 %), Australia (8 %), Canada (8 %), New Zealand (8 %), UK (8 %), and Italy (7 %). Twelve scientists did not complete the questionnaire but did express their viewpoints, so in total 92 fully completed questionnaires were collected. Of these, 50 scientists (54 %) said that they had experience of papers on MRs being rejected.1
Journal Editor and Reviewer Bias
Of those scientists who gave a reason for their paper being rejected, most felt that the reasons were methodological or lack of interest rather than bias (see Fig. 4.1). Nevertheless, 11 % believed that their paper had been rejected for ideological (i.e. normative) reasons.
Fig. 4.1
Reasons given by author for their paper’s rejection
1.
Paper rejected for ideological reasons
Whilst Fig. 4.2 should not be over-interpreted due to the very small sample size (n = 10) and statistical non-significance of the ten scientists who thought their paper had been rejected because of ideological bias, seven suggested that this was due to the bias of the reviewer/editor against a negative/non-significant result, and three suggested that it was the result of reviewer’s/editor’s bias against a positive result (see Fig. 4.2). Both sets perceived biases—towards and against MRs—providing some impressionistic evidence for the view of some authors that the scientific community has become politicised with regards to the use of the MR as a management tool (Agardy et al. 2003).
Fig. 4.2
Scientists’ perceived political bias amongst editors for their paper being rejected
Although the quotations pasted below are anecdotal, they are used as qualitative data to illustrate the fact that some scientists surveyed thought they had experienced ideological bias from peer-reviewers or editors. One scientist said:
We were saying MPAs did not prevent declines in coral cover and associated declines in fish richness and abundance. One of the reviews seemed to circulate around the fact that we could not/should not be publishing bad news stories about MPAs.
Another said:
Well, I do think that one reviewer had a publication bias against papers showing that MPAs don’t work…s/he made a remark along the lines that s/he did not feel a paper showing that MPAs don’t work should be published in a high-profile journal.
– though the same scientist did suggest that there may also have been some legitimate methodological reasons why the paper was rejected, and that it was only one reviewer who displayed a pro-MPA bias:
To be fair, there were some issues that we addressed before re-submitting to the next journal and quite frankly, I don’t think of this as a systemic bias, but rather one poor reviewer and that journal typically rejects anything that does not have a unanimous consensus among reviewers.
Another scientist, whose theoretical work challenged some of the assumptions made about the beneficial impact of MRs on fisheries outside the protected area (the spillover effect), expressed his frustration at having his manuscript rejected by three different conservation/ecology journals:
For our theoretical paper on spillover these reviews completely missed the point of our manuscript and provided incorrect technical critique as a justification for why they were not considering our manuscript…I am sure the associated editors fully believe that their critiques are correct, but it seems they have read our theoretical paper with their own preconceptions at the front of their mind and no matter how explicit we are, we cannot break them down. I also get the feeling that because we are challenging beliefs about MPAs that may undermine some of the previous evidence about their benefits (i.e. spillover)…it is especially hard to communicate our point.
Another respondent hinted that a bias against articles which find that MRs do not generate a spillover effect existed in conservation journals which were not interested in whether or not MRs benefitted fisheries:
MPAs have been first implemented with a conservation focus rather than a fisheries focus. 2 Consequently, the MPA literature has had a strong conservation slant for some time. (see Fig. 4.3 and Table 4.2 )
Fig. 4.3
The total number of ecological MPA studies by general journal type
Table 4.2
The top 20 journals where ecological studies of MPAs have been published. Each journal is labelled according to its general audience: E Ecology, C Conservation, and F Fisheries
Journal
Type
Publications
Impact factor (2011)
Marine Ecology Progress Series
E
92
2.7
Ecological Applications
C
52
5.1
Aquat Conserv: Marine Freshwater Ecosystems
C
44
1.9
Biological Conservation
C
41
4.1
ICES Journal of Marine Science
C
40
2
Can Journal of Fisheries and Aquatic Sciences
F
39
2.2
Bulletin of Marine Science
E
39
1.1
Conservation Biology
C
34
4.7
Fisheries Research
F
27
1.6
Environmental Conservation
C
22
1.9
Marine and Freshwater Research
E
18
1.6
Coral Reefs
E
17
3.9
Biologia Marina Mediterranea
E
13
1.4
Ecology Letters
E
13
17.6
Marine Biology
E
13
2.3
Fishery Bulletin
F
12
1.1
Journal of Exp Marine Biology and Ecology
E
12
1.9
Journal of Applied Ecology
E
10
5
Reviews in Fish Biology and Fisheries
F
10
2.5
African Journal of Marine Science
E
10
1
One respondent took a benign view of the whole controversy, arguing from a pluralistic perspective that one bias balanced out another:
I don’t think there is this bias. While each individual carries a bias, this is balanced by others with opposite bias. On average, there is no or very little bias in peer review, taken as a whole. That’s why, if a paper is rejected in one case, another journal with different reviewers will likely respond differently, as long as the underlying science etc, is up to scratch.
This argument about balance receives some support from the fact that one respondent detected an anti-MR bias in one prestigious journal:
A very clear anti-MPA bias on the part of the editors (one of whom publishes anti-MPA papers).
2.
Paper rejected because of insufficient interest
The journal an author initially submits to is likely to have a relatively high impact factor and consequently high rates of rejection, with even methodologically sound papers being turned down if they were not of sufficient interest (see Table 4.3).
Table 4.3
Description of interest criteria (derived from questionnaire responses) that may affect the likelihood of a study being published in a top journal