What to consider as reviewer before dooming a paper

Question

I am conflicted about what should influence my recommendation as reviewer.

A few times, I've been in a situation similar to the current one. Consider a paper that, after receiving a major revision, has been resubmitted, and I am doing a second-round review. Or alternatively, think of a paper currently in the rebuttal phase of a peer-reviewed conference. After the initial round of reviews the situation is the following:

• 2-3 reviewers are somewhat positive about the paper, but not overwhelmingly positive. Their reviews have reasonable quality, but do not go into depth.
• (At most) one reviewer is negative about the paper and highlights one or more considerable but not necessarily critical weaknesses (e.g. correct but outdated methodology, imperfect reproducibility). The review is quite detailed and it is clear they know what they are talking about.
• I am reserved about the overall idea and have identified some shortcomings that could be improved. The revision provided by the authors addresses some of the shortcomings, but not all. I do not feel that the remaining issues are huge problems, but neither are they negligible. Quality-wise, I believe the paper could be accepted, but I would prefer a rejection.
• Experience tells me it is likely that if I raise my rating/recommendation, the paper will be accepted, but if I stick with my rating, then the paper is doomed ( it's a highly competitive venue).

No further round of reviewing is permitted, and the next decision is final.

As stated, I am reserved about the paper's idea and would prefer if the paper is rejected. However, this is science, and it should not be about my "feelings". The paper does not have a major technical flaw and fits to the venue. It could be accepted. But then again, many other reviewers and editors sometimes follow their feeling of gut. Peer-review is inherently subjective. Maybe I should just follow my feelings as well. However, I can also empathize with the author side: More than once have I or my colleagues been in a similar situation. This thought makes me want to help the authors and raise my rating. But again, this is science, and such factors should not matter – at least I think so.

You will probably realize that I am conflicted.

Question: Which of the above factors should influence my decision? Are there perhaps other, more important factors that I should consider?

---

I am aware that the final decision is made by the editor/program chair, and not by me. However, I am still concern about the impact my recommendation might have.

Answer 1

Some random points to consider:

• Does this science deserve to be published, and is this venue appropriate? By "deserve" I mean was the study appropriately conducted. Are the results meaningful to the community?
• Not every paper has to conclude every imaginable question one could have on that topic. Reviewers often lose focus and expect a career's worth of work out of every manuscript. If it answers some questions, and opens up new ones that inspire further research, then that is great.
• Do not fall into the role of gatekeeper but stay as a peer reviewer. You know the difficulties involved with research and that not everything is perfect. Be a peer, approach it in good faith, and then decide if the work lives up to the claims and the venue. If you see other reviews, were they done in good faith?
• All things being equal, if there are no major technical flaws or misrepresentations, does the field benefit more from this article getting published in that venue with its respective visibility, or if this work ends up in a lower tier (but maybe more specialized) journal?

Answer 2

You should consider whether you think the paper itself should be rejected.

If that is from a gut feeling then applying a lower confidence to your review seems appropriate. But tactical considerations on how others rated the paper, or how your rating relates to others, or what the authors deserve, or similar are not the point of a scientific review.