Over the last couple of months the question of how to write a peer review came up quite a few times, and a couple of my colleagues even asked me directly to help them prepare for their first peer reviews. Preparing solid, critical peer review is an essential component of being a good citizen in the scientific community. I generally do about two for every paper I submit. I thought it might be helpful to provide a brief overview of how I personally prepare a peer review, primarily for marine science and conservation journals geared towards population genetic studies. I’d like to think that this advice is broadly applicable to any scientific peer review.
Step 1. Read the paper. It might seem silly to start with this but a lot of people dive into their peer reviews before they’ve even read the submitted paper in its entirety. You start thinking about how you’ll review it as soon as you get a request from the editor with the title and authors. When you get a paper to review, you immediately start reading it with a critical eye. Think about when you read a paper for pleasure or because you are interested in the content. You’re generally not looking for the fine details or nitpicking word choice, you’re looking for the ideas in the paper. You’re trying to understand what the paper is about and you’re trying to understand what the authors concluded with paper. So before you even begin with your peer review just read the paper as if it were any of a dozen other scientific papers that slide across your desk every week..
Step 2. Write down what you think the paper is about. Do this in broad terms, not so much focused on the methodology but rather the ideas behind the methodology, the motivation for the study, the questions the authors want to answer. Use this as a framework to hang the rest of your review on because you’re not just looking for technical precision but to make sure that the study itself is relevant to the broader themes of the paper.
Step 3. Sleep on it. Seriously. Put the paper aside and go on with your day. Don’t think about it too much until the next morning. This means that you actually have to plan ahead when you’re preparing a peer review. You can’t put it off until the last minute. You need to leave time to properly digest the study. Remember this paper probably represents years of someone’s work summarized in a couple dozen pages. You owe them at least a few days thought.
Step 4. Read the methods. Read the methods critically. You are now looking for the specific details of how they want to answer the questions they are asking. You’re getting into the details of their methodology. Did they use the appropriate analyses? Are there other analyses that would have done a better job? Did they sample enough to answer the question? Do they have enough data to perform the analyses they want to perform?
It’s important when going through the methods that you remember that science is hard. It’s hard for you. It’s hard for them. It’s hard for everyone. There is no study in the world that wouldn’t benefit from a few more samples, that a couple more data points wouldn’t help. Data is not the easiest thing in the world to get, so remember when writing your review that perhaps there are practical reasons regarding why they didn’t get 450 samples instead of 350. The question is not “could the study have benefited from more samples?” The study can always benefit from more samples. The question is “do they have enough samples to answer the question?”
Step 5. Analyze the results. Now is the time to go through the data with a fine tooth comb. Read the results make sure the results match their methods. Make sure the answers they’ve got can be gained with the methodology they use. Then, most importantly, and this is the most time consuming part of a peer review, reanalyze whatever data that you can get your hands on. If they provided supplemental raw datafiles, use them. If they provided GenBank accession numbers, look them up. If they generated phylogenetic trees and you have access to the DNA barcode sequences, rerun that data that.
If a result looks fishy, I’ve gone back to the editor and requested they ask for raw data files so that I can verify those results.
Now, you’re not regenerating the data from scratch, and you’re not redoing every single analysis (some of those could take months). You have to assume that the process that generated the raw data is valid. When there are analyses that you can do in a reasonable amount of time to double check their work do it. I never sign off on a paper that I haven’t at least spot-checked a few of the results to make sure they’re reproducible. This is your quality control.
Step 6. Read the discussion. Read it critically but remember, the discussion is the authors’ interpretation of their results. As long as it’s supported by their data and supported by the literature it’s okay if you don’t necessarily agree 100% with their conclusions. If you don’t like a discussion point, but it isn’t directly contradicted by their data or by the relevant literature leave it be. This is also a place to push back against all too frequent tendency for authors to over interpret their data and try to draw a broader conclusions than the data supports. As a reviewer, part of your job is to reign that in.
Step 7. Write up your review. I generally like to do this in paragraph form with a few paragraphs for each section of the paper. If there is a specific issues with a table or a figure I will mention that. If there are specific issues with particular lines I’ll mention a line number, but I generally don’t like writing up a bulleted list. I feel like that does not provide context to the authors.
The one thing we haven’t done yet is talk about language, typos, style, that kind of thing. As a reviewer, that’s not your duty. The editor’s job is to make sure that the paper is typographically correct. If the language is bad to a point that you can’t understand particular passages then yes absolutely mention that. If you’re feeling particularly charitable and you want to provide line edits, that is perfectly fine, but that is not and should not be an expectation of the reviewer. Your review should be as succinct as necessary and under no circumstances should your review be longer than the actual paper.
Step 8. Write up your notes to the editor. If you think there are papers that should be added to this one, and you can always make a case for more literature, this is where you do it. It’s better to have the editor recommend specific new citations directly to the author, particularly if you’re recommending your own papers. Give the editor of the option of deciding to pass on your recommendation on or not. This frees you from the conflict of interest of directly telling an author you are reviewing to reward you in the form of citing your work. Of course, you’re reviewing this paper because you’re an expert. Some of your papers should be relevant to the study.
Step 9. Yell at your computer because the review submission portal that your journal uses is terrible. Seriously they’re all terrible. Plan to lose an hour of your day just trying to get the work you’ve already done submitted to the journal that wants it.
Step 10. Archive all your documents related to this review in a secure place. Peer reviews are technically confidential unless the journal’s policy is to make them open. Regardless, until a decision has been made on this paper you shouldn’t be sharing this information with anyone. On occasion PIs will invite their graduate students or mentees to participate in a review and that’s okay, but generally speaking you should either destroy these documents when the review is submitted (the journal will retain copies, anyway) or store them in a secure file format. I generally like to shred anything as I printed out after I’m finished with a review.
So that’s how I write and prepare a peer review. In general, it usually takes about 3 and a half days of work. If the data requires more analysis, it can take up to a week. Peer reviews are among the most important contributions you can make to the scientific community, so being thorough and deliberate with them is an essential duty. If you don’t have the time, decline to review. No one is served by a hasty, rushed peer review.
Again these are guidelines that I use and that work for me. I’m sure you can find a variety of other people with different approaches to peer review. If you’d like to share your own perspective on peer reviews, please do so in the comments below.