Guidelines for Reviewers

Becoming a Better Reviewer of Academic Papers

By Richard D. Waters, Ph.D., U. of San Francisco

In the 2015-2016 membership survey, Public Relations Division (PRD) members expressed a strong desire quantitatively and qualitatively to see reviews improve for papers that are submitted to the open, student, and teaching research competitions. Based on the research chairs’ evaluation of reviewers’ comments and paper reviews, the PRD created a 2015-2016 goal of reducing the number of conference paper submissions that received only quantitative scores and no written comments about the paper and instituted a reviewer reward program to thank those reviewers that submitted their reviews in a timely manner and had substantial feedback.

To continue improving the overall review process for authors, the following guide was prepared to help conference and journal reviewers improve their critiques of academic papers. To that end, it is important to stress that peer review is not perfect. Flawed studies are accepted for presentation and publication, and peer reviewers may miss critical problems with research designs and evaluation of data. These may be the result of reviewers not having the time, nor the inclination, to dig deeply into a paper’s research methods and data collection instruments, study assumptions, or the supporting materials and literature to find errors or flaws in a paper.

Even though peer review is not perfect, the research co-chairs rely heavily on the evaluations and advice provided by peer reviewers. Decisions around who should review papers for the AEJMC conference and which papers should be accepted for presentation are made after a significant amount of time is spent matching the right reviewers to appropriate topics and reading their quantitative and qualitative evaluations in comparison to those received by other papers.

What skills are useful in reviewing papers for the PRD research competition?

  1. Reveal your conflicts and deficiencies immediately. If you know who the author of a
    paper is due to previous/current work collaborations or working with the author at the same institution and you’ve discussed or edited the work, then please let the research chairs know immediately so that the paper can be assigned to someone else for review. Additionally, if you have a strong intellectual conflict with the paper topic, you should let the research chairs know immediately so the paper can be judged fairly on its own merits by another reviewer. Finally, if you are asked to review a paper that falls outside of your methodological or theoretical expertise, let the research chairs know so that they can find someone else who is in a better position to provide an in-depth and informative evaluation.
  2. Evaluate the research paper you have been asked to review. This may seem logical, but it is important to keep in mind that you should review what has been presented and not what you think would make for a better research project. To that end, you should determine whether the hypotheses, research questions, or theoretical models make sense given the existing body of knowledge on the topic. You should also critique what was done (or not done) in the research design. You should also evaluate whether the conclusions are appropriate and supported by the data—quantitative and/or qualitative. . Your comments should be based on what you see and not what you would have done or what should have been done instead. These critiques can be included in a review secondarily when pointing out issues with what was presented.
  3. Do not be the paper’s editor. You’ve been asked to review the paper for its academic merit. You should strive to avoid providing a lengthy list of typographical errors, formatting mistakes, and pointing out missing citations and style errors. Unless these errors are so numerous that they detract from your ability to read the paper, these errors typically do not play a role in the first stage of review because these issues will be addressed later on in the academic publishing process with revisions and copyediting. Again, your focus should be on the academic merit of the paper!
  4. Be timely. With AEJMC’s competition, the deadline for submission has historically been April 1, and reviewers are asked to complete their reviews by the beginning of May. AEJMC imposes mid-May deadlines on all divisions and interest groups for having their conference sessions finalized, top paper awards selected, and conference program material submitted to the headquarters. Therefore, it is extremely important to get reviews submitted before May so that the research chairs can review the evaluations of more than 125 papers submitted across the open, student, and teaching paper competitions.
  5. Be brief, but not too brief! The AllAcademic system that AEJMC uses to manage the research competition requires that “none” be written in the comments box at the very least. But, “none” isn’t really helpful for authors of papers who are trying to determine why their papers received certain scores across the different quantitative measures. We understand reviewers’ time is precious, and we don’t need an evaluation that stretches on for pages. Stay focused and let the authors know what you think the central contributions of the paper are and what important problems exist. This could easily be accomplished with just a few paragraphs that would significantly help authors understand why their paper was or was not accepted for presentation and what they can do to improve it.
  6. Refrain from judging whether the paper is suitable for publication. While it may seem that you are doing the author a favor by offering advice on publication, it really is not appropriate for a conference review. Provide your critique of the paper so that the authors can make revisions to accepted papers before the final paper upload deadline in July. The authors can then move into the conference and have conversations with panel moderators and high density and scholar-to-scholar session discussants about their publishing possibilities.

Writing up Your Conference Paper Review

  1. Take time to read the paper carefully rather than skimming it casually while carrying
    out other work. If you’ve agreed to serve as a paper reviewer, you’ve really agreed to give a thoughtful review of the project to the authors. To do this, you have to read the paper and pay attention to the development of the paper’s arguments, its research design, presentation of results, and a discussion of the meaning of the work theoretically and practically. While reviewers may grudgingly see conference paper reviewing as a duty to be fulfilled, it can be critical for other colleagues’ jobs that are at stake when it comes to building their research agenda and developing their research stream. To support our colleagues and the discipline, we need to take time to review conference papers and provide meaningful feedback.
  2. Write a brief summary of the article and its contribution first in your review. If you approach the paper review with the goal of helping the author understand your quantitative review alongside your comments, start out by writing a paragraph about what the authors did well with the paper. Try to summarize the article as best as you can in three to four sentences. If you think favorably of the paper and believe it should be accepted for presentation, write an even longer summary and highlight its strengths.
  3. Write out the major criticisms of the paper after you’ve written your favorable comments. When writing out your criticisms, it helps the author by starting out with larger issues and ending with minor details. Here are some major areas of criticism that may be considered.
    – Is the paper well-organized?
    – Does the paper contain the sections you would expect (Introduction, Literature Review,
    – Method, Results, Discussion, Conclusion)?
    – Are the sections well-developed?
    – Is the literature summarized concisely and accurately?
    – Does the author answer the questions that were addressed by the hypotheses or
    research questions?
    – Is the research method and design clearly explained?
    – Does the theory connect to the data?
    – Is the article well-written and easy to understand?
    – Are you convinced by the author’s results? Why or why not?
  4. Write out minor criticisms of the article. While you are not serving as the paper’s editor (see skill #3 above), it is fine to point out if citations are missing, tables are missing or are mislabeled, or if expected sections such as future research or limitations aren’t included. Leave the spelling, grammar, and style errors to the copyeditors.
  5. Finally, review your review to make sure that it makes sense and that you are communicating your critiques and suggestions in as helpful a way as possible. Strive to write your comments in a manner that you would like your own work to be addressed. There is no reason for the tone of a review to come across as mean-spirited. One way to check the review to determine its appropriateness and its helpfulness is to compare the written comments against the quantitative scores. If a paper receives all 4s and 5s but you recommend rejecting it, then something is off. If a paper receives all 2s and 3s but you recommend accepting it, then something is off. Help everyone out by checking to make sure your review makes sense quantitatively and qualitatively!