JUDGING THE RESEARCH PAPER COMPETITION FAIRLY
Suggestions from The AEJMC Elected Standing Committee on Research

I. INTRODUCTION
The AEJMC Standing Committee on Research often gets requests from divisions and interest groups for recommendations regarding research paper judging. The Committee makes several assumptions about the desired outcomes of research paper presentations at conferences. These include:

* Research papers should represent good scholarship; “bad” papers should not be accepted

* Research paper programs should be a forum for new work, new ideas, and new approaches. Groups should welcome controversial ideas and approaches.

* Young scholars and students should be encouraged to present their original work.

* Research paper judging and programs should reflect the diversity of the Association.

The Standing Committee on Research believes that group leadership and research paper competition chairs should run paper competitions that not only are fair, but are perceived to be fair. To that end the Committee has several suggestions that we believe will increase perceptions of the fairness of the process. AEJMC groups run their own competitions in accordance with the Council of Divisions’ ”Uniform Paper Call” guidelines. Groups are free to accept or reject any of the suggestions offered here, but research chairs should recognize that these suggestions are the result of many collective years of experience in paper judging across AEJMC groups and are endorsed by the Standing Committee on Research.

II. FAIRNESS IN THE CALL FOR PAPERS
The Council of Divisions’ “Uniform Paper Call” (UPC) provides 15 points about research paper competition procedures and deadlines. The following points are additional suggestions from the Standing Committee on Research:

1. Multiple submissions. Guideline 7 of the UPC requires that papers (a) are not already under review for other national conferences; (b) can only be submitted to one AEJMC group for evaluation, and (c) should not have been presented to other national conferences or published in scholarly or trade journal prior to presentation at the AEJMC conference. In addition, some groups limit the number of submissions to their group by an author and some groups allow submission of papers presented at smaller regional conferences. Any limits of these types should be made clear to presenters.

2. Student papers. Guideline 8 of the UPC suggests that student papers should compete on an equal footing in an open paper competition. Thus, several groups do not hold separate student competitions but instead draw their top student papers from the open paper competition. Groups taking this approach should remind student submitters to add “Graduate Student” to their paper’s title page, unless the paper is co-authored with a faculty member.

3. Presentation requirement. Guideline 9 of the UPC requires at least one author of a faculty paper to attend the conference and present the paper. Authors should be reminded of this requirement in their acceptance letter and be asked to immediately notify the research chair if no faculty author can present so that other quality papers can be programmed.

4. Recognition. The Standing Committee on Research recommends that groups develop some means of recognizing top papers for both faculty and students. Often this recognition represents the top three faculty and student papers from the open competition. If awards are given only through a special competition research chairs should clearly state if papers are eligible for presentation in the open competition if they do not win an award.

III. SELECTING PAPER COMPETITION JUDGES FAIRLY
Selecting paper competition judges is one of the most important tasks of a paper chair. As with paper call, groups vary a great deal in how they select judges, but in the interest of fairness and perceptions of fairness, the division and the competition chair should consider the following guidelines.

1. Ideally, paper judges should have expertise in the method/theory of the papers they are judging. Paper competition chairs should determine judges’ areas of expertise and match judges with papers as much as possible. Keep files for subsequent paper competition chairs so that those judges who are particularly effective can be used again.

2. Have both senior and junior faculty as judges. Generally don’t use graduate students as judges, unless the graduate student has a unique area of expertise necessary to judge a research paper.

3. Put a reasonable limit on the number of papers each judge must read. Generally this should be about 3 to 5 paper per judge.

4. Judges should represent different perspectives in the group, including different universities. Avoid using judges primarily from the competition chair’s school. Make sure the process includes judges from diverse backgrounds.

5. Avoid real (and appearances of) conflicts of interest when selecting judges. Judges should be people who have not submitted a paper to the group’s competition, if at all possible. Judges should be asked to report any conflict of interest such as papers from those with whom they have recently co-authored, mentioned, or worked. Judges should not be assigned to review papers from others in their own institution. A paper competition chair should not submit a paper to his/her own group. If the paper competition chair has a conflict of interest (real or one that others might perceive) for the top paper(s) award competition, other members of the group would make the top paper(s) selection.

IV. RUNNING A FAIR COMPETITION
1. Each paper should be reviewed by at least two judges; preferably three. If there is a great deal of variance in the evaluations, another judge should be sought. If a judge is not performing responsibly or does not understand a manuscript, then the judge should be replaced.

2. Maintain anonymity of the judges.

3. Require reviewers to provide written comments on papers that they judge to be inadequate for presentation.

4. A paper competition chair serves as the final arbiter. It’s particularly important not to reject controversial work automatically. The chair is in a unique position to observe judges’ lack of consensus on controversial scholarship. Thus, the chair should read all papers where there is variance in the judges’ evaluations, and use his/her own judgment as to whether to accept or reject the papers.

5. If numerical scoring is used, be sure it is fair. Most groups use some form of numerical scoring for paper judges. This is not a problem as long as research paper chairs recognize that judges naturally will use different scoring “systems” for good and bad papers and that the quality of the particular set of papers received by a judge may bias the numerical evaluations. Thus, you cannot simply sum the numerical scores and use the summed numbers to make judgments of quality. The best solution is to use standardized scoring. This is not hard to do, and it helps to compensate for a judge’s idiosyncratic scoring tendencies.

6. If the group has a theme, selection should not be biased by the theme. All paper selections should adhere to the same standard of quality.

V. JUDGING STUDENT PAPER COMPETITIONS FAIRLY
1. Groups should decide whether to ignore the student/faculty status of authors for purposes of accepting papers. Some groups keep all the papers together and only make distinctions for awards, while other groups judge student papers as a separate competition.

2. Competition for student papers should be rigorous. In some groups where student papers are judged without reference to status, student papers fare as well or better than papers from regular faculty.

3. Student members deserve a fair opportunity to present scholarly work. Groups should not unfairly restrict the number of student papers accepted. Student papers should not be relegated to an “all-student” session but should be integrated into the groups’ sessions. Although AEJMC provides complimentary conference registration to three student authors in each group, a group need not limit student paper acceptances to three.

4. If a group limits the number of student authors on a paper, then the same rules should apply to the number of faculty authors. Some groups limit the number of student authors to three; this is unfair unless the same limits apply to the number of faculty authors.

VI. COMMUNICATING WITH AUTHORS
1. Authors should receive timely notification of acceptance or rejection of their papers. UPC guideline 10 specifies the date by which authors should be notified. Many schools provide little or nothing in travel funding, so faculty and students have to do considerable budget planning to attend a conference.

2. Acceptance notices should be specific about the nature of the session. Authors should be told how long they have to present their work, how to order audio or video equipment, when to send their paper to the discussant, and who the other presenters will be in the session. An example handout of information for presenters, moderators and discussants is available from AEJMC.

3. The author acceptance letter should include a summary of papers submitted and accepted in the group’s competition. This information helps the author see where his/her paper stands with the others submitted.

VII. THE PAPER SESSION
1. Paper sessions should allow enough time for presenters and discussants. Generally three or four papers per session are the limit if there is a discussant. A sample handout of information for presenters, moderators and discussants is available from AEJMC. Groups should tailor such a handout to their own needs.

2. Discussants should be eminently familiar with the work they are discussing. Choose discussants that are known to be effective oral communicators.

3. Diversity should be reflected by the choice of moderators and discussants. Also, make sure that the choice of discussants is not biased toward one perspective or one school.

4. Research chairs should make sure that promised audio/video equipment is available.

5. Moderators should treat all presenters fairly, including equivalent time limits. It is not fair for those who present at the end of the session to have to give up part of their time because early presenters went over the time limit. Again, information about time limits should be provided to presenters, moderators and discussants in advance.

IX. THE SCHOLAR-TO-SCHOLAR SESSION
1. Groups may employ different criteria in assigning papers to a scholar-to-scholar session (e.g. amenability to display, author request). However, papers presented in a scholar-to-scholar session should conform to the same standards as papers accepted for other sessions (e.g. clearly stated goals, important to field, relevant topic, well-researched and written, good methodology, etc.). Perceived or evaluated quality of a paper should NOT be a criterion, nor should scholar-to-scholar sessions become dumping grounds for the lowest ranked papers of those accepted.

2. Wherever possible, scholar-to-scholar boards should be arranged to diminish noise and facilitate interaction. Sponsors of scholar-to-scholar sessions also are encouraged to arrange them thematically, and groups are encouraged to co-sponsor scholar-to-scholar sessions.

3. Authors of scholar-to-scholar sessions are responsible for being present at the session to discuss their papers. An example handout of an effective scholar-to-scholar display is available from AEJMC.

4. To encourage creativity, groups should, wherever feasible, create faculty and student awards for best visual display. This award should supplement, not replace, other awards based on the quality of the paper’s content.

5. Groups may vary the format of scholar-to-scholar sessions and are encouraged to be creative in the use of formats and displays (e.g. scholar-to-scholar sessions could begin with 2-4 minute presentations by individual authors, or with round-table discussion, and they might end with discussion and critiques, either informally or formally). Groups should obtain feedback from scholar-to-scholar presenters and attendees to improve future scholar-to-scholar sessions.

Guidelines revised August 2010

By Chuck Lubbers, AEJMC Public Relations Division

If you are scheduled to present a paper during a high density session at the upcoming AEJMC meeting, this format may be new to you. Below is the format for a high density sessions. This information provides the basics of the format and attempts to outline the reasons for its use.

Rationale:
The high density format may be thought of as a cross between a traditional paper presentation panel and a poster session. The hybrid allows more individuals to be placed on the panel (thus the name “high density”). The key is that the individual presentations must be SHORT to allow for individual discussion with members of the audience. To insure this, your chair and discussant will be strictly enforcing the timelines discussed below. This format allows the audience members to hear the detail on those research projects that interest them the most.

Format:
Presenters will be given 4 minutes to provide an overview or summary of their paper. This time limit will be strictly enforced. You will be stopped if you exceed the time limit. No questions are taken between presentations. Since you have a short time for presentation, you are encouraged to limit audio-visual and lengthy discussions. Think of this as an executive summary or an extended abstract.

The discussant for the panel will not comment on the individual papers. The discussant’s function is to facilitate discussions between the presenters and the audience members. To achieve this goal, presenters are asked to spread out in the room so that individuals who would like to speak to them may do so. Audience members will then be able to spend some time hearing more about or asking questions about those research presentations that most interest them. To help the audience members find the proper presenter, a sign will be made with your paper title and authors. These will be taped to the walls around the room (or at tables, if available). After all the presentations are over, you must go to the area with your sign.

Presenters should prepare handouts containing outlines, key points, executive summaries, etc. for their study to distribute to the audience members. This one-page handout will help the members of the audience comprehend your brief presentations and select those papers they would like to hear more about. There will be no audio-visual equipment available.

If the guidelines outlined here are followed correctly, there will be nearly 30 minutes for individual discussion. This will allow audience members to get more information on several papers.

<<Paper Presenter FAQs

By Jack Rosenberry, St. John Fisher College

“Dear Professor: I am pleased to inform you that your paper submitted to our division’s research competition has been accepted for presentation at the AEJMC summer convention. …”

Typical submitter’s response: ALL-Rig-g-h-t! It got ACCEPTED!

“ … It will be presented in the Scholar-to-Scholar Poster Session scheduled for 1:30 p.m. Friday afternoon …”

Continuation of response: Oh. A poster session. Hmph. I wonder what they didn’t like about it.

Many of us who have had papers accepted for the convention have experienced this range of responses. Without a doubt, the idea of presenting in “just a poster session” carries a stigma that the research, while acceptable for the convention, is somehow second-rate.

This is a belief that the Council of Divisions and Standing Committee on Research are hoping to change.

The simple fact is that with the growth of the organization and the convention, it would be a physical impossibility to accept the number of papers that has become typical in recent years and have them all presented orally.

Last year’s convention in Chicago saw about 700 papers accepted for presentation; for that many to be presented orally with four papers to a session, as is typical, would have required 175 sessions. The convention programming “grid” had about 250 available programming slots from 8 a.m. to 6:30 p.m. Wednesday through Saturday. So to schedule oral presentation of all papers would have occupied literally 70 percent of the convention programming, and left only about 75 slots for other programming – with 30 divisions and interest groups scrambling for them.

As important as research is, nobody wants a convention dominated that severely by it. And nobody wants to limit the number of accepted papers to such a small number that all can be presented in the limited number of oral sessions, either. That is why poster sessions and “high-density” sessions have become increasingly common at the convention. (In a high-density session, approximately 10 presenters each give a 3-to-5-minute summary of their work, a time limit that moderators strictly enforce. Then, presenters go off to different parts of the room and listeners can approach them for more in-depth discussion of the work.)

For its part, the Newspaper Division received about 80 paper submissions for Chicago and accepted about 40, a typical acceptance rate across all units that hold research competitions. About half of those 40 accepted papers were presented in five traditional sessions, leaving the other half for two poster sessions: the Scholar-to-Scholar event and a special poster-research session co-sponsored with the Mass Communication and Society Division.

The same number and format of sessions is on the schedule for Boston this year. So, if your paper is accepted by the Newspaper Division, there’s about a 50 percent chance it will be presented in a poster session.

One hypothesis about the stigma against poster research is that it exists because some institutions consider them to be “lesser” accomplishments that don’t carry as much weight in the tenure case because of fears that posters are a dumping ground for secondary work.

But within AEJMC, all papers are accepted according to the same criteria. Only after all acceptances have been made are papers divided into their presentation venues, which ideally is done on the basis of a theme that combines similar papers in a given presentation.

Research chairs are instructed to divide the best papers between the oral sessions and the poster or high-density ones, and especially to make sure some of the best papers are allocated to the Scholar-to-Scholar session. All papers, regardless of venue, are read by a discussant who offers a critique of them.

So within AEJMC, at least, no stigma should be attached to poster or high-density sessions. Absolutely no distinction is made in the judging or standards for acceptance based on the presentation venue.

I have done both traditional research presentations and posters in recent years and actually have come to prefer posters. A bit more preparation needs to go into the presentation materials for a poster session. But once that is done, the rest is very easy. I find it easier – or at least less nerve-wracking – to make a poster showing than to deliver a formal presentation to a room full of colleagues. (Or to a room devoid of them, which is even worse!) The sessions allow for a lot of informal chatting with people who are really interested in your work, and I’ve met colleagues at such sessions who have become close friends and collaborators because of our similar interests.

So, if your work is destined for a poster session, don’t despair. Know that it met the same high standards that every other accepted paper did (and save this column to show your chair or tenure committee if you need validation of that). Enjoy the more relaxed setting to show off your fine efforts. And look for me, because I like attending them as much as I like presenting in them.

<<Paper Presenter FAQs

By Sheri Broyles, AEJMC Advertising Division Executive Committee

• Poster board example (PDF)

“It’s the first poster session I’ve done. In fact, it’s the first time I’ve done a poster since the 10th grade science fair, which I entered under great duress. I was not looking forward to this one either…”

Do you, too, dread being in a poster session? This is just one comment I received in response to an email questionnaire sent out to participants in to poster sessions at AEJMC 2000 in Phoenix. Last August, as I walked through the two poster sessions in which the Advertising Division participated, I was struck by how, at least graphically, the poster sessions seemed a desert with an occasional oasis of a well-presented poster.

A quick content analysis showed that while most (71%) used some color, it generally wasn’t used effectively. Many posters used color only in the header, and that was often subdued, For example, a burgundy header might be paired with black body type. Overall, type was way too small. Only 23% of the body type was big enough to read standing a couple of feet away from the poster.

The email follow up revealed that many of the poster presenters said they received no guidelines. AEJMC has said that poster sessions will be the same format. That means you’ll have a freestanding 8’ x 4’ bulletin board. Pins will be provided, although bringing a few of your own “just in case” is a good precaution. Bring your pages printed out and ready to pin up, but you don’t need to bring them on poster board as you did in those science fair days.

While clearly content should be the draw, even the most interesting research at 12-point type won’t get noticed. The key is to have great content, but present it in such a way that people will stop and see what you’ve said.

With that in mind, here are a few quick tips to help make you a poster prodigy:

• Treat it as you would an outdoor board: be big, bold, and brief. Those of us in advertising should have an advantage because the same things that make an effective ad will make an effective poster.
• Think big, really big. A title in 144-point type may sound huge, but it’s really only 2” type, and the capital letters are just over an inch. Remember, you want to attract attention to draw people into your poster. Section heads should be 42 to 48 points. And don’t make the body copy too small. Your smallest type should be a minimum of 24 point and 28 point is even better.
• Rewrite your title as a headline. Again, use this to draw people in. Adapt your writing style. Use short, informative phrases. Bullet points may be even better. With posters, less is more, especially with text.
• Use graphics. When asked what they’d do differently, past poster presenters said repeatedly, “Less text. More tables and graphs.” Any illustrations, whether photos or clip art, can make your poster more appealing. And try blowing them up for an interesting dominant element.
• Add some color. One way is to take bright construction paper and offset it behind the white paper on which you present your text. Color can also be used in charts, graphs, illustrations, or photos.
• Consider flow. People read left to right, so set up your poster that way. For example, the abstract should go on the left, not in the center under the title with other sections wrapped around it. Remember it’s sometimes hard to get back to the left side to read a second row, so you may want to go top to bottom and left to right. But it should be very clear how the poster flows. If you need to, you can use small arrows directing readers to the next panel.
• Try it out. Tape it to the wall or place it on the floor, then try reading it from 5 or 6 feet away. Check that everything is big enough and that the flow works.
• Prepare a handout. (There will be no audio-visual equipment available.) Include the abstract and perhaps an abbreviated methods, results, and discussion section. You might add any pertinent tables or graphs. Most importantly, put contact information and that this was for an AEJMC poster session with the year and the city. Make it easy for them to cite you, should they choose to do so.

The extra thought and effort will be worth it, and you’ll feel better about your research during the poster session. You don’t want to be in a position of one poster participant who said, “I must admit I was a little embarrassed that my poster looked so shabby compared to some.”

<<Paper Presenter FAQs

The following links are for instructional tutorials in the ALL ACADEMIC system for AEJMC Research Chairs. These videos are designed to help you navigate through our online paper review system quickly.

Each video is 3 minutes or less and includes step-by-step instructions for a variety of features in our paper review system.

If you have any questions about how to use the system, you can contact Felicia G. Brown.

Video Tutorials

• Research Chair Training (Zoom group video)
• Login, Main Menu & Add Reviewers
• Assign Reviewers
• Send Bulk Emails
• Accept / Reject Proposals
• Request Reports