How to Referee a Scientific Paper

Peer Review Process

The task of the referee is to evaluate in a timely manner a paper for publication in a journal of conference proceedings.

Judge:

  • If the paper is correct
  • If the problem studied and the results obtained are new and significant
  • If the quality of the presentation is good
  • What changes might/should be made to the paper.

Publishable paper: SUFFICIENT CONTRIBUTION

  • new and interesting results
  • new life from old results
  • survey of old results

Conference VS Journal

Conferences papers:

  • hard deadlines, no chance of delaying a report
  • very short refereeing time (1 hour ~ a week)

Journal papers:

  • soft deadlines. delays common
  • long refereeing time (4 month ~ infinity)

Suggestions about refereeing:

  1. Be Boolean (either ACCEPT or REJECT)
  2. Be prompt

Refereeing Process

Input: A submitted paper.

Output: A referee report

  1. A recommendation for/against publication. MUST include sufficient justification for the recommendation.
  2. A list of necessary/recommended changes and revisions.

How to do it?

Read the paper carefully, check and evaluate the material, and without preconceived ideas.

Price: use of considerable time and effort.

Referee Reports

Evaluation score: Usually 0~10.

Confidence level: Usually 1~3.

Justification and general comments

  1. Synopsis. (1-5 sentences) summary of the main points of the paper.
  2. Evaluation.
    • Was the goal of the work worthwhile?
    • Were the developments correct?
    • Was the paper well-written?
    • …
  3. Recommendation
  4. Detailed comments for the authors.

Comments for authors

Confidential comments for PC

Issues in Evaluating a Paper

What is the aim of the paper? Is it significant?

Is the paper appropriate for the chosen forum?

Is the approach valid?

Is the execution of the research correct?

Are the correct conclusions being drawn from the results?

Is the presentation satisfactory?

Does the paper do justice to related work?

Attributes of a Paper

Correctness, Significance, Innovation, Interest, Timeliness, Succinctness, Accessibility, Elegance, Readability, Style, Polish

Categories of Papers

Strong accept:

  • Major results; very significant (<1% papers) e.g. S.A. Cook. The complexity of theorem proving procedures

  • Good solid interesting work; a definite contribution (<10% papers)

Weak accept:

  • Minor, but positive, contribution to knowledge (10~30% papers)

Weak reject:

  • Elegant and technically correct but useless
  • Neither elegant nor useful, but not actually wrong.

Strong reject:

  • Wrong and misleading
  • So badly written that technical evaluation is impossible

Ethical Issues, Dilemmas

Anonymity matters! Never abuse it!

Try to be: objective, quick, professional, honest and courteous.

Professionalism

Criticism should be specific rather than vague.

Avoid sentences:

  • The main result of the paper is most likely wrong (ambiguous)
  • The author should cite related literature (ambiguous)
  • This is a nice exercise for a graduate student. (offensive)
  • The results are folklore (ambiguous)

Dilemmas

How many paper should I referee?

How much time should I put into a paper?

What is the relationship between conference and journal versions of a paper.

What if I have a conflict of interests?

Is the referee responsible for checking the technical correctness of the paper? Absolutely YES!

Should the referee improve the writing of the paper? Not necessarily.

Should the referee proofread the paper for the author? Not necessarily.

⚠️Should I ask the author to cite some work of mine? Do it only if your work is truly relevant

  • ✅It increases the number of citations to my work.
  • ❌ It affects anonymity.

Receiving a Referee Report

One must learn to deal with referee reports properly!

Before reading a referee report, take a deep breath, remember that a good report is always valuable, somebody do spent time reading your paper and producing the report.

Use the report to improve your paper.

⚠️If you expect bad reports in light of paper’s quality, don’t even consider submission.

Something to Remember

Our currency is reputation. It takes a lot of hard work and social skills to build, but it takes very little to destroy it.

Try to evaluate your won work using the same standards you apply to somebody else’s.

The job of refereeing papers is hard, necessary, and important.

How to Give Scientific Talks

Why we need talks? Good talks are helpful to almost all scientists, since most give pretty bad talks.

What to Say?

Ask yourself:

  • Can I explain it clearly?
  • If so, how? What is the punch line? What do I want the audience to take home from to the talk?

Good talks should

  1. make only main point
  2. never run overtime
  3. relate to the audience
  4. given the audience something to take home

Conference Talk - Short and limited time (20-25 minutes): Make only one of the points.

Seminar Talk - limited but longer (45-60 minutes)

  • Well-prepared
  • clear about the goals

How to Deliver the Talk

Read from a script (only for very formal occasions)

Use black- or white-board/Slides

On the slides: don’t write too much

Standard Mistakes

Small fonts or handwriting

Invisible color

meaning attached to colors (unfriendly to color blindness)

Long and complete sentences

Overcrowded slides

Unreadable slides (too many symbols, acronyms)

Slides written at the last moment

Interruptions

Answer on the sport (but don’t get carried away)

Say “good point, just wait two slides”

Say “good point, I’ll come back to it at the end of the talk”

Use a secret slide

Minor interruption: don’t panic, straighten out and carry on.

Major interruption

  • About assumptions
    • Non-specialist audience
      1. Situate the nature of the interruption
      2. delay the discussion until after the talk
    • Knowledgeable audience
      1. make your point clearly
      2. discuss it out
  • About point

Running Out of Slides

Short talks are appreciated.

Conclude without hurry, and summarize the main point of the talk.

“Thank you, are there any questions?”

Don’t make personal comment.

Running Out of Time

Don’t skip many slides looking for the right one to put on text.

Conclude by making the main point

Thank the audience

How to Write Papers

A Simplified Progress

  1. Work on relevant CS question
  2. Write a scientific paper
  3. Submit the paper to an appropriate conference/journal
  4. If accepted for publication then
    • Add one line to CV
    • Present the work at scientific meetings
  5. Else (paper rejected or to be modified) go back to step 1

What is a Scientific Paper

A written and published report that describes original research results

  • FIRST publication
  • Peers can repeat the experiments and test the conclusion
  • Readily available within the scientific community

Other Types of Research Reports

Technical report

Conference paper (neat)

Conference/meeting reports

Textbooks and research monographs

IMRAD Approach

  • Introduction

  • Method

  • Results And

  • Discussion

Organization of a Typical Paper

Title

It should be informative

Be concise

Better to be catchy/memorable

Best original

It is a LABEL, not a sentence

Avoid abbreviations

The List of Authors

Alphabetically ordered (usually in TCS)

Ordered by degrees of contribution

Always give

  • Full addresses (both physical and email)
  • URLs for web pages or data replication packages
  • In case of double-blind review process use anonymous links

Abstract

Be brief (max 250 words)

State the main objectives and scope of the investigation

Summarize the results

Possibly state the principal conclusions

Be updated/written last

⚠️Many more people will read the abstract than the paper.

❗It should not contain formular, special symbols or bibliographic references.

Introduction

⚠️The introduction often decides the destiny of a paper (to be accepted or rejected)

It is often the only part of the paper that will be read.

It should not be too technical!

  • First, the nature and scope of the problem investigated (in all possible clarity)
  • Review the related literature to orient the reader.
  • It should say clearly what is the achievement of the paper.
  • It should end with an overview of the rest of the paper.

Some pitfalls

  • Exaggerating
  • Seeking effect for the sake of seeking effect
  • Misspelling (always use a spell-checker)

About Results

Be honest, do not cheat!

Negative results are still valuable and important.

How to Present Results

Technical preliminaries and background

Progressive development of the material

State where your contribution lies

Anticipate and answer the possible questions that a reader might have.

❗Always introduce technical terms before using them.

Mandatory

It situates the novelty and significance of your work

  • Where do the ideas come from?
  • Have similar ideas been published or proposed earlier?
  • What is really new in the paper?

Either as part of the introduction, or part of the conclusion, or stand-alone section.

Pitfalls

  • Forgetting or misrepresenting work done by somebody else.
  • Copying the text from other papers, even when the citation is added, is PLAGIARISM! Always re-elaborate/paraphrase the concepts from related work.
  • Reinventing the wheel.

Use of Bibliographical References

❌[ABC2003] shows that …

✔️… as introduced by ABC in his monograph on the λ\lambda-calculus [2].

The Conclusion

(Not mandatory)

  • recapitulates the problem and the contribution
  • assesses the significance of the contribution
  • suggests and outlines future work

The References

Be accurate (correct year, place of publication…)

Spanning across years, preferably recent ones.

Complete (page number, number of volume)

Always cite the best primary publication for some work.

Acknowledgments

Always acknowledge input from your anonymous referees.

After Writing the Paper

  • Proofread the paper as carefully as you can
  • Let the paper rest for several days, and then proofread it again.
  • Ask other people to read the paper.

If happy with the product, the submit the paper.

How to Write Thesis Proposals

Structure Research Ideas

Abstract: Limited text (max 1 page) including context, objective, tentative methods and expected results

Introduction: very important, clearly explain the importance of your research, and what you plan to achieve with your research

Background: knowledge on the concepts you will use later on

SOTA: clearly position your research w.r.t. related work

Research Proposal:

  • Research problem: what are the challenges and their importance, possibly referring to real-world examples
  • Feasible/Tentative Approaches: how to solve or address the research problem
  • Expected Research Contributions: clearly state how your research advances the current SOTA.
  • Research Plan: timeline on activities and mitigation strategies if something gets wrong/unexpected.

Abstract

Context

Objective

Method

Results

Conclusion

Introduction

Research problem: What is the problem and why it is important to solve it, possibly referring to real-life examples

Existing literature

  • Approaches already developed in the literature to deal with the same problem
  • Look for survey papers on similar topics to get a larger vision on existing research
  • Select the most related approaches and argue on the main open issues that you are interested to work on

Proposed solution

  • What is missing in the literature and how your PhD thesis will fill the identified gap!
  • Since it is a thesis proposal, none is expecting that you have clear ideas on possible solutions, but still you have to elaborate on what you think it is a feasible solution for the identified problem
  • Always think about alternative solutions, if one solution that initially you think suitable then it turns that it does not work, what are the other possibilities you would like to explore?
  • What are the metrics that can be used to assess the validity of your solution? Possible evaluation methodology that can be applied.

Research contributions

  • Expectations, what are the benefits, possibly referring again to real-life examples

Background

Main concepts and their explanation

SOTA

Very important!

You have to demonstrate that you studied the main approaches developed so far in the selected research domain

Organize the content:

  • what are the main fields of investigation
  • group approaches with a rationale that you have to clarify
  • provide a table that summarizes the main approaches that you select as the mostly closed related to your work

Research Problem

Research questions

Concrete applications

Discussion

Requirements

Modeling

Analysis

Research Plan

What are the research questions/objectives

Time management plan

Research Validation

Threats to validity

  • Conclusion validity regards issues that affect the ability to draw accurate conclusions about relations between the treatments and the outcome of an experiment
  • Internal validity regards the influences that can affect the independent variables with respect to causality

How to assess the validity of your research work

  • How to draw accurate conclusion
  • How to avoid causality
  • How to generalize the results
  • How to assess the representativeness of the proposed work

Bibliography

References must be impeccable

  • Accurate
  • Spanning across years, preferably recent ones
  • Complete