ISWC 2020 Reproducibility Track Call for Participation

The second edition of the Reproducibility Initiative is part of the 19th International Semantic Web Conference (ISWC 2020) that will take place in Athens, Greece on 2-6 November, 2020.

Venue: Megaron Athens International Conference Centre, Athens, Greece

HIGHLIGHTS

The Reproducibility Initiative is running for the second time and will evaluate submissions from the accepted papers of the ISWC Research Track.

DESCRIPTION

‘Standing on the shoulders of giants’ is what we say to highlight that scientific progress is heavily relying on previous discoveries. Oftentimes we are willing to build on top of these results but have a hard time continuing the work because the results in the original publication are difficult to reproduce. A recent survey in Nature shows that, in a pool of 1,576 researchers, more than 70% failed to reproduce another researcher’s experiment and—even more telling—more than half of the survey participants could not reproduce their own experiments. The reasons for these results are having to do with selective reporting, cherry picking of experimental results, and lack of information about the experiments, to name a few.

While the aforementioned survey has participants mostly from natural sciences, one might think that the situation in computer science is different. But it is actually not! Not long ago, Collberg et al. reported that only a third of the code and data, in the 13 venues they selected, are available and can be easily built within 30 minutes. And they have not even tried to reproduce the experiments reported in the respective papers. While reproducibility of computational research could be challenging (due to specialized hardware, complicated setups and library dependencies, etc.), reproducing empirical studies, such as observing users to interact with a particular system, is often out of the question.

It is our belief that sharing experimental code, data and setup will benefit scientific progress, foster collaboration and exchange of ideas. We would like to build a culture where sharing results, code, and scripts is the norm rather than an exception. Since we recognize the additional effort, we aim to build technical expertise on how to do this efficiently and conduct better research via creating repeatable and shareable methods and results.

Hence, as an author of a research track paper, we would like to invite you to submit your contribution to the ISWC 2020 Reproducibility Track (In this call the term reproducibility refers to the case when an independent researcher is trying to re-run the same experimental setup and reproduce the most important results of the paper. As there are terminological issues, discussion and comparison of terminology can be found here.).

The ISWC Reproducibility Initiative has the following goals:

To enable easy sharing of code and experimental set-ups (take a paper and reuse it).
To make more code and data available.
To highlight the impact and increase the credibility of the Semantic Web research.
To facilitate the dissemination of research results.

Why should I be a part of this?

To easily repeat your own experiments.
To discover accidental flaws and improve your results.
To increase confidence in your results.
To make it easy for other researchers to compare to, adopt and extend your research.
To increase visibility and impact of your results.

SCOPE & SUBMISSION

This year’s evaluation will include two lines of work:

Reproducing software & computational experiments - Contributions with significant computational experiments as well as system setups are subject to the following guidelines and belong to the Reproducibility Line of Assessment; The submissions under evaluation will be labelled Reproduced if the experimental results in it have been successfully reproduced. In case that the experiments have not been reproduced due to limited computational resources (for instance, number of GPUs) but the authors had supplied enough materials about their work so an interested researcher could re-run the experiments in question, the manuscript will be labeled Reproducible.
Replicating laboratory experiments with users - As reproducing empirical studies need significant resources, contributions describing quantitative laboratory experiments with users (evaluation studies involving users conducting specified tasks in a controlled setting) belong to the Replicability Line of Assessment. This means that the experiments have not been reproduced but the authors had supplied enough materials about their work so an interested researcher could re-run the experiments in question. They are subject to the following guidelines. (Exploratory studies are out of the scope of this evaluation as even their evaluation is challenging.)

For the Reproducibility Line of Assessment, two independent members of the Programme Committee will interact with the authors to check the availability of the data, source code, documentation, configuration requirements and reproduce the most important results of the paper. For the Replicability Line of Assessment, one member of the Programme Committee will interact with the authors in order to assess if the authors have supplied enough materials about their work so an interested researcher could re-run the experiments in question.

Submission: via EasyChair
Please follow the submission guidelines.

Important Dates:

Submission Deadline: August 19, 2020
Assessment Period: August 26 - October 16, 2020
Results Announcement (During ISWC 2020): November 2-6, 2020

Reproducibility Track Chairs:

Dr. Valentina Ivanova,
RISE Research Institutes of Sweden, Sweden

Dr. Pasquale Minervini,
University College London, United Kingdom

Contact: iswc2020-reprod@easychair.org
Website: http://repro.semanticweb.org/ISWC2020/

Reproducibility Committee

Visit the Reproducibility Committe page.

Acknowledgements: The text of this call is partially based on the call for the first edition of the Reproducibility Initiative at ISWC 2019 by Alejandra Gonzalez-Beltran & Michael Cochez.