Replicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories proceedings

TitleReplicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories proceedings
Publication TypeConference Paper
Year of Publication2010
AuthorsRobles, Gregorio
Secondary Title2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)
Pagination171 - 180
PublisherIEEE
Place PublishedCape Town, South Africa
ISBN Number978-1-4244-6802-7
Keywordsdata, literature review, msr, replication
Abstract

This paper is the result of reviewing all papers published in the proceedings of the former International Workshop on Mining Software Repositories (MSR) (2004-2006) and now Working Conference on MSR (2007-2009). We have analyzed the papers that contained any experimental analysis of software projects for their potentiality of being replicated. In this regard, three main issues have been addressed: i) the public availability of the data used as case study, ii) the public availability of the processed dataset used by researchers and iii) the public availability of the tools and scripts. A total number of 171 papers have been analyzed from the six workshops/working conferences up to date. Results show that MSR authors use in general publicly available data sources, mainly from free software repositories, but that the amount of publicly available processed datasets is very low. Regarding tools and scripts, for a majority of papers we have not been able to find any tool, even for papers where the authors explicitly state that they have built one. Lessons learned from the experience of reviewing the whole MSR literature and some potential solutions to lower the barriers of replicability are finally presented and discussed.

URLhttp://gsyc.urjc.es/~grex/msr2010
DOI10.1109/MSR.2010.5463348
AttachmentSize
171MSR_2010_69.final_.pdf128.26 KB