Validity concerns in software engineering research

TitleValidity concerns in software engineering research
Publication TypeConference Paper
Year of Publication2010
AuthorsWright, Hyrum K., and Kim Miryung
Refereed DesignationRefereed
Tertiary AuthorsRoman, Gruia-Catalin, and Sullivan Kevin
Secondary TitleProceedings of the FSE/SDP workshop on Future of software engineering research - FoSER '10
PublisherACM Press
Place PublishedSanta Fe, New Mexico, USANew York, New York, USA
ISBN Number9781450304276
Keywordsflossmole cited

Empirical studies that use software repository artifacts have become popular in the last decade due to the ready availability of open source project archives. In this paper, we survey empirical studies in the last three years of ICSE and FSE proceedings, and categorize these studies in terms of open source projects vs. proprietary source projects and the diversity of subject programs used in these studies. Our survey has shown that almost half (49%) of recent empirical studies used solely open source projects. Existing studies either draw general conclusions from these results or explicitly disclaim any conclusions that can extend beyond specific subject software. We conclude that researchers in empirical software engineering must consider the external validity concerns that arise from using only several well-known open source software projects, and that discussion of data source selection is an important discussion topic in software engineering research. Furthermore, we propose a community research infrastructure for software repository benchmarks and sharing the empirical analysis results, in order to address external validity concerns and to raise the bar for empirical software engineering research that analyzes software artifacts.


"Obtaining a balanced set of data from
open source repositories has been an issue in the open source
research community for some time, and several collections
of data have grown to attempt to solve this problem [11, 13,