Background: The number of reported empirical studies of Open Source Software (OSS) has continuously been increasing. However, there has been no effort to systematically review the state of the practice of reporting empirical studies of OSS with respect to the recommended standards of performing and reporting empirical studies in software engineering. It is important to understand, how to report empirical studies of OSS in order to make them useful for practitioners and researchers.
Research aim: The aim of our research is to gain insights in the state of the practice of reporting empirical studies of OSS in order to identify the gaps to be filled for improving the quality of evidence being provided for OSS.
Method: To that end, we decided to systematically review the empirical studies of OSS. A total of 63 papers reporting empirical studies were selected from the four editions of the Proceedings of the International Conference on Open Source Systems. The data were extracted and synthesised from the selected papers for analysis.
Results and conclusions: We have found that the quality of the reported OSS-related empirical studies needs to be significantly improved. Based on the results of our systematic review and general principles of reporting good empirical research, we present a set of guidelines for reporting OSS-related empirical studies. The suggested guidelines are expected to help the research community to improve the quality of reported studies.