Towards a Unified Definition of Open Source Quality

TitleTowards a Unified Definition of Open Source Quality
Publication TypeConference Proceedings
Year of Publication2011
AuthorsRuiz, C, Robinson, W
Secondary TitleOpen Source Systems: Grounding Research (OSS 2011)
Date Published10/2011
Keywordsliterature review, measurement, open source, quality, Software

Software quality needs to be specified and evaluated in order to determine the success of a development project, but this is a challenge with Free/Libre Open Source Software (FLOSS) because of its permanently emergent state. This has not deterred the growth of the assumption that FLOSS is higher quality than traditionally developed software, despite of mixed research results. With this literature review, we found the reason for these mixed results is that that quality is being defined, measured, and evaluated differently. We report the most popular definitions, such as software structure measures, process measures, such as defect fixing, and maturity assessment models. The way researchers have built their samples has also contributed to the mixed results with different project properties being considered and ignored. Because FLOSS projects are evolving, their quality is too, and it must be measured using metrics that take into account its community’s commitment to quality rather than just its software structure. Challenges exist in defining what constitutes a defect or bug, and the role of modularity in affecting FLOSS quality.


"In order to answer the research question, how is quality defined in the FLOSS literature, we performed a literature review."

"we searched Google Scholar for journal articles and conference papers containing the terms “open source” and “quality”"

"This process left us with 24 papers, to which we then added 16 from the quality and defect-fixing categories in [34] that met the above stated criteria.
This left us with 40 papers that defined quality and performed some form of empirical validation of that definition."

"there is little consensus in the FLOSS literature when it comes to defining quality."

defect resolution versus modularity: "Defect resolution rates (amount of defects resolved, speed of resolution) are the best way to measure a community’s commitment to quality, because they recognize that FLOSS is not a static product, but ever evolving. These rates should be calculated per release, and not cumulatively, because the cycle of FLOSS evolution is the release. Researchers should be careful to only include defects and not new feature requests, duplicates, or poorly reported bugs into their calculations.
Modularity is being touted as the main driver of FLOSS quality success, but it needs to be further defined and studied in order to understand how it works."

Full Text