A Large-Scale Study on the Usage of Testing Patterns that Address Maintainability Attributes

Test case maintainability is an important concern,
especially in open source and distributed development environments
where projects typically have high contributor turnover
with varying backgrounds and experience, and where code
ownership changes often. Similar to design patterns, patterns
for unit testing promote maintainability quality attributes such
as ease of diagnoses, modifiability, and comprehension. In this
paper, we report the results of a large-scale study on the usage
of four xUnit testing patterns which can be used to satisfy these
maintainability attributes. This is a first-of-its-kind study which
developed automated techniques to investigate these issues across
82,447 open source projects, and the findings provide more insight
into testing practices in open source projects. Our results indicate
that only 17% of projects had test cases, and from the 251
testing frameworks we studied, 93 of them were being used.
We found 24% of projects with test files implemented patterns
that could help with maintainability, while the remaining did
not use these patterns. Multiple qualitative analyses indicate that
usage of patterns was an ad-hoc decision by individual developers,
rather than motivated by the characteristics of the project, and
that developers sometimes used alternative techniques to address
maintainability concerns.