Amassing and indexing a large sample of version control systems: towards the census of public source code history

TitleAmassing and indexing a large sample of version control systems: towards the census of public source code history
Publication TypeConference Paper
Year of Publication2009
AuthorsMockus, A
Secondary Title6th IEEE Working Conference on Mining Software Repositories
Date PublishedMay 16–17
Keywordsbazaar, cvs, flossmole, git, mercurial, source code, sourceforge, subversion, version control

The source code and its history represent the output and process of software development activities and are an in- valuable resource for study and improvement of software development practice. While individual projects and groups of projects have been extensively analyzed, some fundamental questions, such as the spread of innovation or genealogy of the source code, can be answered only by considering the entire universe of publicly available source code and its history. We describe methods we developed over the last six years to gather, index, and update an approximation of such a universal repository for publicly accessible version control systems and for the source code inside a large corporation. While challenging, the task is achievable with limited resources. The bottlenecks in network bandwidth, processing, and disk access can be dealt with using inherent parallelism of the tasks and suitable tradeoffs between the amount of storage and computations, but a completely automated discovery of public version control systems may require enticing participation of the sampled projects. Such universal repository would allow studies of global properties and origins of the source code that are not possible through other means.


Related work: "For example, FLOSSMole [8]
collects the list of projects and their metadata from SourceForge,
FreshMeat, Free Software Foundation (FSF), RubyForge,
and ObjectWeb. "

"The list of
home pages from open source directories such as FSF or
FLOSSMole could provide a focused set of web pages for
the search"

"To get the list of projects for SourceForge we used
FLOSSMole project [8] that collects the list of SourceForge
projects and their metadata. We use project ID, because it
was used to specify the VCS URL within the SourceForge

Full Text
PDF icon 11amassing.pdf108.04 KB