OSS 2009 Trip Report

6/4/09 - Day 1

Session 1 - Governance of OSS Projects (panel)

Francesco Bolici

Governance is usually understood as management. It's about accountability and eliminating problems with principal-agent from economics. In OSS, this can help with resource allocation, coordination.

Direction, coordination, control - these 3 characteristics of governance cannot be separated as well as in other sector. Two important aspects are coordination and accountability. Coordination - assumed that direct communication is the main coordination mechanisms. Accountability - close to trust, related to recognition - how does it work when the community becomes large scale?

Coordination - low levels of communication and hence coordination. Stigmergy - effort required to make sure the code can be understood by everyone. Stuff from the work with James.

Paul de Laat - University of Groningen

Governance tools - several parameters create an organizational design, typically assumed to fall into typical clusters/configs (Mintzberg). Organization design can't be random; values influence these configurations.

Several tools for internal governance in OSS:
-modules
-division of roles
-delegation of decision-making
-training and indoctrination
-formalization
-autocracy/democracy

Notable differences between employees and volunteers: control structure (sign a contract, do as you're told, get paid) - in FLOSS, you're a volunteer and have only a psychological contract, can choose own contributions, but you must be liked and accepted or you will be ignored. Fundamental difference: earning pay and risking being fired versus earning esteem and risking being ignored.

Is too much governance imminent? Legal foundations to handle donations and take care of IP issues. However, some foundations start interfering with internal governance - appointments, release dates. Will the iron law of bureaucracy also apply here? Projects may be pressured to abandon democratic leadership elections and decentralized decision-making?

Thorbiorn Fritzon - Sun Microsystems

Does it matter that commercial actors make money off individual nonprofit contributions and the individuals can't make money off the company's use of the software?

Jan Ljungberg - Gothenburg University

Open source usage and development - coping with ideological tensions
Hierarchic utilizers - proprietary companies with in-house programmers who value and support open source
Tailors on the market - pure play companies who utilize external consulting to customize software; value on openness versus customer demands, using community resources as high quality input to help solve customer problems

Governance challenges - itch worth scratching/customer demands; licenses and IP; company/customer versus community; taking part in an ecology

Discussion -

Forks are not to be taken lightly. Forking and managing a fork is quite challenging. It's only the tough people who fork and can really do it.

Why don't companies like Sun donate their OSS-based code to nonprofits or public organizations? There has been discussion of what will become of OpenOffice, partly because there are so many "outside" contributors to the project - though this seems mostly to be localization.

--------

Session 2 - Mining OSS Data

1. Analysis of OSS development iteration by means of burst detection techniques (Rossi & Succi)

OSS development based on a set of practices leading to an efficient bug-fixing process and quick release cycles. Domain and application make a difference. Goal is to evaluate whether the activity level in bug-fixing and releases are related.

2. Our paper - Heartbeat

Classifying release types - look at changes to LOC to evaluate the response - are users being conservative or eager to update based on the type of release.

Compare on DebPop. Did analysis only on apps that have to be downloaded - if they became packaged then the measure would go to hell.

3. Estimating Commit Sizes Effectively (Hoffman & Riehle)

Commit size used as a proxy for "amount of work". Gives a careful definition of commit size - commit can encompass multiple files and a diff per file can encompass multiple sections.

Diff data pair - per commit, how many SLoC were added, how many removed, and number of lines changed - add up the number of events for diff size, these are all different lines that are touched. Limited by sample size; samples might be biased though the dataset creators don't think so. Their assumption of linearity could be off, but the nature of the model suggests linearity.

--------

Session 3 - OSS Communities

1. Reassessing Brooks' Law for the FLOSS Community - Capiluppi & Adams

Applicability of The Mythical Man Month to FLOSS is debated - particularly the high modularity and lack of wealth-based motivation

Issues of coordination - growing projects require restructuring - for n developers, n^2-n possible connections - how many are used?

Issues of early productivity - new developers require time to meet their potential, need to learn the community before they can contribute at full potential - not discussed here.

What are actual coordination tasks - weekly "community cohesion graph" where links are shared artifacts (in SVN - code, binaries, sound, images, other text-based info like documentation) and weights are frequency. Used Floyd-Warshall algorithm for shortest paths; Kamada-Kawai shows "onion" model. Kamada-Kawai places nodes with high edge weights nearer to one another.

Data - SVN XML formatted - KDE, over 1 GB, 1500 accounts, 350 active, 300 commits/day, 10+ years of development, 85 GB repository (all artifacts)

Coordination issues - explosion in KDE is explored and explained (ask for more - "refactoring" community, working practices changed with technology change); issues of early productivity - different projects require dramatically different "ramp up period" before developers can perform to potential.

Evidence that Brooks' Law holds for core teams with Evince - growth caused reduced cohesion based on utilized coordination paths; KDE restructuring allowed cohesion growth.

Restructuring to improve cohesion doesn't have to be format; changes in pair counts can be a factor, move between technology in use can also be a factor.

Comments:

Do you have results to compare KDE to anything else? KDE to GNOME? KDE has one repository, GNOME has a number of separate repositories that could cause some problems for analysis. Case studies shouldn't be generalized anyway.

----

2. "Peeling the Onion" - Masmoudi, den Besten, de Loupy, Dalle

Using text mining on Bugzilla data, this is problem-solving data, with a focus on organizational processes. Using textual data with metadata.

Importance of metadata, hasn't been used much yet for examining community organization. Used core/periphery to simplify the model for the current analysis.

Preliminary results - more work needed as a number of questions are opened up by their approach. Specific words based on "insider" (core), "outsider", and people contributing at least one unconfirmed bug. Insiders use "we" the most! Outsiders use "I" and "my" while no other roles do this. This supports our group maintenance results with inclusive pronouns.

Typical peripheral activities: duplicate busting, bug reporting, creating attachments. But pure text mining wasn't enough, so they add in metadata about actions. Peripheral contributions: listen to discussion, provide contextual elements, eliminate dupes, don't interfere with technical metadata management, priority management.

"Wedding Table" effect - either you know people at the table or not, and if you don't know the people who already know each other, it's hard to join the conversation.

One kind of action with a negative correlation between core-periphery - QA contact.

Comments:

So if you look at the text of a message you may be able to identify core/periphery status?

----

3. Group Maintenance - Scialdone et al.

Comments:

Some guy: How did your coding deal with investigating artifacts? Acceptance policies - strict or lenient - as a factor influencing group maintenance behavior. Investigating byproducts of interaction rather than contributors?

Response - Does not deal with code acceptance or rejection, just email messages. Valid concern - seeing an artifact that is an outcome of other group norms, or is what we're observing the cause of success? We don't claim it causes success, we need more data on other factors, but it seems to suggest a relationship.

Olivier: What were the origins of the participants and their level of mastery of English? Negative politeness and so on should only be studied within native speakers because those with poor English skills may not have that mastery.

Response: The group is not homogeneous, not all Americans or Brits, so that is a challenge - saw many times foreigners with broken English apologizing profusely for their poor language skills. It could have affected apologies if it was coded, they tend to apologize more, and they tend to be in the periphery because language skills could prevent joining the core. Saw no instances of misunderstandings due to language barriers.

Sulayman: How do you identify core and peripheral participants - use developer mailing list, use other mailing lists?

Response: Used whole developer mailing list, but project's listings of members were the core.

Some guy: This is using a bug report?

Response: This was the developer mailing list, wanted to capture decision-making.

Some guy again: Another weird question that doesn't make much sense to me.

John Knoll: Just a clarification - suddenly confused - what do you mean by group maintenance?

Response: Behaviors between members that keep the group together - maintenance of the social group, not the code.

Jean-Michel Dalle: Suggestion - one of the main findings is that peripheral member in Fire were less polite, both positively and negatively. If the figures mean something, they have very low frequency, this could be interpreted as peripheral members paying less respect to the core in Fire than in Gaim. I think this supports applying this to a larger set of projects, would do a lot toward validating applicability and evaluating other characteristics.

Response: I would not say that the periphery had less respect, but that they didn't feel like they were part of the group, more distant. What's your recommendation?

JMD: Apply to more projects to validate.

Sulayman: This is one of the measures you can add to evaluate success.

-------------

5 June 2009

Session 1 - Empirical Research on OSS

1. Reporting empirical research in OSS: the state of practice - Stol & Ali Babar

RQ1: What has been the focus of empirical studies of OSS?

RQ2: What is quality of reporting? High number of studies scored well on several factors; most scored poorly on: motivation of the study, research design, justification for sampling, relation between researchers and object of study (reflexivity), and limitations of the study. Reflexivity - a number of researchers are involved in the projects they study, and need to indicate that relationship.

RQ3: How can we improve the quality of reporting? Motivation of the study: set the scope of the work; Research design: research method != data collection method, selection of research method based on the problem being researched. Justify research design; describe the sample and why the sample was selection; data collection methods have limitations and these must be described so that the work can be evaluated. Research context is important for interpreting results; researcher relationship to object of study; and study limitations to evaluate the validity.

Limitations of the review: from the first 4 OSS conferences - working to extend this collection. Selection was performed by a single researcher but cross-checked with another, leading to acceptable inter-rater reliability.

Comments:

Paul - How did you do categorization? In our future work we will come up with a more detailed taxonomy.

Olivier - suggestion to allow tagging for categorization?

To what extent are the categories different from software engineering in a more general context - not only the categories but also the results?

----

2. What does it take to develop a million lines of open source code? - Fernandez-Ramil, Izquierdo-Cortazar, & Mens

Cost estimation is a long-standing research topic in software engineering, but primarily in proprietary software. Currently OSS communities don't seem to use effort estimation models - may be useful for comparing projects, providing basis for baselines to evaluate changes.

Looking at this from the perspective of size of code base, combined with effort, duration, and core team size. Initial exploration, more to be done. Initial findings - some big "jumps" in growth trends in 9 of 11 projects. They look like step functions.

----

3. An empirical study of the reuse of software licensed under the GNU GPL - German & Gonzalez-Barahona

Goal was to understand what people are doing when they use GPL software with non-GPL software. Methods to "circumvent" apparent restrictions of GPL when creating derivative works - complex interactions of licenses.

Dependency analysis in distributions, e.g. Debian - found GPL components required by non-GPL components. Describes derivative works and license compatibility. Number of strategies, e.g. making it "compatible" for others to use, such as mysql client libraries.

Another strategy is licensing as - one component "licenses as" another, usually the main component, e.g. Perl modules.

Attempting to classify schemas for license compatibility; as far they know, none have been tested in court but are probably sound; some authors clearly have interest in allowing certain types of compatibility.

------

Session 2 - User Involvement in OSS

1. FLOSS UX design: An analysis of user experience design in Firefox and OpenOffice.org - Bach & Carroll

UX as combination of HCI, HFE, UX, ID, IA, UID - more from the design arts

Motivation: sociotechnical structures for trust, merit & participation; FLOSS is unfamiliar environment for UX designers

Theory: activity awareness - common ground + CoPs + social capital + human development; methods = virtual ethnography

Firefox & OOo have employed UX professionals for end-user apps, plus obvious UX strategies

Common ground issues: different knowledge sets in community - clear evidence of misunderstandings
Social capital: build across team groups
Human development: evidence of learning in blog comments; reflection and evaluation

Variation across cases - particularly with respect to design strategies and control of the work. Evidence of user research is not particularly apparent, but OOo is moving in that direction.

Comments:

Brian: Do either of those two areas use focus groups or anything like that as an input into what they do? Do they ever watch complete newbies use their software?
Response: Did not see evidence of this - some evidence of this on Firefox with certain components. OOo's way to do that was the user survey - a different way to get the info.

Some guy: Do you have any recommendations for the Firefox and OOo developers? What's the take-away? Are these good or bad ways of working?
Response: Hard to make a judgment - it's a new space. Would see Firefox as more successful, but was very focused, strategic and aggressive about UX integration. OOo seemed more ad hoc; it's almost easier to go into one product than a whole suite. The take-away depends on the kind of application and what the community looks like. If it's more like Firefox, and you can do a strategic team approach, that can work well. Thinks that in these cases the benevolent dictatorship looked more effective.

Some woman: Firefox has also done a great job of focusing on UX, whereas OOo's UX people aren't very well integrated into the development team and process.

Brian: Thinks this is very useful.

----

2. Integrating HCI specialists into OSS development projects - Hedberg & Iivari

There is room to improve OSS usability, but involving users in OSS is challenging. Non-technical users don't have the same background from which to contribute. Focus is on project structure and the integration of UX.

Usability + decisionmaking + HCI -> proposed model -> evaluation

Lots of high quality OSS, but generally usability is weak; produced by engineers for engineers; overrated expectation of user's technical skills or no knowledge of users at all; non-technical end-users aren't able to participate in the usual communication channels. Companies may be able to ease the integration of HCI into OSS, which is otherwise challenging.

-----

Session 3 - Educational usage of OSS

1. Using FLOSS project metadata in the undergraduate classroom - Squire & Duvall

FLOSSmole metadata about open source projects used for courses in databases, information architecture, and artificial intelligence. Starts students with small data sets, then graduates them to large ones when they learn SQL querying.

Query example 1: show growth in new project registrations at SF over time, or how has the use of "free" and "open" in project names changed over time. Goals: learn how to write good queries, use the data that they gathered, learn something about FLOSS. The free/open query helps teach about open source history.

Query example 2: compare team sizes on each repository; show how many developers per projects and graph them. Students get curious which project is the biggest one -

Data integration example: How many projects on each repository share a name with a project on another repository? Have to write sql queries, design diagram to show pair-wise repository comparisons & number of shared project names.

Data integration 2: How many projects on each repository list a project URL in common? Learning data integration when they compare the contents of multiple repositories.

Next part: undergraduate research 1

Data collection - spidering, polite and obeying robots.txt, learning about software engineering by comparing different forges. Also starting to collect non-forge data, Apache. Have to think through how that will work and how to integrate the data.

Undergrad research 2

User-facing documentation, data displays, and data formats. Especially in combination with querying. Student creating charts with Processing Engine, which users can consult. Wants to get students working on data formats, e.g. DOAP.

Undergrad research 3

AI applications - extending ID3 - duplicate identification.

Comments:

Anecdotally, requests for direct database access at Teragrid is also primarily students. Also could have students work on the data release and generation processes.

Sulayman: God bless you. Thinks this is very important for FLOSSmole. What he would like next (for others) with wiki attached to query tool, so students' queries can be pasted - researchers could use them.

Does FLOSSmole store release data?
Megan has a student analyzing release histories. We would take the donated data if people have it, would love to have that. Issues of integration and attribution. Now have a schema!

Sulayman: As there is more data, this becomes really useful for researchers.

Question about mailing lists - requires lots of data and lots of spidering.

Asked SF about data on deprecated features because they're using hosted apps. Asked for data from old stuff, said no. Takes 10 days for SF runs now.

Sulayman: SF trackers producing more interesting info that Bugzilla.

----

2. Undergraduate research opportunities in OSS - Boldyreff, Munro, Knowles & Capiluppi

Two undergrad research projects - CODEX and SoMOSS. Looking for suggestions on future work.

Most SE students deal with toy examples - Java classes, unrealistic UML diagrams. In SE degree, students have to study processes in one/two/three FLOSS systems. Define how conflicts are resolved in decision-making, make contributions to the project. UROS projects are summer work - using this as a way to "filter" for PhD students - building on their prior course work.

Role of web-based communities - the label community is used everywhere in forges. Truly web-based communities, software development is a CSCW for SE.

Case study project - OLPC - software is based on OSS with Sugar interface. Student project, became a Sugar package maintainer for Ubuntu. Student has confidence with wider community, has built community relationships, now planning Unix distribution and release management. Student actively blogs his activities, documentation, reporting.

CODEX - development of "serious games" for the XO, extra work with git-hub, and more research on OSS communities and development. Main message is to be effective with giving back to the community.

Comments:

Sulayman: Evangelism - welcome to our workshop tomorrow. Particularly interested in getting students involved in SE, this is done in the context of a course. First question, how do you manage creating the right environment? Was going to ask how you evaluate it, but wants to know how CS departments react to this.

Response: In terms of assessment, they are already doing that in a CS department. The difficult part is how to scope the work - some do nothing, some do design documents and give them to community, others do more interesting things. They maybe fix bugs or creating a plugin for Pidgin/Gaim. Then leads to asking why creating a plug-in is so easy, leads to them exploring the modularity of the software structure.

S: Wants to align this to with ACM/IEEE software engineering curriculum. Traditional resistant profs already in place, hard to get them to teach with OSS; wants to learn how students moved from traditional software engineering classes to the OSS-based engineering. In Greece, they have an exam with traditional SE elements, then 40% is project work - it can either be OSS or the traditional thing with toy programs. Initially poor response, now everyone is going to OSS. Divided students into groups for coding, documentation, and testing - this is the most popular. Trying to run both traditional and OSS in parallel.

Response: BCS, British Computer Science, is creating exams based on OSS with Cornelia so that the official exams include OSS. This year was the first time they did the exam.

Kris: Are research projects extracurricular? (yes) How do you motivate them? (money)

Response: In year 3, they have to do a project, they continue forward with the work from the prior years.

Kris: Does it raise interest in more research? (yes, both are PhD students)

S: What does the group think about assessment? How do you create student participation in OSS? Can grade based on bugs reported and the like, but is this the most useful way to evaluate? Trying to do group assessment by more senior students, but junior students complain. Students pick their own projects, but many have a hard time finding an "interesting" project, many ultimately pick games because of familiarity.