What to Do When People Start Hacking Your Open Source Culture

When organizational rules exist, people will try to game them. The Apache Foundation is a case in point.

Simon Phipps

I've previously written about the fact the Apache Software Foundation offers an exemplar of large-scale open source governance. Even with those supreme qualities, things can still go wrong. Apache offers some of the best protections for open source contributors but its mature rules can be manipulated by skilled politicians and/or determined agendas. What can we learn from their experience?

In a very small proportion of Apache projects, there have been issues that seem to arise from Apache's rules and culture being intentionally gamed. They are a non-representative fraction of the work of Apache, but they provide valuable lessons on the way open source communities can be gamed. In this article I mention two such projects: Apache Harmony, an implementation of Java SE created independently of Sun that's now in the Apache Attic, and Apache OpenOffice, one of the successors to the OpenOffice.org project that closed after Oracle bought Sun Microsystems.

All rules are gameable

I posit that any stable system of organizational rules creates a gaming space proportional to the number of rules and the length of time they have been in operation. In Apache's case, the modes of failure that result from this include the following...

Consensus can mean a blind eye

A core principle of Apache is consensus. That's a good thing. Culturally, Apache members are reluctant to obstruct consensus, believing every project has the right to choose its own path within the rules. When members do speak up—and it does happen—it is assumed that expressing dissent indicates a willingness to personally engage to address the issue.That's a fine approach for most circumstances, but it means that when the mode of failure is complex and political, Apache members can tend to opt for silence rather than speaking up with an objection, especially when the project involved is at the edge of their interests. I've spoken with many Apache Members about the few failing or failed projects, and almost all recognized serious issues with them. But none wanted to speak up—"it's a matter for the project," they say.The participants in the affected projects have a variety of motivations for living with the issues rather than addressing them, so they are reluctant to speak up for fear of fuelling further criticism. Even when they do (as a PMC Chair at AOO did once) there's a tendency to punish the messenger. Many Apache processes treat silence as approval, which is usually the right thing to do. But in the exceptional cases where there's active gaming in progress, toxic situations tend to be left for too long.

Diversity is hard to measure

The expectation of diversity can be satisfied rather too readily when there's a consensus among Apache's senior members that a project deserves to exist. Apache Harmony is a great example. While the individuals who proposed and initiated it did so with excellent motivations and represented a true diversity of affiliations, after a while, the project became an effective monoculture that remained unchallenged in Apache's governance.

The principle of recognizing individuals without reference to their affiliations meant that few questions were asked about the motivations of many of the contributors. Both IBM and Intel were known to be behind the project, and I believe many—if not all—of the apparently independent developers involved in the project were employed to satisfy contractual obligations ultimately arising from one of those companies. After the project became established, I believe producing a complete, running implementation was no longer the primary objective of those companies. It seems more likely that the goal for both companies in sustaining Apache Harmony was the business-political objective of pressuring Sun Microsystems (where I worked at the time and actively advocated on Apache's behalf for them to obtain a TCK license for Apache Harmony).

Through skilled political engagement, the employees of those companies ensured that Apache's rules were prima facie satisfied even though an objective study would have shown the project lacked both real and diverse community or the realistic expectation of full working and sustainable code. The hollowness of the community effort became inescapable when Oracle acquired Sun Microsystems and entered negotiations with IBM over Java licensing. While Sun previously worked from a weak negotiating position, Oracle's macho market modus operandi meant IBM quickly reached a new accommodation. Its direct and immediate consequence was that the contributors to Apache Harmony vanished and before long the project was consigned to the Apache Attic.

Private politics vs. transparency

While transparency is a given, in a politically-charged environment it may not actually happen in practice even if the final actions the project take are public. In some cases there may be a reliance on the private mailing lists for the project although I believe gaming prefers the NDAed safety of bilateral corporate relationships.

The engagement around Apache Harmony was clearly orchestrated behind the scenes by IBM, with only the conclusions of inter-company negotiations—and indeed technical decisions—appearing in public. For transparency to be effective, the full lifecycle of each decision needs to be under public scrutiny.

That's also why open standards need to be created in the open as well as maintained there, for example. In some cases private conversations are unavoidable, but there is great wisdom in the Apache axiom "if it didn't happen on the mailing list, it didn't happen". Even if all decisions eventually emerge in public with plausible rationalizations, gameability arises from a tolerance of the rationale and full negotiation process behind decisions being kept private.

Over-use of Apache's private mailing lists is less of a problem as every Apache member has a right to see every mailing list. When paranoia about external interference led the Apache OpenOffice project to conduct many of its conversations on the private list for the PMC, there were eventually interventions from mentors and other members. The earlier private conversations about bringing the project into Apache Incubator were of far more concern.

Affiliation has layers

To expand on one important dimension of the transparency issue, the hidden industry politics of contributors' employers can easily hide behind the convention that contributor affiliations are not relevant to Apache. That was clearly visible in the situation around Apache Harmony; it was also an important dynamic behind Apache OpenOffice.

In both cases, IBM had an undisclosed motivation to see the project proceed and be perceived as healthy. I believe they negotiated with partners to ensure that, at critical points in the lifecycle of both projects there were many contributors participating apparently independently of IBM. This gave an appearance of diversity that reinforced other biases of project participants to consider the project healthy. But in both cases when IBM stopped participating the project was left moribund.

Games are inevitable

I have to emphasize and re-emphasize that I have long been—and remain—a huge fan and advocate of the Apache Software Foundation, which has contributed massively to the growth of open source use in business. At Sun I risked my career defending and advocating for them and I'm still a supporter because of the excellence they exhibit in the large majority of projects.

At Apache's size—today it has more than 350 top-level projects and nearly 40 more in incubation—there are bound to be corner cases that expose the problems of scale, maturity, and political importance. It's unlikely Apache could do much to prevent these corner cases arising at scale; creating more rules would simply increase the gameable surface while hampering new projects. But it's important we understand the limitations of even the very best community process and incorporate those lessons into governance design for new communities as they arise.

Lessons learned

How can the issues Apache faces be addressed in governance design by others? Here are four ideas.

• Politics happens; design for it. Before the rise of open source, the Java Community Process (whatever you may think of it) was designed primarily to prevent the politics of larger companies taking the design of Java out of Sun's hands. The same forces Sun saw threatening Java will arise in any community that develops mission-critical code. Apache has superb measures in place to deal with this and even they have proven gameable, so this is clearly a hard design requirement.
• Transparency is critical, everywhere. As Justice Louis Brandeis once noted, "Sunlight is said to be the best of disinfectants; electric light the most efficient policeman." No formal process will ever embody it properly, but it's important that there's an expectation of transparency of motivation and affiliation as well as in decision-making. The journey to a decision can be as important as the decision itself and it needs remembering too. A formal process can't insist on documenting every conversation, but having an expectation of openness as Apache does probably needs backing up with an ombudsman role for those who suspect trust is being breached.
• Motivations do matter. It's not just OK to ask who people work for—it's important that everyone has a basic trust of each other's affiliations and motivations. Apache creates a "brand-free room" where everyone is treated purely on their personal choice of identity, but that leaves too much room for abuse. The Four Freedoms protect community members from each other's vested interests, but that has to be taken further when political games are afoot. You can't force people to tell you why they are there, but it's reasonable to expect it to be common knowledge and to treat those without a declaration of their extrinsic interest as suspect.
• Projects have lifecycles; plan for them. End-of-life is not the same as failure, and you can prepare for it. Apache has a fine process for doing just this. All the same, sometimes people have motivations for denial and you need to make sure you design for that denial as well as for the end of the lifecycle of the project itself and other predictable pathologies.

This article originally appeared at Meshed Insights and is reproduced here with the author's permission.