Hello. I am the manager of the Mesa i965 team at Intel® Corporation, a privilege I have had since 2011. In this series of blogs, I plan to talk about my experiences as a software manager working with a pioneering team of open source graphics driver developers to push the state of the art in user mode open source graphics drivers. A lot has changed since I joined the team 6 years ago, and given what the Mesa community has accomplished recently, I thought it would be a good time to share some of what I’ve learned from my team and the community over these years. To go along with this introduction, I will be presenting an overview of the state of Mesa i965 at the X.Org Developer’s Conference (XDC 2017).
I’d like to emphasize that all the opinions expressed here are my own and do not represent the views of my employer, Intel Corporation.
Let’s start with the name of the blog, which is a play on a quote from Linus Torvalds in 2006 (see sidebar). Often when I talk to people new to open source methodologies, I notice that it almost sounds like magic to them. It’s either unbelievable or sounds too unrealistic to be true. So this is my attempt to describe the “alchemy” of open source development as we practice it in developing Mesa i965.
One reason that got me to think about writing this blog is that it became increasingly clear to me that I have had a relatively rare experience in being part of a team with (what I consider to be) historical achievements in open source software development for commercial products. If you’re a regular reader of sites such as Phoronix, you might already be aware of those accomplishments. If not, you might be surprised to learn about a (very) quiet revolution taking place in the world of Linux 3D graphics drivers. Graphics is one of the most important components in any client device, with the driver being a major contributor to the user experience. This component has traditionally been opaque and mysterious, as all sorts of technologies are buried in large closed source code bases. Mesa has attempted the (revolutionary) opposite approach of being completely open, running on an equally open Linux kernel and kernel graphics drivers. The revolution’s success is in having feature parity among code bases with these two starkly different approaches: the traditional closed and Mesa’s completely open source methodology. Read on for more details, but if you’re interested in the history of Mesa, be sure to read this article in LWN (published in 2013) covering recollections of “The Father of Mesa” Brian Paul, on how he started the project back in 1993.
February 3, 2017
This day will be remembered as the day that open source 3D graphics drivers really made their presence felt in the industry. These drivers had already proven to be commercially viable for years through deployment on millions of client Linux PCs and Google Chromebooks. But on this day, for the first time ever, an open source driver became an industry leader by receiving certification from the Khronos Consortium for compliance to OpenGL® 4.5, adding to its previous certifications for OpenGL® ES 3.2, and Vulkan® 1.0 completing top-spec certifications for all open 3D API standards. That driver was Mesa i965 for Intel® graphics running on upstream Linux kernel i915 driver. You can get an overview at Intel’s Linux Graphics site including my introductory blog for people who would like to get started with deploying this driver in their OS.
There are many factors that enabled Mesa to be where it is at today, whereas just a few years ago it seemed to be hopelessly behind (in spec features) in comparison to closed source drivers for leading GPUs. There are many factors that contributed to this achievement ranging from structures and processes, to commercial investment, and the nature and motivation of people and how they interact each other. I’d like to start with the one factor that is often overlooked because it’s hidden in plain sight, namely stable specifications and stringent conformance tests.
Standardization creates Stability, which allows Focus, which brings Efficiency
It’s clear to most people how standard APIs help application developers minimize the time it takes to get their apps running on different OSes and GPUs. A similar effect can be seen on the other side of the API. Mesa drivers primarily implement standard industry APIs. Compare this to many other comparably sized software components where a standard industry-defined spec is not available, not to mention any conformance tests. Focusing on agreed-upon specs removes one of the main reasons that such large projects often get bogged down in implementation. In my experience, having worked on many big projects that ended up going nowhere, one of the recurring major contributing factors was the vague or rapidly changing specification of the software that was being implemented. As the industry has gained velocity and development cycles have shrunk, this has become more of a problem especially with older waterfall models. While many agile methodologies have been devised to lessen the effect of this issue, there is no real substitute for having specs that everyone agrees to implement. The undeniable fact is that it takes time to develop complex software. If the external specifications change faster than the software can be written, the developers end up locked in an infinite loop until either some version of the feature-set is acceptable to the intended customer, or the project is cancelled. More often than not, the latter is the case. So, having specifications that basically don’t change (except by going through rigorous change management processes) is a huge productivity booster for a team that implements those specifications. For most of my involvement with the Mesa i965 team, our developers were implementing specifications that had been written years prior and did not need to worry about changes to those specs. That was a tremendous help in focusing the development work.
In the last couple of years, this has changed for our team. As we have caught up to the most advanced revisions of the 3D APIs, our team members who had always been active at Khronos in API definitions, have actually developed the code as new specs were being discussed and designed. Those spec discussions can be messy and changes are made along the way, which adds some cost in development bandwidth, but those costs are more than justified by having drivers available for day 1 of the public spec release, which is now a goal for our team and the community.
Of course this is not an advantage that only helps open source driver developers. Closed source drivers are helped by this fact as well. But if you’re way behind the industry, as we used to be, having stable specifications greatly helps in optimizing the time it takes to do the work, when you find enough bandwidth to do it.
Open Methodologies are Powerful and Efficient
The other major underlying structural factor in Mesa’s recent successes is the use of true open source methodologies. I find it ironic how little-understood a methodology based on complete transparency is, but I guess I’m conveying my experiences in the hope of demystifying some of this!
If you’ve never been part of a longstanding open source project (as I had not until I joined our team) you would be hard pressed to believe how much work can get done at what speed by small teams using the well-established open source processes. I am not at liberty to disclose the number of developers that Intel allocates to Mesa i965, but suffice it to say that it is far smaller than you would find for a comparable closed source project. Our colleagues who hear about the size of the team for the first time, usually think we’re kidding or mistaken about the scope of the project. It is only after they have interacted with us for a while and deployed Mesa in a full Linux stack, that they realize that we weren’t under-reporting the size of our team. If you’re a long time open source developer, that’s not news to you. It’s probably what you’ve been telling everyone for years! So I’m here, as a manager, to confirm that when open source processes are used properly and consistently, the efficiency and velocity of teams go well beyond anything that is usually possible with traditional closed source processes. With higher efficiency and velocity, the cost of development becomes much lower. But there are other factors as well.
To understand other factors contributing to cost-effectiveness of a sizable open source project, you have to step back and look at the project at a much higher level. You need to account for the entire initial-code-to-device path. This is the path taken from when a feature is first developed, until the time it goes out on a production device destined for an end user. On a project like Mesa, this path shows that substantial savings are derived from the fact that the cost of development and validation is shared among different companies and even volunteer developers. While Intel is the largest single contributor to the Mesa source code base, others contribute substantially as well. Much validation work gets done by enthusiasts or OS vendors that integrate the driver, all in the open. So, by having the source code open and accessible early on, the whole industry avoids a lot of duplicate development and validation effort, with each entity taking on the costs voluntarily and willingly in their own self-interest. In other words, the open source processes enable a virtuous economic cycle that lowers costs for everyone. Having also worked on multi-company closed source software projects in the past, I can tell you that the closed source world is hard pressed to duplicate the savings from this type of cost-sharing even if they choose to do joint projects. I have witnessed instances where much time is spent in figuring out who spends how much every year on what and why. Long negotiations are held on just how to do the accounting of all this. No such discussion is ever held in an open source project. If something works for the participants, they keep pushing it forward. The companies and people who contribute the most, have the most influence, as they should. If someone wants to stop contributing, the project isn’t halted. It might slow down, but it doesn’t stop as others can push it forward at the pace they choose to. Customers depending on that code also aren’t left out in the cold if investment stops as the code gets archived away, never to resurface. They can choose to continue enhancing it if they choose. So, in summary, everyone wins because they have the choice to invest or walk away without pulling the rug from under everyone else in the community and the user base.
Coming up on the next blog: Backward Compatibility, and Continuous Integration.
If you have a question or topic that you’d like addressed in future blogs: Please send me a note using the address on the Contact page.
Open Alchemy - Kaveh Nasri 