Hi! This is a long blog post, but it has a few important action items at its conclusion. If you care about the evolution of the Chef Community, or are considering going to the Community Summit, you should totally read it.
Here are the actionable parts:
- If you like building Chef (through contributing, advocating, or advising), I invite you to join us in #chef on irc.freenode.net.
- We will have a bi-weekly meeting of developers in #chef-hacking, moderated by myself and Nathen Harvey. The first is scheduled for 9am PDT on July 11, 2014. An agenda will be sent to the chef-dev list the evening before.
- We will develop an explicit process for how the project is managed, as an RFC on Github, which we will formalize at the Community Summit in October. Topics we will cover include: how a patch gets approved, how to become a maintainer that approves patches, how disputes are resolved, how the roadmap is built, and more. This process begins at our first meeting.
Where Chef came from
Chef was created six years ago – the first commit was on March 5, 2008. At that time, my friends and I had created a small consulting firm called HJK Solutions. Our business model was pretty simple: we created fully automated infrastructures for startups. They would pay an up-front fee, we would automate everything, and then take a monthly retainer to help run their operations. It was good work, but hard – we were struggling to maintain anything sane about the code base we used to manage 15 totally different infrastructures for 15 totally different clients.
When we created Chef, our first objective was to see if we could solve our own maintenance problem. Could we actually maintain that many different infrastructures with a relatively small team? Could we do it and stay sane? It was either build Chef, and see if it helped solve our problems, or go get other jobs.
The rise of Chef and the transformation of HJK into Opscode (and now Chef Software) were and continue to be inextricably linked together. There was a company before there was software, and the work of building that software is completely intertwined with that of the company.
Open Source software creates better outcomes
From the first lines of the Chef codebase, I wanted it to be open source. Our job was building an automation tool that would power our clients’ infrastructure – and as someone who was already 10 years in to my career at that point, I had no interest in helping people run a company on software that was not open source. One of the things I had come to understand from running HJK was that, while I was amazing at building automation, only the people inside an organization understood their problems deeply enough to come up with the best solutions to them. I understood the fundamentals of how to build great infrastructure, and through teaching those foundations to others we got great results. Translated to Chef, it meant that we needed to build something that was right from our point of view, and that empowered its users to build what they needed: and that meant Chef had to be open source.
Rise of the Builders
As soon as Chef worked at all, I showed it to my friend AJ Christensen. We had worked on some hard problems together, and I knew he understood my point of view on what a better solution would be. His first commit came in September of 2008, and at that moment, it wasn’t just HJK working on Chef. We were soon joined by others: Bryan McLellan (1 day after AJ), Ezra Zygmuntowicz (in October, and the first company to use Chef commercially – Engine Yard), and Joshua Timberman (the first HJK employee other than me to ever commit).
While AJ, Bryan, Ezra and I worked to get Chef ready for the world, our consulting company hooked up with Jesse Robbins. We created the plan for what would become Opscode – a Hosted Configuration Management service, with an Open Source reference implementation. Jesse announced Chef, and Opscode, to the world on January 15, 2009.
Meanwhile, more people started to hack on Chef. New functionality was added daily, sometimes multiple times a day. Thom May, Sean Cribbs, Joshua Sierles, and Matthew Landauer all joined within 24 hours of each other in February, as did our first Opscode committer and long time engineering leader, Christopher Brown.
This group all existed inside of the #chef IRC channel on irc.freenode.net. Conversations, requests for code review, and requests to release a new version would all happen there. As someone who builds software for a living, this was probably my favorite time in my entire career: suddenly a global community of like-minded people were sharing my passion for creating Chef. It was awesome, it grew quickly, it was very close-knit, and it stayed this way for years. (Dan DeLeo would go from community builder in this era to the most prolific Chef author ever).
It was also completely implicit. We all hung out in IRC, Christopher or I would make decisions when necessary. Sometimes we passed the baton for things like merging changes or cutting releases. It worked because we were small, and we taught newcomers the ropes. If we had disagreements, we worked them out.
Arrival of the Early Adopters
Around this time we got our first real early adopters – people who , even though Chef was rough around the edges, were using it for real work. Engine Yard certainly counts here, as did early folks like Mark Imbriaco.
They fixed bugs, they even sometimes added functionality – but for the most part, they used Chef, and they gave feedback about their experiences to those of us who were building it. They were integral to our growth, and to the feeling like we were doing something truly useful. They didn’t mind the rough edges much, as long as it worked for the core of what they were trying to do. And their feedback was invaluable.
You can also see the rise of the Chef mailing list in this phase of our growth. With the addition of early adopters, we had to communicate outside of real time, and we could not guarantee that everyone was always running an IRC bouncer with a backlog.
While the early adopters might not have contributed as much code as the early builders, they still felt like them. They understood the same concerns, and they were willing to dig in if they needed it.
These early builders and early adopters also became our most vocal evangelists – they told others about Chef, and patiently taught and explained to anyone who would listen about how use it to build awesome infrastructure.
Growth means Consumers
It wasn’t long before the mix of high quality software and first rate evangelism caused consumers to appear. These folks wanted to just use Chef. Rough edges were not an invitation to join an IRC channel and help polish things up – they were reasons to choose different software.
This was great for Chef as a project. The consumers pushed the community to document more, to make installation easier, and to fix confusing end user experiences. As builders and early adopters, we were fine with the warts: they were, after all, our warts. It was the consumers that pushed us to actually turn from an inward facing group solving just our problems, to a fully outward facing one that tried to solve them for others.
It may go without saying, but consumers were also great for Chef Software as an open source company. Consumers pay money for your software, if you provide enough value. Over time, the consumers grew to out-number the builders and early adopters. Thousands of companies, all over the world, began building with and relying on Chef. Increasingly more of those were also paying commercial customers of Chef Software.
The birth of an ecosystem
You could see the impact of this new class of needs in the software, and in the way the builders were contributing. The focus shifted from new builders fixing Chef (the core functionality was getting quite stable, or the consumers wouldn’t have come at all) to creating more and more things to fill niches in the ecosystem. This started with the huge rise in community cookbooks, resources, and providers. It continues today with the huge amount of great software that surrounds Chef: Test Kitchen, Berkshelf, Librarian-Chef, goiardi. Builders will build, and they have built one of the strongest ecosystems in Open Source – both other open source tools and commercial offerings that include Chef.
Chef Software grows, too
Meanwhile, Chef Software took on many of the tasks required to satisfy our growing (woot!) customer base. Maintenance, backward compatibility, ease of use, bug fixes, and scalability. It’s a symbiotic relationship – as more people use Chef, more people build Chef, and Chef gets better: Chef Software grows. We sharpened our customer focus on making the experience of Chef more appealing to new users.
A sidebar on communities
I think it’s worth pointing out that people make up a community, not corporations. Companies, even those that are very well managed, often show emergent properties – it is difficult to predict how they will behave in a given circumstance. Unlike a human, a company feels nothing for anyone; it simply acts as an emergent outcome of those who participate in its day-to-day existence.
While Chef has had a company behind it from the beginning, it was never the company that was actually participating in the community of builders. It was the people who were doing it – it was me, Christopher Brown, Joshua Timberman, Dan DeLeo, and Bryan McLellan (and many, many others.) Our bills may have been paid by the company (and it may well have been in our company’s best interest) but it was us, as people, who prioritized that engagement. The same continues today with people like Seth Vargo, who is one of our community’s most prolific contributors – and happens to be employed at Chef Software.
This is important to fostering a healthy community. It does not matter where you work or what you do: when it comes to participating in the community, we are all equals. We give a piece of ourselves to each other in a spirit of fellowship and common interest. If we want to be respected and heard inside that community, it would be that gift of ourselves that creates it, not the origin of our paycheck.
Healthy communities, in open source as in the rest of life, are stronger than simply the original reason they came together. They grow and change with the people who make them up, and as the industry they participate in evolves. The community is and always will be bigger than any single individual. This is what makes communities so powerful and resilient.
However, a common source of pain and anguish is a lack of perspective. It is easy to think, as an individual, that the community comes from you – that if your role were to ever change, the community would fall apart. That may be true in a small, or weak community – it’s not true in a large, healthy one: communities will find ways to support themselves.
The Chef community, as a group composed of builders, early adopters, and consumers, is incredibly diverse and extraordinarily healthy. You see it every time we get together, and you can feel it when something happens (good or bad) to any of us. We have come together with a common cause, regardless of the roles we decide to play within that community.
The failure of implicit organizing
We did, however, lose something as we grew. In the space between being a small group of like-minded builders hacking on the solution to our problems, and being a thriving community with thousands of active members with a hugely diverse set of motivations, we allowed our implicit process to wither away. As the humans involved in it have grown and changed, the implicit sets of behaviors the builders relied on to get work done went with them.
The side-effect has been a community that is supportive, innovative, and easy to work with in many contexts. At the same time, it is potentially confusing and occasionally byzantine in the middle where the builders lived according to an atrophied, implicit code of conduct born of our earliest, swashbuckling roots.
Lets get explicit
I recently had the opportunity to talk about how to build open source communities at DockerCon. It was a fun talk to construct, because it let me use the benefit of hindsight to help another fast-growing community of dedicated builders not make some of the mistakes we did. I was asked what things I would have done differently in building the Chef community, and my answer was simple:
I would have gotten much more explicit about how we build Chef at the same time as we got so much better at serving the needs of our growing consumer base.
(I’m paraphrasing – there is probably video of my actual answer, but hey – that was my point.)
I boiled down what I have come to believe builds a healthy community to five things:
- A clear social contract helps align our interests, and allows us to evolve over time
- Transparency allows us to make decisions quickly, and create systems when they are necessary
- Inclusiveness encourages diversity, and from diversity we gain strength
- Trust is earned through small, consistent actions and clarity of communication
- People join the community for different reasons, and we accept them on their terms
I am going to lead us, along with our Community Director, Nathen Harvey, on a journey as a community to get explicit about these five things. We will work to make clear the process by which everyone who joins us can participate: as a builder, as an early adopter, and as a consumer.
It is long overdue.
Next steps
- If you like building Chef (through contributing, advocating, or advising), I invite you to join us in #chef on irc.freenode.net.
- We will have a bi-weekly meeting of developers in #chef-hacking, moderated by myself and Nathen Harvey. The first is scheduled for 9am PDT on July 11, 2014. An agenda will be sent to the chef-dev list the evening before.
- We will develop an explicit process for how the project is managed, as an RFC on Github, which we will formalize at the Community Summit in October. Topics we will cover include: how a patch gets approved, how to become a maintainer that approves patches, how disputes are resolved, how the roadmap is built, and more. This process begins at our first meeting.
Thank you for reading this far, and for caring about how the Chef community grows and evolves. See you on IRC.
Adam