This post is part 4 of a 4 part series. Read Part 1 for an overview of the current state of the ecosystem, Part 2 on Policyfiles, and Part 3 on Push Jobs.
I spent a good portion of my career tackling provisioning problems. Chances are, if you met me at a Chef event, we probably talked about Chef Provisioning. Before diving in any further, I’d like to point out that Chef works well with just about any modern provisioning tool. Bootstrapping a Chef node is straightforward and a number of clean abstractions for those steps exist in various forms when you look across the landscape of provisioning tools. Many options exist for provisioning your infrastructure and which of those is right for you depends on a number of factors. In this post, I’m only going to cover the Chef Provisioning framework.
Chef recipes allow you to compose your application infrastructure in flexible and powerful ways. We spend time testing and perfecting our recipe code by applying it to a single test node. We end up with a robust code base that can build or rebuild our application infrastructure as needed. Then, it comes time to do just that: let’s build our application stack on a cluster of new machines. Okay, but how should we do that? Knife plugins only focus on single machines. We can wrap knife in brittle scripts, use some homegrown code to do the job, or we can learn how to use a new tool.
The Chef Provisioning framework grew out of a desire to provide Chef users a way to manage entire clusters of machines by extending the language and practices that they already know: Chef recipes. To do that, Chef Provisioning provides new base resources to manage common infrastructure components (machine and load_balancer), resources to manage basic orchestration (machine_file and machine_execute), and resources to manage provisioning-specific tasks (machine_batch and machine_image). Beyond those shared strategies, Chef Provisioning uses a plugin architecture (drivers) to implement specific strategies unique to various infrastructure ecosystems.
If you’re used to writing Chef cookbooks, using recipes and resources to manage your infrastructure makes a lot of sense. As with your Chef managed nodes, with Chef Provisioning your entire infrastructure is now testable, versionable, and convergent. Chef Provisioning provides an abstraction layer to simplify complexity by providing a common interface across multiple ecosystems.
Drivers for Chef Provisioning provide infrastructure-specific functionality for individual providers. The three most active drivers for Chef Provisioning manage infrastructure in AWS, Azure, and via fog (the Ruby multi-cloud services library). Each driver provides a different subset of customization options for shared resources (e.g. machine) along with custom resources for the specific needs of the ecosystem. For example, the AWS driver provides resources to manage VPCs, Auto Scaling Groups, RDS instances, Elasticsearch domains, IAM roles, and lots more. By using these custom resources, it’s possible for users to manage their production application stacks from the same interface by which they manage server configurations: Chef recipes.
Chef Provisioning not only provisions your infrastructure, it will also register and deregister machines against your Chef server as appropriate. Chef Provisioning is open-source and additional drivers (some maintained by the Chef community) are available for use. Chef Provisioning is a community driven project. Open, weekly office hours for Chef Provisioning have been running since 2015 (thanks, Tyler!) and you can always pop into the the Chef community slack channel to discuss Chef Provisioning.
If you’re not used to writing Chef cookbooks, using recipes and resources to manage your infrastructure adds in a lot of complexity. If you’re not already a savvy Chef user, you’ll likely have a hard time with this community-driven project. Depending on the driver, docs aren’t always up to date and often times you may find yourself digging into code internals to figure out how something works or to debug obscure issues.
Chef Provisioning drivers also have varying degrees of activity. For example, the Vagrant and SSH-based provisioning drivers haven’t been actively maintained. Yet, the driver for VMware vRealize Automation was just recently released. Some, like the AWS driver, have received a lot of attention and have a full CI testing pipeline set up. Referring back to our original post in this series, Chef Software Inc. will support use of Chef Provisioning and its drivers, but this is not where we are currently investing development resources. Chef Provisioning relies on support from the community for the development of new features.
Providing an abstraction layer to simplify complexity across multiple infrastructure providers has also proven to be elusive. Individual infrastructure ecosystems are each becoming increasingly complex (with no signs of slowing) and their management cannot be effectively hidden behind one single interface. The complexity of individual infrastructure providers ultimately bubbles up to the surface. As a result, the single interface itself becomes an additional piece of complexity. In the end, Chef Provisioning does not reduce complexity as much as it provides a familiar interface.
Should you be using Chef Provisioning to manage your infrastructure? You should understand the benefits and cost before making that decision. Successful users of Chef Provisioning practice the following behaviors:
Chef Provisioning fits the needs of a specific subset of Chef users. This is not a definitive list, but rather a guideline to help you more readily determine if you can be successful with Chef Provisioning. However, for example, you may be comfortable using Chef Provisioning with a less maintained driver because it meets your needs sufficiently. That works too.
Chef Provisioning has interesting applications and I personally, like others in the Chef community, find many of the patterns postulated by its convergent infrastructure approach fascinating and useful. But as we’ve seen above, there are some significant hurdles for adoption and a significant amount of work is necessary to help drive this community project forward.
Chef Provisioning could benefit from a substantial code refactor. While a wealth of coverage exists for native cloud provider services in AWS via custom resources, that’s a perpetually moving target and additional resources would expand the reach of use cases Chef Provisioning can help solve. If you’d like to help develop Chef Provisioning further, engage with the Chef Provisioning community to explore options for doing so.
Chef Provisioning has a small, but passionate, following in the Chef community. If you’d like to learn more about how you can use Chef Provisioning successfully, consider joining the #chef-provisioning channel in the Chef community slack.
All of the projects discussed in this series (Policyfiles, Push Jobs, and Chef Provisioning) are not currently slated for further development by Chef Software, Inc. For clarity, these tools are fully supported by Chef and your investments in these projects are safe. This series is meant to help you understand their current benefits and drawbacks so you can clearly decide if these tools are right for you.
Chef is an open-source company and we are dedicated to responding to the needs of our customers. While we do not expect these tools to receive new features at this time, new features may be implemented if they’re contributed by the community. If you’re an avid user of these tools and would like to see them developed further, please engage with their user communities and with our Product team so that we can explore your use case and options together.
We hope this series has provided clarity for the current state of these tools. If you have additional questions or concerns, please feel free to reach out. Find me on Twitter or the Chef community slack.
Many thanks to the contributors in this series: Megan Gleason, Charles Johnson, Julian Dunn, Tyler Ball, Marc Anderson, Tim Smith, Thom May, Shanku Niyogi, Adam Jacob, Nathen Harvey, Yvo Van Doorn, Nell Shamrell-Harrington, and Stuart Preston.