Chef and Orchestration

When talking about the management of complex systems, orchestration is always a hot topic. This is because orchestration is often seen as the easiest way to represent and model complex systems, as well as provide a path to delivering complex systems.

Most often orchestration is represented through a topology model. What is a topology model you ask? A description of the order-of-operations across a group of machines. A common example is provisioning a database, cache layer, multiple application servers, web servers, and load balancer(s). This model will include distinct technology components that must interact, are interdependent, and more often than not the provisioning is accomplished through a very specific order.

Although orchestration solutions have been around for a long time, very rarely have they fit the bill. This is one of the reasons why Chef has opted to take a different approach, which we see proving more functional in the real world. In other words, we’ve worked hard to create a solution that actually solves the problem consistently in a wide range of complex, real world implementations.

At the core of ensuring a robust orchestration mechanism is following the principles that have made our core product Chef great: scale, idempotency, flexibility, and test-driven automation.

In this post we’re going to take a look at how Chef can make orchestrating complex systems easier, faster, and more reliable.

For those that might be relatively new to Chef let’s do a quick recap of some of the core principles.

Here’s how this diagram breaks down:

1. Having an open syntax to provide flexibility and consistency to how the product may need to be configured. Whether at a single system / application instance level or in the case of multi-component orchestration this is critical.
2. Driving everything from a single source of truth. Source control has been the tried and true de facto standard for modeling, tracking, testing, and allowing for visibility and collaboration on state.
3. Desired state management as well as providing a consistent mechanism for applying change (idempotency) so that getting and tracking execution of state is intuitive.

So how do we achieve orchestration with Chef? An extension to Chef known as chef-metal allows you to use the core Chef principles to manage other aspects of your infrastructure. In this example we will use AWS as a platform, however, we are continually expanding our overarching orchestration concepts across different platforms.

As you’ll see chef-metal allows us to manage core AWS components such as ELBs, SQS queues, compute resources, and more. Chef-metal makes it extremely simple to provision these core AWS components, as well as ensure that they stay within compliance to the policy you define with Chef.

As always, everything starts from a recipe (policy).

1. In this case we can define our multi-component service and the order it should be provisioned in. Very intuitive and obvious. We’ve defined everything within a datacenter and start with creating an “sqs queue” named mariopipes.
2. We define our machine bowser along with optional bootstrap and dynamic configuration parameters

1. Next we define our load balancer webapp-elb and options such as availability zone, protocol and ports.
2. We include the machine bowser defined in our previous step to be added to the load balancer.

1. Lastly we’ve included some additional services such as our “sqs queue” named luigipipe and “sns topic” named us\_west\_topic.

This is just one example of how chef-metal can enable you to orchestrate complex systems. At the core of ensuring a robust orchestration mechanism is following the principles that have made our core product Chef great: scale, idempotency, flexibility and test driven automation.

Reach out directly to our partner team for feedback and questions at bizdev@getchef.com and don’t be afraid to test and collaborate on the development, which you can find here

If you are in the Bay Area, Chef is partnering with AWS in the Pop-up Loft in downtown San Francisco. Join us for free technical training courses and one-on-one technical appointments where we’d be happy to discuss Chef-metal in more detail.