ChefDK 0.3.0 Released! Introducing Policyfiles

We’re excited to announce that ChefDK 0.3.0 is now released. You can get
it from our download page now.Today’s release contains a preview release of the new Policyfile
feature. This is a significant milestone on our journey to improve the
way you develop and distribute Chef code to manage your infrastructure.

What’s this Policyfile Stuff?

Before we talk about Policyfiles in-depth, I want to be clear that the
feature isn’t complete and some aspects of the design haven’t been
finalized yet. Though one of our goals is to
make Chef a lot easier to get started with, it’s definitely not a good
idea to start using it if you’re new to Chef. If you’re an experienced
user, we’d love for you to use it and provide feedback. If you choose to
try it, please do so in an isolated organization (or separate server, if
you’re using the 11.x Open Source Server). Aside from being good
practice when trying pre-release features, there are some specific
reasons why you need to be cautious when using the Policyfile preview
release, which we’ll describe more below.

Using Policyfiles Today

The rest of this article has a thorough explanation of what Policyfiles
are, why we made certain design decisions, and so on, but before we get
to that, let’s go though a simple example. You can download the example
code from GitHub
if you’d like to see it all at once. We’ll walk through it here:

Firstly, install ChefDK 0.3.0.

Now, create an app cookbook layout with chef‘s generator, and cd
into that directory:

[code language=”ruby”]chef generate app policyfile_demo
cd policyfile_demo
[/code]

Now create a file named Policyfile.rb with the following content:

[code language=”ruby”]name "jenkins"

default_source :community

run_list "java", "jenkins::master", "recipe[policyfile_demo]"

cookbook "policyfile_demo", path: "cookbooks/policyfile_demo"
[/code]

Run chef install. This will solve dependency constraints, install 3rd
party cookbooks to your cache, and emit a Policyfile.lock.json file.
If you inspect this file, you’ll see that it contains information about
each cookbook used, including its name, version, source, content hash,
and more.

To see the policyfile applied to a node, you’ll need a Chef Server
organization you can use for demo purposes. The example repo includes a
knife config file
you can use with knife serve to set up a temporary server with Chef
Zero, or you can provide your own.

To upload your policy, run chef push demo. ChefDK uses your normal
knife config file by default, use -c PATH to specify an alternate if
necessary. You should see output like:

[code language=”ruby”]WARN: Uploading policy to policy group demo in compatibility mode
WARN: Uploading cookbooks using semver compat mode
Uploaded policyfile_demo 0.1.0 (f04cc40f)
Uploaded java 1.28.0 (299d981f)
Uploaded jenkins 2.1.2 (f5d46bcd)
Uploaded apt 2.6.0 (278be58f)
Uploaded runit 1.5.10 (b400659b)
Uploaded build-essential 2.0.6 (88a112f9)
Uploaded yum-epel 0.5.1 (631202c9)
Uploaded yum 3.3.2 (0a0d0426)
Policy uploaded as data bag item policyfiles/jenkins-demo
[/code]

You can inspect the uploaded data with knife if you’re curious.

To configure chef-client for Policyfile mode, add the following
directives to the client config file:

[code language=”ruby”]use_policyfile true
deployment_group ‘jenkins-demo’
[/code]

The use_policyfile true directive enables policyfile mode and the
deployment_group 'jenkins-demo' part specifies both the Policy name
(“jenkins”) and the policy group (“demo”) together.

There’s currently a bug in chef-client policyfile mode for which the
fix hasn’t been released. The next minor release of Chef 11, and an
upcoming Chef 12 preview release will contain the fix. To patch it,
change line 160 of
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.2/lib/chef/policy_builder/policyfile.rb
from

[code language=”ruby”]Chef::Cookbook::FileVendor.on_create { |manifest| Chef::Cookbook::RemoteFileVendor.new(manifest, api_service) }
[/code]

to

[code language=”ruby”]Chef::Cookbook::FileVendor.on_create { |manifest| Chef::Cookbook::RemoteFileVendor.new(manifest, http_api) }
[/code]

Now you can run chef-client to apply your policy to the node. Chef
will run as normal, except:
* It uses the run list from the Policyfile.lock.json
* It downloads the cookbooks specified by the Policyfile.lock.json. No
dependencies are computed at run time.

Ok, Tell Me More

The Policyfile is a Chef feature that allows you to specify precisely
which cookbook revisions chef-client should use, and which recipes
should be applied, via a single document. This document is uploaded to
the Chef Server, where it is associated with a group of nodes. When
these nodes run chef-client they fetch the cookbooks specified by the
policyfile, and run the recipes specified by the policyfile’s run list.
A given revision of a policyfile can be promoted through deployment
stages to safely and reliably deploy new configuration to your
infrastructure.

A Policyfile.rb is a Ruby DSL where you specify a run_list and tell
ChefDK where to find the cookbooks. It looks like this:

[code language=”ruby”]# Policyfile.rb
name "jenkins"

default_source :community

run_list "java", "jenkins::master", "recipe[policyfile_demo]"

cookbook "policyfile_demo", path: "cookbooks/policyfile_demo"
[/code]

When you run chef install, ChefDK caches any necessary cookbooks and
emits a Policyfile.lock.json that describes the versions of cookbooks
in use, a hash of the cookbooks’ content, the cookbooks’ sources, and
other relevant data. It looks like this (content snipped for brevity):

[code language=”ruby”]{
"name": "jenkins",
"run_list": [
"recipe[java]", "recipe[jenkins::master]", "recipe[policyfile_demo::default]" ], "cookbook_locks": { "policyfile_demo": { "version": "0.1.0", "identifier": "f04cc40faf628253fe7d9566d66a1733fb1afbe9", "dotted_decimal_identifier": "67638399371010690.23642238397896298.25512023620585", "source": "cookbooks/policyfile_demo", "cache_key": null, "scm_info": null, "source_options": { "path": "cookbooks/policyfile_demo" } }, "java": { "version": "1.24.0", "identifier": "4c24ae46a6633e424925c24e683e0f43786236a3", "dotted_decimal_identifier": "21432429158228798.18657774985439294.16782456927907", "cache_key": "java-1.24.0-supermarket.getchef.com", "origin": "https://supermarket.getchef.com/api/v1/cookbooks/java/versions/1.24.0/download", "source_options": { "artifactserver": "https://supermarket.getchef.com/api/v1/cookbooks/java/versions/1.24.0/download", "version": "1.24.0" } [/code]

You can then test this set of cookbooks in a VM or cloud instance. When
you’re ready to run this policy on a set of nodes attached to your
server, you run the chef push command to push this revision of the
policy to a specific policy group. We haven’t determined the specifics
of how Chef Server will implement policy groups, but at a basic level,
they are containers for nodes, and you’ll probably give them names like
“staging” or “prod-cluster-1”–these are groups of nodes that all apply
the same revision of a given policy. Your policies will have names like
“jenkins-master” or “webapp” or “database”–the policy specifies a
machine’s functional role. Individual instances of chef-client will
configure both a policy group and a policy.

The chef push command will also upload cookbooks to a new cookbooks
storage API which stores each cookbook according to a SHA-1 hash of the
cookbooks’ content, so that chef-client is guranteed to apply a
consistent set of cookbook code to your infrastructure (more on this
below). For more information on the new cookbooks API, see
this Chef RFC.

When chef-client runs, it reads its policy name and policy group from
configuration, and requests the current policy for its name and group
from the server. It then fetches the cookbooks from the new storage API,
and then proceeds as normal.

Motivation and FAQ

Policyfiles are a pretty big change from the way Chef currently delivers
configuration code to your machines today. Our design goals and answers
to some of the more common questions about Policyfiles are below.

Focus Workflow on Configuring Machines to do Useful Work

Chef’s current tooling (knife in particular) maps very closely to Chef
Server’s REST API and therefore is centered around manipulating
individual objects and uploading them to the Chef Server. chef-client
assembles these pieces at run time (more on that below) to configure a
host to do some useful work for you organization. With the Policyfile
feature, we want to focus the workflow on creating and configuring
entire systems, rather than individual components. For example,
Policyfiles describe whole systems and individual revisions of
Policyfile.lock documents are uploaded with all required components as
a unit to the Chef Server.

Code Visibility

In Chef currently, the exact set of cookbooks that a node will apply is
defined by:

• The node’s run_list property;
• Any roles present in the node’s run_list or recursively included by
  those roles;
• The environment, which restricts the set of valid cookbook versions
  available to a particular node according to a variety of constraint
  operators;
• Dependencies specified in cookbook metadata;
• The dependency solver implementation, which tries to pick the best
  set of cookbooks that meet the environment and dependency criteria.
• Whether or not you or people in your organization have updated a
  particular version of a cookbook (see “Cookbook Mutability” for more
  about this).

These conditions are re-evaluated each time chef-client runs, so it’s
not always easy to tell exactly which cookbooks chef-client will run,
or what the impact of updating a role or uploading a new cookbook will
be.

The Policyfile feature solves this problem by computing the cookbook set
on the workstation and producing a readable document of the solution.
chef-client runs re-use the same precomputed solution until you
explicitly update their specific policy group.

Role Mutability

Roles are currently global objects and changes to existing roles are
applied immediately to all nodes that contain that role in their
run_list (either directly or via another role’s run_list). This
means that updating existing roles can be very dangerous, so much so
that many users advocate abandoning them entirely.

The Policyfile feature improves the situation in two ways. Firstly,
roles are expanded at the time that the cookbook set is computed (i.e.,
the chef install step). Roles never appear in the
Policyfile.lock.json document. As a result, roles are “baked in” to a
particular revision of a policy, so that changes to a role can be tested
and rolled out gradually. Secondly, Policyfiles offer an alternative
means of managing the run_list for many nodes at once, since there is
a one-to-many relationship between policies and nodes. Therefore users
can, if desired, stop using roles without needing to use role cookbooks
as a workaround for managing the run_list of their nodes.

Cookbook Mutability

The Chef Server currently allows an existing version of a cookbook to be
mutated (in other words, you can change the code and re-upload without
changing the version number, so that the “apache2 1.0.0” cookbook has
different code at different times). While this provides convenience for
users who upload in-development cookbook revisions to a Chef Server
(this is common among beginners and some Ci patterns), it presents the
same problems as Role mutability. Some users account for this by
following a rigorous testing process so that only fully integrated
(i.e., all contributors’ changes are merged) and well tested cookbooks
are ever published to the Chef Server. While this process enforces good
development habits, it is not appropriate for everyone, and should not
be a prerequisite for getting safe behavior from the Chef Server.

The Policyfile feature solves this issue by using a new Chef Server
cookbook publishing API which does not provide cookbook mutability. In
order to avoid name collisions, cookbooks are stored by name and an
arbitrary ID, which is computed from the content of the cookbook itself.

One particularly frustrating cause of name/version collisions is when
users need to temporarily use a fork of an upstream cookbook. Even if
the user contributes their change and the maintainer is very responsive,
there may be a period of time where the user needs to use their fork in
order to make progress. However, this presents a versioning quandry: if
the user doesn’t update the version, they must overwrite the existing
copy of the cookbook on their server. Contrarily, if they do update the
version number, they might conflict with the version number of a future
release, which they could only fix by overwriting the newer version on
their server. Using content-based IDs with sourcing metadata makes this
use case easy.

But Opaque IDs are Confusing!

It’s definitely true that such opaque IDs are less comfortable than the
name, version number scheme that users are used to. “my-cookbook 1.2.3”
feels much more meaningful than “my-cookbook ddf827”, for example. In
order to ameliorate the problem, ChefDK and the new server APIs do a few
things:

• When working with the Policyfile.rb, you deal with cookbooks
  mostly in terms of names and version numbers. For cookbooks from an
  artifact service like supermarket, you use names and version constraints
  like you’re used to; for cookbooks from git, you use branch/tag/revision
  as you’re used to; for cookbooks from disk, you specify paths. The
  opaque IDs are mostly behind the scenes.
• Extra metadata about cookbooks is stored and included in API
  responses, as well as the Policyfile.lock.json. This includes the
  source of the cookbook (supermarket, git, local disk, etc.) and the
  upstream ID of the cookbook (such as git revision); for cookbooks loaded
  from local disk, the Policyfile implementation detects if they are in a
  git repo and collects the upstream URL, current revision ID, whether the
  repo is dirty, and whether the commits are pushed to the remote.
• Cookbooks uploaded to the new cookbook storage API can have extended
  SemVer version numbers with prerelease sections, like 1.0.0-dev.

Limit Expensive Computation on Chef Server

In order to determine the cookbook set for a given chef-client run, Chef
Server has to load dependency data for all known versions of all
cookbooks, and then run an expensive (NP expensive) computation to
determine the correct set. Moving this computation to the workstation
and running it less frequently makes Chef Server more efficient.

Where Does the Policyfile Live?

At the moment, we see three main ways to organize your Policyfiles:

• Store the Policyfile and related cookbooks in the same repository as
  the application you’re deploying. If you’re deploying custom
  applications written in-house and your software developers are
  comfortable working with Chef, you can put the Policyfile in the same
  repo as the application’s source code, along with any
  application-specific cookbooks, and version everything together.
• Store the Policyfile with a cookbook. If you’re following the single
  cookbook per repo workflow, you can include the Policyfile in the
  highest-level cookbook’s (i.e., the cookbook ultimately responsible for
  deploying a server’s primary application) repository.
• Store all of your Policyfiles in a single directory. This is likely to
  be the most common way to use Policyfiles with the monolithic cookbook
  repo workflow. There are still some details to be worked out for this
  case.

Am I Going to be Forced to Use This?

This change will be rolled out by adding new functionality to Chef
Client, ChefDK, and Chef Server alongside the existing
functionality. There are no plans to remove or even deprecate the
existing APIs. The plan is to get people to switch by offering an
alternative with both more safety and more freedom than the current
situation.

That said, if adoption is large enough then eventually we may remove the
dependency solver from Chef Server.

Does this Replace Berkshelf?

The Policyfile is definitely a replacement for the “environment
cookbook” pattern in Berkshelf. It also provides a dependency solver and
fetcher (thanks to some code from berks), so it may replace some other
berkshelf use cases. However it is much less opinionated than Berkshelf,
and may not replace Berkshelf for all use cases, so we’ll have to see
how things turn out.

Does this Force Me to Use the Single Cookbook per Repo Thing?

No. We’re still figuring out the optimal way to support the “megarepo”
workflow, but it will be supported. In particular we have to study the
tradeoffs of versioning your Policyfile.rb files (we’ll support other
names) with your chef-repo vs. outside of it. We plan to do some dogfood
testing to inform the design here.

Users who use the “megarepo” workflow may see some benefit to using
single repos for third-party cookbooks, but this will be optional and
users can convert from vendor branches piecemeal if they decide to do
so.

Do I Have to Change My Workflow to Use This?

The answer to this depends on how you define “workflow.” As noted above,
you can choose to have a chef-repo or not, and you can fetch third party
cookbooks using the Policyfile or an out of band mechanism (vendor
branches). You and your team can decide to publish only completely
integrated “release” cookbooks to the server if that works for you, but
you can also safely publish development versions of cookbooks to the
server without risk of mutating the production versions and without
needing a versioning scheme (devodd and friends) to workaround cookbook
mutability issues.

That said, the mechanics of how you get configuration code from your
workstation to production will be different. In particular, when using
the Policyfile feature in the recommended way, you cannot publish an
updated cookbook or role and have it applied immediately to all
machines. Tools that use the old APIs will need to be updated.

Are Policyfiles Versioned?

There currently isn’t any detailed design for the Chef Server policy API
(which will store the Policyfile.lock.json documents). One design
decision we have made is that the documents will be namespaced (by
policy group). This means that at minimum it will be possible to
independently update the policy for different stages of your release
process independently. For example, if you have policy groups for “dev,”
“stage,” and “prod,” you can iterate on new feature work in “dev” and
release a critical hot fix to “prod” independently of each other.

If this is all that’s implemented, then you will be able to version your
policies by committing your Policyfile.rb and Policyfile.lock.json
documents to revision control and using a branching policy that fits
your release requirements. That said, features to support operations
such as reverting or undo and/or tracking changes over time will be
considered.

What About Environments?

We have not made a final decision about how environments will work with
Policyfiles. In compatibility mode, you cannot use environments and
Policyfiles together, but this choice could be reversed. Policyfiles
do completely replace the cookbook version constraint portion of the
environments feature. However, environments do offer a useful way to set
environment-wide attributes, which some users rely on heavily. The main
sticking point is that environments provide the same double edged sword
as many other Chef features where updates to environments are propagated
immediately to all nodes in an environment. When done correctly, this is
very convenient, but it also allows mistakes to propagate to all nodes
immediately. Contrarily, if environment attributes are rolled into the
Policyfile, you can more easily test the effects of changes and control
the way these updates are applied, but it’s more difficult to apply
changes globally.

Compatibility Mode

The Policyfile feature depends on new APIs in Chef Server that don’t yet
exist. In order to provide a preview of the feature, the current
implementation operates in a compatibility mode that uses existing Chef
Server APIs to demonstrate the Policyfile behavior. This makes it
difficult to safely use Policyfiles and the current code publishing
mechanisms in the same organization (or 11.x Open Source Server), so we
recommend you use a separate organization when trying the policyfile
features.

Cookbook Artifact Storage

In compatibility mode, ChefDK must implement content-hash-based storage
of cookbooks using the existing /cookbooks endpoint. To do so, it maps
hash IDs to X.Y.Z version numbers. While this works to demonstrate the
Policyfile behavior, it is certainly a kludge. If you are trying the
Policyfile feature in compatibility mode, beware:

• Cookbooks uploaded by the policyfile commands will have very large
  version numbers with no sort order. Any chef-client that is not
  operating in policyfile mode will prefer these cookbooks to ones
  uploaded normally unless you are dilligent about using environment
  version constraints.
• The /cookbooks endpoint is not designed to be used this way, so it
  doesn’t show you the “real” version numbers or additional metadata about
  these cookbooks. While we have plans to make arbitrary cookbook IDs
  easier to manage in the final implementation, there’s little we can do
  about it in the exiting API.

Policyfile Storage

In compatibility mode, ChefDK uses data bag items to store
Policyfile.lock.json documents. To minimize the chance of conflict
with other data bag items, ChefDK stores all of these documents in the
“policyfiles” data bag; individual Policyfile.lock.json revisions are
given IDs of the form $policyname-policygroup.

Upcoming Features

The implementation of the Policyfile feature is still very
incomplete. Here’s a list of features we’ll be working on as we make
progress towards a production-ready release:

• Conservative Updating: ability to update a subset of the cookbooks in
  your Policyfile.lock.json
• Role Support: roles aren’t supported in Policyfile.rb run lists yet,
  but will be soon.
• Policyfile Attributes: you will be able to set role-level attributes
  directly in the Policyfile.rb.
• Multiple Run List Support: Policyfiles don’t support override run
  lists, so named run lists will be allowed in the Policyfile.rb as a
  replacement.
• In-House Cookbook Hosting: Currently you cannot host cookbooks behind
  your firewall; support will be added for using Chef Server’s
  /cookbooks endpoint as a “mini supermarket”, or you will be able to
  run a full supermarket instance.
• Server API Support: As mentioned above, “native” support for
  Policyfile documents and cookbook storage will be added to the Chef
  Server.

Chat Us Up at the Summit(s)!

If you’d like to chat about anything Policyfile-related, the
Community Summit is a great venue. If
you can’t make it, we’re available on the mailing list and IRC if you
have any questions.