If you watched the ChefConf keynote,
attended last years’ community summits,
or follow our open source mailing lists, you’ve probably heard about
Policyfiles.
If you haven’t, here’s the deal: Policies are a new feature of Chef that
combine the very best parts of Roles, Environments, and client-side
dependency resolvers like Berkshelf into a single easy to use workflow.
Policies are built by defining a Policyfile, which looks similar to a
Chef Role combined with a Berksfile. When a Policy is ready for upload,
a workstation command included with the ChefDK compiles the Policyfile
into a Policyfile.lock file. This locked Policy, along with all of the
cookbooks it references, are treated as a single unit by the Chef
tooling. The bundle of Policyfile.lock and cookbooks are uploaded to the
server simultaneously. They are also promoted simultaneously through the
deployment lifecycle, from dev to QA to production.
Policies make your chef-client runs completely repeatable, because
cookbooks referenced in a Policy are identified by a unique hash based
on their contents. This means that once the lock file + cookbook bundle
has been generated, the code underlying it will never change.
For more information, see the
Policyfile README in the ChefDK repo.
Previously we’ve recommended that you only use Policyfiles in
specialized testing environments because using Policyfiles in
compatibility mode along side existing infrastructure could cause
unexpected behavior. With Chef Server 12.1 and ChefDK 0.7, Policyfile
data is now stored via specialized APIs, making it safe (and a lot
easier) to use Policyfiles in your existing Chef Server setup.
To help you get familiar with the workflows Policyfiles make possible,
we’ll walk through deploying a simple demo application using Policyfiles
to manage our dependencies across the application’s lifecycle.
The code here is based on
a demo my colleague created for the London Chef meetup.
It deploys the “Awesome Appliance Repair” Python application.
Getting Started With Local Development and Testing
To follow along with this example, you’ll need:
- ChefDK 0.7+
- Hosted Chef or Chef Server 12.1+
- Vagrant (for local testing with
kitchen
) - To have configured your
~/.chef/config.rb
(or~/.chef/knife.rb
) to
communicate with your Chef Server.
Initialize Shell
If you have previously installed ChefDK or Test Kitchen on your machine
using rubygems, you might have older versions of these tools in your
PATH. If that’s the case, you can use chef shell-init
to setup your
environment variables, like so (I use zsh
, be sure to update the
command for your shell):
eval "$(chef shell-init zsh)"
which kitchen
# => /opt/chefdk/bin/kitchen
Generate:
For this example, we’ll structure our code as if we were developing our
infrastructure code alongside the application, in the same source repo.
Note that Policyfiles don’t require you to manage your code this way,
you can use individual git repos per cookbook or a monolithic repo if
you prefer. Just follow along for now :)
We use chef generate
to create the required files and directories for
us:
mkdir aar
cd aar
chef generate app .
# Policyfiles will be the default someday, 'till then:
chef generate policyfile
Commit
We’ll commit our work now so we can roll back to a fairly blank slate if
we make a mistake later:
git add .
git commit -m 'initial policyfile demo commit'
Edit Policyfile:
We describe how we want Chef to compose our cookbooks to configure a
machine to run our application by editing the Policyfile.rb
. This file
defines a few things:
name
: This describes the kind of machine we are creating. We name
this “aar”, which is our abbreviation for “Awesome Appliance Repair.”default_source
: The place where we get shared cookbooks. The default
source is:community
, which is the Chef Supermarket site. You can
also use an internal supermarket instance or a monolithic Chef Repo.run_list
: The list of recipes, in order, that you want Chef to
evaluate to configure your system. When using Policyfiles, you set therun_list
in the Policyfile instead of on the node.cookbook
: APolicyfile.rb
can have multiplecookbook
statements;
these can configure specific cookbooks to be loaded from alternative
sources or set additional version constraints on them.- Default and override attributes: these define attributes at the ‘role’
precedence level. We’ll look at these later.
Edit the Policyfile.rb as follows:
# Policyfile.rb - Describe how you want Chef to build your system.
#
# For more information on the Policyfile feature, visit
# https://github.com/opscode/chef-dk/blob/master/POLICYFILE_README.md
# A name that describes what the system you're building with Chef does.
name "aar"
# Where to find external cookbooks:
default_source :community
# run_list: chef-client will run these recipes in the order specified.
run_list "aar::default"
# Specify a custom source for a single cookbook:
cookbook "aar", path: "cookbooks/aar"
Chef Install
With our basic Policyfile.rb
, we run chef install
to fetch
dependencies and generate a Policyfile.lock.json
. We haven’t specified
any dependencies yet, so we don’t need to fetch anything, but we will
need the lockfile to be generated before we can proceed to the next
step.
chef install
Let’s take a peek at the Policyfile.lock.json
we just created. We’ll
go over each part individually:
Revision ID
"revision_id": "5f750bf464100b487cd7c276c5d532341b79fbeb5e8accd29538ae972896992b",
Each time we create or update the lock, chef
will automatically
generate a revision_id
based on the content. These values are used to
automatically version your policies, so that you can apply different
revsions of a policy to different set of servers. We’ll see this in
action a little later.
Name and Run List
"name": "aar",
"run_list": [
"recipe[aar::default]"
],
The lock includes the name and run list we specified previously. The run
list is normalized to the least ambiguous form.
Cookbook Locks
"cookbook_locks": {
"aar": {
"version": "0.1.0",
"identifier": "cff2d37260c04b21053ad30b68aa20e674e52e6c",
"dotted_decimal_identifier": "58532310150070347.9294424438433962.36174175743596",
"source": "cookbooks/aar",
"cache_key": null,
"scm_info": {
"scm": "git",
"remote": null,
"revision": "3455fb415d56f9a7cabbb76f2063942a6547b2eb",
"working_tree_clean": true,
"published": false,
"synchronized_remote_branches": [
]
},
"source_options": {
"path": "cookbooks/aar"
}
}
},
For each cookbook we use, there is a corresponding entry in thecookbook_locks
section. The exact data collected about each cookbook
is dependent on the cookbook’s source. In this case, we have a cookbook
sourced from the local disk which happens to be in a git repo. In the
event we need to debug this cookbook later, ChefDK has collected
information about the cookbook’s git revision. If we’d setup a remote,
git would tell us the cookbook’s git URL and whether we’d pushed this
commit to a branch on the remote.
Attributes
"default_attributes": {
},
"override_attributes": {
},
Policyfiles have attributes that replace role attributes. We’ll see
these a little later.
The Rest
"solution_dependencies": {
You can ignore the solution_dependencies
section. It’s used to keep
track of dependencies in your cookbooks so ChefDK can check whether
changes to your cookbooks are compatible with their dependencies without
having to download the full cookbook list from supermarket every time.
Commit the lockfile
We’ll want to compare it to an updated version
later, to see what changed.
git add Policyfile.lock.json
git commit -a -m 'updated Policyfile and created lock'
Edit .kitchen.yml
ChefDK ships with a policyfile_zero
provisioner for Test Kitchen that
allows us to test our policies in local (or cloud) VMs. Note that
currently Chef Zero doesn’t fully support “native” Policyfile APIs, so
instead it runs in compatibility mode. This isn’t a problem on an
isolated server like Chef’s local mode, but it’s something you might
notice when debugging.
---
driver:
name: vagrant
network:
- ["forwarded_port", {guest: 80, host: 8080}]
provisioner:
name: policyfile_zero
require_chef_omnibus: 12.3.0
platforms:
- name: ubuntu-14.04
suites:
- name: default
attributes:
Run TK with Empty Cookbook
To verify our test rig works, we’ll run kitchen
with our empty
cookbook:
kitchen converge
If you get an error like Message: Could not load the 'policyfile_zero' provisioner from the load path
then you didn’t run the chef shell-init step
above. Run that and try
again.
If that worked without a problem, you can throw that VM away:
kitchen destroy
Develop your Cookbook
This is where we’d normally run a TDD testing loop, but for the purpose
of this walkthrough, we’ll just import the aar
cookbook fully formed:
cd cookbooks
curl -LO https://github.com/danielsdeleo/aar/releases/download/draft-1/aar-cookbook.tgz
tar zxvf aar-cookbook.tgz
rm aar-cookbook.tgz
cd ..
If you inspect cookbooks/aar/metadata.rb
, you’ll notice that our
cookbook now has some dependencies, but we haven’t yet downloaded them.
We’ll do that next.
Chef Update and Commit
Now that we’ve added some dependencies to our cookbook, we need to runchef update
to fetch them.
chef update
This will recompute our dependencies and cache all the cookbooks we
need for our Policy.
To see what’s changed since our last commit, we could just run git
, but
diffchef diff
gives us itemized output, listing added, removed,
and changed cookbooks. Let’s give it a go:
chef diff --head
Now that we’re satisfied with our changes, we’ll commit again.
git add .
git commit -m 'Update aar cookbook and deps'
Run Kitchen Again
We can run test kitchen again to see the result of our changes:
kitchen converge
Visit the Site
With the port forwarding, we can visit
http://localhost:8080 and see the site. For
more info on using the application, see the awesome appliance repair README on github.
Deploy
We can use the chef provision
feature to create a “staging” and then a
“production” node. We’ll use these to see how we can deploy different
revisions of a policy to different machines. Since chef provision
is
new and somewhat experimental, it’s not yet integrated with chef
, however, we can generate a cookbook like this (make sure it’s
generate
named “provision”, that name is special):
chef generate cookbook provision
Then overwrite the generated provision/recipes/default.rb with the
following. NOTE: It’s vital that you set the convergence_options
as shown here. Nodes currently don’t have any attributes to set
Policyfile options; instead you must set policy_group
andpolicy_name
in the config file. The provisioning convergence options
will take care of that for you automatically.
If you’d like to extend this example to use something other than
Vagrant, you can learn more about Chef Provisioning on docs.chef.io.
context = ChefDK::ProvisioningData.context
# Set the port dynamically via the command line:
target_port = context.opts.port
with_driver 'vagrant:~/.vagrant.d/boxes' do
options = {
vagrant_options: {
'vm.box' => 'opscode-ubuntu-14.04',
'vm.network' => ":forwarded_port, guest: 80, host: #{target_port}"
},
convergence_options: context.convergence_options
}
machine context.node_name do
machine_options(options)
# This forces a chef run every time, which is sensible for `chef provision`
# use cases.
converge(true)
action(context.action)
end
end
Notice that we are making the forwarded port configurable via the
command line. This lets us run multiple VMs on the same host without the
forwarded ports colliding with each other.
We can sync the policy to the server and create our “staging” node with
a single command:
chef provision staging --sync -n aar-staging-01 -o port=8000
That will sync our local policy lock to a policy group called ‘staging’
(creating that policy group in the process), then run an embedded Chef
Client which creates a VM (via Chef Provisioning), configures it and
converges it.
We can see the site running by visiting http://localhost:8000
(notice that’s port 8000 this time).
Since it works in staging, we’ll create a “production” node with the
same policy:
chef provision production --sync -n aar-production-01 -o port=8888
Update the Attributes via Policyfile
Policyfiles allow us to set attributes. Since Policyfiles don’t support
roles, these attributes replace role attributes in the precedence
hierarchy. In our Policyfile.rb, we set attributes using the same syntax
we use in cookbooks. In this example, we’ll change the version number
that appears on the home page. Add the following line to your
Policyfile.rb
default['aar']['version'] = "19.7.4"
To apply the changes to the Policyfile.lock.json, use chef update
:
chef update --attributes
We can see the effect of our changes with chef diff
:
chef diff --head
And we can see that our local policy differs from what we’ve deployed to
our staging group:
chef diff staging
Now that we’re satisfied with our changes, commit to git again:
git commit -a -m 'update aar version'
Deploy it to Staging:
In a normal TDD workflow, we’d run kitchen again to see our changes, but
this time we’ll just deploy it to staging by running chef provision
again:
chef provision staging --sync -n aar-staging-01 -o port=8000
If we visit the site at http://localhost:8000
we see “Awesome Appliance v.19.7.4” right under the login dialog.
Let’s suppose we’re not ready to apply the change in production, but we
want to run chef-client on our production machine. We can do this by
using the --policy-name
option instead of the --sync
option:
chef provision production --policy-name aar -n aar-production-01 -o port=8888
Note that since we did not update the policy, nothing is updated. If we
visit the web page at http://localhost:8888
we’ll see that nothing has changed. Though we only changed the
attributes, the same is true if we updated the cookbooks, since they’re
locked down by content in our policy.
Oh No! A Bug in Our Cookbook
To demonstrate how cookbook code is automatically versioned and
sandboxed by policyfiles, let’s introduce a “bug” into our cookbook. Add
this line to the top of cookbooks/aar/recipes/default.rb
:
raise "OH NO THIS IS A BUG"
Since this cookbook is local, chef
will automatically pull in updates
when we upload (no need to run chef update
). Lets upload to staging
and run chef-client again. This time, we’ll use chef push
to upload
our changes without invoking provisioning (which is probably what you’d
want to do in your normal workflow).
chef push staging
We can invoke provisioning without syncing the policy like so:
chef provision staging -p aar -n aar-staging-01 -o port=8000
This will cause an error like this in the chef run on the VM (which will
cause another error in chef provision
on your workstation):
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/aar/recipes/default.rb
================================================================================
RuntimeError
------------
OH NO THIS IS A BUG
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/aar/recipes/default.rb:1:in `from_file'
But if we run Chef on our production node, everything is roses and
sunshine:
chef provision production -p aar -n aar-production-01 -o port=8888
# => Chef Client finished, 0/41 resources updated in 10.634577518 seconds
We can also confirm that the policies applied to each group are
different with the show-policy
subcommand:
chef show-policy aar
The output should be similar to:
aar
===
* production: 8312cd89c9
* staging: eb0fedf311
Cleaning Up
To shut down the Vagrant VMs, you can use the -d option to chef provision
to set the default action to destroy:
chef provision production --policy-name aar -n aar-production-01 -o port=8888 -d
chef provision staging -p aar -n aar-staging-01 -o port=8000 -d
Fin
Policyfiles give us a consistent and repeatable description of how we
want Chef to configure our machines, with minimal hassle. Because
versioning is built-in and automatic at both the cookbook and policy
level, we can make changes to our infrastructure code safely and
explicitly. While we still have work to do to fill out the feature set
around Policyfiles, enough of it exists for you to get started today.
Currently, the most complete documentation is in the
Policyfile README in the ChefDK repo,
but we’ll be moving documentation to docs.chef.io
as we complete work on Policyfiles.
If you give it a try and find that any missing feature is a deal breaker
for you, let us know and we’ll do our best to make you successful.
Happy Cheffing!