Chef 0.10 Preview: Environments

Chef 0.10 recently hit the release candidate phase of the release
process, so I’d like to give you all a preview of the new features in
Chef while we’re finalizing the code. The biggest new feature in Chef
0.10 is environment support, so we’ll start there. If you’d like to
play along at home, you’ll need an Opscode Platform
Account or have a Chef 0.10
server installed, and have Chef 0.10 (RC) installed on the client.

Environments Today

With previous versions of Chef, you face a tough choice when automating
the various infrastructures (e.g., production, staging, qa, dev, etc.)
you have: you could run a unique instance of Chef Server (or use a
unique Opscode Platform organization) for each one, which keeps each
environment nicely isolated, but you’ll have to maintain a configuration
for each Chef server. Alternatively, you could try to keep your
environments separate by assigning nodes in each environment an
“environment role”, but in this configuration, an update to cookbooks
for your development environment could impact production.

Chef Environments

Chef 0.10 solves this problem by allowing you to set policies to dictate
which versions of a given cookbook may be used in an environment. To get
a feel for how this works, let’s take a look at an example environment
file using the Ruby DSL:

[sourcecode lang=”ruby”]
# `name’ and `description’ are what you’d expect
name "prod"
description "OMG production"

# Use version 11.0.0 *only*
cookbook_versions "couchdb" => "= 11.0.0",
# use versions greater than 0.99.0
# and less than 0.100.0
"application" => "~> 0.99.0"
[/sourcecode]

The name and description fields are pretty standard fare. It’s in the
cookbook_versions field that we set our allowed versions policy. For
each cookbook you have, you may set a version constraint to describe the
allowed versions, where the version constraint is an (in)equality
operator combined with a version. The available operators are:

• = – Equality
• > – Greater than
• >= – Greater than or equal to
• < – Less than
• <= – Less than or equal to
• ~> – Pessimistic greater than

These are mostly self explanatory, though the pessimistic greater than
may be new to some of you. This constraint operator is borrowed from
rubygems’ dependency syntax; its exact behavior depends on the version
you specify in the version constraint. This is best illustrated with
some examples:

• ~> 1.0 matches any version greater than or equal to 1.0, but less
  than 2.0.
• ~> 1.1 matches any version greater than or equal to 1.1, but less
  than 2.0
• ~> 1.1.0 matches any version greater than or equal to 1.1.0, but
  less than 1.2.0
• ~> 1.1.5 matches any version greater than or equal to 1.1.5, but
  less than 1.2.0

Trying it Out

Now that you’ve got the gist of how environments specify cookbook
version constraints, let’s see it in action. In this example, I’m using
the Opscode platform with a node I’ve already set up for testing. I
start by upgrading my test node to Chef 0.10:

dan@chef-client$ sudo gem install chef --pre

Do the same for your laptop you run knife from. Back on my laptop, I
create the production environment:

dan@laptop$ knife environment create production

For now we’ll leave the version constraints blank:

[sourcecode lang=”javascript”]
{
"name": "production",
"description": "lets pretend it’s prod",
"cookbook_versions": {
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
}
}
[/sourcecode]

I’ve also edited my node to assign it to the production environment and
have the tmux recipe in
its run list:

dan@laptop$ knife node edit ghost.local

[sourcecode lang=”javascript”]
{
"name": "ghost.local",
"chef_environment": "production",
"normal": {
"tags": [
]
},
"run_list": [
"recipe[tmux]"
]
}
[/sourcecode]

When I run chef, I see that tmux is installed:

dan@chef-client$ sudo chef-client
INFO: *** Chef 0.10.0.rc.0 ***
INFO: Run List is 
]
INFO: Run List expands to [tmux]
INFO: Starting Chef Run for ghost.local
INFO: Loading cookbooks [tmux]
INFO: Processing package[tmux] action install (tmux::default line 19)
INFO: package[tmux] installed version 1.1-1
INFO: Chef Run complete in 11.682407 seconds

So far this is pretty basic chef. But now let’s introduce a bug into the
tmux recipe:

dan@laptop$ vim repo/cookbooks/tmux/recipes/default.rb

[sourcecode lang=”ruby”]
# This is a bug:
raise "oops"

package "tmux" do
action :install
end
[/sourcecode]

And we change the cookbook’s version from 1.0.0 to 1.1.0:

dan@laptop$ vim repo/cookbooks/tmux/metadata.rb

[sourcecode lang=”ruby”]
maintainer "Opscode, Inc."
maintainer_email "cookbooks@opscode.com"
license "Apache 2.0"
description "Installs tmux"
long_description IO.read(File.join(File.dirname(__FILE__), ‘README.md’))

###
# Changed this from "1.0.0" to "1.1.0"
###
version "1.1.0"
[/sourcecode]

Upload the cookbook, and run chef-client. chef-client fails because of
the bug in the cookbook:

dan@laptop$ knife cookbook upload tmux
Uploading tmux...
upload complete

dan@chef-client$ sudo chef-client
INFO: *** Chef 0.10.0.rc.0 ***
INFO: Run List is 
]
INFO: Run List expands to [tmux]
INFO: Starting Chef Run for ghost.local
INFO: Loading cookbooks [tmux]
INFO: Storing updated cookbooks/tmux/recipes/default.rb in the cache.
INFO: Storing updated cookbooks/tmux/metadata.rb in the cache.
ERROR: Running exception handlers
FATAL: Saving node information to /var/chef/cache/failed-run-data.json
ERROR: Exception handlers complete
FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
FATAL: RuntimeError: oops

Oops, indeed. Now let’s use environments to lock production down to the
cookbooks that we know are good:

dan@laptop$ knife environment edit production

[sourcecode lang=”javascript”]
{
"name": "production",
"description": "lets pretend it’s prod",
"cookbook_versions": {
"tmux": "= 1.0.0"
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
}
}
[/sourcecode]

When we run chef-client again, we’ll use the 1.0.0 version of the tmux
cookbook (without the bug):

dan@chef-client$ sudo chef-client
INFO: *** Chef 0.10.0.rc.0 ***
INFO: Run List is 
]
INFO: Run List expands to [tmux]
INFO: Starting Chef Run for ghost.local
INFO: Loading cookbooks [tmux]
INFO: Storing updated cookbooks/tmux/recipes/default.rb in the cache.
INFO: Storing updated cookbooks/tmux/metadata.rb in the cache.
INFO: Processing package[tmux] action install (tmux::default line 19)
INFO: Chef Run complete in 3.805665 seconds
INFO: Running report handlers
INFO: Report handlers complete

Freezing Cookbooks

If you followed the example closely, you probably noticed that we could
have uploaded the broken version of our tmux cookbook at version 1.0.0
which would have overwritten the “good” version. This certainly throws a
wrench in our carefully crafted cookbook policy. Luckily, in Chef 0.10,
you can freeze a version of a cookbook by using the --freeze option to
knife cookbook upload. When a cookbook is frozen, you can only upload
a cookbook with the same name and version by using the --force option:

dan@laptop$ knife cookbook upload redis --freeze
Uploading redis...
upload complete

dan@laptop$ knife cookbook show redis 0.1.6 |grep frozen 
frozen?:        true

dan@laptop$ knife cookbook upload redis
Uploading redis...
ERROR: Version 0.1.6 of cookbook redis is frozen. Use --force to override.

By combining frozen cookbook versions with environments, you can make
sure that production is safe from accidental updates when testing out
changes in your development infrastructure.

Environments Workflows

Environments are clearly a powerful feature for keeping changes
isolated, but chances are you’ll want to use them differently than our
contrived example. Here are some factors to consider when designing your
workflow:

• How important is it to keep your environment files in source control?
• Do you want to edit environments in the management console (Web UI)?
• Do you want to be able to use the -E ENVIRONMENT flag to cookbook
  upload to automatically set cookbook version constraints on an
  environment?

If you place supreme importance on always keeping every last bit of data
in version control, then, you’ll want to drive your use of environments
by editing files only. On the other hand, if you’re already managing
your cookbooks for separate environments using git branches, you already
have your versioning policy information stored in your cookbooks’
metadata, so you might opt for a lighter weight approach.

Below are two recommended strategies for using Chef’s environments as
you develop cookbooks and move your changes into production. For
simplicity, we’re assuming you have only a development and production
environment; you can extend them to include staging, QA or whatever you
might have. We anticipate that you’ll find lots of other ways to use
environments as you get familiar with them—that’s why we made them
so flexible.

Branch Tracking Strategy

In the branch tracking strategy, you have a branch in your source
control repo for each environment, and your cookbook versioning policy
tracks whatever is in the tip of each branch. This strategy is simple
and lightweight. For development environments that track the latest
cookbooks, you continue to work as you have before environments, except
that you need to bump the version before you upload the cookbook for
testing. For environments that need special protection, i.e.,
production, you always upload cookbooks using the -E ENVRIONMENT and
--freeze flags.

In development:

1. Bump the version number as appropriate.
2. Hack.

3. Upload and test:

   # Upload for dev is the same as always:
   knife cookbook upload my-app

4. Repeat 2 and 3 as necessary.

When you’re ready to move your changes into production:

# Upload for prod with automatic version constraints:
knife cookbook upload  my-app -E production --freeze

Adding the -E ENVIRONMENT option will cause knife to automatically set
the version constraint on that environment to match the version you’re
uploading.

Maximum Version Control Strategy

If you prefer to version control absolutely everything, then you need to
use a file-editing based approach to environments. In your development
environment, you work the same as the branch tracking strategy:

In development:

1. Bump the version number as appropriate.
2. Hack.

3. Upload and test:

    # Upload for dev is the same as always:
    knife cookbook upload my-app
   

4. Repeat 2 and 3 as necessary.

When you’re ready to move your cookbooks into production, you’ll do the
following:

1. Upload and freeze your cookbooks:

    knife cookbook upload my-app --freeze
   

2. Modify your environment to prefer the new version you uploaded:

    (vim|emacs|mate|ed) YOUR_REPO/environments/production.rb
   

3. Upload the updated environment:

    knife environment from file production.rb
   

4. Deploy!

Of course, version control stickler that you are, you’ll commit and push
your updates to the right branches during the process.

Further Reading

This is a far from comprehensive overview of Chef environments—I
didn’t get to cover environment attributes, the webui, or
environment-specific run lists in Roles. You can find
more information on the chef wiki,
and ask further questions on our mailing list
and on our IRC channel (irc.freenode.net#chef).

Be sure to keep an eye on the Opscode blog, as we’ll be previewing more
Chef 0.10 features in the upcoming days.