Turn it to 11: Upgrading From Chef 10

Editor’s Note: A previous version of this blog post recommended setting cookbook\_path to nil in the knife.rb file. That line has been removed, as it causes an error in newer version of knife/chef-client. Either do not set the cookbook\_path or do not set it to nil when following these instructions. 

You can find instructions on upgrading from Chef server 10 to Chef server 11 and newer in the Chef docs. These docs are the recommended instructions for upgrading, as they are updated as tooling changes, whereas this blog post is less likely to be updated.

So you’re on Chef 10, and you can taste the shiny radiating from Chef 11. It’s faster, scales better, and is easier to install and manage. There’s no question: you want to go to there.

But how? Installing the server is super easy, but all your cookbooks, your node definitions with their run lists, and your roles are in Chef 10’s CouchDB. And Chef 11 all that data is in PostgreSQL. One thing’s for sure, this ain’t no file copy.

Today we’ll use the knife-essentials plugin to download everything as JSON from the old Chef server, and then upload it all to the new Chef server. Because Chef 11 is REST API-compatible with Chef 10, the knife plugin works against both, and we’ll exploit that to the fullest.

Prerequisites

To upgrade, you need a live Chef 10 and Chef 11 server, and a knife workstation that can hit them both, loaded with the latest knife-essentials.

1. To install Chef 11, follow the instructions here.
2. To install the latest knife-essentials:

   [sourcecode gutter=”false” language=”bash”]gem install knife-essentials[/sourcecode]

NOTE: You might want to firewall the Chef 10 server off from clients, to prevent data from getting stale while you upgrade.

Step 1: Set things up to talk to Chef 10.

First, we’ll set up knife.rb files to point at the old and new servers.

1. Make a transfer directory you will download the stuff to. We chose ~/transfer.
2. Make a .chef/knife-chef10.rb file in the transfer directory that points at the Chef 10 server (replacing the server URL with your server). You must use an admin client for this; we chose chef-webui because it’s present by default. Any admin client will do; replace it in the appropriate places. The file should look like this:

   transfer_repo = File.expand_path(‘..’, File.dirname(__FILE__))
   chef_server_url “http://chef-10.example.com:4000”
   node_name ‘chef-webui’
   client_key “#{transfer_repo}/.chef/chef-webui.pem”
   repo_mode ‘everything’
   versioned_cookbooks true
   chef_repo_path transfer_repo

3. Grab the admin key for the Chef 10 server (the chef-webui key is typically at /etc/chef/webui.pem on the server):

   [sourcecode language=”bash” gutter=”false”]
   cp <your webui.pem> .chef/chef-webui.pem
   [/sourcecode]

You can check if this worked by running knife list /clients. This
will show you a list of all the clients, including clients/chef-webui.json
and clients/chef-validator.json.

Step 2: Download all the Chef 10 data.

The first task is to download everything from the Chef 10 server. Turns out our hard work setting things up has paid off:

[sourcecode language=”bash” gutter=”false”]
knife download -c .chef/knife-chef10.rb /
[/sourcecode]

Congratulations! Every last piece of data in your Chef server is now in the transfer directory, ready to be massaged and uploaded at will, to any place you want.

Step 3: Set things up to talk to Chef 11.

1. Make a .chef/knife.rb file in the transfer directory that points at the new server (replacing the server URL with your server):

   transfer_repo = File.expand_path(‘..’, File.dirname(__FILE__))
   chef_server_url “https://chef-11.example.com”
   node_name ‘admin’
   client_key “#{transfer_repo}/.chef/admin.pem”
   repo_mode ‘everything’
   versioned_cookbooks true
   chef_repo_path transfer_repo

2. Copy the Chef 11 admin PEM file to admin.pem in the transfer directory.

   [sourcecode language=”bash” gutter=”false”]
   cp <your admin PEM> .chef/admin.pem
   [/sourcecode]

You can check if this works by running knife list /users. This
should print users/admin.json.

Step 4: Massage the data.

There are two differences between Chef 10 and 11 that we need to pave over before we upload.

1. The special chef-validator client is no longer special: Chef 11 requires you to set the chef-validator flag to create a validator.To fix this, edit clients/chef-validator.json and add “validator”: true as a JSON property. It should look a lot like this:

   {
   “name”: “chef-validator”,
   “public_key”: “—–BEGIN PUBLIC KEY—– \nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8l0+sy05G6YX/SaVsu2k\ndwOTIZKLhvfuhp/VcBU4hJoU8ADTMWyxTR9qEqRq+mgUqkF4ox/zIwhLG5nyHMLa\nFKsKPxUQlS1Jsf2gaoP+RhnswmspJffhF2l593DwSsglTLNtDw5cqhF6YYo7b7cB\nywHaWL+O3cSFLd0US7tSoOTeOdnAAwPbAy7mKQ4nJUHZOCV3Ottn83V8BUCfpnbi\nNetytGDnE1Ms9lvYswsW2EqEnzQ+afvlDq5tXu72b1XBs7Y/8JqQz8+3lVHNGKys\nh5U6VdI5Br0u1leO0LcO2FjhHaMjyqZ7/T2MVztXujUN9CoX1a+3siu3HAa8lslo\noQIDAQAB\n—–END PUBLIC KEY—–\n”,
   “_rev”: “1-72a9f16a92108bd794704c075261aeb5”,
   “validator”: true
   }

2. Chef 11 has a new user named “admin.” Many Chef 10 instances have an admin client named “admin.”If you have knife clients that use the admin key, you can either copy the new admin.pem to those client workstations, or you can make them work silently by replacing the Chef 11 admin’s key with the old one. NOTE: if you don’t have knife clients or are able to get the new pem over to knife workstations easily, I would skip the following section and just remove clients/admin.json.

   knife download users/admin.json
   grep public_key clients/admin.json
   

   You may want to verify that knife list /users works at this
   point; if it returns users/admin.json, then you’re gold.

   Either way, you won’t want to move the admin client to Chef 11, so remove
   the file:

   [sourcecode language=”bash” gutter=”false”]
   rm clients/admin.json
   [/sourcecode]

3. This is also a perfect opportunity to clean up roles, cookbooks and cookbook versions you aren’t using, Just delete the files / directories from your transfer dir before you upload, and they won’t go :)

Step 5: Upload everything to Chef 11.

Once again, previous steps have made this easy:

[sourcecode language=”bash” gutter=”false”]
knife upload /
[/sourcecode]

Profit!

After this procedure, you can point your DNS or load balancer at the new
servers. All clients and knife installs should continue to work exactly as
before, only now your server is better, stronger and faster.

If you have issues, you can ask in the IRC channel or email the list. If knife-essentials itself is causing a problem, file an issue in github or check on IRC.