Test Kitchen makes it easy for you to test cookbooks on a variety of platforms. With Test Kitchen, you can quickly create test nodes, converge them, and then run tests against them to verify their state.
In this recorded webcast, Chef engineer Fletcher Nichol explains how to use Test Kitchen to write and maintain Chef cookbooks with confidence. Fletcher shares:
- How to set up your workstation
- How to create a new Chef cookbook that runs on your target platform
- How to deal with new operating system and Chef client releases
- How to play in the world of Windows
Q&A from the live webinar, including the questions we didn’t have time to answer live is available below.
Q&A From the Live Webinar
Q: How is Test Kitchen different in terms of development work flow compared to uploading a cookbook to the chef server and run chef client on your designated node? In other words what are the benefits of Test Kitchen?
A: In Test Kitchen, we were trying to make the development of code to running it as direct and errorless as possible. I’ve had my fair share of minutes and hours wasted trying to debug a cookbook issue before realizing that I forgot to upload my cookbook updates to the Chef Server. The idea here is to save code in your editor and run kitchen converge … to see the result of your changes as quickly as possible. I’d say that’s the primary advantage.
Q: Will Test Kitchen eventually provide a way to upload the cookbook to a Chef server and the cookbook dependencies?
A: Uploading cookbook to a real Chef Server would be supported if/when we add a chef_client Test Kitchen Provisioner (similar to the existing chef_solo and chef_zero Provisioners). However, if you are using a Berksfile in your project, Test Kitchen is essentially performing a berks vendor --path [TMP_PATH] before uploading the cookbooks to your remote instances. You could run a similar command to upload or apply a set of cookbooks onto your actual Chef Server. The kitchen converge process is going to generate your Berksfile.lock anyway, so berks upload and berks apply [ENVIRONMENT] would work.
Q: Question on the EC2 driver… Were credentials for your AWS account needed to be provided somewhere in the cookbook to enable a new instance to be created?
A: Yes, when I switched the Driver name over to "ec2", my workstation had a global config file under $HOME/.kitchen/config.yml which provides defaults for any project your user is running on that system. It looked very similar to the following:
[code language=”xml”]
–––
driver:
name: ec2
region: us-west-2
instance_type: m4.large
iam_profile_name: fnichol-provisioning
aws_ssh_key_id: fnichol-aws-ssh-key
tags:
x-project: test-driven-infrastructure
x-sub-project: feedback-factory
created-by: test-kitchen
transport:
ssh_key: /Users/fnichol/.ssh/id_rsa-aws
[/code]
That provided me with enough default configuration to pick an AMI and ignore the other credentials details in the webinar. Hopefully that fills in some more blanks—for other settings it’s worth checking out the kitchen-ec2 README for more configuration attributes.
Q: It is possible to have multiple drivers inside a single .kitchen.yml file?
A: It absolutely is! Each and every Test Kitchen Instance (which are created by multiplying all the Platforms and Suites together) has its own independent Driver, Provisioner, Verifier, etc. which leads to interesting combinations like the following:
[code language=”xml”]
–––
driver:
name: vagrant # a default for all instances
platforms:
– name: ubuntu-15.04
– name: centos-7.1
– name: amazon-2015.03
driver:
name: ec2 # only available on AWS
image_id: ami-XXXXX
– name: windows-8.1
driver:
name: hyperv # if on Windows, use the builtin hypervisor!
suites:
– name: client
provisioner:
run_list: recipe[foo::client_only]
– name: server
provisioner:
run_list: recipe[foo]
[/code]
Q: Have you seen Test Kitchen used on a CI server – is that a good pattern to implement?
A: Yes, there are people who are running Test Kitchen in their CI environments and it seems like the logical next step once you have an automated battery of tests that are providing you value. The simplest setup might be to run a kitchen test --destroy=always in as your CI task or perhaps with the --concurrency flag added to make things quicker (the default is to run all the instances in serial).
There are a growing number of open source cookbooks that are running Test Kitchen in CI such as the Poise Ruby Cookbook using the Docker Driver. Even the Chef project has started testing itself though Test Kitchen, using the EC2 driver.
Q: Do you recommend .kitchen.yml or .kitchen.local.yml ?
A: The preference would be to start with a reasonable .kitchen.yml that you could commit into version control with your project, suitable for sharing with your peers, teams, or the public in the open source case. You can use a .kitchen.local.yml when you want to overlay or inject your own user credentials locally without accidentally committing them into the .kitchen.yml file. The .kitchen.local.yml is also where you might override the Driver choice of the cookbook author. In my case, I might want to try and run Test Kitchen with the Docker Driver whereas someone else might prefer to run everything in the cloud, using the Digital Ocean Driver, for example.
Q: How do you get Test Kitchen to run “verification tests” (like Serverspec tests)?
A: The one of easiest ways to see this in action, is with the ChefDK; if you generate a cookbook with generate cookbook sample, you should see a long directory path of test/integration/default/serverspec containing a default_spec.rb file. The presence of files under test/integration//serverspec is enough to make the kitchen verify action install software, upload test files and run Serverspec.
From any point you should be able to run kitchen verify which is the step at which Test Kitchen runs the verification tests. Note that kitchen test will also do this, but will start from a cleanly created instance first.
Q: Do you have any advice for using Test Kitchen after cookbooks are written? We have a big ball of mud with a large monorepo, interdependencies, etc. Any best practices here?
A: I’ve used a few strategies here, but if you happen to have a chef-repo setup (i.e. a git repo/project with cookbooks/, data_bags/ and other directories), you can drop a .kitchen.yml directly into the root of that project. If no Berksfile or Cheffile exists at the root, Test Kitchen will default to using your ./cookbooks/ directory as well as any other directories for roles, data bags, etc. That way you could have a suite per node type, customizing each run-list:
[code language=”xml”]
–––
driver:
name: vagrant
platforms:
– name: centos-7.1 # we’ll assume only 1 platform
suites:
– name: db
provisioner:
run_list:
– role[base]
– recipe[mysql::server]
– name: web
provisioner:
run_list:
– role[base]
– recipe[apache2]
[/code]
You can also try to drop a .kitchen.yml into a sub-cookbook and treat that cookbook independently. By default, Test Kitchen is going to use that cookbook’s metadata.rb file to deal with dependencies (which are almost never enumerated if it’s a really old project). From that point you can try adding cookbook dependencies into the metadata.rb until you get a running instance. This feels like good blog post content—it is by no means simple, but Test Kitchen can be viciously good at detecting undeclared dependencies :)
Q: Can we use Test Kitchen instead of vagrant directly to help with testing our app development locally? We do things like vagrant up etc to get a box and pass variables to get that vBox instance to run a branch from our code base. Could I use Test Kitchen to do that stuff instead?
A: Yes, this is something you could do and I know that at least a couple of internal Chef projects use this pattern. If you’re using Chef to prepare the Vagrant system then you’re already most of the way there. I’d recommend using Chef’s attributes to inject your desired version of your application which could either be a static value, or injected using an environment variable. Sidebar pro-tip: by default, Test Kitchen passes your kitchen.yml/kitchen.local.yml files through ERb before loading them. So, I’m thinking something like this:
[code language=”xml”]
driver:
name: vagrant</br></br>
provisioner:
name: chef_zero
attributes:
my_app:
git_ref: <%= ENV.fetch(“MY_APP_GIT_REF”, “master”) %>
[/code]
The example above would take the Git reference in the MY_APP_GIT_REF environment variable or default to master if no variable was set.
Q: Should Test Kitchen only run on the Cookbook your developing? Or can you have test kitchen converge lots of different cookbooks which then build a final box locally? I’ve never been sure if I should use test kitchen just for making sure the one cookbook works or if I should use it in more of a role style and always have it converge my end product server that might need and use a multitude of cookbooks.
A: In an ideal world, you can use Test Kitchen for both use cases. When I’m working on a single cookbook, I’m thinking of the cookbook as a unit of software and running it accordingly. However, when I’m deploying full infrastructures, I might want to think at the full node or “role” level. That is, try out your high level wrapper cookbooks or even traditional Chef roles. Having these two uses will hopefully keep your individual cookbooks more honest (i.e. isolated, single purpose, orthogonal to other cookbooks, etc.), more compossible, and re-usable among other teams.
Q: What platforms does ChefDK support by default? In case of unix, what all flavours ?
A: While this answer is for a point in time (Summer 2015), you can currently get pre-packaged downloads of ChefDK for the following platforms:
- Mac OS X
- Red Hat Enterprise Linux/CentOS
- Windows (Desktop and Server)
- Ubuntu Linux
- Debian
Q: I am trying to install web server on one vm, app server on two vms. Can I use Test Kitchen to spin off three vms (1 for web server and two for app servers). If so, how can I specify it in .kitchen.yml file?
A: There’s nothing currently built into Test Kitchen which makes multi-node support first class, but there are ways to approximate things in the meantime. You could construct a .kitchen.yml that uses Suites to describe different node role types:
[code language=”xml”]
–––
driver:
name: vagrant
provisioner:
name: chef_zero
platforms:
– name: centos-7.1
suites:
– name: db-1
driver:
network:
– [“private_network”, {ip: “192.168.33.10”}]
provisioner:
run_list: recipe[my_app::db_server]
– name: app-1
provisioner:
run_list: recipe[my_app::app_server]
driver:
network:
– [“private_network”, {ip: “192.168.33.11”}]
– name: app-2
provisioner:
run_list: recipe[my_app::app_server]
driver:
network:
– [“private_network”, {ip: “192.168.33.12”}]
[/code]
This would yield the following Test Kitchen instances:
[code language=”xml”]
> kitchen list
Instance Driver Provisioner Verifier Transport Last Action
db-1-centos-71 Vagrant ChefZero Busser Ssh <Not Created>
app-1-centos-71 Vagrant ChefZero Busser Ssh <Not Created>
app-2-centos-71 Vagrant ChefZero Busser Ssh <Not Created>
[/code]
Bringing these instances up in the correct order would approximate a multi-node deployment, but you can see how this isn’t totally ideal. Having said this, I’ve seen projects add a Rakefile or command line tool to abstract the kitchen commands away and have good success—your mileage may vary here.
Q: Do we also need virtual box as a pre-requisite ? Last time when I tried test kitchen it was pre-req.
A: If you are using the Vagrant Driver (which ships with the ChefDK), then yes: VirtualBox is a required dependency. Having said that, I typically use the VMware Fusion Vagrant provider which means I need the vagrant-fusion plugin (a paid licence required) and VMware Fusion installed (a paid license also required).
You can get by without VirtualBox or Vagrant if all you want is to use a cloud-based Driver like EC2, Digital Ocean, Rackspace, etc.
Q: Can you integrate with existing rvm system?
A: If you’re referring to Test Kitchen then yes, it works nicely with Ruby switchers such as RVM, rbenv, shrubby, etc. Most of my Test Kitchen code development (and even some cookbook development) uses a chruby setup which allows me to scrub back and forth between Ruby versions.
The ChefDK packages contain their own version of Ruby which means that the installed /usr/bin/kitchen symlink will use a Ruby under the /opt/chef/embedded path. Joshua Timberman has a good blog post covering how he uses ChefDK’s Ruby for his primary development environment.
Q: What about –concurrency=4 so I don’t run 100 at once and blow out my memory?
A: This is a great point—by default kitchen converge --concurrency will run every instance on a thread a once. If you have 100 instances, you get a lot of chaos, and potentially a lot of RAM eaten if you are using the Vagrant Driver or similar. The --concurrency flag supports a number which will limit the number of concurrently executing instances, giving your system some more breathing room.
Q: You showed defining the dependency of other cookbook in the netadata.rb. But you did not show the modification or addition of the same in berkshelf. So how did the cookbook you ran understood the path from which it needs to resolve the dependency?
A: Yes, you are completely correct that Berkshelf was help to resolve and download the cookbook dependencies. When I generated the cookbook with chef generate cookbook, the generated Berksfile looks like this:
[code language=”ruby”]
source ‘https://supermarket.chef.io’
metadata
[/code]
That is one reason (of many) why I generated those cookbooks from scratch using the chef command from the ChefDK: it makes the initial experience that much quicker.
Q: Is there any easy way to cache kitchen package downloads to save converge time? I’ve only seen a couple of hacks on github to install a local caching proxy.
A: There hasn’t been a solution that works consistently across the various Drivers and plugins to date, which is why I’ve been reluctant to bring too much in the core tool. You’re correct that there are some tricks to enable http proxy caching which can speed things up (I have just such a gist myself) and I’m hoping to spend more time later this year investigating ways to speed up converges.
Q: Test-Kitchen offers no hints to a recipe that it is running under test-kitchen (node.kitchen?). Can you talk about best practices when using Test-Kitchen as a Dev environment which involves mocking services.
A: You’re correct that Test Kitchen doesn’t have any explicit hints and this is somewhat by design. I tend to think of these differences as behavior modes which could be selected before Chef executes. For example, if the intention of detecting a run inside a Test Kitchen instance was to enable service mocking, you could tell Chef this directly with:
[code language=”xml”]
–––
driver:
name: vagrant
provisioner:
name: chef_zero
attributes:
my_cookbook:
mock_services: true
platforms:
– name: ubuntu-14.04
suites:
– name: default
run_list:
– recipe[my_cookbook]
[/code]
Just make sure that node["my_cookbook"]["mock_services"] defaults to false or nil and the cookbook operates in “default” or “normal” mode as opposed to “mocked” mode. Just remember that this mode of operation will be different than production meaning that your cookbook is exercising a slightly different code path.
Q: Is the docker provider tied to its own docker implementation, or can it use docker client against your existing boot2docker?
A: The Docker Driver currently shells out to the docker binary on your workstation and will respect the DOCKER_HOST environment variable. In the Windows workstation part of the session, I setup a Docker machine (using docker-machine) on my Mac, then exported its configuration into the Windows VM so I wouldn’t have to run a Docker VM inside the Windows VM.
Q: Is it possible to use vagrant-lxc driver?
A: It is possible by way of using the Vagrant Driver. You’d need to set the :provider to "lxc" or export DEFAULT_VAGRANT_PROVIDER=lxc and your details under the :driver block like this:
[code language=”xml”]
–––
driver:
name: vagrant
provider: lxc
customize:
container_name: mysql
backingstore: lvm
backingstore_options:
– [vgname, schroot]
provisioner:
name: chef_zero
platforms:
– ubuntu-15.04
suites:
– name: default
[/code]
Q: You mentioned about docker driver and issue with services and there is a way to fix that. Where I can find more resources about that?
A: Depending on the platform type/version (remembering that many Linux distributions are in different stages of systemd adoption), you may be able to set run_command: /sbin/init in your :driver block which would run the init system rather than running sshd as the pid=1 process.
Q: I noticed that you generated cookbooks using ‘chef’ but one could also gen cookbooks with Berkshelf, which is the preferred method?
A: That is absolutely correct! The choice is ultimately yours although these days I tend to use the chef generator out of the ChefDK cause I know what I’m going to get. There’s also a nice way of using your own custom cookbook generator, which is described on the docs site.
Q: Your thoughts on packer in the context around test kitchen?
A: I quite enjoy Packer and its ability to create one or more resulting images concurrently. In fact, we’re aggressively using Packer in the Bento project to make the Vagrant base boxes which are used by default with kitchen-vagrant.
There might be some overlap with Packer if you’re using Test Kitchen to prepare images, although you’d have to tool around Test Kitchen default flow to make that happen. Although now I’m wondering what a kitchen-packer driver might look like :)
Q: Where can all of the attributes for the .kitchen.yml be found?
A: There is a growing .kitchen.yml page on the Chef Docs site which goes into more great detail with examples. In addition to this effort I’m planning some time to add generic .kitchen.yml reference documentation on the Test Kitchen website but that’s still pending work.
Finally, I use the kitchen diagnose tool a lot to discover the configuration attributes available to me for the various Drivers and plugins. In general, the rule goes: “if I can see an attribute in kitchen diagnose, I should be able to override it”.
Q: Can you please share the procedure to setup those 3 environments?
A: Good idea! I pushed my cookbook project to fnichol/feedback-factory-webinar-cookbook on GitHub. Hope you get some crazy ideas!
Q: I would like to spin off different VMs to run different roles/recipes, using different environments and data bags. How can I write .kitchen.yml for this requirement?
A: There is fallback detection logic when Test Kitchen computes your path to roles/, environments/ and other directories. For our example, let’s say that we have a suite called “server” which would be setup in a .kitchen.yml like so:
[code language=”xml”]
–––
driver:
name: vagrant
provisioner:
name: chef_zero
platforms:
– name: ubuntu-15.04
suites:
– name: server
[/code]
Assuming we want to test a cookbook which uses roles, Test Kitchen will use the first directory to exist on this list:
./test/integration/server/roles./test/integration/roles./roles
This way you can have separate role or environment setups for each Suite if necessary.
Q: What should we do if we want windows or redhat OS in my VM
A: The quickest way to run a licensed (i.e. non-free) operating system such as Windows or Red Hat Enterprise Linux would be to use the EC2 Driver (or another cloud-based Driver) as they typically charge you per-hour and not for a full license.
If your needs are a bit more specific or custom, you could try build a Vagrant base box with Chef’s Bento project. We have templates to build a few non-free operating systems such as Mac OS X, Suse Enterprise Linux, Red Hat Enterprise Linux, and Solaris. We’re hoping to have fresh Windows templates available in the not-too-distant future as well.
Q: Are those local Virtual machines or on the cloud
A: The virtual machines I used in the webinar session were all over the place:
- The first example spun up local virtual machines using Vagrant under the hood and were hosted on my Mac workstation.
- The second example was running Windows in a Vagrant virtual machine (hosted on my Mac workstation) and spinning up Docker containers which were running in a VMware Fusion virtual machine (also hosted on my Mac workstation).
- The final example was running Debian on an Amazon EC2 instance and spinning up other EC2 instances on Amazon’s AWS cloud.
It seems like a diagram might help for the next go-around :)
Q: I will want to know sometime during the presentation if anyone is or should be using Chef to install and manage complex licensed Windows software like Exchange / Sharepoint / AD DCs, etc. Stuff that is more likely to remain a ‘puppy’ than a ‘cattle’.
A: I can tell you that yes, there are many companies that manage Windows-related services using Chef. A lot of effort went into Test Kitchen to make testing Chef cookbooks on Windows nodes much easier. It’s my hope that over time with these tool improvements, we’re helping to make infrastructure automation on Windows much more effective, quick, and pleasurable.
