How to Work with Chef Data Bags

Often, engineers find themselves in situations where they do not want to hard code certain data into recipes or cookbook attributes.  User information, external server details, and database connections are examples of such data. Chef offers a unique way to deal with such scenarios in the form of ‘data bags’ that help manage arbitrary collections of data, which can be used with your cookbooks. Data bags can encrypt data, have global access across your infrastructure, and store sensitive information.

Installing Chef Workstation is the easiest way to work with data bags and all other Chef tools from one control centre.

What is Chef Workstation?

It is a collection of developer tools that enable devices in your fleet to interact securely with your Chef Server. It includes Chef Knife, InSpec, Cook style, Habitat, and Test Kitchen. It also contains embedded Ruby and other dependencies, so you don’t have to install anything else to get started with Chef tools.

Setting up the Environment

Chef Client

Download and install Workstation here. Verify the installation in your terminal with the command.

$ chef -v

In this scenario, the server is trying to communicate to the HTTP endpoint URL. For this, you need to create a data bag, which will hold the endpoint URL detail and use it in your recipe.

One way to establish an HTTP endpoint is to run nc –l 80 on any server accessible by your node and use its IP address below.

Another way to establish an HTTP endpoint, which shows us the requests we make, is a free service called RequestBin. You can read about RequestBin and its usage here.

As a best practice, always create a separate directory for Data bags within your cookbook.

$ mkdir data_bags/hooks

Create a data bag item for RequestBin. Make sure to use your own RequestBin URL.

$ subl data_bags/hooks/request_bin.json

{
"id": "request_bin",
"url": "https://eo5t76tpa13fhfyo.m.pipedream.net"
}

Create the data bag on the Chef Server:

$ knife data bag create hooks

Upload your data bag item to the Chef Server:

$ knife data bag from file hooks request_bin.json

Edit the default recipe of cookbook to retrieve the RequestBin URL from your data bag:

$ subl cookbooks/databags_cookbook/recipes/default.rb

Upload your modified cookbook to the Chef Server:

$ knife cookbook upload databags_cookbook

Run Chef Client on your node to test whether the HTTP request to your RequestBin gets executed:

$ sudo chef-client

Check your RequestBin or your HTTP endpoint. The request should show up there. You define each data entry in a JSON file, called a data bag item. You can search for data bag items from your recipes to use the data stored in the data bag.

In the above example, you created a data bag called hooks. A data bag is a directory within your Chef repository, and you can use $ knife to build it on the Chef Server. Then, you created a data bag item with the name request_bin in a file called request_bin.json inside the data bag's directory and uploaded it to the Chef Server.

Your recipe retrieves the data bag item using the data_bag_item method. Therefore, the first parameter will be the data bag name, and the second parameter will be the item name. You then create an http_request resource passing it the URL attribute of the data bag item. You can retrieve any attribute from a data bag item using the Hash notation hook['url'].

Learn more about Data Bags here.