Note: This post originally appeared on Padgeblog.
Chef-vault is a tool created by Nordstrom and adopted by Chef as the de facto way to handle secrets management using the Chef platform. Chef-vault builds on the original Chef encrypted data bags concept¹. Rather than a single shared decryption key, chef-vault creates a separate copy of the data bag item for each node that is granted access, using the existing RSA public/private key pair normally used for Chef API authentication. According to Noah Kantrowitz, who sums it up nicely, “This means you no longer have to worry about distributing the decryption keys, but it also moves chef-vault into the gray area between online and offline storage systems. Once a server is granted access to a secret, it can continue to access it in an online manner. However granting or revoking new servers requires human interaction. This means chef-vault is incompatible with cloud-based auto-scaling or self-healing systems where nodes come/go frequently. It also inherits the same issues with lack of audit logging as all other data-bag driven approaches”.
Summary of Chef-Vault Functionality:
Problem:
How to have teams of developers manage secrets in a common manner using chef-vault?
Of course you could say that each individual team has designated individual(s) with knife access to Chef servers and they use knife vault to get the secrets up to Chef and call it a day. However, this is not always as cut and dry as one might hope. For example, in my company we chose years ago to tightly restrict knife write access to chef servers only from a CI/CD server via a pipeline. In other words, there are SDLC process limits and security process limits put in place to limit the risks of “wild west knife access”. Another reason is that members of teams in larger orgs can come and go, and often you need systems where things are well documented and stored in ways that others can pick up and work where another left them.
Due to the need to have a consistent process as well as limit knife access to the pipeline, we needed a mechanism to get secrets into Vault without funneling them all to a central team every time a new secret is added or removed. Because Chef Vault uses data_bags which are predominantly stored in version control, the seductive answer is to just throw them on your private version control system like GitHub Enterprise. However, the moment you do that, your secrets no longer become secret. What was needed was a way to encrypt secrets in GitHub in a seamless manner such that a CI/CD pipeline server can ingest them for the express purpose of using knife vault to get it securely to Chef.
The design solution was to use a combination of technologies: git-crypt + GPG + github + Jenkins + Chef vault. Let’s dig into the design details.
Solution Summary:
MyOrg Web of Trust (WoT) Using GPG
In order to sanely place secrets like passwords, certificates, and keys in a DVCS like GitHub with only authorized people having access, we need to use GPG keys. The design is such that the authorized “web of trust” user accounts like an Infrastructure CI/CD server ldap user, Infrastructure Chef admins, or specific Security Engineering team members get a copy of a given team’s public GPG key. An authorized user account will then be able to grab the data and unencrypt it using the GPG keys. Depending on your Web of Trust preferences, you may desire to only allow access to your GPG public key to a smaller group.
Essentially, the design is such that each team can have a designated email address or name as an identifier which is used with the GPG public key. This public key is then sent to ONLY those in your Org Web of Trust (WoT) that are authorized to get the public keys. In this case we are not actually freely giving out team public GPG keys, but selectively choosing who can get them. I realize that the beauty of the GPG key in the public internet use case is to make the public key widely available. Here, we are using it selectively within a given org for purposes of limiting who has access to decrypt. You are taking some steps to protect your public keys within a team for purposes of getting the shared secrets into a data_bag store (usually Git) securely. The trick here is to have the data_bag items encrypted in the version control system like a private Github repository such as Github enterprise using GPG keys + Git-Crypt.
The process of getting them into the Chef server as an knife vault encrypted data_bag item then can be handled by a secured job on a CI server such as a private Jenkins instance.
The CI/CD server job must do 2 things:
This job workflow allows a distributed team to centrally store secrets in a way that a central job can grab it, decrypt it, and then re-encrypt it using native Chef vault processes. Of course, when people leave the team, it behooves you ideally to generate a new team based GPG key and then re-encrypt your team secrets using the new GPG key and ensure the CI/CD server job gets the new key for its purposes. This design also assumes you own a private DVCS system like Github enterprise or a private Github account. I don’t recommend implementing this design for any open public internet projects, on a non-private Github or bitbucket for example, where mistakes in your workflow and WoT could potentially expose your secrets. This is a weakness of the design in terms of security and management, but a fair trade off from the alternative of non-standard secrets management among large teams before they get it into chef vault.
Pre-requisites:
[code class=”ruby”]cd /chefRepo/data_bags/chef-vault/
mkdir myApproleProd[/code]
[code class=”ruby”]cat > .gitattributes << EOL
secretfile filter=git-crypt diff=git-crypt
*.key filter=git-crypt diff=git-crypt
EOL[/code]
Workflow Summary : Common Chef Repo Data_bags or Chef 12 Organizations
Chef-Vault Directory Design Standards
The data_bags directory structures allow data_bags that contain data_bag items. This can exist in a sane root directory structure such that we can organize things in a Github repo in a way that is visually appealing. Once the data_bags and their items are ingested by Chef to SOLR, the root directory structure in GitHub is not present. Instead there are data_bags (keys) and data_bag items (values).
See details on how Chef views the data_bag directory structure. Per the Chef docs, “A data bag is a container of related data bag items, where each individual data bag item is a JSON file. knife can load a data bag item by specifying the name of the data bag to which the item belongs and then the filename of the data bag item.”
Essentially, the form is:
data_bag directory
data_bag_item json file
You can chose the name the json data_bag_item in Chef vault with a .key extension, but the content should have “id” and “key” section.
[code class=”jscript”]{
"id": "my_secret_item_name",
"key": "value"
}
[/code]
An example directory structure is below:
The bag_name should be updated based on your application role.
[code class=”ruby”]
myChefrepo
├── data_bags
│ ├── chef-vault
│ │ └── bag_name_matching_role
│ │ └── fooapp-secrets.key
│ │ └── barapp-secrets.key
[/code]
[code class=”ruby”]
├── data_bags
│ ├── chef-vault
│ │ └── myApproleProd
│ │ └── myApp-secrets.key
[/code]
Design for sharing encrypted Items in a DVCS for Chef using chef-vault
Here are some common use cases or knife-vault that the CI/CD server job logic can execute.
Chef-vault commands used by the CI Server automation (Examples)
Create Vault Item
[code class=”ruby”]vault/apps/fooapp/fooapp-secrets.json \ -A "adminuser1,adminuser2" -S "role:fooapp-server"[/code]
Show Vault Item
[code class=”ruby”]knife vault show fooapp fooapp-secrets -Fjson [/code]
Delete Vault Item
[code class=”ruby”]knife vault delete fooapp fooapp-secrets[/code]
Delete old node
[code class=”ruby”]knife vault update fooapp fooapp-secrets \ -S "role:fooapp-server"
[/code]
Update list of Admins
First see who has access:
[code class=”ruby”]knife search fooapp ‘id:fooapp-secrets -a clients[/code]
Next change the membership:
[code class=”ruby”]vault/apps/fooapp/fooapp-secrets.json \ -A "adminuser3,adminuser2" -S "role:fooapp-server"[/code]
Update/Rotate/Refresh Keys/secrets
Rotate keys for a single vault item:
[code class=”ruby”]knife vault rotate fooapp fooapp-secrets[/code]
Rotate all keys for all vault items:
[code class=”ruby”]knife vault rotate all keys[/code]
References: