On November 22, 2016, I presented a webinar with Seth Vargo on managing secrets with Chef and HashiCorp’s Vault. Our very large and highly engaged audience came prepared with great questions.
We started off talking about generic secrets, and why you should start rotating them. From there, Seth went into the traditional ways you do secrets with Chef, such as encrypted data_bags and the like. He introduced Vault as another option, leveraging some basic Custom Resources being able to pull directly from his Vault. Seth also showed us how to leverage Vault in different use cases, with the reveal of a short leased Dynamic Secret for Postgres.
After Seth’s presentation, I did a quick demo of leveraging Habitat to build and run a Consul backed Vault system on his local laptop. For those that don’t know, Habitat is our new open source project for application automation platform. The demo was a quick teaser for a more in-depth explanation and demo we’re planning for early next year.
I encourage you to watch the recording of our webinar to learn:
- The fundamental advantage of Dynamic Secrets
- The fundamental design of Vault
- How to leverage Chef and Vault via Custom Resources
- Options for secrets management leveraging Chef
Finally, a recap of our Q&A from the live session is available below.
Q&A From the Live Presentation
A typical security requirement in financial, government, military, etc. environments is that secrets cannot be stored in the clear on disk. How can I use Vault + Chef and still abide by this requirement?
Vault encrypts data in transit (via TLS) and at rest (storage), so it would definitely aid in these requirements. Vault has been audited multiple times by the NCC group, and data is never stored in plain-text by Vault. As long as you are communicating to Vault through TLS, you’re covered.
Additionally, depending on the storage backend, Vault may never write data to disk. For example, the “Consul” storage backend runs entirely in memory, so the encrypted data only lives in memory, and is still encrypted the entire time.
What’s the Chef -> Vault authentication configuration best practice? In AWS, I would presume IAM, if that’s supported, but what’s recommendation for non-AWS environment?
This is something that HashiCorp is actively working on improving. If you are running in AWS, you have access to the EC2 authentication backend. If you are not running in AWS, you can utilize the AppRole authentication mechanism.
Will you store the unseal token in a chef data bag and run the unseal before you run the postgres cred retrieval? If not, would you just leave the the vault unsealed?
Storing the initial token in an encrypted data bag is definitely an option, but there are other “authentication” options as well. See here for more information: https://www.vaultproject.io/docs/auth/index.html
For example with PostgreSQL, what if a user has a query running that runs past the expiration of its credentials. Will vault revoke the creds during the running query and will that break the query? Would vault or PostgreSQL notice this situation and wait until the query is done to revoke creds?
Vault attempts to gracefully kill the user, and then after a timeout, it will force-close the connection. Long-running processes should investigate increasing timeouts or using periodic secrets instead.
Does vault need a plugin to work with databases? Could it work with a MicrosoftSql database with .net?
Vault has a number of built-in database authentication mechanisms. See here for more information: https://www.vaultproject.io/docs/secrets/index.html
RE: generating AWS credentials for developers via Vault – is it possible to generate/store multiple static developer credentials behind the same aws/creds backend so each developer only has access to their own set of credentials? (e.g. Joe has access to aws/creds/joe but not aws/creds/mary, Mary has access to aws/creds/mary but not aws/creds/joe)
This is certainly possible, but it’s generally recommended to group things into “roles” or “responsibilities” instead of per user. See the AWS docs for more information: https://www.vaultproject.io/docs/secrets/aws/index.html
Does the vault command have to take place in the context of some exiting Chef-Server user?
No, but keep in mind that you are writing secrets to disk and those files should be secured.
Is it secure to leave the vault unsealed for a large time range (a month)?
The Vault operates in an unsealed state for the majority of it’s lifetime. The only time the Vault should be sealed is in the event of a data breach or when a new Vault server is added to the cluster.
If I already have HSMs can vault act as a mediator? Does vault only work with specific brands of HSMs?
The OSS version of Vault does not integrate with HSMs. Integration with HSMs is one of the enterprise features we offer. Please note this is not to “get you to buy something”, but we have found, in analyzing market segments, that HSM integration is not something that is widely-requested and generally applies to larger organizations that care about SLAs and other enterprise features. If you’re interested, I can put you in touch with the sales team, or you can contact them directly at sales@hashicorp.com.
I joined a bit late, but did you discuss the secure introduction problem? For instance, if I have a server that is bootstrapped with chef and i want the chef run to provide secrets to applications. How can i securely give chef client the initial creds or token to get credentials distributed in the run.
We did discuss the secure introduction problem a bit toward the end, but there are currently a number of ways to introduce the first secret onto the system. If you are running in AWS, you have access to the EC2 authentication backend. If you are not running in AWS, you can utilize the AppRole authentication mechanism.
The default setup for your sample is set up to allow insecure TLS. — While your automated build of vault looked like you had tls disabled. So is the production system a ‘built by hand system’, or did you purposefully make the demo ‘apps’ entry tls-insecure?
The Vault server itself is not running in TLS, but it’s fronted by an nginx server that is running with TLS in my setup. There are a number of techniques for securing the Vault server, but for simplicity I chose to offload the TLS negotiation to nginx. This is not because Vault cannot use custom TLS, but because I chose to use Let’s Encrypt as my CA and it was easier using their nginx plugin. Both are perfectly valid options.
The vault endpoint I was communicating with was “https://vault.hashicorp.rocks”, which was communicating via TLS.
Could we change the lease duration of the dynamic secret?
It is possible to tune the lease duration for secrets on a per-secret and a per-backend basis. You can find more information in the Vault documentation: https://www.vaultproject.io/docs/secrets/postgresql/index.html
In the enterprise, AuthT/AuthN lifecycle is typically managed through approval processes (e.g. systems like SailPoint, which also have connectors for dynamic provisioning). A point that can be confusing is that these workflow systems also support connectors to resources like databases for dynamic provisioning, but most customers would prefer to integrate with an Open Source API like Vault, and use Vault’s dynamic connectors instead. Are you aware of any Vault-fronted SailPoint integrations?
Vault has a number of integrations, but I do not think SailPoint is currently one of them. I definitely agree that people enjoy using Vault’s API. I think the best course of action here is to open an issue on GitHub or reach out to your sales rep to get more information. Depending on the SailPoint technology, I don’t see why we cannot add such an integration, but it would depend on custom demand and the complexity.
When you create the Vault, all the unseal keys are visible in the same place. Are there mechanisms or patterns for distributing those to multiple people automagically such that no operator ever has full access?
This is a great question, and something I had in my notes to discuss but ran out of time. Vault can optionally output gpg-encrypted unseal keys using a series of supplied public keys for operators. The person who sets up the Vault will only see the GPG-encrypted keys, and then it would be safe to send those directly to the operator who decrypts with his/her GPG private key.
What is the difference between chef-vault and HashiCorp vault?
chef-vault is a Chef cookbook and plugin system for per-node secrets. HashiCorp Vault is a secret management solution and provides “secrets as a service”.