Automating Cloud Compliance with AWS OpsWorks for Chef Automate

AWS OpsWorks for Chef Will be Discontinued in All AWS Regions on May 26, 2024.

Progress Chef SaaS is a purpose-built offering to manage your infrastructure for OpsWorks customers with an easy three-step migration process.

Find More Information Here

Chef, InSpec, and Habitat provide workflows for automating everything you manage, from infrastructure configuration to security auditing to application releases. Chef Automate ties these projects together with a powerful dashboard that aggregates data from all of your environments and provides a shared world view of the configuration and security of your estate for everyone within your organization. AWS’s OpsWorks for Chef Automate makes it easier than ever to get started with your own Chef Automate server, with push-button installation and AWS-managed backups and updates. Recently, I had the opportunity to host a webinar with Jonathan Weiss, Senior Manager of AWS OpsWorks, to show off just how easy it is to get started quickly with OpsWorks for Chef Automate. Take a look!

Launching your Automate Server

After a brief overview of Continuous Automation, Jonathan kicked things off with an overview of OpsWorks for Chef Automate (5:15) and demonstrated launching a new instance within AWS (8:45). A full rundown of prerequisites and installation instructions can be found in the AWS OpsWorks for Chef Automate getting started guide.

Detecting Issues with InSpec Scan Jobs

I split my own demo portions into a few parts. The first (19:40) focused on detecting issues with InSpec. There are a few ways to initiate an InSpec scan, but one of the easiest is via the “scan jobs” feature of Chef Automate which is what we showed off in the webinar (21:55). Once you have a Chef Automate server, all you need to follow along is a target node to scan, and the ability to connect to it over SSH or WinRM. Once you’ve identified a good target, you can scan it by following these steps:

• Install an InSpec Profile: The easiest way to do this is via the Profile Store in Chef Automate. In our webinar, we used the Linux Security Baseline profile, and there is a corresponding Windows profile as well.
• Create a login credential: Before we can scan a node, we’ll need to tell Chef Automate how to connect to it by creating a credential, providing a username and password or SSH key with which to connect.
• Create a Node & Scan Job: Once your profile and credential are in place, you can add nodes to scan and create jobs using your profile(s) via Chef Automate’s scanner. In the webinar, we created a one-time scan, but you also have the option to set the scan to recur regularly for continuous evaluation of your security and compliance.

Extra Notes: Detecting Issues

Scan Jobs are a great way to start auditing existing servers without requiring you to pre-configure a client. That said, if you’re managing nodes’ configuration with Chef, you can have InSpec audits created automatically at the end of each chef client run via a special Audit Cookbook. AWS provides step-by-step instructions for configuring audits on chef-managed nodes.

Correcting Issues with Chef

When I scanned my nodes during the webinar (25:45), I found that they had some security issues that needed to be remediated. Just as InSpec can tell us whether our systems are securely configured, Chef can remediate any configuration issues we encounter. The Security Baseline profile we used has a corresponding os-hardening cookbook that can be used to harden our configuration according to the rules laid out in the associated profile. To remediate our servers, I made use of a new feature in Chef Workstation called chef-run. Chef-run allows us to perform ad-hoc configuration tasks by executing local chef resources or recipes against remote targets over SSH or WinRM. This provides us a simple method for quickly configuring single machines, or groups of machines in parallel, with a single command:

chef-run USERNAME@SERVERNAME /PATH/TO/COOKBOOK/recipes/RECIPE_NAME.rb -i /PATH/TO/SSH/KEY 

That’s it! Chef-run will ensure that the chef client is installed, and execute whichever recipes or resources we’ve provided. In the webinar, however, I also had my results sent to my Chef Automate server, which can be done in a few simple steps:

• Find your Data Collection Token: A data collection token is used to authenticate with Chef Automate when sending client data. To find your Automate Server’s token, log into your OpsWorks for Chef Automate instance, and run the following command as root:

  automate-ctl show-config

  This will display a JSON hash of the server’s configuration, and the string following token: is what we’ll want. Make a note of it for the next step.

• Update your config.toml: Chef Workstation has a config.toml file used for any optional configuration paramaters for using chef-run, located in ~/.chef-workstation/config.toml. To configure chef-run to send its data to Chef Automate, add the following lines to that file:

  [data_collector]
  url="https://YOUR_OPSWORKS_URL/data-collector/v0/"
  token="YOUR TOKEN FROM STEP 1"
  

That’s it! Now when you run a chef-run command, the results should show up in the “nodes” view within Chef Automate!

Extra Notes: Correcting Issues

As with audits, there are a number of ways to execute Chef on machines. Chef-run is great for ad-hoc tasks, but to configure nodes for regular configuration with Chef, you’ll likely want to formally bootstrap them. Again, AWS provides a great guide for bootstrapping your nodes with knife (included in Chef Workstation).

Automatically Bootstrapping Autoscaled Nodes

One of Amazon’s most popular cloud features is AWS Auto Scaling, wherein you can define groups of EC2 instances that will automatically add or remove nodes based on metrics like CPU load or network traffic. With OpsWorks for Chef Automate, you can easily define autoscaling groups that will be automatically bootstrapped and managed by Chef. In the webinar (35:00) we did exactly this, making use of the OpsWorks CM API. When you download a starter kit from OpsWorks for Chef Automate, it includes a custom User Data file that will handle this for us with minimal modification. A full overview of the process we used in the demo can be found in the Add Nodes (to Chef Automate) Automatically docs on AWS. This will walk you through everything from IAM profile creation to how to configure your launch configuration for unattended bootstrapping of autoscaled nodes.

Extra Notes: Auto Scaling

One thing that’s worth noting about my demo environment was that my configuration targets were running Ubuntu, which requires a few extra things in the autogenerated user data file in your starter kit. Most notably, the stock Ubuntu AMIs don’t have zip or python pre-installed, which we’ll need to configure things. If you open up userdata.sh in your editor of choice, you’ll want to start by adding a function to install these components:

prepare_os_packages() {
  # ubuntu
  apt-get update
  apt-get -y install unzip python
}

Then, near the end of the file, you’ll need to invoke this function before it runs the install_aws_cli function. Here’s what the list looked like in my final userdata file:

prepare_os_packages
install_aws_cli
node_association_status_token="$(associate_node)"
install_chef_client
write_chef_config
install_trusted_certs
wait_node_associated "${node_association_status_token}"

With those updates, you should be able to follow along with the AWS docs called out above!

What’s Next

Now that you’ve gotten a taste of what you can do in OpsWorks for Chef Automate, here are some extra resources to take your automation even further!

• Check out the full documentation for OpsWorks for Chef Automate for even more insight into how to configure and manage your estate.
• The Chef Workstation Getting Started Guide will guide you through more use cases for chef-run to get the most out of your ad-hoc tasks with Chef.
• Get a taste for Chef on AWS with these hands-on, guided tutorials. Manage your first node, test Chef cookbooks on temporary cloud instances, and deploy changes to a production-like environment, all on AWS.
• Hear how University of Kansas is succeeding with OpsWorks for Chef Automate.
• Listen to Arun Gupta, Principal Open Source Strategist at Amazon Web Services, talk about the benefits of OpsWorks for Chef Automate.